Trust-Store and Chain/Intermediate certificates

[Stefan Klotz]

Hi there,

who can explain, how the certificate trust process is working?

Background is the following issue:

We have some APIs with backend/provider in the cloud (e.g. Azure). They are using the following server certificate "*.azurewebsites.net". We installed the intermediate and root certificate in the trust-store of the gateway, both with the option "Signing Certificates for Outbound SSL Connections", but only the root certificate with the "Certificate is a Trust Anchor" option enabled.

During last weekend Microsoft changed the server certificate and also its issuer. Only the root certificate is the same. Due to the different intermediate certificate, the gateway was not able anymore to trust these SSL connections. Although the intermediate and root certificate will be provided from the server during SSL handshake, we had to install the new intermediate certificate first to get it working again.

I was expecting, that in such case the root certificate should be sufficient to be available in the gateways trust-store.

Is this behavior really correct or do we miss some settings?

As Microsoft will not inform the customers about such certificate changes, this will always cause an impact due to missing certificate trust on the gateway as we are not aware of all potential intermediate certificates.

Is there any way to avoid this?

I mean, if the intermediate and root certificates will not be provided from the server during SSL handshake, then this behavior would be ok for us, but as long as the chain is available, the root certificate should be sufficient.

Thanks for your help!

 

Ciao Stefan

[Stefan Klotz]

CA-support confirmed in the meanwhile that the described behavior is normal and that there is currently no option to workaround this. Based on this it makes no sense to import any Root-CAs or at least "higher" chain-certificates other than the direct issuer-certificates from the server-certificate.

To get this behavior changed I've raised the following idea.

Hope that could be changed asap.

Thank you!

 

Ciao Stefan

[Alan Baugher]

Thanks for the information & link for enhancement request.

#### ####

I found this error message in the API GW logs (enabled verbose logging)

I have seen this issue for Google Apps (GCP) connector as well; and resolve it with the intermediate public root CA certificates as a "trusted anchor".

I have up voted your enhancement request.

####   Details using openssl to trace #####

config@vapp0001 VAPP-14.1.0 (192.168.242.146):~ > openssl s_client -connect googleapis.com:443 -showcerts | grep -e "subject" -e "issuer" -e "s:" -e "i:"
depth=2 OU = GlobalSign Root CA - R2, O = GlobalSign, CN = GlobalSign
verify return:1
depth=1 C = US, O = Google Trust Services, CN = GTS CA 1O1
verify return:1
depth=0 C = US, ST = California, L = Mountain View, O = Google LLC, CN = www.google.com
verify return:1
0 s:/C=US/ST=California/L=Mountain View/O=Google LLC/CN=www.google.com
i:/C=US/O=Google Trust Services/CN=GTS CA 1O1
1 s:/C=US/O=Google Trust Services/CN=GTS CA 1O1
i:/OU=GlobalSign Root CA - R2/O=GlobalSign/CN=GlobalSign
subject=/C=US/ST=California/L=Mountain View/O=Google LLC/CN=www.google.com
issuer=/C=US/O=Google Trust Services/CN=GTS CA 1O1

​​
Verbose logging to trace SSL verification error message:

​​

------------------------------
Sr. Principal Architect
------------------------------

[Stefan Klotz]

Are there any news/updates on this topic? Is this already planned for one of the upcoming releases or is there maybe any workaround available?
We had another outage last week as Microsoft changed again some intermediate certificates in their Azure cloud-environment.
This behavior is not industry-standard and needs to be fixed asap!!!
Thank you!

Ciao Stefan :)​

[Zhijun He]

Dear Ciao, Alan,
For security concern, by default, the gateway only trust the (manually) imported certificates, and the gateway doesn't trust wildcard certificate.

Ciao, your problem could be resolved by "trust well known public CA", add a cluster wide property pkix.useDefaultTrustAnchors as true.

​​Alan, the google uses wildcard certificate, (you can see from your screenshot), you may need to set cluster wide property io.httpsHostAllowWildcard to true.

Use caution when enabling those options in a trusted zone.

Regards,
Mark

[Stefan Klotz]

Hi Mark,
thanks for the answer. But where can I check, what is included in the "trust well known public CA" list? And is it also possible to edit/maintain this list?
Any further details here would be very helpful!
Thank you!

Ciao Stefan :)

[Zhijun He]

It's on java side.
You can export trusted root certs (to /home/ssgconfig/calist) by command,
/opt/SecureSpan/JDK/bin/keytool -list -v -keystore /opt/SecureSpan/JDK/jre/lib/security/cacerts -storepass changeit >/home/ssgconfig/calist

If you want to install more trusted root, the website below may give you some ideas,
https://knowledge.digicert.com/solution/SO4085

Regards,
Mark

[Stefan Klotz]

Hi Mark again,
I made some testing with this CWP and first of all, I can confirm this is working fine as long as the CA is included in the Java trust-store. There is no need to have anything else imported in the Policy Manager trust-store.

But then I noticed the following behavior:
- CWP pkix.useDefaultTrustAnchors is set to false
- Intermediate CA of the Server certificate is NOT installed in the Policy Manager trust-store
- But Root CA is installed in the Policy Manager trust-store
-> Connection is working without "path validation checking failed" error!!!

So was there any behavior change in one of the last updates? I performed the tests with 9.4 CR05.
Does this maybe belong to this?

DE371400

Corrected an issue with the Trusted Server Certificate option when used with the Route via HTTP(S) Assertion that caused verification of certificate path even after specifying the selected Trusted Server Certificate option. This correction stops verification of the complete certificate path if a subset of Server Certificate(s) is selected from the Trust Store.

Does this mean, it would be sufficient if ANY of the certificates from the chain (doesn't matter if the first issuer, a second issuer or the root ca) is imported in the Policy Manager trust-store? Because this would be exactly the behavior we would expect and were looking for all the time.
Thank you for official confirmation or any further details on that topic.

Ciao Stefan :)

[Stefan Klotz]

Hi all,
the issue seems to be still there. We have both:

• pkix.useDefaultTrustAnchors
• io.httpsHostAllowWildcard

set to true, but still getting the "path validation" error.
Again the intermediate certificate for "*.azurewebsites.net" was changed and currently we only get it working again with importing the latest "Microsoft RSA TLS CA 01" cert into Policy Manager trust-store.
I checked the java keystore and it correctly contains the "Baltimore CyberTrust CA". The name is mentioned differently "Baltimore CyberTrust Root", but the PEM is identical. Is this maybe the issue?
Do you have any ideas, why it's still not working and how to further analyze the root-cause?
This behavior with the trust-store is really annoying and needs to be fixed asap!!!
FYI: we are running 9.4 CR05 with latest security patch (Layer7_API_PlatformUpdate_64bit_v9.X-RHEL-2020-11-19.L7P).
Thank you!

Ciao Stefan :)

[Jay MacDonald]

Hi Stefan,

  Can you contact me directly regarding this?

Cheers!

JayMac
[e-mail]

------------------------------
Jay MacDonald - Adoption Architect - Broadcom API Management (Layer 7)
------------------------------