Recently a customer asked for help with acquiring an Active Directory server for the CA Identity Manager (CA Identity Suite). The customer already had one (1) ADS domain acquired, and wished to acquire a 2nd ADS domain (Company had merged with another).
This is usually a straight forward process.
Minimal information needed is
- hostname of DC server,
- service ID & password (if not Administrator, then a delegated service account with CrUD permissions),
- the public root CA certificate from that AD server or AD domain (which had signed a server cert).
I typically have customer use a 3rd party ldap client tool, e.g Jxplorer or SoftTerra LDAP browser tool to confirm they have the correct service ID/password & public root CA cert. Other tool is the IMPS\bin\adsldapdiag.exe to validate serviceID/password.
If there are issues with the SSL/TLS security, I will have the customer navigate to the IMPS\bin folder (or IAMCS\ccs\bin) and use the openssl.exe binary.
1) Save the public CA certificate (and any intermediate CA) as a PEM format (base64 - that you can open in notepad to see BEGIN END statements). [May also do pks format]
2) Execute openssl s_client -connnect hostname:636 -showcerts -CAfile c:\temp\ads-ca-file.pem
If the above returns success, then we know we have the complete TLS chain.
However, for this one customer, they were successful with all the above, but their remote Active Directory domain, still reports that "confidential required" error message. We ensured that the public CA and the customer intermediate CA were being referenced correctly.
To isolate this issue, we reviewed the customers'
1) Server Certs
- One delta was use of SANS and no CN
- Validate in my own lab this is not an issue
- Had to install MS CertSrv to generate the correct SANs with a <null> CN, then used openssl to sign it.
- No issues for openssl, 3rd party ldap client tools, nor Identity Manager/Suite via IAMCS to CCS.
2) ADS Group Policies - CIS harden
- Customer used various CIS recommended hardening policies
- Had customer export their GPO and I used the MS tool SCM v4.0 to load them into my lab.
- validate they had no impact on the issue.
- note: MS SCM is a great tool to compare GPO instead of doing this manually via eye-ball 1.0.
I needed to dive deeper; and the debug logs of IAMCS, CCS and Active Directory were not enough to isolate this issue.
###### The use of Wireshark to monitor all traffic #####
I have used the CA Identity Suite vApp (Virtual Appliance/IDVA) as my new base lab solution; deploying the IAMCS/CCS service twice; updating the IAMCS configuration file to use the remote CCS service (to allow use of Wireshark); enable the debug logs; changed the default Cipher Suite for Active Directory.
Below image of lab:
Wireshark was able to capture when a NEW ADS endpoint is acquired for the 1st time. I wanted to see WHEN the CCS service validated the SSL/TLS certificate and how that was viewed.
There are four (4) TLS transaction between the CCS service and the ADS endpoint.
1) Initial Client Hello - Listing all the available Cipher protocols that CCS has access to.
- No decryption at this stage yet.
2) Initial Server Hello - Selects the highest (or order) Cipher protocol provided, that it shares with the client.
- Shares the server certificate, that will display "server authentication" use, and the SANS value for the DNS entry
- No decryption at this stage yet.
3) Certificate Key Exchange - Able to view the successful negotiation.
- Able to view de-crypted data with server private key.
4) Change Cipher Spec - Appears to be the final step; little value
- All other communication is now seen as "LDAP" over TCP 636 and can be viewed as decrypted data.
Note: By default, Active Directory Cipher Suite uses the Diffie-Hellman key exchange process. Diffie-Hellman key exchanges can NOT be decrypted with the RSA private server key, as the session key is built in memory between the client and server.
Example of RSA Cipher Used for Active Directory:
Example of Server Certificate with <null> CN and use of SANS DNS entry
A view with using OpenSSL to check Cipher Suite. Below is an example with Diffie-Hellman Cipher Suite, that is not possible to use Wireshark with. When Cipher Suite order is changed, the MS Windows Server must be rebooted (or schannel service restarted).
After a reboot of the server, and re-validation with openssl
Where to change the ADS Server Cipher Suite order:
I kept the following SSL Ciphers Suites to allow Wireshark to use the server's private ssl key. Note that both are RSA cipher suites.
How to read a SSL Cipher Suite? Cipher Suites in TLS/SSL (Schannel SSP) (Windows)
A view of all Active Directory (2012) SSL Cipher Suite protocols:
What to enable within Wireshark, to monitor ADS over TLS?
How to export your Active Directory server key w/private key? See below using MS tool "certlm.msc"
Save as PFX files and add to Wireshark as is, with the password.
Or convert from PFX to PEM (with openssl), and add as PEM files to Wireshark with no password
Note: If the ADS Cipher Suite has NOT been updated, then you will receive this error message in the Wireshark logs
How to export your de-crypted ADS traffic to a text file? Use the below Wireshark Export process, select the following check boxes.
Hopefully, you will find some value of this process and research.
I am enclosing attachments of an ADS endpoint acquired for the first time (over TCP 636), in various Wireshark export formats. You will likely find value within the TXT format.
Edit: 3/5/2018. Move to CA Security community.
Summary: Avoid false error message of "Confidential Required" due to increase delays.
1. Remote Active Directory servers may increase transaction delay
2. New elliptical Cipher Suites may increase transaction delay during Hello / Exchange Cycle
3. Introduction of the IAMCS as the intermediate tier between the IMPS (prov server) and the CCS service may increase transaction delay
A follow up & remarks.
Using various tools, I was able to determine that the im_ccs service does a hand-off to the MS lsass service to determine the encryption communication to the remote MS Window server. The MS lsass communicates to the MS schannel process, to identify the Cipher Suites. You may limit what is to be used from only the client site, and not have to touch the MS Active Directory domain. (This assumes that the CipherSuite you select, is one that is offered by MS AD).
During troubleshooting exercise with the customer, we did capture they were using default CipherSuites, and the process was failing not during the Certificate Exchange, but immediately after. The root issue was not CipherSuite, but a timing issue. The IM solution was closing the connection prior to allowing the connection to finish. We recorded an average timing of 10 seconds for the Cipher Exchange, then 40-45 seconds for the initial application data to flow. Since this timing was over the default of 30 seconds, we were seening a false error message of "confidential required".
As a test, we changed the CipherSuite, on the IMPS server, to a single one, and validated the connection process again. We were able to reduce the timeout, some what, and did received a different error message regarding IMPS server timeout.
To address the above issue, we increased the timeout, to ensure that the communication would complete, within the IMPS server, using the IMPS Manager GUI; from a default of 30 seconds to 120 seconds; and ensure the partial results list was checked.
After restarting the im_ccs server, we were able to see a successful completion of the initial acquisition of the Active Directory server; and explorer it to completion.
To confirm, this was NOT an issue with the CipherSuite, we returned all values back to default; bounced all services; including the MS Windows Server with CCS.
We monitor the communication from CCS server to MS Active Directory, to ensure we saw the four (4) TLS hello / exchange transactions; then the TLS application transactions.
We are now able to manage the remote Active Directory servers using the OOTB CCS service managed by IAMCS.
An interesting exercise.
Hmm - that's a really difficult one to troubleshoot! Nice job.
It sounds like the most-contributing factor was the time required for the data to flow seems to be the remote AD server (perhaps because of additional round-trip latency delay)?
Did it help at all to change the CiperSuite to the single one? Or is that something we shouldn't recommend, and instead just need to recommend increasing the timeout values and using partial results?
With regards to the question about changing the CipherSuite to a single one, we did see the error message change from "Confidential Required" to a timeout message, "ETA_E_2540 Timed out waiting for IM Provisioning Server ...".
Even when this message occurred, we did see the ADS endpoint was built; and we were able to explore and correlate the endpoint.
Since this message indicated a possible root issue, we did monitor the timeline during the connection with Wireshark, and were able to see the length of time taken. The use of a single Cipher Suite, reduced the duraction during the Hello / Certificate Exchange period.
Then we investigated areas that have impact on timeout, which lead us to the IMPS Manager GUI (aka an LDAP client) own default timeout settings.
Which then, when we change the default value from 30 seconds to 120 seconds, all error message disappeared.
Top Notch as always!!
sent via mobile
Sr Principal Architect, Practice Services, Security and Compliance
CISSP, ITIL, Master Certified IT Architect
Tel: +1 636 336 6605 / 1 636 578 6072
CC Bridge # 18663766162 x6365786072
To the information here, It was a great read. So your work in general top shelf. Just wanted to say it is all.
I tried to respond a smiley emoticon via Mail, and but Jive translated it to a question mark. :-)