Need help troubleshooting LDAPS configuration on vCenter 8

[pricemc1]

I am trying to configure an identity source in vCenter 8 using LDAPS with Active Directory on a new vCenter 8 implementation. I have been able to do this successfully many times in the past on vCenter 7 instances so I'm familiar with the procedure and requirements. This is my first time doing it though on a vCenter 8 VCSA, but nothing really looks different than how it was done in earlier versions.  Unfortunately, I seem to be running into some issues when I try to add the identity source using LDAPS. Adding it using standard LDAP works fine though.  I am receiving the following error when trying to configure the identity source for LDAPs though:

"Cannot configure identity source due to Failed to probe provider connectivity [URI: ldaps://domainl ]; tenantName [vsphere.local], userName [User] Caused by: Can’t contact LDAP server."

For troubleshooting I have done the following:

• I was able to contact and query the LDAP servers (Windows Server 2022 DCs) using 'ldp.exe' to test LDAPS from Windows clients that I tested from.  I was able to bind and authenticate using the domain account credentials I specified in the identity source configuration.
• I was able to confirm that the LDAPS servers are presenting the correct certificate by using Openssl to display the certificates being presented on port 636/3269.
• I was also able to verify network connectivity and proper name resolution from the VCSA to the LDAPs servers from the VCSA CLI using 'ping', 'dig', and 'nslookup'. Both forward and reverse lookups seem to be fine from the VCSA and Windows servers. All of these systems are on the same network. There is no router or firewall between any of them (other than OS level firewalls built into Windows and VCSA).
• When I ran 'ldapsearch' command from the VCSA's CLI it returns the no errors that indicate LDAPS is not working when being queried.
• I confirmed that the CA certificates for the CAs that issued the LDAPS certificates to the Windows servers have been added to the trusted list in vCenter.

I'm thinking my next troubleshooting step should be to review any relevant VCSA log files that would have detailed information on what is going on when I try to add the LDAPS identity source. I'm not sure what log files I should be checking though. If anyone can point me to the correct logs or has some additional troubleshooting advice it would be greatly appreciated.

 

Thanks in advance,

Mike

 

[pricemc1]

Shocked that nobody has been able to reply to this request. Is there something that I need to make more clear to get help for it?

[a_p_]

Can you confirm that you've specified the LDAPS server(s) under "Specific domain controllers"?
"Any domain controller in the domain" will not work with LDAPS.

André

[pricemc1]

I have specified the specific AD domain controllers I want to use for LDAPS lookups. What is so odd is that the only error I can find in the logs and in web interface  is:

"Cannot configure identity source due to Failed to probe provider connectivity [URI: ldaps://ad01dc01.ad1.lab:3268 ]; tenantName [ad1lab.local], userName [[e-mail]] Caused by: Can't contact LDAP server."

I know it is contacting the server though because if I enter invalid credentials for the account to be used for the LDAP lookup I get this error: 

"Cannot configure identity source due to Failed to probe provider connectivity [URI: ldap://ad1.lab:3268 ]; tenantName [ad1lab.local], userName [[e-mail]] Caused by: Invalid credentials."

If it really couldn't contact the LDAP server then it wouldn't be reporting invalid credentials when I try an incorrect password versus when I try the correct password and it reports that it can't contact the LDAP server.

[a_p_]

>>> tenantName [ad1lab.local], userName [[e-mail]]
Can you confirm that this is correct? ad1lab.local vs. ad1.lab?

André

[pricemc1]

"ad1lab.local" is my vsphere SSO domain name that the VCSA is configured for. There are no other VCSA's that are part of the SSO environment.

"ad1.lab" is the Active Directory name the Identity Source will perform LDAPS lookups from. I have a user level service account named "svc_ldaps" that is used for the LDAP lookups from vCenter. This account and its password works just fine when the identity source is configured for standard LDAP instead of LDAPS. It seems to me to me that the values for tenant and username as shown in the error are correct since they would be the same when using LDAP.

[a_p_]

I just saw that you try to connect on port 3268 instead of 3269.

André

[pricemc1]

Sorry when I copied that I had been testing all the various LDAP/LDAPS related ports just to be thorough and I copied in the wrong results. I did get the same results though when using 3269 and 636. See below:

"Cannot configure identity source due to Failed to probe provider connectivity [URI: ldaps://ad01dc01.ad1.lab:3269 ]; tenantName [ad1lab.local], userName [[e-mail]] Caused by: Can't contact LDAP server."

 

"Cannot configure identity source due to Failed to probe provider connectivity [URI: ldaps://ad01dc01.ad1.lab:636 ]; tenantName [ad1lab.local], userName [[e-mail]] Caused by: Can't contact LDAP server."

[RajeevVCP4]

are you able to ping LDAP server from VCSA putty 

[pricemc1]

Yes, I am able to ping the AD DCs directly from the VCSA's CLI that I am attempting to use for VCSA LDAPS lookups. They are in fact on the same subnet as the VCSA.

[Han Welvaarts]

Was struggling with this as well.  For my issue, the solution was to export the certificates from the DCs in base-64 format instead of DER format.  (both formats export as .CER file)

Then I could specify those certificates, without error, as certificates with LDAPs connection to domain controllers.

Also, just for info,  I tested and you need to specify FQDN  for the DCs, and not just the IP address.

[pricemc1]

Thanks for your suggestions. The certs I was using were in Base64 format and I always use the FQDN as well so unfortunately neither of those are the the issue in this case. 

[andrew789]

Hi

I'm not sure this will help but these guys had the same error message - Migrate from Active Directory Integrated Windows Authentication VMware vSphere 7.0 – TheSleepyAdmins

Regards

 

Andrew

 

[pricemc1]

I took a look at the link you provided and I didn't see anything there that mentioned the error I am seeing. Are you sure this was the correct link?

 

[SureshKumarMuth]

Do you have any identity source created already using AD Integrated Windows Authentication ? or this is the first one you are trying to create using ldaps ? If you have IWA type identity source already, then try to remove that and add an ldaps based Identity source.

[pricemc1]

No, there has never been any IWA identity source configured. This is vSphere 8 so IWA is supposed to be deprecated in theory...

[SureshKumarMuth]

I can still see that option in the lab. Anyway, you can check this log file for detailed information 

/var/log/vmware/sso/ssoAdminServer.log

 

 

[pricemc1]

Thanks for the log location. I did a quick test again and then checked the log you mentioned to see what was there. I copied the stuff at the end of the logs into the attached file that shows the errors being reported. Pretty much as shown in the GUI it seems to just indicate that it can't contact the LDAP server.

The only thing I noticed beyond that was an error relating to validating the expiration date of the certificate for the LDAP server I used. I'm not sure why it would have any issues with that though. The server's LDAP cert is issued by a CA that the vCenter server trusts and the expiration date on the server's cert is 3/18/2053 so its clearly not expired. The Root CAs expiration date is 3/5/2063 and the issuing  intermediate CA's expiration is 3/5/2058 so they are not expired either.

 Maybe you can find something in the log that I am missing...

Attachment(s)

ldaps_id_error.txt   47 KB 1 version

[SureshKumarMuth]

Could you please also check this log for related info

vmware-identity-sts.log

[SureshKumarMuth]

I am seeing two errors

1. Java exception , unparseable date. Not sure if this is the contributing factor of this issue, in that case code fix might be needed, only vmware can help here

2023-06-06T02:13:13.851Z ERROR ssoAdminServer[141:pool-2-thread-6] [OpId=lgph43x9-1041449-auto-mbl8-h5:70238925] [com.vmware.identity.interop.ldap.OpenLdapClientLibrary] Error when trying to parse validity date
java.text.ParseException: Unparseable date: "20530319022108Z"

2. After the above exception, I can still see the process goes on and failed with following error

2023-06-06T02:13:13.868Z WARN ssoAdminServer[141:pool-2-thread-6] [OpId=lgph43x9-1041449-auto-mbl8-h5:70238925] [com.vmware.identity.interop.ldap.LdapErrorChecker] Error received by LDAP client: com.vmware.identity.interop.ldap.OpenLdapClientLibrary, error code: -1
2023-06-06T02:13:13.868Z WARN ssoAdminServer[141:pool-2-thread-6] [OpId=lgph43x9-1041449-auto-mbl8-h5:70238925] [com.vmware.identity.idm.server.ServerUtils] cannot bind connection: [ldaps://ad01dc01.ad1.lab:3269, [e-mail]]
2023-06-06T02:13:13.869Z ERROR ssoAdminServer[141:pool-2-thread-6] [OpId=lgph43x9-1041449-auto-mbl8-h5:70238925] [com.vmware.identity.idm.server.ServerUtils] cannot establish ldap connection with URI: [ldaps://ad01dc01.ad1.lab:3269] because [com.vmware.identity.interop.ldap.ServerDownLdapException] with reason [Can't contact LDAP server] therefore will try to attempt to use secondary URIs, if applicable

 

It looks like a connectivity issue but you have already confirmed no firewall in place and the same machine with ldap works fine. Since it is not working for ldaps connection might be something due to the certs but could not derive anything.

Lets check vmware-identity-sts.log to see if we get any messages there.

Can you run nc command on VCSA and check if the connections are fine on ports related to ldaps, just to double check. 

[pricemc1]

I checked the vmware-identity-sts.log log file and there is nothing really showing any errors or anything suspicious. I can't really even find any reference to the ldaps service account I'm using for testing the LDAPS identity source except for when its being used for regular LDAP instead of when I try to use it for testing LDAPS. If you really want I can attach the log file but I don't think there's anything to be found in that one.

I'm not much of a netcat person. Could you give me the exact nc command syntax you would like me to run and I'll run that on the VCSA?

If your looking to verify the port is open I'm pretty sure at some point I ran a curl -v from the VCSA to confirm I could reach 636/3269 from the VCSA to the AD LDAPS host. 

Thanks for your assistance...

[SureshKumarMuth]

If you have already checked using curl -v then thats fine. Since we are running out of options, I can suggest one thing just to isolate the issue. Check if you can quickly deploy a VCSA 7.x and try to add identity source with same settings. This would help us to conclude if this is due to the version changes ... may be based on the result we can find next course of action. You may also raise a vmware support ticket where they can deep dive into logs.

[pricemc1]

I will give that a shot.  Before I try that I think I will also try issuing another server LDAP cert with a shorter expirationation date and testing with that as well. Unfortunately it will take me a few days to get either of these done due to other life activities so if you think of anything else in between please let me know. Once I have those done I'll post my results.

[Ajay Achary]

Share the inputs provided on identity source .

 

From vCSA: share the output. 

openssl s_client -connect <ad-domain>:636 -showcerts

 

Test the ports from vCSA

VCSA to AD ports : IP is for AD 

nc -zv -w 5 <IP> 88

nc -zv -w 5 <IP> 135

nc -zv -w 5 <IP> 389

nc -zv -w 5 <IP> 464

nc -zv -w 5 <IP> 3268

nc -zv -w 5 <IP> 3269

[pricemc1]

Here are the netcat command outputs:

root@vcenter0 [ ~ ]# nc -zv -w 5 ad01dc01.ad1.lab 88
AD01DC01.ad1.lab [10.0.1.11] 88 (kerberos) open

root@vcenter0 [ ~ ]# nc -zv -w 5 ad01dc01.ad1.lab 135
AD01DC01.ad1.lab [10.0.1.11] 135 (epmap) open

root@vcenter0 [ ~ ]# nc -zv -w 5 ad01dc01.ad1.lab 464
AD01DC01.ad1.lab [10.0.1.11] 464 (kpasswd) open

root@vcenter0 [ ~ ]# nc -zv -w 5 ad01dc01.ad1.lab 3268
AD01DC01.ad1.lab [10.0.1.11] 3268 (msft-gc) open

root@vcenter0 [ ~ ]# nc -zv -w 5 ad01dc01.ad1.lab 3269
AD01DC01.ad1.lab [10.0.1.11] 3269 (msft-gc-ssl) open

[pricemc1]

Here's a couple others I may have left out.

root@vcenter0 [ ~ ]# nc -zv -w 5 ad01dc01.ad1.lab 389
AD01DC01.ad1.lab [10.0.1.11] 389 (ldap) open

root@vcenter0 [ ~ ]# nc -zv -w 5 ad01dc01.ad1.lab 636
AD01DC01.ad1.lab [10.0.1.11] 636 (ldaps) open

[Ajay Achary]

Okay..So no issues with ports . Seems I missed checking the first update ....

So you are able to add with ldap. only ldaps is giving issues . 
This is surely with the certs.
Can you get the certs duration for less validity ? Keep it expiring before 2049 and check.

I recall we had an issue with more than 20 yrs cert in 6.5 and that was increased .  

[pricemc1]

Sorry, missed this reply for some reason. I will try this out in the next 24-48 hours and reply with the results. I wouldn't be surprised if that is the issue though. 

 

[pricemc1]

AJ,

You appear to have found the issue. I issued a new certificate with a 10 year expiration length and it worked just fine. Seems like whatever issue they had earlier with cert expiration lengths has either come back in vSphere 8 or was never truly fixed earlier. Perhaps being a VMware employee, you can file a bug report for this?

Thanks very much for your help...

Mike

[Ajay Achary]

Great. Doesn't qualify for a bug....I have to check what they increased it to from 20 yrs. Your certs are 30 and 35 yrs. 
Which means we need to limit ldaps certs under the said limits. 
I will try to get this KB updated >> https://kb.vmware.com/s/article/2041378

[pricemc1]

Well if its not a bug then it definitely needs to be documented... I actually think the issue should have its own KB but it might be nice to have it mentioned in the one you cited as well. Is there a reason you wouldn't consider it a bug? Is there something in the LDAPS RFC or other PKI  standard that prohibits a certificate expiration length beyond 20 years? (granted, I know its bad practice, but it's a lab). If there isn't something like that then I would consider it a bug. The longer expiration certs had worked with everything else in my lab (AD, other apps, etc...) with no issues so that seems to point to this being something specific to vCenter, implying a bug...