vCenter

 View Only
Expand all | Collapse all

Need help troubleshooting LDAPS configuration on vCenter 8

  • 1.  Need help troubleshooting LDAPS configuration on vCenter 8

    Posted Mar 28, 2023 03:35 PM

    I am trying to configure an identity source in vCenter 8 using LDAPS with Active Directory on a new vCenter 8 implementation. I have been able to do this successfully many times in the past on vCenter 7 instances so I'm familiar with the procedure and requirements. This is my first time doing it though on a vCenter 8 VCSA, but nothing really looks different than how it was done in earlier versions.  Unfortunately, I seem to be running into some issues when I try to add the identity source using LDAPS. Adding it using standard LDAP works fine though.  I am receiving the following error when trying to configure the identity source for LDAPs though:

    "Cannot configure identity source due to Failed to probe provider connectivity [URI: ldaps://domainl ]; tenantName [vsphere.local], userName [User] Caused by: Can’t contact LDAP server."

    For troubleshooting I have done the following:

    • I was able to contact and query the LDAP servers (Windows Server 2022 DCs) using 'ldp.exe' to test LDAPS from Windows clients that I tested from.  I was able to bind and authenticate using the domain account credentials I specified in the identity source configuration.
    • I was able to confirm that the LDAPS servers are presenting the correct certificate by using Openssl to display the certificates being presented on port 636/3269.
    • I was also able to verify network connectivity and proper name resolution from the VCSA to the LDAPs servers from the VCSA CLI using 'ping', 'dig', and 'nslookup'. Both forward and reverse lookups seem to be fine from the VCSA and Windows servers. All of these systems are on the same network. There is no router or firewall between any of them (other than OS level firewalls built into Windows and VCSA).
    • When I ran 'ldapsearch' command from the VCSA's CLI it returns the no errors that indicate LDAPS is not working when being queried.
    • I confirmed that the CA certificates for the CAs that issued the LDAPS certificates to the Windows servers have been added to the trusted list in vCenter.

    I'm thinking my next troubleshooting step should be to review any relevant VCSA log files that would have detailed information on what is going on when I try to add the LDAPS identity source. I'm not sure what log files I should be checking though. If anyone can point me to the correct logs or has some additional troubleshooting advice it would be greatly appreciated.

     

    Thanks in advance,

    Mike

     



  • 2.  RE: Need help troubleshooting LDAPS configuration on vCenter 8

    Posted Jun 03, 2023 04:25 PM

    Shocked that nobody has been able to reply to this request. Is there something that I need to make more clear to get help for it?



  • 3.  RE: Need help troubleshooting LDAPS configuration on vCenter 8

    Posted Jun 03, 2023 06:47 PM

    Can you confirm that you've specified the LDAPS server(s) under "Specific domain controllers"?
    "Any domain controller in the domain" will not work with LDAPS.

    André



  • 4.  RE: Need help troubleshooting LDAPS configuration on vCenter 8

    Posted Jun 04, 2023 04:57 PM

    I have specified the specific AD domain controllers I want to use for LDAPS lookups. What is so odd is that the only error I can find in the logs and in web interface  is:

    "Cannot configure identity source due to Failed to probe provider connectivity [URI: ldaps://ad01dc01.ad1.lab:3268 ]; tenantName [ad1lab.local], userName [svc_ldaps@ad1.lab] Caused by: Can't contact LDAP server."

    I know it is contacting the server though because if I enter invalid credentials for the account to be used for the LDAP lookup I get this error: 

    "Cannot configure identity source due to Failed to probe provider connectivity [URI: ldap://ad1.lab:3268 ]; tenantName [ad1lab.local], userName [svc_ldaps@ad1.lab] Caused by: Invalid credentials."

    If it really couldn't contact the LDAP server then it wouldn't be reporting invalid credentials when I try an incorrect password versus when I try the correct password and it reports that it can't contact the LDAP server.



  • 5.  RE: Need help troubleshooting LDAPS configuration on vCenter 8

    Posted Jun 04, 2023 05:04 PM

    >>> tenantName [ad1lab.local], userName [svc_ldaps@ad1.lab]
    Can you confirm that this is correct? ad1lab.local vs. ad1.lab?

    André



  • 6.  RE: Need help troubleshooting LDAPS configuration on vCenter 8

    Posted Jun 04, 2023 05:11 PM

    "ad1lab.local" is my vsphere SSO domain name that the VCSA is configured for. There are no other VCSA's that are part of the SSO environment.

    "ad1.lab" is the Active Directory name the Identity Source will perform LDAPS lookups from. I have a user level service account named "svc_ldaps" that is used for the LDAP lookups from vCenter. This account and its password works just fine when the identity source is configured for standard LDAP instead of LDAPS. It seems to me to me that the values for tenant and username as shown in the error are correct since they would be the same when using LDAP.



  • 7.  RE: Need help troubleshooting LDAPS configuration on vCenter 8

    Posted Jun 04, 2023 05:23 PM

    I just saw that you try to connect on port 3268 instead of 3269.

    André



  • 8.  RE: Need help troubleshooting LDAPS configuration on vCenter 8

    Posted Jun 04, 2023 07:31 PM

    Sorry when I copied that I had been testing all the various LDAP/LDAPS related ports just to be thorough and I copied in the wrong results. I did get the same results though when using 3269 and 636. See below:

    "Cannot configure identity source due to Failed to probe provider connectivity [URI: ldaps://ad01dc01.ad1.lab:3269 ]; tenantName [ad1lab.local], userName [svc_ldaps@ad1.lab] Caused by: Can't contact LDAP server."

     

    "Cannot configure identity source due to Failed to probe provider connectivity [URI: ldaps://ad01dc01.ad1.lab:636 ]; tenantName [ad1lab.local], userName [svc_ldaps@ad1.lab] Caused by: Can't contact LDAP server."



  • 9.  RE: Need help troubleshooting LDAPS configuration on vCenter 8

    Posted Jun 04, 2023 02:45 AM

    are you able to ping LDAP server from VCSA putty 



  • 10.  RE: Need help troubleshooting LDAPS configuration on vCenter 8

    Posted Jun 04, 2023 04:27 PM

    Yes, I am able to ping the AD DCs directly from the VCSA's CLI that I am attempting to use for VCSA LDAPS lookups. They are in fact on the same subnet as the VCSA.



  • 11.  RE: Need help troubleshooting LDAPS configuration on vCenter 8

    Posted Jun 05, 2023 11:46 AM

    Was struggling with this as well.  For my issue, the solution was to export the certificates from the DCs in base-64 format instead of DER format.  (both formats export as .CER file)

    Then I could specify those certificates, without error, as certificates with LDAPs connection to domain controllers.

    Also, just for info,  I tested and you need to specify FQDN  for the DCs, and not just the IP address.



  • 12.  RE: Need help troubleshooting LDAPS configuration on vCenter 8

    Posted Jun 05, 2023 10:22 PM

    Thanks for your suggestions. The certs I was using were in Base64 format and I always use the FQDN as well so unfortunately neither of those are the the issue in this case. 



  • 13.  RE: Need help troubleshooting LDAPS configuration on vCenter 8

    Posted Jun 05, 2023 02:50 PM

    Hi

    I'm not sure this will help but these guys had the same error message - Migrate from Active Directory Integrated Windows Authentication VMware vSphere 7.0 – TheSleepyAdmins

    Regards

     

    Andrew

     



  • 14.  RE: Need help troubleshooting LDAPS configuration on vCenter 8

    Posted Jun 05, 2023 10:23 PM

    I took a look at the link you provided and I didn't see anything there that mentioned the error I am seeing. Are you sure this was the correct link?

     



  • 15.  RE: Need help troubleshooting LDAPS configuration on vCenter 8

    Posted Jun 06, 2023 01:34 AM

    Do you have any identity source created already using AD Integrated Windows Authentication ? or this is the first one you are trying to create using ldaps ? If you have IWA type identity source already, then try to remove that and add an ldaps based Identity source.



  • 16.  RE: Need help troubleshooting LDAPS configuration on vCenter 8

    Posted Jun 06, 2023 01:36 AM

    No, there has never been any IWA identity source configured. This is vSphere 8 so IWA is supposed to be deprecated in theory...



  • 17.  RE: Need help troubleshooting LDAPS configuration on vCenter 8

    Posted Jun 06, 2023 02:02 AM

    I can still see that option in the lab. Anyway, you can check this log file for detailed information 

    /var/log/vmware/sso/ssoAdminServer.log

     

     



  • 18.  RE: Need help troubleshooting LDAPS configuration on vCenter 8

    Posted Jun 06, 2023 02:36 AM
      |   view attached

    Thanks for the log location. I did a quick test again and then checked the log you mentioned to see what was there. I copied the stuff at the end of the logs into the attached file that shows the errors being reported. Pretty much as shown in the GUI it seems to just indicate that it can't contact the LDAP server.

    The only thing I noticed beyond that was an error relating to validating the expiration date of the certificate for the LDAP server I used. I'm not sure why it would have any issues with that though. The server's LDAP cert is issued by a CA that the vCenter server trusts and the expiration date on the server's cert is 3/18/2053 so its clearly not expired. The Root CAs expiration date is 3/5/2063 and the issuing  intermediate CA's expiration is 3/5/2058 so they are not expired either.

     Maybe you can find something in the log that I am missing...

    Attachment(s)

    txt
    ldaps_id_error.txt   47 KB 1 version


  • 19.  RE: Need help troubleshooting LDAPS configuration on vCenter 8

    Posted Jun 06, 2023 03:07 AM

    Could you please also check this log for related info

    vmware-identity-sts.log



  • 20.  RE: Need help troubleshooting LDAPS configuration on vCenter 8

    Posted Jun 06, 2023 03:13 AM

    I am seeing two errors

    1. Java exception , unparseable date. Not sure if this is the contributing factor of this issue, in that case code fix might be needed, only vmware can help here

    2023-06-06T02:13:13.851Z ERROR ssoAdminServer[141:pool-2-thread-6] [OpId=lgph43x9-1041449-auto-mbl8-h5:70238925] [com.vmware.identity.interop.ldap.OpenLdapClientLibrary] Error when trying to parse validity date
    java.text.ParseException: Unparseable date: "20530319022108Z"

    2. After the above exception, I can still see the process goes on and failed with following error

    2023-06-06T02:13:13.868Z WARN ssoAdminServer[141:pool-2-thread-6] [OpId=lgph43x9-1041449-auto-mbl8-h5:70238925] [com.vmware.identity.interop.ldap.LdapErrorChecker] Error received by LDAP client: com.vmware.identity.interop.ldap.OpenLdapClientLibrary, error code: -1
    2023-06-06T02:13:13.868Z WARN ssoAdminServer[141:pool-2-thread-6] [OpId=lgph43x9-1041449-auto-mbl8-h5:70238925] [com.vmware.identity.idm.server.ServerUtils] cannot bind connection: [ldaps://ad01dc01.ad1.lab:3269, svc_ldaps@ad1.lab]
    2023-06-06T02:13:13.869Z ERROR ssoAdminServer[141:pool-2-thread-6] [OpId=lgph43x9-1041449-auto-mbl8-h5:70238925] [com.vmware.identity.idm.server.ServerUtils] cannot establish ldap connection with URI: [ldaps://ad01dc01.ad1.lab:3269] because [com.vmware.identity.interop.ldap.ServerDownLdapException] with reason [Can't contact LDAP server] therefore will try to attempt to use secondary URIs, if applicable

     

    It looks like a connectivity issue but you have already confirmed no firewall in place and the same machine with ldap works fine. Since it is not working for ldaps connection might be something due to the certs but could not derive anything.

    Lets check vmware-identity-sts.log to see if we get any messages there.

    Can you run nc command on VCSA and check if the connections are fine on ports related to ldaps, just to double check. 



  • 21.  RE: Need help troubleshooting LDAPS configuration on vCenter 8

    Posted Jun 06, 2023 03:26 AM

    I checked the vmware-identity-sts.log log file and there is nothing really showing any errors or anything suspicious. I can't really even find any reference to the ldaps service account I'm using for testing the LDAPS identity source except for when its being used for regular LDAP instead of when I try to use it for testing LDAPS. If you really want I can attach the log file but I don't think there's anything to be found in that one.

    I'm not much of a netcat person. Could you give me the exact nc command syntax you would like me to run and I'll run that on the VCSA?

    If your looking to verify the port is open I'm pretty sure at some point I ran a curl -v from the VCSA to confirm I could reach 636/3269 from the VCSA to the AD LDAPS host. 

    Thanks for your assistance...



  • 22.  RE: Need help troubleshooting LDAPS configuration on vCenter 8

    Posted Jun 06, 2023 04:08 AM

    If you have already checked using curl -v then thats fine. Since we are running out of options, I can suggest one thing just to isolate the issue. Check if you can quickly deploy a VCSA 7.x and try to add identity source with same settings. This would help us to conclude if this is due to the version changes ... may be based on the result we can find next course of action. You may also raise a vmware support ticket where they can deep dive into logs.



  • 23.  RE: Need help troubleshooting LDAPS configuration on vCenter 8

    Posted Jun 06, 2023 10:43 AM

    I will give that a shot.  Before I try that I think I will also try issuing another server LDAP cert with a shorter expirationation date and testing with that as well. Unfortunately it will take me a few days to get either of these done due to other life activities so if you think of anything else in between please let me know. Once I have those done I'll post my results.



  • 24.  RE: Need help troubleshooting LDAPS configuration on vCenter 8

    Broadcom Employee
    Posted Jun 06, 2023 03:26 PM

    Share the inputs provided on identity source .

     

    From vCSA: share the output. 

    openssl s_client -connect <ad-domain>:636 -showcerts

     

    Test the ports from vCSA

    VCSA to AD ports : IP is for AD 

    nc -zv -w 5 <IP> 88

    nc -zv -w 5 <IP> 135

    nc -zv -w 5 <IP> 389

    nc -zv -w 5 <IP> 464

    nc -zv -w 5 <IP> 3268

    nc -zv -w 5 <IP> 3269



  • 25.  RE: Need help troubleshooting LDAPS configuration on vCenter 8

    Posted Jun 06, 2023 05:19 PM

    Here are the netcat command outputs:

    root@vcenter0 [ ~ ]# nc -zv -w 5 ad01dc01.ad1.lab 88
    AD01DC01.ad1.lab [10.0.1.11] 88 (kerberos) open

    root@vcenter0 [ ~ ]# nc -zv -w 5 ad01dc01.ad1.lab 135
    AD01DC01.ad1.lab [10.0.1.11] 135 (epmap) open

    root@vcenter0 [ ~ ]# nc -zv -w 5 ad01dc01.ad1.lab 464
    AD01DC01.ad1.lab [10.0.1.11] 464 (kpasswd) open

    root@vcenter0 [ ~ ]# nc -zv -w 5 ad01dc01.ad1.lab 3268
    AD01DC01.ad1.lab [10.0.1.11] 3268 (msft-gc) open

    root@vcenter0 [ ~ ]# nc -zv -w 5 ad01dc01.ad1.lab 3269
    AD01DC01.ad1.lab [10.0.1.11] 3269 (msft-gc-ssl) open



  • 26.  RE: Need help troubleshooting LDAPS configuration on vCenter 8

    Posted Jun 06, 2023 06:07 PM

    Here's a couple others I may have left out.

    root@vcenter0 [ ~ ]# nc -zv -w 5 ad01dc01.ad1.lab 389
    AD01DC01.ad1.lab [10.0.1.11] 389 (ldap) open


    root@vcenter0 [ ~ ]# nc -zv -w 5 ad01dc01.ad1.lab 636
    AD01DC01.ad1.lab [10.0.1.11] 636 (ldaps) open



  • 27.  RE: Need help troubleshooting LDAPS configuration on vCenter 8
    Best Answer

    Broadcom Employee
    Posted Jun 07, 2023 01:33 AM

    Okay..So no issues with ports . Seems I missed checking the first update ....

    So you are able to add with ldap. only ldaps is giving issues . 
    This is surely with the certs.
    Can you get the certs duration for less validity ? Keep it expiring before 2049 and check.

    I recall we had an issue with more than 20 yrs cert in 6.5 and that was increased .  



  • 28.  RE: Need help troubleshooting LDAPS configuration on vCenter 8

    Posted Jun 08, 2023 02:28 PM

    Sorry, missed this reply for some reason. I will try this out in the next 24-48 hours and reply with the results. I wouldn't be surprised if that is the issue though. 

     



  • 29.  RE: Need help troubleshooting LDAPS configuration on vCenter 8

    Posted Jun 09, 2023 12:49 AM

    AJ,

    You appear to have found the issue. I issued a new certificate with a 10 year expiration length and it worked just fine. Seems like whatever issue they had earlier with cert expiration lengths has either come back in vSphere 8 or was never truly fixed earlier. Perhaps being a VMware employee, you can file a bug report for this?

    Thanks very much for your help...

    Mike



  • 30.  RE: Need help troubleshooting LDAPS configuration on vCenter 8

    Broadcom Employee
    Posted Jun 09, 2023 01:14 AM

    Great. Doesn't qualify for a bug....I have to check what they increased it to from 20 yrs. Your certs are 30 and 35 yrs. 
    Which means we need to limit ldaps certs under the said limits. 
    I will try to get this KB updated >> https://kb.vmware.com/s/article/2041378



  • 31.  RE: Need help troubleshooting LDAPS configuration on vCenter 8

    Posted Jun 09, 2023 01:23 AM

    Well if its not a bug then it definitely needs to be documented... I actually think the issue should have its own KB but it might be nice to have it mentioned in the one you cited as well. Is there a reason you wouldn't consider it a bug? Is there something in the LDAPS RFC or other PKI  standard that prohibits a certificate expiration length beyond 20 years? (granted, I know its bad practice, but it's a lab). If there isn't something like that then I would consider it a bug. The longer expiration certs had worked with everything else in my lab (AD, other apps, etc...) with no issues so that seems to point to this being something specific to vCenter, implying a bug...



  • 32.  RE: Need help troubleshooting LDAPS configuration on vCenter 8

    Broadcom Employee
    Posted Jun 09, 2023 07:07 AM

    KB definitely needs an update and documented. I will try to reach the contacts to get it updated. 
    Well in short for me a bug is something which doesn't work the way it is coded or developed... (Could be a product limitation) .
    Here until 6.5 version some update they had the ldaps cert support only for 20 yrs and increased it later. But I don't recall what is was increased to.. ( I think may be less than 25 or 30 yrs). 



  • 33.  RE: Need help troubleshooting LDAPS configuration on vCenter 8

    Posted Jun 06, 2023 05:25 PM
      |   view attached

    Attached is a screen shot of the identity source config.