LDAP login slow

[Chris Perry]

I created a new hub for logins to take some load off our primary hub. This new server is in the same subnet as the primary hub. The specs are less than the primary hub but it's only purpose is IM logins so I didn't think it would need much. I've noticed that login times on this new hub are a lot longer than the primary hub. To give you an idea on the primary hub it takes about 19ms to login according to the logs whereas the new server takes 6000ms to login. I've also noticed in the logs on the new server that it runs a command (nim_ldap_get_connection) which seems to fail but the server can run ldap queries successfully. Any ideas?

[Gene Howard]

what is the version of HUB on each?
Does the problem happen only with LDAP users?
Did you check your latency times from the new hub to the LDAP server?

------------------------------
Gene Howard
Principal Support Engineer
Broadcom
------------------------------

[Chris Perry]

We are using 9.30.

Yes it still happens with local accounts. 

Ping response is under 1ms unless there is another check you'd like me to run?

[Gene Howard]

update the controller and hub on the primary and this secondary hub to the lastest HOTFIX for robot and hub from below:
https://support.broadcom.com/external/content/release-announcements/CA-Unified-Infrastructure-Management-Hotfix-Index/7233

------------------------------
Gene Howard
Principal Support Engineer
Broadcom
------------------------------

[Chris Perry]

I don't see anything for the hub in that link? We are currently running 9.3.

Also when I look at the notes for robot_update I don't see any updates around logins. Do you have any documentation around why this would fix login issues?

[Gene Howard]

the robot_update has some performance improvements in it, So I would still suggest applying it.
also take a look at:
Enabling LDAP configuration slows down hub performance (Knowledge Base Articles - 6451)
https://ca-broadcom.wolkenservicedesk.com/external/article?articleId=6451

------------------------------
Gene Howard
Principal Support Engineer
Broadcom
------------------------------

[Chris Perry]

I've applied the robot update along with the hub settings. It seems to take the same amount of time. Any other ideas?

[Gregg Stillwell]

Hi Chris,

If nimsoft users have a delay as well as LDAP users I would guess hub to hub communication. Typically with an issue like this I would check Windows event logs or /var/logs/messages. Verify firewalls, antivirus, SSL and queues/tunnels. Try to isolate UIM as the issue. Are there any errors on LDAP server? can you install open LDAP client or have one(windows) to test connection. You may need issue if have not opened one already

[Gene Howard]

I would suggest setting your hub loglevel to 5 and logsize to 95000 on both hubs and do a test on both hubs.
compare the logs and see where the slow down is.
Without detailed logs such as the above not sure we can provide insight to a performance issue such as this.

------------------------------
Gene Howard
Principal Support Engineer
Broadcom
------------------------------

[Chris Perry]

Thank you for the updates. If I upload the logs is this something only support sees or will everyone in the community see?

[Gene Howard]

anything uploaded here is open to anyone with a community login

------------------------------
Gene Howard
Principal Support Engineer
Broadcom
------------------------------

[Gregg Stillwell]

Be careful to black out sensitive information per your companies policies.

[Chris Perry]

hub01 is our primary hub whereas hub02 is the authentication hub. cper is my user.

Attachment(s)

hub02.txt   5 KB 1 version

hub01.txt   4 KB 1 version

[Gene Howard]

so looks like something is not configured correctly or there is a problem as you are getting a connection failure:
Jun 16 08:50:43:531 [6164] 0 hub: (nim_ldap_get_connection): LDAP server spec 'SERVERNAME.DOMAIN.COM' failed (secure=2)
Jun 16 08:50:43:541 [6164] 3 hub: login [LDAP] - basic login took 6023 ms

whereas the working one is not:
Jun 16 08:50:21:637 [5056] 3 hub: (nim_ldap_get_connection): successful contact with LDAP server 'SERVERNAME.DOMAIN.COM', secure=2
Jun 16 08:50:21:644 [22084] 3 hub: Sent heartbeat on queue route 't_24'
Jun 16 08:50:21:647 [5056] 3 hub: login [LDAP] - basic login took 19 ms

are both hub.cfg exactly the same for the LDAP section?
since you are using DNS names both hubs resolve to the same IP address for the LDAP server?

------------------------------
Gene Howard
Principal Support Engineer
Broadcom
------------------------------

[Chris Perry]

Yes they are both configured the same. I used notepad++ to run a compare against them. DNS resolve comes back with the same results.

[Chris Perry]

Here is the config:

<server>
active = yes
host = SERVERNAME.DOMAIN.com
template = Active Directory
hub_user = DOMAIN\USER
hub_pass = XXXXXXXXXXX
base = OU=GROUP,DC=DOMAIN,DC=com
user_base = DC=DOMAIN,DC=com
use_ssl = yes
proxy = 0
timeout = 10
retries_count = 3
auth_sequence = 0
</server>
<templates>
<Active Directory>
tag = ad
filter_group = (objectClass=group)
filter_user = (&(objectClass=person)(|(userPrincipalName=$loginname)(sAMAccountName=$loginname)))
exclude_regexp = /(@)|(\\)|(^(C|c)(N|n)=)/
ldap_dn_regexp = /^(C|c)(N|n)=/
attr_grp_name = name
attr_grp_member_name = member
attr_usr_firstname = givenName
attr_usr_lastname = sn
attr_usr_mail = mail
attr_usr_cellphone = mobile
attr_usr_phone = telephoneNumber
attr_usr_www = wWWHomePage
attr_usr_office = physicalDeliveryOfficeName
attr_usr_company = company
attr_usr_title = title
attr_usr_department = department
attr_usr_description = description
attr_usr_name = displayName
attr_usr_id = userPrincipalName
attr_usr_member_of = memberOf
attr_usr_restrict_view = restrictViewToUserAssets
format = $username
lookup = no
paging = yes
member_lookup_reverse = yes
</Active Directory>

[Chris Perry]

I removed the config I added in https://ca-broadcom.wolkenservicedesk.com/external/article?articleId=6451 when I found out it wasn't resolving my issue.

[Gene Howard]

two things to try.
1) change the host entry and an IP address.
2) try disabling SSL.

------------------------------
Gene Howard
Principal Support Engineer
Broadcom
------------------------------

[Chris Perry]

We may have figured it out. I tried a different domain controller and the login time was reduced significantly. The difference between the two logs are as follows. I'm wondering if this has something to do with your second point around SSL? I will be testing disabling SSL shortly.

New DC:
(nim_ldap_get_connection): successful contact with LDAP server 'SERVERNAME.DOMAIN.com', secure=2

OLD DC:
(nim_ldap_get_connection): LDAP server spec 'SERVERNAME.DOMAIN.COM' failed (secure=2)

[Gene Howard]

it could be or a white list of some type.
I am glad to hear you are making progress

------------------------------
Gene Howard
Principal Support Engineer
Broadcom
------------------------------

[Chris Perry]

Disabling SSL does not make a difference.

[Gene Howard]

you would need to check with your AD admin to understand what is different between the domain controllers.

------------------------------
Gene Howard
Principal Support Engineer
Broadcom
------------------------------

[Chris Perry]

What is Nimsoft requesting from LDAP when it makes this call? --> (nim_ldap_get_connection) 

[Gene Howard]

I believe it is just doing a simple LDAP open bind call using the open LDAP libraries based on the configurations in the .cfg for the hub.

------------------------------
Gene Howard
Principal Support Engineer
Broadcom
------------------------------