If Enterprise Management is connecting to an LDAP user store over SSL, it might experience a slower connection than when connecting to a non-SSL user store. The following tech doc was written by an Identity Manager engineer, it explains why the issue occurs and provides steps to improve the connection by enabling LDAP connection pooling for SSL. The same steps can be applied for Enterprise Management since both products are built upon the same framework.
How to enable LDAP connection pooling over SSL to gain increased performance for Identity Manager.