I have a scenario where internal users should be authenticated against an ldap server (not internal identity provider) and external user against an external SQL Server(Password in the database stored in MD5 format)
Can the OTK be modified to accommodate both and differentiate between external and internal users?
OTK can be configured to use an alternate IDP (ie: LDAP). I have included the doc below for OTK 4.2 and will vary slightly on earlier versions.
By default it will use only one of the available IDPs. Each in it's own 'All assertions must evaluate to true' branch. If you want multiple IDP's to be used I suppose you could just wrap those in an 'At least one assertion' branch. You would be responsible for getting user attributes to populate the needed variables as well. You may want to have a some logic to identify between internal and external users, maybe something like the restrict by IP assertion.(Restrict Access to IP Address Range Assertion - CA API Gateway - 9.3 - CA Technologies Documentation )
Otherwise if the same user name exists in both IDP's (ie: Sshyam) it would first try to authenticate against your internal LDAP, then fall over the the external SQL server. A concern only for the extra overhead. If you do a check for IP first then you can update the logic to only go to the correct IDP.
Finally, we have a 'Generate Security Hash' assertion (Generate Security Hash Assertion - CA API Gateway - 9.3 - CA Technologies Documentation ) that can be used for the password hash.
Support Optional Authentication Mechanisms - CA API Management OAuth Toolkit - 4.2 - CA Technologies Documentation