Venkat
Difficult to suggest with less info.
I initially thought it could be URL encoding-decoding issue. However since you've suggest that the Custom Auth Scheme works fine with other SAML/WSFED SSO, that is unlikely the case. When you say Custom Auth Scheme what is it, a HTML Custom Login Page?
Have you checked the WA-WAOP Trace logs on IdP Side to see which component is throwing the HTTP 500 error. If the error is thrown by SiteMinder on IdP, the WA or WAOP trace logs should have some additional indicators.
Regards
Hubert