Are you trying to do user authentication for a API, or for a browser based application? APIM (Layer7) is our API management tool, CA Single Sign-On is our tool for browser based user authentication and authorization. Your note indicated it was within the browser so you may not be using the correct tool. CA SSO can setup a forms based authentication which you can customize to your look and feel and send that to the browser when users have to authenticate. It also offers numerous other ways to login such as windows authentication, kerberos, certificates, SAML, Oauth2, and basic authentication.
We are seeing both solutions used together for browser based applications that make REST calls. (Single page apps) SSO authenticates the browser to the app, and then can generate an OIDC token that can be sent on the API requests that go to the APIM server.