This is intended to be a living document. Please prefer editing the document (adding, revising, etc.) over posting comments.
Do not skip steps. The steps are ordered to identify any problems early in the flow and to concentrate efforts on the right area.
Kerberos Troubleshooting Procedure
Perform Kerberos troubleshooting by following the procedure below. It is intended to follow the authentication flow chronologically starting with the configuration and then progressing from the client, to the CA SSO web agent (or CA SSO application gateway), and finally to the CA SSO policy server.
1. Check Configuration
The items to check below are the ones most commonly misconfigured. Please refer to the CA Single Sign-On product documentation for detailed steps to configure Kerberos authentication.
CA Single Sign-On Agent Configuration Object
ACO Option |
ACO Value Format |
Description |
HttpServicePrincipal |
HTTP/www.example.com@EXAMPLE.COM |
This option is used by the web agent when authenticating to the KDC. It is always in the form HTTP/web-server-name@kerberos-realm where web-server-name is the name of the web server (as used by the HTTP user agent), and kerberos-realm is the Kerberos realm. For example, there might be multiple web servers behind a load balancer virtual IP. In that case, you would specify the name of the load balancer rather than a specific server. |
KCCExt |
.kcc |
Extension for the CA SSO Kerberos credential collector, typically .kcc . |
SmpsServicePrincipal |
smps@pserver.example.com |
This option is used by the web agent when delegating authentication credentials to the policy server. It is always in the form smps@policy-server-name where policy-server-name is the name of the CA SSO policy server. As with the HttpServicePrincipal setting, this might be a name shared among multiple policy servers. |
CA Single Sign-On Kerberos Authentication Scheme
Authentication Scheme Option |
Authentication Scheme Value Format |
Description |
Principal Name |
smps/pserver.example.com@EXAMPLE.COM |
This option is used by the policy server when authenticating to the KDC. It is always in the form smps/policy-server-name@kerberos-realm where policy-server-name is the name of the CA SSO policy server and MUST match the value in the ACO's SmpsServicePrincipal option. |
Check that the policy server name as specified in the SmpsServicePrincipal
of the agent configuration object matches the policy server name as specified in the Principal Name
of the authentication scheme.
Check that the SmpsServicePrincipal
format is as described above (similar to an email address) and NOT in Kerberos service principal name format.
2. Check Client
Check HTTP Browser Settings
Web browsers must be configured to attempt integrated (automatic) authentication only to specific servers or domains. The way this is done is different for each browser.
Internet Explorer
Check that the specific web server or the domain is part of the Intranet security zone. This is configured under Internet Options on the Security tab.
Chrome/Chromium
Verify that both the AuthServerWhitelist
and AuthNegotiateDelegateWhitelist
settings are configured. After Google Chrome 86 these settings were renamed to AuthServerAllowlist
and AuthNegotiateDelegateAllowlist.
These setting names are case sensitive.
See https://www.chromium.org/developers/design-documents/http-authentication for more information.
Mozilla Firefox
Verify that both the network.negotiate-auth.trusted-uris
and network.negotiate-auth.delegation-uris
settings are configured.
These setting values do not support wildcard characters like *.
See https://developer.mozilla.org/en-US/docs/Mozilla/Integrated_authentication for more information
curl
Verify that both the --negotiate
and --delegation always
options are used.
See https://curl.haxx.se/docs/manpage.html for more information.
Check HTTP Status Codes
Check that the client receives the expected HTTP status codes. These codes include HTTP 302 redirect when first accessing the Kerberos-protected resource and HTTP 401 after following the redirect to the creds.kcc
Kerberos credential collector URL.
Request for Kerberos-Protected Resource Receives HTTP 302 Response
When the client first accesses a Kerberos-protected resource and the request does not contain a valid CA Single Sign-On session, then the client is redirected to the Kerberos credential collector URL via an HTTP 302 response.
If HTTP 302 is not received, then either the resource is not protected or the client request contains a valid CA Single Sign-On session.
If HTTP 302 is received but does not redirect to the Kerberos credential collector URL, then the resource is not protected by a Kerberos authentication scheme.
Request for creds.kcc Receives HTTP 401 Response
When the HTTP user-agent follows the HTTP 302 redirect to the creds.kcc
page, verify the HTTP response is HTTP 401 with the response header contains WWW-Authenticate: Negotiate
.
If HTTP 500 is received, then there is a problem on the CA SSO web agent (or access gateway). Skip to the next section for help troubleshooting the web agent and then return back here once the client is receiving HTTP 401 responses.
Check for Kerberos HTTP Service Ticket
Check that the client was able to obtain an HTTP service ticket. This is checking that the HTTP user-agent is properly configured for integrated authentication and automatically retrieves an HTTP service ticket to authenticate to the web server.
Checking for Kerberos HTTP Service Ticket on Windows
Use the klist.exe
program to check for an HTTP service ticket, searching the output for the text "HTTP".
> C:\windows\system32\klist.exe | findstr HTTP
Server: HTTP/www.example.com @ EXAMPLE.COM
Be sure to use the Microsoft Windows klist.exe
program and not the Oracle Java klist.exe
program by using the fully-qualified path to klist.exe
under C:\windows\system32
.
Checking for Kerberos HTTP Service Ticket on Linux
Use the klist
program to check for an HTTP service ticket, searching the output for the text "HTTP".
$ klist | grep HTTP
10/23/17 21:30:39 10/24/17 07:30:39 HTTP/www.example.com@
10/23/17 21:30:39 10/24/17 07:30:39 HTTP/www.example.com@EXAMPLE.COM
If the client does not have an HTTP service ticket, then either the HTTP browser is not configured correctly or there is some problem other issue. Check the HTTP browser configuration again and then continue to the next step of monitoring the network traffic.
Check for HTTP Authorization Header
If the HTTP browser is able to successfully retrieve an HTTP service ticket, then the next request for the Kerberos credential collector URL will contain an HTTP Authorization request header. The value of the HTTP Authorization request header will be "Negotiate YII..." where "YII" is the start of the BASE64-encoded Kerberos HTTP ticket.
Use a proxy like Fiddler to capture the HTTP traffic from the browser, especially if the HTTP traffic is encrypted.
Sometimes the developer's tools feature of an HTTP browser does not show all the traffic or all the request/response headers. Use a proxy designed for capturing browser traffic instead.
Verify the browser sends an HTTP request for the Kerberos credential collector containing an HTTP Authorization request header with the value "Negotiate YII...".
Capturing Client Network Traffic
While tools like Fiddler are useful for capturing browser traffic, they do not capture raw network traffic. Use a tool like Wireshark to capture network traffic. Focus on DNS, Kerberos, and HTTP traffic by setting an appropriate capture filter, e.g. "port 53 or port 80 or port 88".
Check DNS
Check that DNS queries are returning A records and not CNAME records. Use tools like "nslookup.exe" on Windows or "dig" on Linux to check DNS. Also, if capturing DNS and Kerberos traffic in Wireshark you might see requests for a Kerberos ticket for the server name returned in a DNS A record rather than the CNAME used to access the web server. Confirm the server name in the Kerberos request from the client matches the expected name.
Using DNS CNAME records complicates the authentication flow. When using a proxy the CNAME record will be used but, when not using a proxy, the A record that the CNAME points to will be used instead. Prefer using DNS A records if possible to reduce complexity and to support troubleshooting using proxy tools like Fiddler.
Check Key Version Number
Check the Kerberos HTTP ticket version number retrieved by the client. It is in TGS response messages and called "kvno" for key version number. Note the number and compare to the key version number in the web agent's keytab file when you start to focus on the web agent side of the architecture.
Client Check Summary
If the client performs all of the following successfully, then move on to troubleshooting the web agent component:
- HTTP browser configured to trust the authentication system for integrated authentication and delegation
- Request for Kerberos-protected resource responds with HTTP 302 redirect to creds.kcc Kerberos credential collector
- First request for creds.kcc page responds with HTTP 401 and "WWW-Authenticate: Negotiate" response header
- Client systems requests and receives expected DNS queries
- HTTP browser successfully retrieves an HTTP service ticket
- HTTP service ticket key version number (KVNO) matches the one in the web agent's keytab file
- Second request for creds.kcc page send "Authorization: Negotiate YII..." request header
3. Check Web Agent
Enable web agent log and trace log in the CA Single Sign-On agent configuration object to help troubleshoot the web agent.
Check Kerberos Configuration File
Check the settings in the Kerberos configuration file. In general Kerberos can locate KDCs (key distribution centers) automatically using DNS SRV records. The critical items to be configured in the Kerberos configuration file are:
default_keytab_name
: The location of the keytab file containing the credentials to perform the initial authentication.
default_client_keytab_name
: The location of the keytab file containing the credentials to perform the initial authentication (set to same value as default_keytab_name
).
default_realm
: The name of the default Kerberos realm (also used as the DNS suffix when locating KDCs via DNS SRV records)
default_ccache_name
: The name of the credential cache (Required on Windows only - the default on Linux and MacOS is OK). This value can be set to FILE:%{TEMP}\krb5cc_%{uid} on Windows.
forwardable
: Set to true
to request forwardable tickets by default.
Check Keytab File
Check that the web agent's keytab file is correct:
- HTTP Principal Name: Check that the name matches the one configured in the CA Single Sign-On agent configuration object's
HttpServicePrincipal
.
- KVNO (Key Version Number): Note the value of the key version number.
Listing Keytabs on Windows with KTPASS.EXE
The contents of a keytab file can be listed using the KTPASS.EXE command with either the "/in" or, as shown below, the "-in" option:
> ktpass -in C:\windows\krbsvc-smwa.keytab
Existing keytab:
Keytab version: 0x502
keysize 74 HTTP/www.example.com@EXAMPLE.COM ptype 1 (KRB5_NT_PRINCIPAL) vno 8 etype 0x1 (DES-CBC-CRC) keylength 8 (0x...)
keysize 74 HTTP/www.example.com@EXAMPLE.COM ptype 1 (KRB5_NT_PRINCIPAL) vno 8 etype 0x3 (DES-CBC-MD5) keylength 8 (0x...)
keysize 82 HTTP/www.example.com@EXAMPLE.COM ptype 1 (KRB5_NT_PRINCIPAL) vno 8 etype 0x17 (RC4-HMAC) keylength 16 (0x...)
keysize 98 HTTP/www.example.com@EXAMPLE.COM ptype 1 (KRB5_NT_PRINCIPAL) vno 8 etype 0x12 (AES256-SHA1) keylength 32 (0x...)
keysize 82 HTTP/www.example.com@EXAMPLE.COM ptype 1 (KRB5_NT_PRINCIPAL) vno 8 etype 0x11 (AES128-SHA1) keylength 16 (0x...)
WARNING: No principal name specified.
Listing Keytabs on Windows with Java KLIST.EXE
The contents of a keytab file can also be listed using Java's KLIST.EXE command.
> "C:\Program Files\Java\jdk1.8.0\bin\klist.exe" -e -k -t C:\Windows\krbsvc-smwa.keytab
Key tab: C:\Windows\krbsvc-smwa.keytab, 5 entries found.
[1] Service principal: HTTP/www.example.com@EXAMPLE.COM
KVNO: 8
Key type: 1
Time stamp: Dec 31, 1969 16:00:00
[2] Service principal: HTTP/www.example.com@EXAMPLE.COM
KVNO: 8
Key type: 3
Time stamp: Dec 31, 1969 16:00:00
[3] Service principal: HTTP/www.example.com@EXAMPLE.COM
KVNO: 8
Key type: 23
Time stamp: Dec 31, 1969 16:00:00
[4] Service principal: HTTP/www.example.com@EXAMPLE.COM
KVNO: 8
Key type: 18
Time stamp: Dec 31, 1969 16:00:00
[5] Service principal: HTTP/www.example.com@EXAMPLE.COM
KVNO: 8
Key type: 17
Time stamp: Dec 31, 1969 16:00:00
Listing Keytabs on Linux with MIT Kerberos klist
The contents of a keytab file can be listed on Linux systems using the MIT Kerberos klist command. If Java is installed, then Java's klist command could also be used as described above.
$ klist -e -k -t krbsvc-smwa.keytab
Keytab name: FILE:krbsvc-smwa.keytab
KVNO Timestamp Principal
---- ----------------- --------------------------------------------------------
8 01/01/70 00:00:00 HTTP/www.example.com@EXAMPLE.COM (des-cbc-crc)
8 01/01/70 00:00:00 HTTP/www.example.com@EXAMPLE.COM (des-cbc-md5)
8 01/01/70 00:00:00 HTTP/www.example.com@EXAMPLE.COM (arcfour-hmac)
8 01/01/70 00:00:00 HTTP/www.example.com@EXAMPLE.COM (aes256-cts-hmac-sha1-96)
8 01/01/70 00:00:00 HTTP/www.example.com@EXAMPLE.COM (aes128-cts-hmac-sha1-96)
Check that the HTTP principal name in the keytab file matches the HttpServiceName setting in the CA Single Sign-On agent configuration object.
Confirm the HttpServiceName setting actually used by the agent by checking the web agent log.
Check Web Agent Authentication to KDC
The web agent will perform Kerberos authentication on the first request for the creds.kcc Kerberos credential collector. Monitor the network traffic with a tool like Wireshark, capturing DNS and Kerberos traffic.
Kerberos errors like KRB5KDC_ERR_PREAUTH_REQUIRED and KRB5KRB_ERR_RESPONSE_TOO_BIG are expected status codes returned by the KDC and typically not a symptom of problems with Kerberos authentication.
Check for successful AS-REQ/AS-REP messages between the web agent machine and the KDC. This phase is called "obtaining initial credentials" where the web agent, using the saved credentials in the keytab file, authenticates to the KDC and acquires a TGT (ticket granting ticket).
Check that the key version number in the AS-REP response message matches the key version number observed on the client system.
If the account that the web agent process runs under has been changed, then stop the web agent, delete the credential cache (C:\windows\temp\krb5cc) and service cache (C:\windows\temp\HTTP) files, and start the web agent again.
Check Policy Server Service Ticket from KDC
At this point, the web agent should be authenticated to the KDC and the client should be sending Kerberos credentials in the HTTP request to the Kerberos credential collector. When the web agent receives the HTTP request it will ask the KDC for a service ticket using TGS-REQ/TGS-REP messages.
Note the service ticket key version number returned by the KDC and ensure it matches the key version number in the policy server keytab.
Check the network capture for TGS-REQ/TGS-REP messages for the "smps" ticket.
If no TGS-REQ message is observed, then check the packet capture for DNS traffic and confirm that the name in the SmpsServicePrincipal is resolved by DNS.
If no TGS-REQ message is observed, then check the web agent account on the KDC (usually Active Directory) and verify the account is trusted for delegation. Configure unconstrained delegation using "Trust this user for delegation to any service (Kerberos only)" first while troubleshooting. If that works, try to configure constrained delegation using "Trust this user for delegation to specified services only" and add the "smps" account/service to the list.
Web Agent Check Summary
If the web agent performs all of the following successfully, then move on to troubleshooting the policy server component:
- Kerberos configuration file specifies default_keytab_name, default_realm, and default_ccache_name (on Windows)
- Web agent authenticates to KDC with AS-REQ/AS-REP messages
- Web agent retrieves "smps" service ticket from KDC with TGS-REQ/TGS-REP messages
4. Check Policy Server
Enable the CA Single Sign-On profile log (also known as the "trace" log) to help troubleshoot Kerberos authentication problems.
Check Kerberos Configuration File
Check the Kerberos configuration file as described above for the web agent.
Check Keytab File
Check the policy server keytab file as described above for the web agent.
Check that these principal names match exactly:
- Policy server keytab
- Authentication scheme principal name
Check than the server portion of the principal name matches the agent configuration object's SmpsServicePrincipal setting (smps@pserver.example.com
format)
Check Policy Server Authentication to KDC
The policy server will authenticate to the KDC using the credentials stored in the keytab file the first time it receives a request from the web agent containing the "smps" service ticket.
Check for successful AS-REQ/AS-REP messages between the policy server machine and the KDC. This phase is called "obtaining initial credentials" where the policy server, using the saved credentials in the keytab file, authenticates to the KDC and acquires a TGT (ticket granting ticket).
Check that the key version number in the AS-REP message matches the key version number in the TGS-REP message sent to the web agent when requesting an "smps" service ticket.
Check Client User is Authenticated
Check the CA Single Sign-On policy server profile (trace) log to see if the user was successfully authenticated.
If user is not authenticated, check that the authentication scheme's User DN Lookup setting is correct.
If user is authenticated, then the web agent should set a session cookie, SMSESSION, in the HTTP 302 response sent to the client redirecting the browser back to the Kerberos-protected resource.
Check Client User is Authorized
Check that the client user can access the Kerberos-protected resource.
If the client user is authenticated and receives a CA Single Sign-On session cookie, but still cannot access the Kerberos-protected resource, then check that the user account is included in the CA Single Sign-On policy granting access to the resource.
Policy Server Check Summary
If the policy server performs all of the following successfully, then the user should be authenticated with Kerberos:
- Kerberos configuration file specifies default_keytab_name, default_realm, and default_ccache_name (on Windows)
- Policy server authenticates to KDC with AS-REQ/AS-REP messages
- User is disambiguated in user directory associated to CA Single Sign-On domain
Frequently Asked Questions (FAQ)
If I have multiple servers, e.g. multiple policy servers, do I need to create an account (and keytab) per server?
No. You generally only need to create an account and keytab file per server type even when there are multiple servers. One of the reasons why is the way that Kerberos authentication is configured in CA Single Sign-On. For example, the authentication scheme configuration specifies a single SPN for the "smps" principal and this configuration is shared among all policy servers so only a single account (and keytab) can be used.
Epilogue
If Kerberos authentication is still not working after trying the troubleshooting steps above, then open an issue with Support at https://support.broadcom.com. As new troubleshooting steps are identified, they will be added to this document.