I have my policy server and SPS both on Red Hat Linux Machine. My Active Directory is configured as a user directory using the LDAP namespace.
If I wanted to use IWA auth scheme on the SPS for a particular resource, is it possible, with the SPS being on Red Hat? Or would I absolutely have to configure a windows IIS server or SPS on IIS?
What about your Login page server?
For IWA authentication, it is IIS that does the authentication.
SPS on windows is able to do this authentication, substituting for IIS.
I was wondering if SPS on Linux will be able to do the same?
The SPS system must be joined to the AD domain, so IWA with SPS is windows only.
If your SPS is on Windows, then you can use Integrated Windows Authentication and it would do NTLM handshake.
If your SPS is on Linux, then you can setup kerberos to do the same andit would do Negotiate/Kerberos handshake.
Configure CA SiteMinder® SPS to Support Integrated Windows Authentication
For validating the Kerberos ticket the Siteminder Proxy Engine service should run using a domain user and not a local user.
That user should have SPN defined as well (using setspn command).
The following KB can help:
Kerberos Authentication problems – Service Principal Name (SPN) issues – Part 3 | Ask the Directory Services Team