Top 10 Siteminder performance tuning

[Anon Anon]

Hi all,

Just wanted to start a thread regarding Siteminder performance tuning. Experts can share their experience regarding the same.

If there is thread already present please let us know.

Thanks,
Jagadeesh.K

[Anon Anon]

Tuning the backend stores has been the biggest bang for the buck for us. Adding multiple banks of directories in the directory object, and ensuring that our directory response times are are under 10ms (queries are under 1ms and updates are ~3ms avg). Having a solid backend ensures the policyserver gets the information it needs to make decisions in a timely fashion.

We also tuned databases for replication times and paid attention to what should be setup in a failover config, and what should be setup in a loadbalancing config at each tier.

Other than that, making sure file descriptors and sockets are set high enough for the loads, and we continually monitor the wait queues and high priority - adjusting the worker threads if required. We also tuned the ASA/PS/Webagent caches - but we did not see huge gains here, once we had the backends performing.


We use Wily with the siteminder powerpack to monitor the policyserver and agents, and find it extremely useful.

[Anon Anon]

Thanks Andrew for your input.

I agree that siteminder will perform better when backend directory response time is quick. Also we extensively used wily for monitoring of infrastructure.

I would also like to know are there any specific parameters for tuning in policy server/web agent, by which siteminder can work better.

Regards,
Jagadeesh.K

[banst03]

I also have to recommend Andrew's suggestion around the backend user directory.

Number of worker threads is also a good start, but be careful. It is easy to assume that a larger number equates to better performance. Using a tool like Wily or our Support tool that Mark wrote (in the Tools and Scripts section) would be useful to get a baseline and see if there is an improvement.

If you think your directory server can handle (most cannot), you can open more connections to the directory server by doing this trick (assuming Solaris OS).

/etc/hosts
10.0.0.1 MyRealLDAPHostname
10.0.0.1 ldap1
10.0.0.1 ldap2
10.0.0.1 ldap3

Then when you define your user directory definition, do not use the IP, use the dummy hostnames in a load balanced configuration. With this simple trick you can triple your throughput to the directory server. But again, I caution, make sure your directory server can handle it. Use a monitoring tool to get real numbers.

-Steve
(As an aside, using the same IP address listed multiple times might work, but there is internal code on when we mark a connection as bad, we remove all IPs. So if you have three IPs listed that are all the same, and have a blip with the directory server, all connections would be marked as bad. Using difference hostnames (which map to the same IP) circumvents this problem.)

[Anon Anon]

Steven is 100 percent correct. Most directories cannot sustain the volumes that people think they can, and it is the most important piece of the puzzle.

We spent years tuning directories, then got CA directory, and have not had issues. We then tuned back all of our worker threads and saw another increase in throughput by decreasing context switching

User store performance is the key to tuning...



Thanks, Andrew

_________________________________
Sent from my BlackBerry handheld.
Envoyé à partir de mon BlackBerry.

From: CA Security Global User Community (Distributed) [mailto:[e-mail]]
Sent: Friday, June 22, 2012 05:56 PM
To: [e-mail] <[e-mail]>
Subject: [CA SiteMinder General Discussion] RE: Top 10 Siteminder performance tuning

I also have to recommend Andrew's suggestion around the backend user directory.

Number of worker threads is also a good start, but be careful. It is easy to assume that a larger number equates to better performance. Using a tool like Wily or our Support tool that Mark wrote (in the Tools and Scripts section) would be useful to get a baseline and see if there is an improvement.

If you think your directory server can handle (most cannot), you can open more connections to the directory server by doing this trick (assuming Solaris OS).

/etc/hosts
10.0.0.1 MyRealLDAPHostname
10.0.0.1 ldap1
10.0.0.1 ldap2
10.0.0.1 ldap3

Then when you define your user directory definition, do not use the IP, use the dummy hostnames in a load balanced configuration. With this simple trick you can triple your throughput to the directory server. But again, I caution, make sure your directory server can handle it. Use a monitoring tool to get real numbers.

-Steve
(As an aside, using the same IP address listed multiple times might work, but there is internal code on when we mark a connection as bad, we remove all IPs. So if you have three IPs listed that are all the same, and have a blip with the directory server, all connections would be marked as bad. Using difference hostnames (which map to the same IP) circumvents this problem.)
Posted by:Steven_Bankowitz
--
CA Communities Message Boards
98378621
[e-mail]
https://communities.ca.com

[Anon Anon]

Hi People

I'm working on a performance issue here in Chile, but I think in this case the User Store is not the problem.

I have a Web Agent on IIS Server and in the same server is installed Siebel Object Manager.
I'm using Windows Authentication Scheme and we have two robusts Active Directory Severs supporting this environment.


Without SiteMinder the login process is about 2,5 seconds, using SiteMinder the login takes about 8 seconds.
I test the connection from SiteMinder to Active Directory using SiteMinder Test Tool, and the responses are the following:

Is Authenticated 0:00:00.034
Is Authorized 0:00:00.007

I think the AD performance is good, but I think the IWA integration could be the problem.

How can I improve the performarnce in the Agent Side in this case??


Thanks for your tips.

Paulo

[SungHoon_Kim]

I would suggest that you use a "header trace + WebAgent Trace + Policy Server Trace + AD Tracing" to identify where the delay is.
If you find any delay, that should appear in the trace logs or you need to look between the trace logs.

For example, if the agent has sent a request to the policy server, has the policy server received the request immediately or had there been any gap when the policy server received the request.
Easier way to track this is if you add TransactionID and SessionID to the Agent Trace and Policy Server Trace log so that you can spot the matching transaction quickly.

There should be a delay somewhere in between the components involved.

You might also want to check if the DNS server is causing any delays.

[Anon Anon]

Hi all,

Making this post active. Anybody who did performance tuning for their environment, you can share info with our community so that others can benefit.

Thanks,
Jagadeesh.K

[Anon Anon]

Hi all,

This is regarding semaphores accumulation during webagent restarts. We encounterd a situation where webagents(6QMR5) on any webservers, gets restarted, LLAWP process should clear all the semphores it created during startup. But webagent was not clearing all the semphores and few were left and after restarting the web server, those semaphores were getting accumulated. It drastically reduces performance. If attention is not paid on this, it will bring down that webserver causing application outage. As i mentioned earlier, we have seen this in 6QMR5 environment. We have to see how it is performing in R12.

Anybody experienced the semaphores issue in R6/R12 environment?

Thanks,
Jagadeesh.K

[deactivated user]

Hi Jagadeesh, We faced with this problem long ago. LLAWP process takes extra few seconds to die, even after apache is dead. So if you start apache when before the semaphores are nt cleared, you will run into issues with Webagent. So, we always use apache restart command(not stop and start), and the sleep between stop and start is increased to 10 seconds, by default I think it is 2 seconds.

In the past, as an additional precaution, we updated our restart script to delete all shared memory during startup and but soon it became a problem when we had multiple instances running using the same binaries.
So today, with sleep time to 10 seconds, I had this issue once(with 12.x agent) in last 6 years. Hope this helps, please let me know if it solved your problem.

[Anon Anon]

Hi SamWalker and Mark,

Thanks for details and explanation. Yes, We already implemented the solution of keeping sleeping time to 10 sec in restart script and issues were resolved. But i just wanted to see if R12 version is improved to handle the issue, rather than having 10 sec as sleeping time in restart scripts. Otherwise CA engineering team can look in future releases.

Thanks,
Jagadeesh.K

[Mark ODonohue]

Hi Jagadeesh

I thought I should add these here, since some of the discussion in the article :
Blog entry on SM Spiral of Dealth.
May be of use.

Usually locating what is the bottleneck for performance by load testing a (test setup ) to breaking point, and then some analysis to figure out what broke, re-configure and then re-run the load test.

Also the PolicyTraceTool (link is to the scripts section of the ca community site) :

http://tiny.cc/SMTraceAnalysisTooll
can be useful for identifying where the bottlenecks are.

Cheers - Mark

[Anon Anon]

Hi,

 

Great post. Can any of you please share your thoughts on Capacity planning like how many agents can connect to one Policy server, so we can plan for number of Policy Servers.

 

Thanks.