How to crash a policy server?

[Pavansai C]

Hi There,

The question seems awkward but we are trying to test the possible hiccups to our api services going live.

We have 6 policy servers in a cluster, CA API Gateway has a service which calls these policy servers for authentication.

Let's say one of the policy servers in a cluster goes wrong or has abnormal behavior or got crashed or something happened,
we need to know what is the impact on the api service.
will the requests get fail or it'll result in high response time and how much time does it take to recover from?

To get answers for all these questions -- we tried killing the process

ps -ef | grep smpolicy
kill -9 <<PID>>   

smpolsrv -status

and made sure that policy server is stopped.

We tried this multiple times like 7 to 8 times but have seen nothing abnormal from api response.

Can someone help us on testing the same. (probably crashing the server or killing the process or something which impacts)


Thanks in advance!!!

[Sung Hoon Kim]

I haven't tried this but how about causing it to HANG for a while:

kill -STOP <PID>
and
kill -CONT <PID>

If the Policy Server process crash, that is actually not too bad as the watchdog will start it up again.
But the worst use case is when the policy server runs with reduced number of threads and slowing down.

You can also try making policy server run with 1 or 2 worker threads to cause bottle neck and see what happens.
At the same time, you can adjust the max connections to cause an outage quicker or delayed.
Turning off AZ Cache may also add more load to policy server.

Based on my experience, customers do run an actual load test (without above negative tweaks) to simulate such conditions.

------------------------------
Support Engineer 5
Broadcom
------------------------------

[Patrick Dussault]

Hi,

You mentioned :

"Let's say one of the policy servers in a cluster goes wrong or has
abnormal behavior or got crashed or something happened, we need to
know what is the impact on the api service."

So from the CA API Gateway, the accessibility to the Policy Server
relies on the Policy Server host and port availability. There are 2
status. The Policy Server port is closed (Policy Server crashed), or
the Policy Server port takes long time to answer (Policy Server
hangs).

As the Policy Server run on linux, you can use Linux tools on
the OS to simulate these 2 states :

To simulate a crash :

# iptables -I INPUT -p tcp –-dport policy_server_port -j DROP

and restablish the service :

# iptables -I INPUT -p tcp –-dport policy_server_port -j ACCEPT

To simulate slow responses :

To set a delay of 100ms on the network interface :

# tc qdisc add dev eth0 root netem delay 100ms

to verify the configuration :

# tc -s qdisc

to remove the configuration

# tc qdisc del dev eth0 root netem

where the eth0 is the network interface.

I hope this helps,

Best Regards,
Patrick