ICAP Queue Issues

[Arpit Dave]

Hi Team,

One of my customer see sudden spike in ICAP Queue and they were not able to access internet. When i disable the CPL Layer for ICAP and Enable its start working. Can anyone please suggest what could be RCA for this. how i can check and prevent in future. 

Thanks
Arpit

[Matthias Geiser]

I guess the cause is some "hanging" ICAP connections. I.e. objects that are never downloaded completely and block an ICAP connection during that time. Check the "current connections" on the CAS. I assume there are quite a few connections in the receiving state with only a few bytes transmitted. Check if you can block these requests in your proxy ruleset or if you can do an ICAP exemption for these.

Best regards, Matthias

[Perry Crabtree]

Hello Arpit,

I can echo what Matt is saying about stale content leaving sessions open. I would also check the existing connections and verify that something like streaming content didn't sneek its way in. While it's rare, I have seen that happen because the CPL needed to be tweaked.

Also, I have run into situations where the connection table is being oversaturated and its causing the CAS to fail in communication with the Proxy. When troubleshooting we would systematically shut down tabs, and turning off the DLP integration resolved the issue. It helped us narrow down the issue when we had to involve Support.

Either was this leads to ICAP messages for the end users (which is what I assume you are seeing). If you see high counts in the Duration column it might be worth investing what that content is and whether it needs to be added to a bypass or blocked.

Large durations and small Sizes usually mean a hung session.

Just curious if this is still occurring or not.

Regards,
Perry

[AKH_BC]

A review and additional of the ICAP Best Practices may also be in store, and determining whether a "Fail Closed if no ICAP" is a business approved setting for your environment.  You have been given many recommendations by the other Knights here, hope this sets you on the right path.