Hi JeMann,
You understand correctly. Either you are sending too much traffic to the internal CAS of your ASG, or something is causing the service to go down. If some transactions are working and you are getting responses back, you know it is do to the amount or type of traffic. If all ICAP RESPMOD queries are failing, or you see the healthcheck for
icap.local_response failing, there may be something causing the service to crash. This is much less likely, but if it does happen, I would recommend having support take a look.
Assuming what you posted is the only policy for ICAP, if we revisit your response policy, it is essentially saying to scan anything that is not in your
ICAP_BYPASS_GROUP. I would recommend seeing what is actually in that condition.
Another situation you may be encountering is unsupported traffic being sent to the CAS. CAS is file based, and so things that don't have End of File can cause issues, such as streams, or stock tickers. When a stream or stock ticker is sent to the CAS, the CAS continues to wait for the end of file, which never comes. This ties up the connection. See
this KB for reference.
My recommendation would be to review and implement the ICAP Best Practices. They are updated periodically, and so you will want to check back from time to time for updates. The CPL policy itself can be found
here, and a brief overview of ICAP Best Practices can be found
here.
Thanks!
Original Message:
Sent: 09-21-2020 10:45 AM
From: Unknown User
Subject: ICAP service queue backing up
Thank you, Sakkarin. The link you provided was helpful in understanding the flow. The way I understand this configuration flow to be now is that the requests are being sent to the external server but the responses are being handled locally on the proxy. Since the error message is regarding the queue for the "bluecoat-local-response", this is most likely an issue with too many connections for scanning locally on the proxy. Am I thinking of this correctly?
response.icap_service(bluecoat-local-response,fail_closed) <--- local
request.icap_service(external_icap,fail_open) <---"external_icap" is the external server
Original Message:
Sent: 09-21-2020 12:55 AM
From: sakkarin pichetskul
Subject: ICAP service queue backing up
Hello JeMann
So, If the policy using the condition "condition=Posts_and_Puts", if I understanding corrected, I know this condition is about HTTP methods PUT and method POST.
In the all website should have this behaviour the both HTTP method very much in category Social Network, field Login user, submit password and upload file.
All these are send to request_ICAP_Service then the ICAP server will have not enough performance.
Or your setting on ICAP service in ProxySG was not match the ICAP server performance.
In the ICAP service setting on the ProxySG has the setting about maximum connection between ProxySG and ICAP Server.
You should checking on the ICAP server performance scanning about maximum connection or maximum file scan threshold and should setting on the ProxySG configuration with that threshold.
However, If upper step not work, you should decrease scope the destination of category or domain you concern or add more file extension for bypass request ICAP scanning.
------------------------------
Thank you and BR
Sakkarin Pichetskul
System Engineer
nForce Secure Co.,Ltd. [Thailand]
Original Message:
Sent: 09-21-2020 12:32 AM
From: sakkarin pichetskul
Subject: ICAP service queue backing up
Hello JeMann,
You should understanding Traffic flow when ProxySG and ASG is integrated with ICAP Service on this link : https://knowledge.broadcom.com/external/article?articleId=169837
After that, you look at your policy and the both layer you told was different about action object request.icap_service() and response.icap_service().
------------------------------
Thank you and BR
Sakkarin Pichetskul
System Engineer
nForce Secure Co.,Ltd. [Thailand]
Original Message:
Sent: 09-20-2020 01:38 PM
From: Unknown User
Subject: ICAP service queue backing up
I'm trying to understand how our ICAP service is configured on our proxies. It was setup by someone no longer at my workplace. I thought the ICAP service was offloaded to an external service but looking at the VPM I'm not so sure after all. It looks like we had a CPL ICAP Best Practices layer but it is currently disabled and the following layers are active after that disabled layer.
;; Tab: [ICAP Best Practice(NEW)]
<Cache> condition=!__is_notify_internal
condition=ICAP_BYPASS_GROUP response.icap_service(no) ; Rule 1
response.icap_service(bluecoat-local-response,fail_closed) ; Rule 2
condition=OracleCloud cache(no) ; Rule 3
;; Tab: [Vontu]
<Proxy> condition=!__is_notify_internal
request.header.User-Agent="Shockwave Flash" url.domain="api.photoshop.com" request.icap_service(no) ; Rule 1
condition=FileExtension_NODLP request.icap_service(no) ; Rule 2
condition=Posts_and_Puts request.icap_service(external_icap,fail_open) request.icap_service.secure_connection(auto) ; Rule 3
This reason I am asking is because we have been encountering many errors like this: "ICAP Notification: Queued requests in a service above permissible" While investigating the CAS requests, there are many queued requests backing up in the current requests and this eventually causes disruption in users traffic with an icap error splash page. To fix, we manually restart the service to clear out the requests and this seems to work for several hours or days until this happens again. Can you help me understand what this configuration is doing and any recommendations on a best process here?