DLP Web Prevent high cpu usage

[chong]

Hi, 

We noticed recently our DLP web prevent cpu usage consistently stay on 90% and above during peak hours. We are using bluecoat + DLP web prevent. 

Bluecoat concurrent connection set as 500
DLP web prevent request and response we set as 700. (we only have 1 dlp web prevent)
The bandwidth available maximum in the environment is 1.14 Gbps and no bandwidth controller. 

During the peak hours, it can reach more than 600 Mbps. 

May we know if we need to allocate more than 1 DLP Web prevent? As during peak hours, the browsing speed is slowing down and take long time to load. We have contacted the Support, however did not get response since a week ago.

[Fady azab]

Hi Chong 

if its a virtual machine you can increase the CPU and see what happens as Network Prevent for Web is CPU bound

Thanks

------------------------------
Fady Azab
Senior Consultant
CCIT GMBH
------------------------------

[chong]

Hi Fady,

Thank you for your response. 

The DLP web prevent previously was 8 core and 16 ram. The usage is maintaining 90% and above during peak hours

We have proceed to increased from 8 core to 16 core. However, the usage is still maintaining 90% and above during peak hours as well. 

The messagechain we set to 16 at the beginning and yes it is virtual machine.

[Fady azab]

Hi Chong 

Please check that messagechain.numchains and messagechain.cachesize = 16 

Thanks

------------------------------
Fady Azab
Senior Consultant
CCIT GMBH
------------------------------

[chong]

Hi fady, 

yes. the value is 16.

[Fady azab]

Hi Chong 

Then you can add another Web prevent server it should decrease the CPU usage significantly 

Thanks

------------------------------
Fady Azab
Senior Consultant
CCIT GMBH
------------------------------

[chong]

Hi Fady,

Any method we can use to check if this slowness issue is caused by the DLP Web prevent? Like from the logs of DLP Web Prevent. 

Recently we implemented web isolation and we want to see if this is cause by DLP Web prevent or other product. 

As just now when were we checking in the server, we found that the connection was only 100 out of 500 and the network utlization only 1 Mbps. But stil when browsing it was slow and take long time to load. 

[Fady azab]

Hi Chong 

Are you isolating all the Websites ? 


------------------------------
Fady Azab
Senior Consultant
CCIT GMBH
------------------------------

[chong]

Hi Fady, 

All the external website will be go through the web isolation. But not for intranet. However, when we browsing the intranet. We are also facing the browsing slowness issue. .

[Barnabas Toth]

Hi,

You should check what traffic is sent over to DLP on the proxy and also check for ICAP errors there.
There might be also a lot of Queued Requests, which the users will experience as a lag.
It is advisable to filter what traffic you send over to DLP, category PUT/POST requests, etc.
You can also play with the Connection Backlog parameter on the Web Prevent too.
Further tuning recommendations: Data Loss Prevention Network Prevent for Web ICAP Performance Tuning further(broadcom.com)

Adding another Web Prevent is always a good idea to provide redunancy and load balancing.

Good luck!
Barnabas

------------------------------
DLP expert
------------------------------

[chong]

Hi Barnabas,

Thank you for your response. 

Yes there are a lot of queue requests we found in proxy. However, if the filtering is not an option for us due to the policy of organization. The only option we have is to add another Web Prevent? Do correct me if i am wrong. 

We can see that the Queue Request will spike up to 20k+ during peak hour.

[Barnabas Toth]

Make sure you have only PUT and POSt enabled in the proxy policy:
How to create an DLP REQMOD ICAP Policy (broadcom.com)
I would filter traffic, like telemetry to microsoft, google api and so, because it is taking slots from meaningful connections. This can help with eg. morning peaks.

You can open a case with Broadcom and try to tune, but for this amount of traffic I would add another Web Prevent, for sure.

------------------------------
DLP expert
------------------------------

[Xxsnsxx]

Keep in mind that cores directly relate to the number of connections the ICAP can receive from the proxies.  If the Network Prevent cannot accept any more connections that causes queueing on your proxies.  Meaning slow browsing.  I always use more than 1 Network Prevent server because of these reasons and try to run capacity with 20 to 25 % available at all times.  A mathematical example is if you have 8 cores you can receive 32 'Max Number of Requests'.  This setting is under the configuration of your Network Prevent for the Web server.  It sounds to me like you need more cores and most likely memory to alleviate this slow down. 

This command will show you how many connections your taking in during peak times.   Change 1344 to 11344 for sicap.  If the numbers continue to be equal to your Max number of requests you need cores.

netstat -anp | grep ':1344 ' | grep ESTABLISHED | wc -l

[chong]

Hi Barnabas, 

Thank you for your suggestion. However, we will need to inspect other method of traffic request as well such as GET due to the organization policy. We have added another Web Prevent yesterday and will continue to monitor the traffic. We also have an existing case with Broadcom, their suggestion was to add another Web Prevent. May i know if there is any calculation i can do to determine what is the number of Web Prevent boxes i need? 
​


Hi Xxsnxx,

Yes, we have changed the value to 700 as suggested from the Broadcom KB which match or greater than proxy concurrent connection. At first, we have only 8 core. Then we proceed to increase until 20 cores and seems still experiencing the slow down. Unfortunately, the server team told us that the maximum core they can increase is 20. By the way, this is a vm environment. Thus, we added another Web Prevent box to see if we can resolve the slow down issue. From what we see, during peak hour. The connection received Web  Prevent received is always 500, this is the value that we set in proxy for concurrent connection that can be sent to Web  Prevent.

[Xxsnsxx]

If you are running get requests ask the proxy team to uncheck that box.  Get requests are specifically noted in the admin guide as causing high load.  I would start small using req mod and the proxy reference below using put push stor.

https://knowledge.broadcom.com/external/article?articleId=166373

After you have this working and blocking I would then move into the get world. 

FYI, tune your ICAP reference = guessing with connections doesn't work as cores can only handle a max number.  

https://knowledge.broadcom.com/external/article/159666/data-loss-prevention-network-prevent-for.html

Hope this helps you

[Xxsnsxx]

Also I believe Barnabas mentioned this but filtering out traffic of specific sites also helps.   I couldn't find the article on this but in the past I had to remove some sites causing issues holding connections. I will post it if I find it.  Also build a strategy and be very careful when selecting what to intercept with the proxy.   Bc has categories.  Last, are the proxies separated for users vs servers.  The loads make differences.

[chong]

Hi all, 

Thank you for your sharing and information. 

I have added another NPW box and load balanced the traffic. It seems the issue resolved.