Excesive TCP 8014 traffic from SEP Clients to SEPM Server

[Migration User]

 

I've 20,000 clients with SEP. All was working fine but suddenly a lot of clients started to generate excesive traffic with port TCP 8014 to SEPM Server. 

 

I even had to cancel the port traffic (by firewall) to the server because the network was saturated in several sites.

 

The heartbeat was increased to 24 hours in pull mode, but it doesn't work.

 

For example a netstat command each 3 seconds in a SEP client (10.X.Y.Z) to SEPM (10.A.B.C) Server:

 

 C:\>netstat -an 3 | find "8014"

  TCP    10.X.Y.Z:24138    10.A.B.C:8014    ESTABLISHED

  TCP    10.X.Y.Z:24138    10.A.B.C:8014    ESTABLISHED

  TCP    10.X.Y.Z:24138    10.A.B.C:8014    ESTABLISHED

  TCP    10.X.Y.Z:24138    10.A.B.C:8014    ESTABLISHED

  TCP    10.X.Y.Z:24138    10.A.B.C:8014    ESTABLISHED

 

[pete]

limit the number of connection on the IIS site.

have the pull setting s been applied on the groups and whether the clients have taken the new policy?

[ℬrίαη]

Make sure you're in pull mode and you will also want to setup GUPs at your remote locations.

[Migration User]

Throttling is preferred to limiting connections. With 20,000 clients, be careful with this, though... both restrictions may have unintended consequences (you have to leave enough to let everyone get access). 

Since this isn't something you experience on a regular basis, I'd be more interested in what is generating the spike in traffic. Did a number of clients go beyond the number of content revisions kept by the SEPM? We just had a long weekend... if you're only setup for 3 days of revisions, you may have a number of people who are 4 days out-of-date and are forced to request a full definition package (100 MB instead of 100 KB).

 

Also, +1 to GUPs.

[Migration User]

We have seen issues like this when the definition files on a client get corrupt.  They will continually try to grab the Full.zip file containing all defs from the SEP server.  We noticed they would retry to download the Full.zip every couple of hours.  So I wrote this Snort sig to catch any host trying to download it twice within 9000 seconds.  Works great to find them.

 

alert tcp $HOME_NET any -> SEPM.IP.Address.here 8014 (msg:"Possible Symantec Client Corruption";flow:established,to_server;content:"Full.zip";threshold: type both, count 2, seconds 9000, track by_src;classtype:unknown;sid:1081209192;rev:1;)

[ℬrίαη]

This would be nice to add as a custom IPS signature in SEPM

[Migration User]

Ok, I've been investigating and the problem seems to be with the Microsoft SQL Server. I don't know what is happening but the SQL server (same the SEPM) is overloading.

The server features are:

Memory 1(RAM) 16 GB

Processor Intel Xeon X5450 @ 3.00 GHz (2 processors)

Windows 2008 Server 64-bit OS

Hard Disk 1.5 TB (more trhan 800% free)

Now the performance:

Network NIC performance 100% (a lot of connections with sep clients)

Memory 97% used - a sqlservr.exe process is running and has a 13Gb of use.

 

What could be?

 

[Migration User]

oh, so does this problem fixed in 11.0.6 MR6 MP2 ?

[Mithun Sanghavi]

Hello,

I completely agree with Posthums.

The definition files on a client get corrupt.  They will continually try to grab the Full.zip file containing all defs from the SEP server.

However, personally I would see few things here.

a) What version of SEPM and SEP are they on?

b) Any GUP's on the Network?

 

I would then also recommend tring Rolling back of the definitions by the following the steps below and check if that works.

 

1) Login to the Symantec Endpoint Protection Manager

2) Click on Policies

3) Click on Liveupdate

4) On the Right Hand side, click on the Liveupdate Content TAB

5) Double click on the Liveupdate Content Policy.

6) Click on Security Definitions

7) Click on the Bullet Button that states Select Version

8) Click on Edit and

9) Click on the Drop Down arrow and Select the Right previous versions of Definitions.

10) Click on OK

[Migration User]

a) What version of SEPM and SEP are they on? SEPM 11.06 and SEP 11.05 b) Any GUP's on the Network? Yes, there are a lot of GUPs' on the network (more less 160)

[Mithun Sanghavi]

Hello,

Since you are carrying  Older version 11.0.6000

We recommend you to upgrade the Symantec Endpoint Protection Manager and all clients to Latest version.

Since you are carrying old version, there would be 2 upgrades you will have to go through. Please check the steps below:

1) From Symantec Endpoint Protection 11.0.6000  to Symantec Endpoint Protection 11.0.6005 (RU6a)

http://service1.symantec.com/support/ent-security.nsf/854fa02b4f5013678825731a007d06af/6f5427dfdd4a0cc18825770400611b5a?OpenDocument

2) From Symantec Endpoint Protection 11.0.6005 (RU6a) to Symantec Endpoint Protection 11.0.6200 (RU6 MP2)

http://www.symantec.com/business/support/index?page=content&id=TECH145428&actp=search&viewlocale=en_US&searchid=1293479302456

 

Reason:---> check the Release Notes below:

Release notes for Symantec Endpoint Protection 11.0.x and Symantec Network Access Control 11.0.x

http://service1.symantec.com/support/ent-security.nsf/854fa02b4f5013678825731a007d06af/2c62c4ee0e697aae882573b000034ded?OpenDocument

[Migration User]

We had a similar issue.  We had one server (the first on in the list for notifications) that was just pouding our SQL server and making everying else SEP was doing just crawl.  Take a look at your alerts table in the SEP DB.  We have a large number of clients as well and out alerts table had grown to almost 2 million rows.  

And, as it turns out, SEP was performing several full table scans on the alerts table every minute (due to a query that SEP is making on a non-indexed column).  The query was killing our SQL server.  We were able to resolve th issue by tweaking the log settings so that the size of the table was greatly reduced.  We have notified Symantec of the issue and hopefully they will analyze the SQL and table structure in furture releaseses.  

For good measure, we also retired the server that was causing the issue and spun up a new one to takes it place.  All seems well now.

[Migration User]

There are some known issues in 11.0.6000, 11.0.6005 and 11.0.6100 that cause extra load on SQL. They aren't fully resolved in 11.0.6200, but from what I understand, there is an improvement.

[Migration User]

For obvious reasons, I can't give details, but I think it's safe (as in, I won't lose my job) to say SQL in 12.1 will be significantly more optimized. The development team spent a lot of time/effort eliminating bottlenecks in the database. 

[Migration User]

We did upgrade to 11.0.6200 at that same time.  That could have easily contributed to the soultion as well.  Its good to hear that the next version has been addressing these concerns.