Chicago Security User Group

 View Only
Expand all | Collapse all

What did SEPMs push out?

ℬrίαη

ℬrίαηSep 20, 2013 07:58 PM

  • 1.  What did SEPMs push out?

    Posted Sep 20, 2013 05:46 PM

    Guys,

    I need to find out the best way to determine what was pushed out from the SEPMs to clients on

    9/19/13 2:00:00 PM CDT - 9/19/13 3:00:00 PM CDT. Networking is telling me that there was a spike in traffic coming from the SEPMs. I need report back on what it was. Can anyone point me in the right direction? Running SEPMs 12.1.2015 on server 2008.

     

    Thanks in advance.

     



  • 2.  RE: What did SEPMs push out?

    Posted Sep 20, 2013 05:50 PM

    Did you retain logs for the day?  I would go to Monitor, the logs tab, and take a look at the server and client-server activity logs to see if anything shows up during that time frame.



  • 3.  RE: What did SEPMs push out?

    Posted Sep 20, 2013 06:16 PM

    Fine tune the settings in Time Range to narrow down your search to the date and time desired.

    Capture.JPG

     



  • 4.  RE: What did SEPMs push out?

    Posted Sep 20, 2013 07:58 PM

    What port was the spike in traffic? 8014?



  • 5.  RE: What did SEPMs push out?

    Posted Sep 22, 2013 12:19 PM

    you may also check the reporting.log. If you still have the entries for the particular date then you would be able to see what was downloaded from server to clients.

    %Program Files%\Symantec\Symantec Endpoint Protection Manager\apache\logs\reporting.log

    Do let us know, if you have any questions.



  • 6.  RE: What did SEPMs push out?

    Posted Sep 23, 2013 03:39 AM

    Hello,

    I doubt you will be able to identify the reason of that spike by only looking at the logs in the SEPM console, you may see there were content updates, client upgrades or something else not the size of the transferred files. More details about what has been transferred between clients and manager, including the size of the files, would be in the Apache access logs which are disabled by default.

    Common reasons are:

    + SEPM => clients:
    - SEP client package upgrade (did you schedule any client upgrade?)
    - definitions (they are bigger when clients are offline for longer time)
    - policies (changing a policy for large amount of clients may cause a spike)

    + clients => SEPM:
    higher client logging (firewall, applications, outbreak, etc.)

     



  • 7.  RE: What did SEPMs push out?

    Posted Sep 23, 2013 11:21 AM

    We had a very similar problem happen a few times.   Our network team was monitoring bandwith and found that at clients were counsuming 10-100x the normal bandwith during random times.  SEPM logs, and client logs were useless.  The best tools we had to monitor and track down workstations were bandwith reports from the network team, wireshark packet captures on known bad clients, and/or the SEPM server, and Symantec Critical System protection policies applied to the SEPM's to track clients using up more then one connection.

     

    After diligent work we discovered that some clients were stuck in a loop downloading definitions.  The problem stemmed from a resource problem on the client machine.  Something on the client was consuming all of the available paged pool memory.  On the client when this resource was exausted the SEP client tended to get into a loop.  When observing a "bad" client in wireshark one would see the client connect to the SEPM, then start to download definitions.  During a certain point in the download the SEP client would reset the connection and try over again.  On the SEPM if one were to check the number of half open connections on the SEPM one would see hundreds, from the same client.

     

    So after working with Symantec an issue was found with the way the client did some error handling and this was resolved in 12.1.3.  See http://www.symantec.com/docs/TECH206828 See fixid 3160786.

     

    Untill 12.1.3 clients could be distibuted we found that by agressive use of the GUP's could at least keep the bandwith consumption local to the site and didnt affect our WAN anymore.

     

    Let me know if you have any other questions.  I hope 12.1.3. resolves your problem. 



  • 8.  RE: What did SEPMs push out?

    Posted Sep 23, 2013 11:53 AM

    RichBagurdes,

    Where on the SEPM would you be able to ..."check the number of half open connections on the SEPM...?"



  • 9.  RE: What did SEPMs push out?

    Posted Sep 23, 2013 12:37 PM

    Where on the SEPM would you be able to ..."check the number of half open connections on the SEPM...?"

    I'm not sure I understand your question.  If you're using a pull communication to update clients, the SEPM would see "half open" connections.

    Are you using any GUP's, or is all communication right from the SEPM to the clients?
     



  • 10.  RE: What did SEPMs push out?

    Posted Sep 23, 2013 12:38 PM

    Another thing you can do is use wireshark on the SEPM to detect full.zip download to the clients. This could be indicative of corrupt defs on the client if they continuously ask to download over and over even though they show as up to date. See article here:

    https://www-secure.symantec.com/connect/articles/using-wireshark-detect-fullzip-downloads-sep-client-machines



  • 11.  RE: What did SEPMs push out?

    Posted Sep 23, 2013 01:25 PM

    Before our bandwith problems all our clients were set to pull updates from the SEPM.  The command we used to search for half open connections on the SEPM was:

    netstat -anbo > netstat.log



  • 12.  RE: What did SEPMs push out?

    Posted Sep 23, 2013 01:33 PM

    Also - once we observed the half open connections via the netstat command we pushed a CSP policy out to the SEPM's that logged all connections to the SEPM port.  Then I created a query to create a list of "problem" clients.    When we were deep into the problem we found that some clients had thousands of connections to the SEPM in the three hour period we were reporting on.

    If you plan to use this query - update the Policy with the name of the policy you decided to use. In my example I removed the policy name so  below it says POLICY_NAME = 'redacted'

     

    Select
     Sum (event_cnt)as "Events",
     max(value1)                       as "Process Set",
     max(Process_name)                 as Process,
     dbo.FmtEventType(Event_Category,'','')   as "Event Type",
     max(CASE event_type WHEN 'PNET' THEN CASE OPERATION
     WHEN 'Accept'  THEN CASE DISPOSITION WHEN 'D' THEN 'Inbound ' + target_info + ' Connect from ' ELSE 'Inbound ' + target_info + ' Connect from ' END + VALUE5 + ':' + VALUE6 + ' to local IP ' + VALUE4 + ':' + VALUE3 + REPLACE(' ('+ VALUE10 + ')','()','')
     WHEN 'Connect' THEN CASE DISPOSITION WHEN 'D' THEN 'Outbound ' + target_info + ' Connect to ' ELSE 'Outbound ' + target_info + 'Connect to '   END + VALUE5 + ':' + VALUE6 + REPLACE(' ('+ VALUE10 + ')','()','') + ' from local IP ' + VALUE4 + ':' + VALUE3
     ELSE 'Unknown network operation (' + COALESCE(OPERATION,'') + ')' END
     ELSE  dbo.getpathleaf(target_info) END) as "Resource"
     From CSPEVENT_VW WITH (NOLOCK) WHERE EVENT_TYPE LIKE 'P%' AND event_dt > dateadd(hh,-3, getutcdate()) AND POLICY_NAME = 'redacted'  AND Process_name = 'Smc.exe'
     GROUP BY
     SUBSTRING(PROCESS_NAME,1,50),SUBSTRING (dbo.getpathleaf(target_info),1,75),EVENT_CATEGORY,SUBSTRING(VALUE1,1,25),
     OPERATION, VALUE2, DISPOSITION,EVENT_SEVERITY,
     SUBSTRING (HOSTNAME,1,15),HOSTADDR,VALUE5,OSTYPE
     Order by 1 DESC



  • 13.  RE: What did SEPMs push out?
    Best Answer

    Posted Sep 24, 2013 03:38 AM

    Would not be easier to enable the Apache Access log to trace all clients' connections?
    Once you get it, you may import it in Excel and apply filters and counters as you wish.