SEP 12.1.5: question about content optimization

[MIXIT]

Hi all.  Just wanted to ask a question, but first, to say thanks for addressing the content optimizastion issue.  I have 3 SEPM's so far brought to 12.1.5 and am seeing that the ImportPackages folder is indeed emptying. 

 

I admit however I don't understand the part about deltas as mentioned in the release notes.  What I am seeing is that about every few hours, my SEPM's are downloading a file approxiimately 450MB in size.  During when this file appears in the ImportPackages folder, I find that the SEPM console is quite unresponsive, often not even being able to load pages without giving database timeout messages from within the console page itself.  Then the 450MB dfisappears, and suddenly the SEPM console is responsive again. 

 

This seems like while processing the 450MB file the SEPM processes slow to a crawl, yet the rest of the system appears to be ok so if it were say, just pure disk I/O as the bottleneck I'd expect everything to be slow but that's not the case.  Perhaps Symantec is looking at optimzing this?  Perhaps RU5 was the initial fix but the performance issues are step 2 for RU6? 

 

And so my question also was, what are deltas in this case?  Since my SEPM is getting a 450MB file every few hours, is it parasing that entire file for the deltas or something?  The end result is still a big download every few hours and a lot of I/O to parse or load it (or however ImportPackages functions). 

 

Sorry I"m a bit distracted as I write this but hopefully the points are there.  Thank you! 

[Mithun Sanghavi]

Hello,

Check the Comment from "SMLatCST" - https://www-secure.symantec.com/connect/forums/gup-space

The amount of space used on the GUP should remain the same, regardless of how defs are stored on the SEPM (in fact, I think it might use more).

The 12.1RU5 SEPM switches round how it stores defs, but it doesn't change the fact that it must have a client's defs in its repository in order to update it using deltas.  This means the SEPM still has to hand down the full defs to some clients (i.e. those running defs that the SEPM doesn't have a copy of anymore).

With full def sizes increasing everyday, the GUP cache size must increase to accommodate them.  Therefore, I'd recommend the cache should be large enough to store 1 copy of the full 32 bit AV defs, 1 copy of the full 64bit AV defs, plus a bit pf space for the other def types.

Leaving it at 500MB means only a single copy of the full AV defs can be cached.  Therefore it's possible to end up in a situation where a GUP must purge current defs to make room.

Hope that helps!!

[Migration User]

MIXIT - Did you ever get your SEPM non-responsive while getting content definition problem figured out.  I upgraded servers, installed 12.1.5 - SEPM does nothing during content updates (can't log in and occasional unknown server error) - Symantec just wanted me to run a bunch of diagnostics and acted like I was the only one with this problem.  Good to know that I am not the only one.

 

My server has tons of disk space 32GB of ram, and brand new Xenon processors with multiple cores.  I do believe there is something wrong with 12.1.5, and am thinking about downgrading to 12.1.4.  I have noticed really high disk cache while getting content updates.  Daily.

Hope they provided a solution to the problem.

[MIXIT]

Hi, and first off thanks to Mithun.  I apologize that I forgot to reply here initially.  I marked your reply as solution but I'll admit I don't know the back-end architecture of the defs storage but your answer sounds good to me in this case. 

 

AdamUSA:  I had written a good 8 paragraphs citing my knowledge of how global support organizations work, having supervised in them myself, and complaining in accordance with that knowledge, but in the end I wiped them out and instead just will say this:  If you can live with the problem until they come out with another release that might fix it, try that.  Calling tech support is very hit and miss - often they just try to fix something that is clearly an escalation issue and can't be fixed.  I don't know if it's from lack of information they have, as is common with outsourced tech support centers, or that they have to pretend the issue doesn't exist so management can avoid upsetting their larger accounts, I don't know.  I'm sure it's both actually.  But anyway, yes I have the problem still, it seems to be worse on my very slow test SEPM whereby I have a very lame disk system set up (standard Dell server, Dell array controller, but oh no!, non-Dell SATA drives - not paying $400 for a damned 1TB Dell SATA drive, give me a break). 

 

On my customer systems where they've got say, 8 SAS 15K drives, the issue I think occurs but dissapaers quickly.  So I think it's disk I/O related moreso than CPU or memory). 

 

 

[SMLatCST]

As the marked Solution is just a link to my post, I thought I might chime in here:

The post I wrote was specifically around the amount of disk space required on a GUP, and so may not be applicable to your question, which I interpret as: "Why is my SEPM grinding to a halt whenever it's processing new defs?"

I'm not personally aware of any problems with the def processing at this time, but they are handled differently and the SEPM is doing more during def processing in 12.1RU5 than it was in previous versions.

To help illustrate the differences:

OLD: The SEPM would download the defs and parse them into its folder structure and DB as is.

12.1RU5: The SEPM now downloads the defs, parses the latest defs into its folder structure and DB, then proceeds to create a delta between the defs just downloaded, and the ones previous.  It then deletes the ones previous, keeping only the newly created delta and a full copy of the latest defs.

Perhaps it's this additional processing that's causing the slow down?

[Migration User]

The issue is... Tech support says THIS IS A ONE TIME CONVERSION.  It isn't, the parsing is happening every time the definitions files are updated.  I have an extremely fast server with at least 4 times the recomended RAM, Processor, Disk Space, etc... I keep getting told that I am the only one with the problem and support would like to troubleshoot with me.  I am not going to be the Beta tester for a product that should not have been released.  I support 200+ physicians and their offices.... Unfortunately I have told them to go symantec... McAfee is looking kind of nice right now to me... and may be my recommendation if there is not a fix soon.

[Migration User]

One more thought... this is definitely an I/O issue on the disk.  Unfortunately I don't think many people will think kindly about having to install multiple hard drives and switch to a hardware RAID or the configuration above  8 SAS 15K drives just to run their virus software.  That is really bad form! 

[Migration User]

Hello AdamUSA,

The one time conversion is when you upgrade your SEPM server to 12.1.5.  In that case we convert all content (definitions) on SEPM to this new format.  This only happens ones.  However when the definition files are updated from LiveUpdate we only need to convert one definition set to the new format since all the rest of the definitions were updated already.

You should not need to have a hardware RAID or the configuration above 8 SAS 15K drives.  It may take a little while for SEPM to do the conversion, which is true, but overall it should be less than it was in the previous versions of SEPM.  The main difference here is that SEPM will create the delta files ahead of time during the LiveUpdate process.  This saves disk space and saves future work for SEPM, as it is much quicker for SEPM to prepare delta files for the clients later on when they request new content.  In other words SEPM will do prework to reduce the overall load of SEPM during normal activities.

I have a few questions for you:

1. Why does it matter that SEPM runs slower during the LiveUpdate prcoess?
2. Do you have any other applications running on SEPM?
3. Is the LiveUpdate process on SEP 12.1.5 worse than previouse versions of SEPM?
4. Have you thought about running LiveUpdate only during off hours?

Thanks,

Elisha

[Migration User]

If overall CPU is getting pegged during LiveUpdate in 12.1 RU5, you may try the following steps: 

1) Goto SEPM folder, and edit %SEPM%\tomcat\etc\conf.properties

2) Add the following line, and save it.

           scm.content.incoming.delta.cpu.limit=1

3) Restart SEP Manager service, you can do this by opening command prompt, and type

           net stop semsrv & net start semsrv

SEPM machine - either having just 2 cores (logical/physical), or that run other resource intensive applications can leverage above steps, however note that LiveUpdate process would end up running few minutes more.

[Migration User]

Elisha

You say it is a one time process... However on my machine, it seems to do the "one time" conversion every single time it updates.  You also have another user on this thread saying the same thing.  There is a problem.  You guys seem to want me to beta test it, and I don't have time.  I downgraded to 12.1.4 and everything is great.

 

1. - Not being able to log in to the SEPM for 2 hours + every time it grabs an update file is obnoxious especially if this is happening during server administration.

 

2. Yes... 1 piece of custom software - it was never an issue with older versions.

 

3. 12.1.4 runs just fine - 12.1.5 sounds like it is going to explode my hard drive

 

4. Sounds to me like you are asking me to compromise security on my server.  I know you release updates more than once a day.

 

Thanga - This is a disk issue - CPU is at less than 6% memory less than 18% DISK pegged at 100%

[MIXIT]

Not sure why people got confused here but I wasn't suggesting you go buy an array of SAS drives for SEPM.  And obviously that kind of hardware is not dedicated solely to SEPM, the server does other things. 

 

Thanks to both of the Symatnec people that posted on here, the information is helpful.  I'll leave Adam to answer the questions because they're not intended for me, but I will say that question #1 that Elisha posted can be answered on my end by me saying that just launching the SEPM icon, waiting for the username/pw screen to come up, and then the process of logging in locks up, sometimes completely, sometimes just delayed for an excessive period of several minutes. 

If I have the ImportPackages folder view open, I see this behavior stop as soon as that folder empties, indicating in my mind clearly the processing of this 450MB def file is what's killing the Symantec processes, and yet, the rest of the server does not appear to be severely affected.  This tells me there's probably a software setting for SEPM or the db where something is being throttled, probably to prevent the Symatnec stuff from potentially killing a production server's I/O.  That probably means engineering knows about this issue and is working on finding ways to optimize things.  I'll trust in that getting done.  If this issue were affecting my customer's servers I'd b eupset, but I see it mainly on my own test server which is not equipped with high-end disk subsystems. 

 

On the othe rhand, I Have one client runing their SEPM on a Wion 7 single-disk box without much issue either, OR, I haven't tried to log in at that magical moment when it's doing the def file thing. 

[Migration User]

Pretty funny that my comments were deleted. The way Mixit says it, is correct.... SEPM non-responsive during processing... Either locks up or takes forever to log in. 

 

I will paraphrase what I said one last time in responses to the questions above.

Elisha

1. Server administration is why it matters that SEPM runs slow.  It is obnoxious to go in to check client machines via SEPM and not be able to log in for 1.5 hours while SEPM is processing a definition update

2. Yes, however, the server runs fine on 12.1.4 - All other software runs fine while experiencing this issue.  Starting to wonder if this is a conflice between DBSVR16.EXE and SQLExpress?  12.1.4 uses an older version of DBSVR (12 I think??)

3. 12.1.5 is MUCH worse

4. I would much rather have up to date virus definitions.  Running it one time a day seems less secure

@ Thanga

This is a disk issue.  This whole thread is about a disk issue.  CPU is at less than 7% and RAM usage less than 18%  Disk pegged at 100%

[Migration User]

I recommend opening a case with support.  It could be a conflict between some other application on your server.  I am running my SEPM on VM that has a shared drive with other VMs.  The disk is a standard 7200 RPM disk and I and not experiencing that issue.

[Migration User]

@AdamUSA

Although the configuration sounds to deal with CPU, it indirectly helps avoid Content Delta Processing tasks from running in parallel in multiple CPUs, and thereby reducing I/O overheads at an instant.

As Elisha recommended - opening a case will help us understand the problem better.

[Migration User]

Note: The scm.content.incoming.delta.cpu.limit=1 update that Thanga mentioned will reduce both the CPU and the I/O.  It probably has more effect on the I/O then the CPU.  This will reduce the number of threads that SEPM uses when processing content.  The less threads used, the less I/O is used.  In other words this will serialize the work, rather than running them in parallel.

[Migration User]

12.1.4 works perfectly on my server... thanks, but 12.1.5 has some serious problem.  Not sure what it is.  I did open a support case.  Spent over a week on this... Collected info.. collected more info... clean wiped, re-installed, ran symHelp, ran several other tools... I ran out of patience.  I do believe there is a problem with 12.1.5 and some kind of conflict with other software installed... I will just wait for a new release.  It is a great product, but 12.1.5 is not an option for us.  I also believe that the problem is that the Delta Processing is occuring each definition download instead of just once like you have said repeatedly on this forum.  Also, as I am not the only person experiencing this problem, I am sure you will find someone else to test the issue.  I have run out of time and patience and need to get on with the rest of what I do here at the office.  Good luck finding the issue!

 

I have not run the command from Elisha, because 12.1.4 is stable and running just fine.

[MIXIT]

I'm kind of like Adam where I don't have much time right now to focus on this problem but I am fairly sure that since he and I are both having the same issue, and I somehow doubt he and I are the only two on earth that do, that you guys will have people to work with that can help more. 

 

Meanwhile, for my one customer that has their SEPM on the Win7 single-disk system, I will figure out when the virus defs come in, and do some login testing.  I'm working on their SEPM a lot lately anyway so can easily add this in. 

 

I'll post here if the problem does not occur on their machine during when it should.  Otheriwse our default actionjj plan of leaving this to others to solve will have to stand :) It's just the reality of IT management, we don't have unlimited time to spend on certain types of issues though priorities often change depending on customer impact.  Thank you for your help and info so far though! 

[Migration User]

MIXIT - I saw some weird interaction between DBSRV16.exe and Retrospect Backup.  You had mentioned a Dell box.  I have a newer Dell with removable RD1000 drives.  I wonder if there is an issue with Retrospect (backup software that comes with the RD1000 drive.)  Let me know if you see that on the effected systems... if you wouldn't mind.

 

[MIXIT]

No Retrospect anywhere unfortunately.  FWIW my only experiences with that particular software were not great, hence why I went with Backup Exec for my customers.  Is DBSRV16 a 16bit exe?  Or just a carryover filename from the old days? 

[Migration User]

I believe it is 32 or 64 bit.  DBSRV16 replaced DBSRV12 for SEPM 12.1.5 and is the culprit of all of the high disk activity during update.