Endpoint protection client using old (non-existent) policies

[Migration User]

It appears that some of our SEP11 clients are getting policies that no longer exist in our SEPM environment.

We have policies for each region, with basic network location awareness set only to determine if the machine is on the network or off the network. If the machine is on the network the location is set to "on the network" and when off its "Off the network". This has been in place now for quite a few months, but previously before we created these it was still default, where the location is named "Default". Now what we are seeing is the machines are showing green in the correct group in SEPM, but in the help and troubleshooting they are showing a group that no longer exists. The location is also showing as "Default", which does not exist anywhere in our SEPM environment anymore (I have changed all locations in all groups to specific names to help identify the issue, but with no success).

The policy number in the help and support troubleshooting is showing as 59B9-07/15/2009 00:02:45 400, the group for 59B9 still exists but has been modified some months back, it just does not seem to take effect even though the SEPM console is showing as green.

This is also a random problem, one day the machine is showing the correct information and is in the correct group, and then the next day it is back in this old group with the old policies applied. Each time I stop and start SMC services this group changing will happen.

A re-install gets the machine back into the correct group, but soon enough it starts to move around again. I have looked at the policy files on our SEPM, found the folder for the policy 59B9 but the XML files are all showing the correct settings from newer policies, so where is this old policy from 15 July coming from?

[Rafeeq]

whats the version of your SEPM?
try this document as the first step of troubleshooting

Troubleshooting Policy Changes

 http://service1.symantec.com/support/ent-security.nsf/854fa02b4f5013678825731a007d06af/fd6a5c5e2228020b882574c5005d329f?OpenDocument

[Migration User]

Thanks for the reply

SEPM Version 11.0.4202.75

I've been through this document previously, and the problem is that the machines do appear to be getting policy updates, they are just getting ones that no longer exist.

If i move a problem machine to another group on SEPM, and hit update the machine will update its policy and will show the new group. However, once the service is stopped and restarted again (on the client) it appears to get this old policy again that no longer exists.

i have matched up this old policy number to a folder on the SEPM server, but the policy under that number on the SEPM is correct, and up to date, whereas the policy the client pulls down is from July last year and incorrect.

i can't find the XML file that this is coming from anywhere on the server or the machine.

[Migration User]

Get a test machine and try the followiing...

* Check the 'Preffered group' key in the reg location : HKLM\Software\Symantec\SEP\SMC\Sylink\Sylink and make sure its correct, you can also check the policy serial number here.

* stop the SMC service on the client and delete the files : Sylink.bal, SylinkEx.bak(if present)

* Start the SMC service.

And also, when you find the client switching from actual group to an incorrect group, try this.... in the properties of the incorrect group, check the option " block new clients"... and delete the client's entry from the client list ....

Observe and let us know what happened... :)

Cheers,
Visu.

[Migration User]

Hi Visu

The "preferred group" key in the registry is showing the correct group name, but the policy under the "serialNumber" entry is the wrong policy number (The old one).

Even though the preferred group in the registry is correct the client itself shows the incorrect group under help and support troubleshooting.

I could only find the sylink.bak file, no sign of any sylink.bal file anywhere. I stopped the service and deleted the bak file, then restarted. Still whowing the incorrect group.

The incorrect group in this case does not really exist. It is a group that existed in July last year, but has since changed names and policies. The machine account does not move between groups on the SEPM, it only shows different groups in the client settings itself.

I'm not sure about setting a block on the "incorrect" group, since this group is still in use but has been renamed and policies changed since July last year. But since the machine in question has got a policy from July anyway, its possible that even a block would not help.

I have noticed that this is happening on a few more machines now (Although I'm not sure if they always were or not, you can't tell there is even a problem on the SEPM itself you can only see the issue on the client).

I know even when i'm typing this that it doesn't make much sense, so thanks for bearing with me on this one.. :-)

[Migration User]

Ummm, that's interesting...
 
"..since this group is still in use but has been renamed and policies changed since July last year. But since the machine in question has got a policy from July anyway..."

So, if am not mistaken, this incorrect group is nothing but a group name which was presnt sometime back, that has been renamed now... and whenever the client refers to this old group, the policy serial numbers also mimmatch and reflect the old (read it as incorrect) one....

Well, whenever a client registers with the group, it refers to the group with its group ID and when you rename the group, technically, the entry in DB should be modified.... I guess in your case, another entry is being created with same GroupId... Also, there's a problem with SEPM that  the deleted groups will stay in the DB ... this might contribute to our problem... Lets dig into the DB...

Run the follwing queries and let us know the output:

1. Select * from sem_client where computer_name='name of any one affected computer'
(This will basically give us the number of registrations done by that client, this is to check if we have multiple entries)

2. Select * from identity_map where type='SemClientGroup'
( This will list us the client groups with the GroupID, lets check if we have the incorrect group listed)

I will wait for your response before I can suggest you some DB manipulations.. :) .. Let me know if you find any difficulty in running the query ..

Cheers,
Visu.

[Migration User]

Hey Visu

Here are the results

1.  Ordered as Client ID --- Group ID ---- Computer ID

F7798E400A280839018353CFA6C1BD77	9F4D840A0A01024C00A4579E45547626	C90650DD0A280839018353CF1B6B46CE
24CB17FF0A28083900DAC4D42961A52A	995C0C130A01024C0161C5A6FA810FD7	C65DA2E20A28083900DAC4D4E0D69BFF

2. Ordered as Group ID ---- Group Name

59B9D5DD0A032310007DD60C3E8DCD9B	My Company\Hatch Clients
995C0C130A01024C0161C5A6FA810FD7	My Company\Hatch Clients\Australasia
9F4D840A0A01024C00A4579E45547626	My Company\Hatch Servers\Australasia

So we can see that the machines in question appeared only once in the query, and they are listing as being in their correct groups. Machine 1 should be in the group called "Hatch Servers\Australasia" and machine 2 should be in the group "Hatch Clients\Australasia".

On the SEPM they appear in the correct groups. But on the client, under help and support, troubleshooting they are showing this :


So there is no way to check on the SEPM to see if this is affecting anyone, as you can only see it on the clients themselves. There is no way for me to tell how widespread this problem is right now.

[Migration User]

One more thing whether your client is in computer mode or in user mode?
This you can identify with the icon present in the SEPM console.In the console it is possible switching one mode to other mode by right clicking on the client.In your scenario if you find those clients in user mode change it to computer mode 
 If you had doubt regarding the icon refer this doc
Symantec Endpoint Protection: Troubleshooting Client/Server Connectivity 

[Migration User]

The machines are both in computer mode with a green dot to show they are communicating. I have also run the communication tests from each machine and they come back positive.

They are both getting updates from the SEPM as well, as the user liveupdate function is disabled if they are able to see the SEPM server.

[Migration User]

That's getting more interesting... Well, as we could see, the client has just one entry in the database and that portends to correct information!! (surprise?!!) ... and what's more interesting is that the clients are communicating... I am having a bad feeling this is just visually fooling us... Something is cached and is boycotting to go away...

* AFAIK, certain files at the client side are responsible for this "beautiful" piece of info you get under H&S-> TS..  They are Serdef.dat, Serstate.dat and Sylink.xml.... Now, each of these guys have a cache saved under the same name with a .bak extension.. .. Deleting the bak file would clear the cache and get create a new one....  Our plan-of-action... :


* Choose a test client and delete the entry from clients list of 'Clients tab', wait for a few seconds and check if it reappears .. (Just to check the connectivity)

* Also, once it comes back (It actually should connect back), just try to disable AutoProtect from console and check if that reflects on the client... and if it does, enable it on the client and check if that reflects on the console... (Basically, we are checking if client is able to write log...)

* Move that test client to "Default group"... and observe after a few SMC stop/starts and system reboot(if possible)

* From a working client, copy the files i mentioned, (Sylink.xml, Serdef.dat, Serstate.dat) and replace in this test machine.... Care should be taken that the Bak files are already deleted before replace... and ofcourse, SMC should be stopped...

* And oh, do we have a replication partner by any chance?

Lets see how it goes... ;)

Cheers,
Visu.

[Migration User]

Try this

https://www-secure.symantec.com/connect/forums/sep-and-ad#comment-2965551

 

[Migration User]

Hey Hey

Looks like I am having some success!

Visu..

Deleted the client and it appeared back within seconds  ---- ( just curious, but does everyone's SEPM console view defy all logic, with almost 2000 machines it sorts in some strange order, and you have to look through all pages for your target? The search works, but i prefer to see it straight in the group window)

I couldn't stop the auto protect (its not allowed in the current policy), but I did start an active scan from the SEPM console, and the machine immediately went nuts - So that part worked :-)

I then stopped the client completely, and the SEPM console eventually recognized the machine had gone and the green dot on the SEPM disappeared - So i hope that proved what you were looking for. -- It also keeps the definition up to date on the SEPM, so the client is giving it some information...

I moved the machine to another group and forced an update. Immedaitely the client on the machine showed the correct group. I then ran the "smc -stop" and "smc -start" and saw that the client had once again jumped ship and gone into the group that doesn't exist - So back into the "My Company\Clients"

I then moved the machine back to the normal group and forced an update. Immediately the machine showed the correct group again "My Company\Hatch Clients\Australasia". But after yet another "smc -stop" and start the client started showing the "My Company\Clients" group again with the outdated policy.

I then stopped the services, and deleted ALL the bak files from the Symantec folder, and replaced the 3 files (Sylink.xml, Serdef.dat, Serstate.dat) with those from a working machine.

Now I am able to stop the services and start them, restart the machine and the client reports it is in the correct group! It no longer ends up with the old policy, and I think that this has finally solved my problem.

Oh, and yes we have another 3 replication partners to the SEPM here.

Thanks to everyone that chipped in, especially to you Visu for that eventual victory.

By the way, I did find a way to find the machines that were affected by this... The old policy (The one that no longer exists) had the location name as "default", whereas the new policy is set to show as "On the Network". I ran a report on network threat protection, and most of the machines in that report are showing as "Default" which means they are affected. You see we use a simple program in house called "Spark" that generates a Jabber notice from SEP, and we have a policy in place to ignore this. But the clients running the outdated policy have not got this policy and so are ending up in the report. Now I just have to try to work out a way to fix about 100 machines throughout our organization.

[Migration User]

That uber cool man... :) .. Good to know that things are working fine.. (Wth Symantec, you should be really blessed..:P) .... So, Like we suspected its just the cached information that failed to update... Umm... not sure why they couldn't update automatically.... And yes, It could be a lot of administrative task to do the same on more clients... :( .... May be we should write a batch file.... :) ...

And hey, just for yout information, when you have more clients reporting to the same group... It will display only 30 clients in a page by default.. If you want all of them in the same page with a scroll bar, you can do it...

Under group tasks, you have 'Set display filter' ... You can have a maximum of 999, if am not wrong.... :)

[Migration User]

I have what I think is the same problem with a few hundred clients.  The policy serial number in SEPM is incorrect and some of these also show the wrong client version or virus definition date.  I DO NOT want to visit every client.  I cannot get  "SMC -stop" to run remotely and smc.exe is not always in the same location from client to client.  Will there be a fix for this BUG??

[Migration User]

I do not find this an acceptable solution. 

[Migration User]

Well it was not a perfect solution, but it was a solution nevertheless. By deleting the BAK files and replacing the serstate and serdef files my machines are immediately reporting the corrrct group again.

I created a small vbs script to automate the fix. It prompts for machine name, then will read the remote machine's registry to get the symantec home folder, then stops the service, deletes the BAK files and replaces the two dat files and sylink.xml. So far it has been working for me. The SMC service also seems to start right back up again within seconds of the scrpt finishing. So it can be automated, and I guess if you really wanted to you could interface it with a spreadsheet of machine names so you don't even need to type them in.

Just not sure about posting script to the site though, so I won't post it here. If anyone would like to see it let me know, and I'll send you the script directly.

Cheers