VMs Stuck on Applying Computer Settings after upgrade to SEP 12.1 RU1

[dsmith1954]

VMWare Tools v4.0.0 build 392990

XP SP3 with latest patches and updates

Upgraded from SEP 11 RU6 using SEPM v12.1 RU1

 

After the upgrade, SEPM showed that the VM required a restart, so I did. When it started back up, it stuck at "Applying Computer Settings". I left it for over 30 minutes while I checked other VMs and a few of them had the same problem.

I was able to recover the others by going into safe mode and removing SEP using CleanWipe. On the other VMs, I was able to manually install SEP 12.1 RU1 (exported a full install and removed the /qn from setup.ini) and everything was fine. Not with this one.

I noticed that SEP installed a Teefer driver v12.1.808.5 (8/16/2011), which disabled the network card. When I revert back to v11.0.4819.6, my network starts working again - usually after disabling and enabling the card a few times.

When it hangs, there are two DCOM errors in the System log:

• ID: 10005
• Source: DCOM
• Description: DCOM got error "The service cannot be started, either because it is disabled or because it has no enabled devices associated with it. " attempting to start the service BITS with arguments "" in order to run the server:
  {4991D34B-80A1-4291-83B6-3328366B9097}

I'm trying one last install of 12.1 RU1 on this VM, and will be attempting to leave 12.1 with teefer drivers of 11.

Anyone have any suggestions on what may be causing this? Is there an issue with VMs and 12.1 RU1 that I missed?

[dsmith1954]

That didn't work, and uninstalling SEP left behind the teefer drivers, so it still doesn't boot properly. Running CleanWipe again...

[dsmith1954]

Well, it appears that something comes in through LiveUpdate that disables the SEP service.

A somewhat clean installation of v12.1 RU1 installs ok. LiveUpdate runs and fails on one of the updates. All appears to be good until a restart of the system. SEP will no longer start because the service has been disabled. Not sure yet what in the LiveUpdate causes everything to go haywire, and why for only some VMs and not others.

Anybody run across this yet?

[dsmith1954]

Anyone?

[Migration User]

I'm experiencing a similar issue with our Windows 7 clients.

It's hanging on the Applying Security Policy, or Power Policy startup phase. Several reboots or removing SEP resolves the issue. It's not widespreed, probably less than 1% of our 25,000 seat deployment, and is affecting fresh installs as well as upgraded clients.

 

I will post back here with my findings, as it may be related to what you are experiencing.

 

Cheers,

Andy

[dsmith1954]

I'm beginning to think 12.1 RU1 is a bust, although I've only had problems with XP VMs so far. My hardware, XP or 7, all seem to work fine.

My major problem is my servers. About 75% are VMs, and with a 30% failure rate on VMs, I'm not about to upgrade them until this is resolved. One desktop VM failure puts one person out of commission for a short time. One server VM failure puts hundreds of people out of commission. I can't risk that.

I've been working with Support, but the only thing they've been able to determine so far is that they think there may be a problem with the VM's network card. They're not sure, but they've given me some debugging switches to use with the MSI to see if they can determine any problems from there.

[Migration User]

I have removed Network Threat Protection and the issue is resolved, though this is an integral part of the suite.

Our clients are non RU1 at this stage. I don't have any VDI's and servers (VMware) tend not to reboot often so haven't experienced it on servers.

I'm not sure what NTP could be doing during this phase of start-up that could be causing it to hang. 

Tomorrow I'll begin with file/registry logging. Be interesting if we have the same issue.

[dsmith1954]

Support has come up fairly empty so far. The problem occurs after installation and reboot - when the migration is supposed to take place. After reboot, the migration is supposed to startup the Windows Installer to complete the switchover, but doesn't. Our attempts at logging hasn't helped because of that.

Support has a bunch of my logs - before and after upgrade - so hopefully they'll see something soon.

[Migration User]

I'll open a case and submit some logs also. I'm convinced my issue is with NTP - interestingly didn't have it with the last releases of 11.0

[dsmith1954]

I didn't have any problems upgrading previous versions either. One thing I was shown was that current policies didn't all upgrade to the latest versions when I upgraded the SEPM. When I created a new policy - Application and Device Control Policy - I saw that there were more options available. The old policies still don't have all of those options, so now I have to create all new App & Device Control policies. What a pain trying to re-create all of those application controls.

Support had me create a new group and add my VMs that failed into that group. It only had a new default AV and LiveUpdate policy. They still failed on the migration after restart...

[Migration User]

Have you tried this artcile.

It tells how to remove the "Startup scan" as part of best practice for Vitual Environments:

http://www.symantec.com/business/support/index?page=content&id=TECH180229

* * * * *

Specifically, ensuring that the "startup scan" is disabled.

[Migration User]

Ill try the startup-scan in my intance also.

 

In the eventlog I do have a few errors generated before it timesout from its hanging state.

Log Name:      Application
Source:        Microsoft-Windows-WMI
Date:          22/02/2012 10:52:36 AM
Event ID:      10
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      PPCOW022.cloud.local

Description:
Event filter with query "SELECT * FROM __InstanceModificationEvent WITHIN 60 WHERE TargetInstance ISA "Win32_Processor" AND TargetInstance.LoadPercentage > 99" could not be reactivated in namespace "//./root/CIMV2" because of error 0x80041003. Events cannot be delivered through this filter until the problem is corrected.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Microsoft-Windows-WMI" Guid="{1edeee53-0afe-4609-b846-d8c0b2075b1f}" EventSourceName="WinMgmt" />
    <EventID Qualifiers="49152">10</EventID>
    <Version>0</Version>
    <Level>2</Level>
    <Task>0</Task>
    <Opcode>0</Opcode>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2012-02-21T21:52:36.000000000Z" />
    <EventRecordID>1449</EventRecordID>
    <Correlation />
    <Execution ProcessID="0" ThreadID="0" />
    <Channel>Application</Channel>
    <Computer>PPCOW022.cloud</Computer>
    <Security />
  </System>
  <EventData>
    <Data>//./root/CIMV2</Data>
    <Data>SELECT * FROM __InstanceModificationEvent WITHIN 60 WHERE TargetInstance ISA "Win32_Processor" AND TargetInstance.LoadPercentage &gt; 99</Data>
    <Data>0x80041003</Data>
  </EventData>
</Event>

 

 

Log Name:      Application
Source:        Microsoft-Windows-User Profiles Service
Date:          22/02/2012 10:51:21 AM
Event ID:      1530
Task Category: None
Level:         Warning
Keywords:     
User:          SYSTEM
Computer:      PPCOW022.cloud.local

Description:
Windows detected your registry file is still in use by other applications or services. The file will be unloaded now. The applications or services that hold your registry file may not function properly afterwards. 

 DETAIL -
 2 user registry handles leaked from \Registry\User\S-1-5-21-2537138084-603115122-2870846346-1147:
Process 616 (\Device\HarddiskVolume2\Windows\System32\lsass.exe) has opened key \REGISTRY\USER\S-1-5-21-2537138084-603115122-2870846346-1147
Process 616 (\Device\HarddiskVolume2\Windows\System32\lsass.exe) has opened key \REGISTRY\USER\S-1-5-21-2537138084-603115122-2870846346-1147

Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Microsoft-Windows-User Profiles Service" Guid="{89B1E9F0-5AFF-44A6-9B44-0A07A7CE5845}" />
    <EventID>1530</EventID>
    <Version>0</Version>
    <Level>3</Level>
    <Task>0</Task>
    <Opcode>0</Opcode>
    <Keywords>0x8000000000000000</Keywords>
    <TimeCreated SystemTime="2012-02-21T21:51:21.080030700Z" />
    <EventRecordID>1436</EventRecordID>
    <Correlation ActivityID="{848FAA0C-E003-0001-6687-C098E1F0CC01}" />
    <Execution ProcessID="1060" ThreadID="3548" />
    <Channel>Application</Channel>
    <Computer>PPCOW022.cloud.local</Computer>
    <Security UserID="S-1-5-18" />
  </System>
  <EventData Name="EVENT_HIVE_LEAK">
    <Data Name="Detail">2 user registry handles leaked from \Registry\User\S-1-5-21-2537138084-603115122-2870846346-1147:
Process 616 (\Device\HarddiskVolume2\Windows\System32\lsass.exe) has opened key \REGISTRY\USER\S-1-5-21-2537138084-603115122-2870846346-1147
Process 616 (\Device\HarddiskVolume2\Windows\System32\lsass.exe) has opened key \REGISTRY\USER\S-1-5-21-2537138084-603115122-2870846346-1147
</Data>
  </EventData>
</Event> 

 

Log Name:      System
Source:        Microsoft-Windows-DistributedCOM
Date:          22/02/2012 10:41:06 AM
Event ID:      10010
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      PPCOW022.cloud.local

Description:
The server {E10F6C3A-F1AE-4ADC-AA9D-2FE65525666E} did not register with DCOM within the required timeout.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Microsoft-Windows-DistributedCOM" Guid="{1B562E86-B7AA-4131-BADC-B6F3A001407E}" EventSourceName="DCOM" />
    <EventID Qualifiers="49152">10010</EventID>
    <Version>0</Version>
    <Level>2</Level>
    <Task>0</Task>
    <Opcode>0</Opcode>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2012-02-21T21:41:06.000000000Z" />
    <EventRecordID>2985</EventRecordID>
    <Correlation />
    <Execution ProcessID="0" ThreadID="0" />
    <Channel>System</Channel>
    <Computer>PPCOW022.cloud.local</Computer>
    <Security />
  </System>
  <EventData>
    <Data Name="param1">{E10F6C3A-F1AE-4ADC-AA9D-2FE65525666E}</Data>
  </EventData>
</Event>

Have you got any similar events raised?

I'll need to look up that DCOM Guid (http://www.symantec.com/connect/forums/dcom-error)

 

[dsmith1954]

One of the tests I did with support was to create a new group with only a new AV policy. There were no startup scans defined in the policy.

Support is still going over my logs to see if there is anything the can see.

[Benjamin John]

Did you get anywhere with this?

[dsmith1954]

Not really. Support wants a copy of one of my VMs so they can look it over. That isn't going to happen. We are a private company. We don't allow vendors to remote control our computers, nor do we send copies of our VMs out to vendors. So, I guess we're stuck not knowing why.

[dsmith1954]

I got most of my VMs to upgrade by running CleanWipe, and/or moving them to a group with minimal policies that were new.

I don't know why Symantec can't upgrade policies when they upgrade SEPM.

I still have two VMs that won't upgrade. If I remove the VM network card, SEP installs just fine and the VM restarts just fine. Once I re-install the network card, they get stuck on Applying Computer Settings. This still happend on SEP 12.1 RU1 MP1 as well.

I guess I'm just going to have to create new VMs for those two...