DLL corruption during VMWare Tools install (msvcp71.dll)

[StJoes]

We recently upgraded our ESX server to vSphere 4.0. After bringing the vm's back online, and then upgrading the Tools, we had issues with several applications working. Troubleshooting seems to have revealed that during the Tools installation a new version of msvcp71.dll is used, and any existing version is (apparently) removed from the system32 directory. We found three copies of the dll in various directories under Windows\temp all in directories used during the VMWare Tools install.

By copying the file back into the system32 directory, the application (MS SQL was the main application with issues) now works correctly.

Brian

[htwnrver]

Can you post the other applications so that we can be aware of them when we upgrade?

Thanks,

[StJoes]

We also had problems with pcAnywhere (v10), and Symantec Antivirus Corporate Edition (v10), which we believe were associated with the same issue. We haven't yet reinstalled those applications to test whether this fixed there issues or not.

Brian

[TAMW]

We had the exactly same msvcp71.dll problem with MS-SQL 2000.

After the vmware tools update the SQL service refused to start with the error that is was missing the msvcp71.dll

[AndrewAbo]

I encountered multiple VM's where Symantec Endpoint stopped working after the upgrade to 4.0. Posting this due to multiple others with Symantec products experiencing issues.

The problem was fxed by going to Add/Remove Programs, and running Modify-->Repair on Symantec Endpoint.

[adamsb]

One way I "fixed" this was after i finished the tools/updates I assigned a new package to be applied to the clients group. This ran across the guests and reinstalled the clients "fixing" the problem.

[jawad]

I have also had this problem on host that has SAV10 installed. Did a repair on the SAV install and all OK.

But pretty annoying to do this on 40-50 VMs...

[Texiwill]

Hello,

Moved to Virtual Machine and Guest OS.

Best regards, Edward L. Haletky VMware Communities User Moderator, VMware vExpert 2009
Now Available on Rough-Cuts: 'VMware vSphere(TM) and Virtual Infrastructure Security: Securing ESX and the Virtual Environment'[/url]
Also available 'VMWare ESX Server in the Enterprise'[/url]
[url=http://www.astroarch.com/wiki/index.php/Blog_Roll]SearchVMware Pro[/url]|Blue Gears[/url]|Top Virtualization Security Links[/url]|Virtualization Security Round Table Podcast[/url]

[AVK82]

We had the same problem with all SQL instances after upgrading: the msvcp71.dll file was completely missing. Copied from another similar machine: it works like a charm now.

Thank you for posting the solution!

[sbolter]

Apparently when you install the upgraded VM Tools on the virtual machine, it deletes a file called msvcp71.dll from the System 32 directory.

This also impacts IBM\Tivoli (TSM) Backup software. The TSM Service requires this Dll to start. Quick fix is to copy this file from another server with the TSM software back into the system32 folder. I noticed this once the VM Tools upgrade completed logging in produced the error that service could not start due to msvxp71.dll file missing. Once the File is replaced the service starts. Also upgrading the Hardware to v7 has no impact on this dll from what I have seen only during the VM Tools upgrade.

Error on the side of Caution Take a snapshot first!!!!

[gkheigde]

We upgraded last night. The same problem. VMWare Tools installation busted our DATEV installation. You have to do the repair the "Installationspacket" in the DATEV Software.

[apohlgeers]

This also affected our Laserfiche 7 servers. We couldn't get the server to start but luckily when I started the client it reported the missing file. Replacing the .dll in the system32 folder worked but is really is a pain. Anyone know why the VMware upgrade does this?

[chjones]

Hey all,

Has anyone heard anything from VMware about this issue? I experienced it as well today. SAV 10.x failed on 30+ VMs after VMware Tools was upgraded.

-

---

Chris Jones

Canberra, Australia

[Chuck8773]

We have not upgraded the tools on many VM's yet. I have two experiences with this dll.

1. I imported a VCB backup of hardware 4 with old tools onto a vsphere 4 host. The VM was created with hardware 7. The tools were still old. When it came up it complained about this dll andSQL Server simply would not start. We troubleshot it for a while before noticing that the hardware had been upgraded. We restored it again to a 3.5 host and everything worked just fine.

2. Saturday, I upgraded the vmtools and hardware on our virtual vCenter and sql server. They started up just fine. Yesterday, the vcenter server started complaining about that dll at about 10:30 am. Then the vcenter service crashed. I couldnot restart it. I rebooted bothVM's and still nothing. vCenter was reporting the the login to the database failed. I tested the password we have on file and it worked. I ended up resetting the password to the database within vcenter and it started working. vCenter forgot the password. When I spoke with VMWare about this, I asked them to add this dll information to the ticket for future reference as the timing of the events was too close to be coincidence. It seems like the issue with that dll made vCenter forget the password. Not sure how that can be.

Hope this helps someone.

Charles Killmer, VCP

If you found this or other information useful, please consider awarding points for "Correct" or "Helpful".

[jjahns]

Note: 64-bit machines keep the file at C:\Windows\SysWow64

If you have good backups, or copied itt, simply restore it after the VMware Tools upgrade. There's another workaround.

If you don't backup, use the Windows Install CD. It works. Tested it already.

Our "patient zero" ended up being a production instance of SQL Server 2000 that processes utility meter reads for our company on an hourly basis. So... our downtime was significant. Given that this is only the 2nd bug I have ever experienced from VMware in over 2 years, I can overlook this error. Its not that big of a deal because there are smart people on these forums who spotted this problem straight away.

Good luck everyone.

[jjahns]

Also you might want to try uninstalling the old tools before upgrading to new ones.

And... update your virtual machine hardware first before doing that. This has been successful for me.

[rogerfinch]

just found this out by upgrading the tools on a few servers.

why isn't this a known issue?

[jjahns]

Based on the response I received on our support request, it is a known issue.

The problem occurs when the tools are being updated from 3.5 U4 to 4.0. Before upgrading to 4, you're supposed to update 3.5 to Update 5. Then, you have to go in and update the tools to U5 version. Then, you can update them to 4.

This seems to answer the problem. The official workaround is to find the DLL off of another machine (same version... duh) and replace the file, or if you have backups of it, restoring it.

It only affects 1 DLL file, and that is the msvcp71.dll.

Hope this is a help to all of you out there.

[rogerfinch]

thanks for the info.

I've vmtool'ed all our virtual servers now. we lost pcanywhere 11.5 on one, and oracle running on an sms 2003 box, but that was all.

the virtual hardware is interesting. 20 in the dmz were fine, but those in a dhcp environment started getting addresses while the nic was reinstalled and ip settings applied. the static was put back in, but by then dhcp has registered the new address in dns, wiping out the static entry!

only another 70 or so to do. manually.

oh, and wins disappears too

[chjones]

I have upgraded three of my ESX Clusters to version 4.0 and have found something interesting.

The first cluster I did, I put each host into maintenance mode manually and vmotion took care of moving the VMs. Then I used vSphere Update Manager to upgrade the host to 4.0, install the 3 released patches for ESXi 4.0, then exit maintenance mode and move onto the next host. I hadn't done anything to the VMs themselves other than allow vmotion to move them around as I upgraded each host. Those VMs that have the option "Check and upgrade tools before each power-on" all seemed to automatically upgrade VMware Tools on the guest when a vmotion performed that moved the VM onto a version 4 host. I would have thought this wouldn't happen, but only when you shut down the VM and power it back on? Anyway, every one of those VMs had the MSVCP71.DLL issue.

The VMs that didn't have the tools upgrade option enabled, manually had their VMware Tools upgraded. None of these VMs had the DLL issue.

On the second cluster I upgraded, I did it out of hours and shut down every VM in the cluster. I then upgraded the hosts to 4.0 and then powered back on the VMs. Each of the VMs automatically upgraded VMware Tools after Windows booted and then restarted. None of them had the DLL issue.

On the third cluster, I did a mix of the previous two. Some VMs I shut down and others I allowed vmotion to move them online. The VMs that were moved with vmotion automatically upgraded VMware Tools and all had the DLL issue.

I have two more clusters to upgrade and I don't like that I will have to shut down all the VMs if I want to avoid this issue :smileysad:

Chris

[Chuck8773]

I started a support ticket with this and thought I would share what I heard.

"At this point in time, we have narrowed down the problem you are experiencing to a known problem in ESX. Our core product development team is aware of this problem, and have a problem report open in order to have it resolved. The fix will be release in a ESX 3.5 update 5 and ESX 4.0 Update 1. The current workaround is to copy the missing file over to the C:\windows\system32\. For your convenience, we have an RSS feed available to alert you when new knowledge base articles are created. Besides the obvious benefits of keeping current on knowledge base articles (which include descriptions and resolutions to common problems), each product update has a knowledge base article written for its release. These articles describe the problem in depth, and the patch in which the fix is included. This is your best opportunity to track fixes. You may subscribe to the RSS feeds at the link below.

VMware RSS Feed:

http://vmware.simplefeed.net/subscription"

No estimate on when ESX 4 U1 will be released yet.

Charles Killmer, VCP

If you found this or other information useful, please consider awarding points for "Correct" or "Helpful".

[JoFu]

Today I upgraded from ESXi 3.5 U4 to ESXi 4. I ran into the same problem, in my case Panda Security failed to start because of the missing dll.

I'm about to reinstall the Panda protection right now, and I assume it's fixed after that.

[jjahns]

You really don't need to do that. It's a windows system file. If you're backing it up, restore it. If you've got another machine with the same version of OS, use that. It works. Tested.

[iemmadavis]

I also face alot of problems while doing so. But I searched for it on the web and  found websites like dll-me.com , pcpatchedup.com where i found detailed instructions how to solve these issues. The msvcp71.dll can also be downloaded from the same. 

Emma Davis 

[davisjohn2377]

You can solve this by downloading and installing msvcp71.dll files in your system directory