Extremely Slow Performance

[bjustin13]

I was trying to P2V a server tonight and ran into substantial performance issues.   I was getting transfer speeds approximately 500KB/sec on each attempt.

Here are some of the steps I tried in troubleshooting.

1. Installed VMware Converter 5.1 locally on the server and ran it from there.
2. Removed VMware Converter 5.1 from the server and ran it from the Client/Server install from another server, version 5.0. (All subsequent attempts were client/server attempts from this server)
3. Restarted the source server with all services except for Microsoft services, TeamViewer and the McAfee validation trust protection services running.
4. Tried to change the cloning operations to run on ESX0 (previous attempts were to ESX1)
5. Tried to change the destination datastore to the local datastore on ESX1 (was using shared iSCSI storage)
6. Tried without changing the disk type (to avoid volume based cloning) to local datastore on ESX1

I've run P2V migrations successfully with the client/server install on the server attempted above.  So, it certainly points to something on the source server.  I didn't notice any CPU, RAM or Disk bottlenecks during the attempts.  The only thing that I could find was that I couldn't stop the McAfee Validation Trust Protection service from running.  Not sure if that would have caused these issues.

Has anybody encountered anything similar?

ps.  Forgot to mention that I have SSL is disabled in the VMWare Worker service.

[bjustin13]

Found this in the logs:

-----------------

2013-05-21T18:44:58.026-05:00 [04968 info 'Default'] [,0] Partition:Invalid sector magic number.

2013-05-21T18:44:58.026-05:00 [04968 info 'Default'] [,0] Disk number 1 has been skipped because of errors while reading partition table

2013-05-21T18:44:58.042-05:00 [04968 info 'Default'] [,0] Partition:Invalid sector magic number.

2013-05-21T18:44:58.042-05:00 [04968 info 'Default'] [,0] Disk number 2 has been skipped because of errors while reading partition table

2013-05-21T18:44:58.042-05:00 [04968 info 'Default'] [,0] Partition:Invalid sector magic number.

2013-05-21T18:44:58.042-05:00 [04968 info 'Default'] [,0] Disk number 1 has been skipped because of errors while reading dynamic disks header or LDM database is corrupted

2013-05-21T18:44:58.042-05:00 [04968 info 'Default'] [,0] Partition:Invalid sector magic number.

2013-05-21T18:44:58.042-05:00 [04968 info 'Default'] [,0] Disk number 2 has been skipped because of errors while reading dynamic disks header or LDM database is corrupted

2013-05-21T18:44:58.042-05:00 [04968 info 'Default'] [,0] Partition:Invalid sector magic number.

2013-05-21T18:44:58.042-05:00 [04968 info 'Default'] [,0] Partition:Invalid sector magic number.

2013-05-21T18:44:58.042-05:00 [04968 warning 'Default'] [MoveActiveDiskIfNeeded] GetFirstBootDisk failed, mntapi error: 176

2013-05-21T18:44:58.698-05:00 [04968 info 'Default'] [,0] Partition:Invalid sector magic number.

2013-05-21T18:44:58.698-05:00 [04968 info 'Default'] [,0] Partition:Invalid sector magic number.

2013-05-21T18:44:59.354-05:00 [04968 info 'Default'] [,0] Partition:Invalid sector magic number.

2013-05-21T18:44:59.354-05:00 [04968 info 'Default'] [,0] Partition:Invalid sector magic number.

2013-05-21T18:44:59.354-05:00 [04968 info 'Default'] Reusing existing VIM connection to vCenter

2013-05-21T18:44:59.354-05:00 [04968 warning 'Default'] [PopulateCapabilities] Volume-based cloning was disabled due to: <no volumes are recognized>

2013-05-21T18:44:59.354-05:00 [04968 info 'Default'] [LogMessagesForQA] System volume is not detected.

2013-05-21T18:44:59.354-05:00 [04968 info 'Default'] [BaseComputerImpl::Open] Config file: [LUN1] server.domain.local/Serverl.vmx

2013-05-21T18:44:59.354-05:00 [04968 info 'Default'] TargetVmManagerImpl::GetStorageMapping: Source Disk id missing with new target disk disk-2001

2013-05-21T18:44:59.354-05:00 [04968 info 'Default'] [,0] Partition:Invalid sector magic number.

[Rosen Tomov]

What you mean with "extremly slow performance"? What the speed transfer is? What are your expectation? What machine (CPU/RAM/Disks) you want to convert?

This part of log doesn't say anything wrong :smileyhappy:

[bjustin13]

Converting at about 700KBps.

The machine is a Xeon E5506 server, with 4GB RAM.  Not sure internal RAID configuration of the disks but the size of the disks are about 850 GB.

I had talked with VMware support and they had me run convert to workstation format to locally attached USB and to a network share.   The USB conversion was happening around 20MB/s and the network share was 50MB/s.

[Rosen Tomov]

What's are your expectations? As I see your USB stick is slow, but network transfer is good, isn't it?

[bjustin13]

Those transfer speeds are when I use workstation formats.    I need these converted as VMWare Infrastructure virtual machine.

I would expect somewhere in the ballpark of 40-70MB.

[Rosen Tomov]

It's always depends on physical limits of disks bandwidth, networks bandwidth, and total loading of source and destination machines. You should try and will see the speed of transfer.

[bdeltel]

Bonjour,

I had a few performance issues during P2V. Most of the time the network was misconfigured. If you have the Physical machine NIC card in autosensing mode and the physical switch in fixed mode, I notice a really slow conversion/copy rate.

Most of the times, I solved that issue by setting both sides either on auto-sensing or on fixed speed.

Hope this helps.

Best regards.

[Clausbc5]

I believe I have the answer to that one.

We currently have a support case running with McAfee about this. If the McAfee Validation Trust Protection Service is running because of one or more McAfee products are installed then you should see a performance degradation that is quite severe.

Without any McAfee product installed we tried copying a file of 3,5 gig to our 2012 server. That took 14 seconds.

Once we installed McAfee antivirus we noticed that the copy took 60 seconds.

With HIPS installed it increased to 90 seconds.

After that we disabled all the McAfee services we could, which left out the McAfee Validation Trust Protection Service. Started the copy and still 90 seconds. As I'm writing this, we (me and my colleagues) are trying to convince McAfee that there must be a bug in this service.

If anyone agrees with my observations please let me know.

We don't see quite the same performance degradation on Windows Server 2008 R2 SP1, but it is there. And it doesn't look good on Windows 7 SP1 as well.

We did the test with McAfee Antivirus 8.8 patch 3 and HIPS 8 patch 3 installed.