Slow Transfer Rate VMware Converter P2V

[kakent138]

I am converting a physical Windows 2012 R2 server to a VM, but the transfer rate is abnormally slow, sub 50KB.

The conversion is only converting 2 drives, the boot volume (~300MB) and the C partition which is ~180GB so nothing huge.  I am also not doing any resizing so it is performing a block-level clone.

I went through the VMware best practices guide for optimizing the conversion rate so SSL encryption is turned off.  I set the "Data connections per task" to maximum, completely removed all AV, and made sure there were no other disk intensive tasks running during time of conversion.

The network is full Gig, and "flat" between the physical server and the VMware hosts/vCenter (one single switch).

The storage I am writing to is a Dell Compellent 8000 SAN connected via 8GB fiber channel, and the particular datastore I am writing this to is all flash.

All of this in my opinion should constitute an optimized (or at least better than average) environment for the conversion.

When I kick off the conversion it estimates ~1 hour to complete.  I can watch the transfer rate initially show ~100KB, but slowly dwindle over an hour down to ~30-40KB.  After about an hour the boot partition cloning shows completed, and then the estimated time remaining jumps up to 60+ DAYS.  All it has left is the C partition which again is only 180GB so I'm not sure why it would be this slow and take this long.

Other factors involved:

The physical server is running SQL 2012.  I am not converting any of the 8 SQL DB/Log/Temp/Backup/etc. partitions, as they are mounted via iSCSI from a SAN.  My VMware environment has direct access to this SAN so my plan is to mount these drives as RDMs once the conversion is complete.

The physical server has multiple physical NICs bonded into a "team" using Microsoft's adapter teaming.

This is also not a production/critical system so any suggestions/changes etc. can be done any time.  I am open to whatever.

Thanks in advance.

[RickVerstegen]

Did you also stop services before starting the conversion? Also remove software not needed on the box.

[kakent138]

I did not.  But I'm happy to do so.  When you say "all services" are we talking all running Windows services?  OS related as well as 3rd party?

I did remove any unnecessary software from the server before attempting the conversion.

[RickVerstegen]

Yes, stop as much services (OS related and third party) as you can and try again.

[kakent138]

Same result unfortunately.  I stopped ~50% of OS related services, the ones I could stop.  All SQL services.  And any unnecessary third party software was removed, and the remaining third party software services were stopped.

I've gone through the Worker/Agent/Server logs, and can upload them here if you think reviewing them would help.

Worker logs shows some Errors/Warnings prior to the Clone Tasks kicking off.  Here is a snippet of those:

018-10-15T08:22:24.520-04:00 warning vmware-converter-worker[03288] [Originator@6876 sub=task-2] [LiveDeviceReaderWriter] FSCTL_ALLOW_EXTENDED_DASD_IO for volume \\.\PhysicalDrive0 failed with error code 87

2018-10-15T08:22:24.520-04:00 info vmware-converter-worker[09320] [Originator@6876 sub=ThreadPool] Thread enlisted

2018-10-15T08:22:24.520-04:00 info vmware-converter-worker[06008] [Originator@6876 sub=task-2] [UsedClusterBitmapImpl] FlushFileBuffers for volume \\.\GLOBALROOT\Device\HarddiskVolumeShadowCopy3...

2018-10-15T08:22:24.520-04:00 info vmware-converter-worker[03288] [Originator@6876 sub=task-2] BlockLevelVolumeCloneMgr::CloneVolume: Using transferSize of 655360, blocksize 0 bytes, Write Cache Disabled.

-->

2018-10-15T08:22:24.520-04:00 warning vmware-converter-worker[06008] [Originator@6876 sub=task-2] [UsedClusterBitmapImpl] FlushFileBuffers failed for volume \\.\GLOBALROOT\Device\HarddiskVolumeShadowCopy3, error: 19

2018-10-15T08:22:24.520-04:00 info vmware-converter-worker[03288] [Originator@6876 sub=task-2] [StreamCacheWriter] - BlockSize 0, CapToFill 0

-->

2018-10-15T08:22:24.520-04:00 info vmware-converter-worker[03288] [Originator@6876 sub=task-2] BitmapBuilderCollection::ConsultModifiedClusterBitmap: the modified cluster bitmap will not be consulted for volume Disk#0_Partition#1

2018-10-15T08:22:24.520-04:00 info vmware-converter-worker[03288] [Originator@6876 sub=task-2] BitmapBuilderCollection::ConsultModifiedClusterBitmap: the modified cluster bitmap will not be consulted for volume Disk#0_Partition#1

2018-10-15T08:22:24.520-04:00 error vmware-converter-worker[03288] [Originator@6876 sub=task-2] BitmapBuilderCollection::ConsultUsedClusterBitmap: cannot consult the used cluster bitmap because this volume (Disk#0_Partition#1) does not support it

2018-10-15T08:22:24.520-04:00 info vmware-converter-worker[03288] [Originator@6876 sub=task-2] BitmapBuilderCollection::ConsultFileClusterBitmap: the file cluster bitmap will not be consulted for volume Disk#0_Partition#1, because the volume does not support it.

2018-10-15T08:22:24.520-04:00 error vmware-converter-worker[03288] [Originator@6876 sub=task-2] [GetVolumeSymlink] Cannot return symlink for detached volume Disk#0_Partition#1

2018-10-15T08:22:24.520-04:00 info vmware-converter-worker[05016] [Originator@6876 sub=ThreadPool] Thread enlisted

2018-10-15T08:22:24.520-04:00 error vmware-converter-worker[03288] [Originator@6876 sub=task-2] AllClusterBitmapBuilder: failed to obtain cluster size, setting the cluster size to 4096 bytes by default for volume Disk#0_Partition#1; error: Requested information is not available

2018-10-15T08:22:24.520-04:00 info vmware-converter-worker[03288] [Originator@6876 sub=task-2] GetStartingBitmapOffset: the starting bitmap offset is 0

2018-10-15T08:22:24.520-04:00 warning vmware-converter-worker[06008] [Originator@6876 sub=task-2] FileClusterBitmapBuilder [AddBadBlockExtents] Unable to get bad blocks extents for \\.\GLOBALROOT\Device\HarddiskVolumeShadowCopy3\. Error: 5

2018-10-15T08:22:24.520-04:00 info vmware-converter-worker[06008] [Originator@6876 sub=task-2] GetStartingBitmapOffset: the starting bitmap offset is 0

2018-10-15T08:22:24.536-04:00 info vmware-converter-worker[02080] [Originator@6876 sub=task-2] [UsedClusterBitmapImpl] FlushFileBuffers for volume \\.\GLOBALROOT\Device\HarddiskVolumeShadowCopy4...

2018-10-15T08:22:24.536-04:00 warning vmware-converter-worker[02080] [Originator@6876 sub=task-2] [UsedClusterBitmapImpl] FlushFileBuffers failed for volume \\.\GLOBALROOT\Device\HarddiskVolumeShadowCopy4, error: 19

2018-10-15T08:22:24.536-04:00 warning vmware-converter-worker[02080] [Originator@6876 sub=task-2] FileClusterBitmapBuilder [AddBadBlockExtents] Unable to get bad blocks extents for \\.\GLOBALROOT\Device\HarddiskVolumeShadowCopy4\. Error: 5

2018-10-15T08:22:24.536-04:00 info vmware-converter-worker[02080] [Originator@6876 sub=task-2] FileClusterBitmapBuilder::FindExtentsToSkip: Skipping file extents at 0x00000002085ac000 length 45097156608 bytes

Please let me know if anything stands out, or you would like me to upload the full log.

[Rosen Tomov]

The full logs are more informative (some of the errors are not error that stops the conversion).

However for slow network speed you'll need to check the path between source computer and destination ESX@902

The slow speed can be result of bad/long/throttled route path or router with low priority of your network traffic.

HTH

[kakent138]

Thanks for the recommendation  I will investigate network path and configuration and get back to you.  In the meantime, the job log is attached.

[Rosen Tomov]

I see big performance degradation: when starting "percentage: 0, xfer rate (Bps): 41119369", but then "percentage: 9, xfer rate (Bps): 75111", then "percentage: 50, xfer rate (Bps): 18533", and so on..., "percentage: 100, xfer rate (Bps): 16497" finally. Then the speed is "stabilized" for next disk around "percentage: 6, xfer rate (Bps): 39610"

If the network is not the root of the problem, then the disks and/or VSS should be.

Verify your VSS with vssadmin command - how many shadows, providers, where shadows are hold, etc. The whole conversion's consistency is based on snapshots, if snapshots are slow, then the conversion will be slow because of reading from them.

There is a (small) chance to upgrade conversion performance with <vstor2VolumesHotplug>true</vstor2VolumesHotplug> in converter-worker.xml - this option will not using big memory cache for destination disks and will load up network. (but this is a wild guess)

There is option "Data connections per task" in administration menu - you can increase the number of simultaneously transferred disks - try it (however there are limitation fro ESX side so put the value between 3 and 5)

HTH

[kakent138]

I want to thank everyone for all of the input on this.

The issue ended up being network related on the Windows OS side.  The server had a Heartbeat NIC attached as it was once part of a SQL AO Cluster, and for some reason (I believe because of subnet overlap) Converter was using that 10/100 NIC to perform the transfer.  As soon as I disabled that NIC the conversion ran at 40MB+ and completed well within an hour.

Thanks again,

Kyle