Could not complete network copy for file - Template deployment

[uncLee]

Greetings,

I've encountered what I believe to be a bug with New-VM -Template cmdlet and was wondering if anyone else could shed some light on this subject. I consistently get this network file copy error when I try to deploy a new vm from a template via powercli (error message below). I can do the exact same operation from the Vsphere web client and it will complete successfully. I have tested powercli version 6.3-6.5 and get the same results on each version. I am on a network with jumbo frames enable and have our hosts directly attached to storage via management NIC interface. Network latency is not an issue as both host and storage networks are dedicated. This is really just to be thorough because this exact same operation works when initiated from the web client. Please let me know if I can clarify anything and your input is appreciated. Thanks!

Error Message:

New-VM : 3/21/2017 10:40:00 AM	New-VM		The operation for the entity "Template - Windows Server 2012r2" failed with the following message: "Could not complete network copy for file
/vmfs/volumes/03195b0e-e6997962/Template - 2012r2/Template - 2012r2.vmdk"

At line:1 char:1

+ New-VM -Name TestTemp -VMHost (Get-VMHost -Datastore esmc4_fucs1_f1c1 ...

+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

    + CategoryInfo          : NotSpecified: (:) [New-VM], NetworkCopyFault

    + FullyQualifiedErrorId : Client20_TaskServiceImpl_CheckServerSideTaskUpdates_OperationFailed,VMware.VimAutomation.ViCore.Cmdlets.Commands.NewVM

This is the one liner I used to recreate the error message:

New-VM -Name TestTemp -VMHost (Get-VMHost -Datastore esmc4_fucs1_f1c1_incoming_01 | select -First 1) -Datastore "esmc4_fucs1_f1c1_incoming_01" -Template (Get-Template "Template - Windows Server 2012r2")

All the best,

Leander

[LucD]

Did you already try changing or disabling the web timeout value with the Set-PowerCLIConfiguration cmdlet and the WebOperationTimeoutSeconds parameter?

Do you find any more information about the error in the vpxd file?

[uncLee]

Thank you for the prompt reply and sorry for my delayed response.

I tried the disabling the web timeout value with a negative integer but that did not resolve the issue. I also set it to an absurdly large number and same result.

I've retrieve the vpxd file but admittedly it does not mean much to me. I've located one of the exceptions and I believe this may be the meaningful part:

-----------

2017-03-21T17:30:16.929Z error vpxd[7F5F38183700] [Originator@6876 sub=VmProv opID=428dece1-d8] Aborting task tracking since task vim.Task:task-51148 failed

-->

2017-03-21T17:30:16.929Z error vpxd[7F5F38183700] [Originator@6876 sub=Datastore opID=428dece1-d8] [VpxdDatastore::UrlToDSPath] Received a non-url [/vmfs/volumes/03195b0e-e6997962/Template - 2012r2/Template - 2012r2.vmdk], instead of a url

2017-03-21T17:30:16.938Z error vpxd[7F5F38183700] [Originator@6876 sub=VmProv opID=428dece1-d8] [WorkflowImpl] Get exception while executing action vpx.vmprov.CopyVmFiles: vim.fault.NetworkCopyFault

------------

I've attached the log file from that day as well and used a find on "Template" to locate the exceptions so you can look at the context of this error. I'm a relatively novice VMware admin so any guidance you can provide for how to interpret this log file would be much appreciated. I'd rather learn how to troubleshoot this effectively, than have someone give me the answer.

Thanks again for your time and assistance.

Best,

Leander

[LucD]

Did you do a Disconnect-ViServer/Connect-ViServer after changing the timeout?
See also Get-Log: The operation has timed out

[uncLee]

I did not originally but I just confirmed $global:DefaultVIserver.ExtensionData.Client.ServiceTimeout was set to -1 and this did not resolve the issue. Still getting the "Could not complete network copy for file" for error. The error message in the vpxd.log file has the same error message about the URL for the DataStore path not being valid.

2017-03-27T13:43:44.904Z error vpxd[7F5F2B46A700] [Originator@6876 sub=Datastore opID=2c9dce6c-a7] [VpxdDatastore::UrlToDSPath] Received a non-url [/vmfs/volumes/03195b0e-e6997962/Template - 2012r2/Template - 2012r2.vmdk], instead of a url

2017-03-27T13:43:44.914Z error vpxd[7F5F2B46A700] [Originator@6876 sub=VmProv opID=2c9dce6c-a7] [WorkflowImpl] Get exception while executing action vpx.vmprov.CopyVmFiles: vim.fault.NetworkCopyFault

[uncLee]

I've had a little time to troubleshoot this further.

After researching "[VpxdDatastore::UrlToDSPath] Received a non-url" I ran across this kb article.

Deploying multiple virtual machines in VMware vCenter Server 5.x and 6.0.x from the same template fails with the error:

While this is not an exact match to my issue it does reference the same error message. I looked at my templates and they all have -ctk.vmdk files because they were created from virtual machines with snapshots. While this makes sense from the perspective that the templates are essentially clones of the existing VMs, it doesn't make sense for a template to have a -ctk file since they are supposed to be static copies. So maybe the presence of the -ctk file is causing this exception... easy enough to test. I created a new virtual machine (no OS loaded), confirmed that no -ctk.vmdk file existed for this VM, cloned the virtual machine to a template. Then confirmed that the new template didn't have a -ctk.vmdk file, updated my one-liner from the original post to reference the new template and fail. Dug into the vpxd.log file again and found the same exception message "[VpxdDatastore::UrlToDSPath] Received a non-url". So, still don't have an answer but thought it was worth mentioning to preventing duplicating effort. Thanks as always for your time and effort.

Code:

New-VM -Name TestTemplate -VMHost (Get-VMHost -Datastore esmc4_fucs1_f1c1_incoming_01 | select -First 1) -Datastore "esmc4_fucs1_f1c1_incoming_01" -Template (Get-Template "NewTemplate")

Error:

New-VM : 3/27/2017 2:48:10 PM    New-VM        The operation for the entity "NewTemplate" failed with the following message: "Could not complete network copy for file /vmfs/volumes/03195b0e-e6997962/NewTemplate/NewTemplate.vmdk"   

At line:1 char:1

[LucD]

Just to make sure we're dealing with a PowerCLI issue, and not a more general vSpshere issue, can you create a VM from that template via the Web Client?

[uncLee]

Correct

[LucD]

Then I would suggest you open a SR for this issue, looks indeed like a bug to me.

[uncLee]

Will do, thanks

[uncLee]

FYI for the community, this issue is not resolved. I submitted a SR (17419352303) for this issue but VMware will not assist because we do not pay for SDK support entitlement. Just wanted to let you all know in case anyone else runs into this issue.

[LucD]

That is not the correct way to handle that SR.
For obvious issues with the cmdlets, and not your code, you don't need to have SDK Support.

I'll ping the PM.

[uncLee]

Thank you. Your time and effort are much appreciated.

[jakerpowercli]

uncLee​ Sorry you're having a bad time with this :smileysad:

Based on your description, the New-VM -template command executes correctly, and the problem occurs when the template copy starts. This is a symptom of an issue with your network settings or switches. I have a hunch that the reason that it is consistent with PowerCLI and works fine with the GUI has to to with the source and target ESXi hosts. As an additional test, can you create a VM from template targeting the same host, once from PowerCLI and once from the GUI?

Some additional reading: Storage migration fails with the error: Could not complete network copy for file path to file 

[uncLee]

No worries at all, these things keep the job interesting :smileywink:. To be fair, I've learned a ton troubleshooting this issue. I also apologize if this post caused anyone trouble.

This was an oversight on my part. I accounted for the specific datastore but not the host in my one-liner to recreate the issue. I was able to create the new vm on the same host & datastore as the template from PowerCLI. I don't quite understand the difference between the PowerCLI version of this command and the Web client since I do not typically specify the host for new builds, but it is entirely likely that the web client picked the same host because storage DRS could see an error I did not. So now I get to go figure out where the network is not supporting jumbo frames. Thanks for the help everyone. I'm very excited to see where this goes.

All the best,

Leander

[uncLee]

Further validation:

I brought this to our senior engineers and jumbo frames was not configured on our storage network. This was an oversight during the upgrade from 5.5 to 6.0 that I had no visibility to. Thanks for all the help identifying this. Take care.

[jakerpowercli]

Yaaay! Glad you got it sorted, uncLee​!