Downloading a datastore folder using Powershell

[bingscott]

I am attempting to download all of the files in a particular datastore to a local USB drive using Powershell.  Some of the files are *.vmdk files.  When an *.vmdk file is downloaded the system splits it into two files - a 1KB descriptor file and a flat vmdx file.  When the GetDatastoreItem command encounters an *.vmdk file, the copy process quits with the following error:

PS C:\> Copy-DatastoreItem -Item ds:\alb-sys-factsprod\* -Destination F:\sbing\Downloads\ESX_NFS_FACTS_SSD\alb-sys-factsprod\Backup\

Copy-DatastoreItem : 9/17/2018 11:20:38 AM VimDatastore Download of file 'https://alb-sys-vcsprd.ad

.cyfd.state.nm.us/folder%2falb-sys-factsprod%2falb-sys-factsprod-e5800be6.vswp?dcPath=ABQ&dsName=ESX

_NFS_FACTS_SSD' failed. Error message: Response status code does not indicate success: 500

(Internal Server Error).

At line:1 char:1

+ Copy-DatastoreItem -Item ds:\alb-sys-factsprod\* -Destination F:\sbin ...

+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

    + CategoryInfo          : NotSpecified: (:) [Copy-DatastoreItem], VimException

    + FullyQualifiedErrorId : Client20_StorageServiceImpl_DownloadFile_DownloadFailed,VMware.VimAut

   omation.ViCore.Cmdlets.Commands.CopyDatastoreItem

Copy-DatastoreItem : Object reference not set to an instance of an object.

At line:1 char:1

+ Copy-DatastoreItem -Item ds:\alb-sys-factsprod\* -Destination F:\sbin ...

+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

    + CategoryInfo          : NotSpecified: (:) [Copy-DatastoreItem], NullReferenceException

    + FullyQualifiedErrorId : System.NullReferenceException,VMware.VimAutomation.ViCore.Cmdlets.Com

   mands.CopyDatastoreItem

What am I missing? I used an exact copy of a Copy-DatastoreItem command used by someone else to copy a datastore folder containing *.vmdk files and everything worked perfectly for them.  Perhaps the Copy-DatastoreItem command cannot handle *.vmdk files.

here is my complete Copy-DatastoreItem command:

PS C:\> Copy-DatastoreItem -Item ds:\alb-sys-factsprod\* -Destination F:\sbing\Downloads\ESX_NFS_FACTS_SSD\alb-sys-factsprod\Backup\

[LucD]

I normally copy a datastore folder like this

$dsName = 'MyDS'

$folder = 'MyVM'

$ds = Get-Datastore -Name $dsName

New-PSDrive -Location $ds -Name DS -PSProvider VimDatastore -Root "\" | Out-Null

Get-ChildItem -Path "DS:/$folder" -Recurse | %{

   Copy-DatastoreItem -Item "DS:/$($_.DatastoreFullPath.Split(' ')[1])" -Destination C:\Temp

}

Remove-PSDrive -Name DS -Confirm:$false

[bingscott]

LucD,

Thank you... I tried your script/procedure provided above. It almost worked.  It did copy the entire folder; however, it did not complete the downloading of the vmdk files.  Typically, a download of a vmdk file spawns two files - a small 1 KB description file and a vmdk.flat file that contains a complete data file dump.   It seems like the GetDatasourceItem command cannot handle this backup file spawning. The small description file downloaded successfully, but all of my vmdk files are empty. 

It gets this error when it attempts to back up the vmdk file to the vmdk.flat file:

Copy-DatastoreItem : Object reference not set to an instance of an object.

I am performing this backup using an active online VM. Does a VM need to be powered off, before performing a backup?

scott...

[LucD]

When I run this, I also get 2 files for a VMDK, but the -flat file is copied as well.

Could it be that there has been a timeout?

You could play with the timeout value via the WebOperationTimeoutSeconds on the Set-PowerCLIConfiguration cmdlet.

[bingscott]

The script work flawlessly when the machine was powered off.  All of the files were copied successfully.  The only thing I noticed is that the reported backup file sizes are slightly off from the original file sizes as reported by VCenter.  It could be that the Copy-DataStoreItem command compresses the resulting copied file. 

[LucD]

Could be, but could also be caused by the blocksize difference on the two sides.
Is it a big difference?

[bingscott]

When I run the same script against a different datastore and folder, the file sizes are identical, but for the datastore that I am working on the sizes are quite different.  The file sizes of the copied/downloaded files are smaller.  The same thing happens when I copy the directory through VCenter.  The sizes for this datastore do not match.  There are several case reported about this in Google.

I do set the timeout value to forever in my download script.

[LucD]

Could it be that one VMDK is allocated as Thick and the other one as Thin?

[bingscott]

I checked and these machines use Thin Provisioning.  I guess that I could try restoring one of the vmdk files to a different name to see if it restores to the original size. I am not even sure that I can rename when uploading files.

[LucD]

I suspect you will have to edit the header file, the -flat should be ok.

[bingscott]

I stand corrected. The file sizes of the backup files are larger than those reported by the VMWare Datastore screen.  When I first looked at the values, I thought that the files sizes of the backups were smaller, but just the opposite is true. All of the associated descriptor files show that the vmdks are ddb.thinProvisioned = "1".

[LucD]

And does the size of the copied file correspond with the allocation (not what is actually in use)?