How to find Orphaned VMDK files

[Kumargtk]

How can we find out Orphaned VMDK files in all the Data-stores.

[LucD]

You could try the script from my Orphaned files and folders - Spring cleaning post.

[vAdmin]

LucD ,

Just to confirm if the script that you created will not delete anything yet unless it is used with -Delete switch? Is it possible to list the result in .CSV file ?

function Remove-OrphanedData {
<#
.SYNOPSIS   Remove orphaned folders and VMDK files
.DESCRIPTION   The function searches orphaned folders and VMDK files
   on one or more datastores and reports its findings.
   Optionally the function removes  the orphaned folders   and VMDK files
.NOTES   Author:  Luc Dekens
.PARAMETER Datastore
   One or more datastores.
   The default is to investigate all shared VMFS datastores
.PARAMETER Delete
   A switch that indicates if you want to remove the folders
   and VMDK files
.EXAMPLE
   PS> Remove-OrphanedData -Datastore ds1
.EXAMPLE
  PS> Get-Datastore ds* | Remove-OrphanedData
.EXAMPLE
  PS> Remove-OrphanedData -Datastore $ds -Delete
#>
  [CmdletBinding(SupportsShouldProcess=$true)]
  param(
  [parameter(Mandatory=$true,ValueFromPipeline=$true)]
  [PSObject[]]$Datastore,
  [switch]$Delete
  )
  begin{
    $fldList = @{}
    $hdList = @{}
    $fileMgr = Get-View FileManager
  }
  process{
    foreach($ds in $Datastore){
      if($ds.GetType().Name -eq "String"){
        $ds = Get-Datastore -Name $ds
      }
      if($ds.Type -eq "VMFS" -and $ds.ExtensionData.Summary.MultipleHostAccess){
        Get-VM -Datastore $ds | %{
          $_.Extensiondata.LayoutEx.File | where{"diskDescriptor","diskExtent" -contains $_.Type} | %{
            $fldList[$_.Name.Split('/')[0]] = $_.Name
            $hdList[$_.Name] = $_.Name
          }
        }
        Get-Template | where {$_.DatastoreIdList -contains $ds.Id} | %{
          $_.Extensiondata.LayoutEx.File | where{"diskDescriptor","diskExtent" -contains $_.Type} | %{
            $fldList[$_.Name.Split('/')[0]] = $_.Name
            $hdList[$_.Name] = $_.Name
          }
        }
        $dc = $ds.Datacenter.Extensiondata
        $flags = New-Object VMware.Vim.FileQueryFlags
        $flags.FileSize = $true
        $flags.FileType = $true
        $disk = New-Object VMware.Vim.VmDiskFileQuery
        $disk.details = New-Object VMware.Vim.VmDiskFileQueryFlags
        $disk.details.capacityKb = $true
        $disk.details.diskExtents = $true
        $disk.details.diskType = $true
        $disk.details.thin = $true
        $searchSpec = New-Object VMware.Vim.HostDatastoreBrowserSearchSpec
        $searchSpec.details = $flags
        $searchSpec.Query += $disk
        $searchSpec.sortFoldersFirst = $true
        $dsBrowser = Get-View $ds.ExtensionData.browser
        $rootPath = "[" + $ds.Name + "]"
        $searchResult = $dsBrowser.SearchDatastoreSubFolders($rootPath, $searchSpec)
        foreach($folder in $searchResult){
          if($fldList.ContainsKey($folder.FolderPath.TrimEnd('/'))){
            foreach ($file in $folder.File){
              if(!$hdList.ContainsKey($folder.FolderPath + $file.Path)){
                New-Object PSObject -Property @{
                  Folder = $folder.FolderPath
                  Name = $file.Path
                  Size = $file.FileSize
                  CapacityKB = $file.CapacityKb
                  Thin = $file.Thin
                  Extents = [string]::Join(',',($file.DiskExtents))
                }
                if($Delete){
                  If ($PSCmdlet.ShouldProcess(($folder.FolderPath + " " + $file.Path),"Remove VMDK")){
                    $dsBrowser.DeleteFile($folder.FolderPath + $file.Path)
                  }
                }
              }
            }
          }
          elseif($folder.File | where {"cos.vmdk","esxconsole.vmdk" -notcontains $_.Path}){
            $folder.File | %{
              New-Object PSObject -Property @{
                Folder = $folder.FolderPath
                Name = $_.Path
                Size = $_.FileSize
                CapacityKB = $_.CapacityKB
                Thin = $_.Thin
                Extents = [String]::Join(',',($_.DiskExtents))
              }
            }
            if($Delete){
              if($folder.FolderPath -eq $rootPath){
                $folder.File | %{
                  If ($PSCmdlet.ShouldProcess(($folder.FolderPath + " " + $_.Path),"Remove VMDK")){
                    $dsBrowser.DeleteFile($folder.FolderPath + $_.Path)
                  }
                }
              }
              else{
                If ($PSCmdlet.ShouldProcess($folder.FolderPath,"Remove Folder")){
                  $fileMgr.DeleteDatastoreFile($folder.FolderPath,$dc.MoRef)
                }
              }
            }
          }
        }
      }
    }
  }
}

Thanks in advance.

[LucD]

Confirmed, without the Delete switch nothing will be removed.

The script shows the results on the console by default.

You can just pipe the result to a CSV.

Remove-OrphanedData -Datastore DS1 | Export-Csv report.csv -NoTypeInformation -UseCulture

[vAdmin]

LucD ,

You are absolutely wonderful :smileyhappy:.

Thanks for the script.

[vAdmin]

Hi Luc,

What about executing the script to go through all of my VMFS datastore ?

I tried the below line:

Remove-OrphanedData -Datastore * | Export-Csv -Path "C:\TEMP\VMDKreport.csv" -NoTypeInformation -UseCulture

but somehow it doesn't work ?

[LucD]

Try like this

Get-Datastore | Remove-OrphanedData | Export-Csv report.csv -NoTypeInformation -UseCulture

[vAdmin]

Yes, it works as expected.

Thanks once again Luc.

[formz]

Hey LucD, I'm trying to run this script but I'm not getting any data out of it.

I copy the function that is pasted in this thread, then after I put:

connect-viserver nameofmyvcenterhere

Remove-OrphanedData -Datastore SGINSCLSV01_DATASTORE1 | Export-Csv 'E:\report.csv' -NoTypeInformation -UseCulture

It runs without error, but just brings me back to the prompt and doesn't export a CSV at all. Any insight?

Thanks

[LucD]

It could mean there are no orphaned VMDK on that datastore.

Can you do a test, create a VM and then unregister it?

[formz]

I know there are a lot. We have a program (Turbonomic) that shows orphaned VMDKs and has many in the list of that datastore, but I'm trying to get to a point where we don't have to rely on third party tools for the information.

EDIT: I should also mention that it returns to the prompt immediately - like it's not searching at all.

[LucD]

The function only works for shared VMFS datastores.

Is your datastore a shared VMFS datastore?

[formz]

Ha! That's why... They're all NFS datastores. Am I screwed at this point?

[LucD]

Try removing lines 42 and 124, in theory it should work for NFS datastores as well

[formz]

Removing 42 and 124 worked perfectly! Thank you so much!

[rstack]

Is there a way to exclude a directory? It works great for NFS with the modifications listed. But as we use a NetApp feature I would like to exclude the folder .snapshot/

[LucD]

Yes, you could add a Where-clause on line 77

[rstack]

As I am still learning here, :smileyhappy:

Would it look something like this?

foreach($folder in $searchResult | Where {$folder.FolderPath -ne ".snapshot/*"})

I am guessing not I still them LOL

[LucD]

I don't have access to my lab right now, but I would suggest a match operator.

Something like: foreach($folder in ($searchResult | Where {$folder.FolderPath -notmatch "snapshot"})

But be aware that this might cause skipped folders if you have some that have 'snapshot' in their name

[rstack]

Thanks Luc.

Tried that Still getting snapshot folders.

Tried

foreach($folder in $searchResult | Where {$folder.FolderPath -notmatch "snapshot"})

foreach($folder in $searchResult | Where {$folder.FolderPath -notlike "*snapshot*"})

Extents	: [DS] .snapshot/daily.2017-10-24_2100/<ServerName>/<ServerName>-flat.vmdk
Name	: <ServerName>.vmdk
Thin	: True

CapacityKB : 52428800

Folder	: [DS] .snapshot/daily.2017-10-24_2100/<ServerName>/
Size	: 18209038336

[LucD]

I'll have to check in my lab.
Hold on

[formz]

I tried this:

foreach($folder in ($searchResult | Where {$folder.FolderPath -notmatch "*.snapshot*"})){

As the snapshot folder is actually called .snapshot. But received a lot of errors:

At line:77 char:52

+ ... $searchResult | Where {$folder.FolderPath -notmatch "*.snapshot*"})){

+                            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

    + CategoryInfo          : OperationStopped: (:) [], ArgumentException

    + FullyQualifiedErrorId : System.ArgumentException

parsing "*.snapshot*" - Quantifier {x,y} following nothing.

At line:77 char:52

+ ... $searchResult | Where {$folder.FolderPath -notmatch "*.snapshot*"})){

[LucD]

The dot and the asterisk are special characters in RegEx expressions.

And the (not)match operator expects a valid RegEx expression.

Try with the -notlike instead of -notmatch

[rstack]

I had tried the -notlike and was receiving the same output as I posted before.

I tried

foreach($folder in $searchResult | Where {$folder.FolderPath -notlike "*snapshot*"})

I have even tried

foreach($folder in $searchResult | Where {$folder.FolderPath -notlike ".snapshot*"})

[rstack]

Oh, also tried with the () in it like

foreach($folder in ($searchResult | Where {$folder.FolderPath -notlike "*.snapshot*"}))

[formz]

Yeah this is my exact line 77

foreach($folder in ($searchResult | Where {$folder.FolderPath -notlike "*.snapshot*"})){

Still getting them listed.

[formz]

Hey LucD​ have you had a chance to check this out in your lab? Thanks!

[LucD]

My flight landed 3 hours ago, but I had a quick check in my lab.

With the following, I don't see any of the .snapshot folders popping up in the result

foreach($folder in ($searchResult | where{$_.FolderPath -notmatch "\] \.snapshot/"})){

[formz]

That did it LucD​ !!! Thank you so much for the help. Can you give a brief explanation as to why that worked and what I was doing wrong previously?

[LucD]

We're taking all the results (in $searchResult) and we 'pipe' them through a Where-clause.

An object in the pipeline is addressed by the $_ variable, not the $folder variable.

That's also why I explicitly used the parenthesis, to show what order everything is done.

[formz]

Perfect thank you!

[formz]

Hey LucD​ do you have any advice in changing the script to run against a datastore cluster?

[LucD]

This should work

Get-DatastoreCluster -Name MyDSC | Get-Datastore | Remove-OrphanedData

[formz]

That mostly worked. I say mostly because I don't think there is an error in the code but rather in my vCenter. It worked perfectly fine for one DS cluster but not for another. I got this error:

Exception calling "SearchDatastoreSubFolders" with "2" argument(s): "An error occurred while communicating with the remote host."

At C:\Users\mhenze_sa\Documents\Scripts\Delete Orphan VMDKs.ps1:76 char:9

+         $searchResult = $dsBrowser.SearchDatastoreSubFolders($rootPath, $searchSpec)

+         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

    + CategoryInfo          : NotSpecified: (:) [], MethodInvocationException

    + FullyQualifiedErrorId : VimException

[LucD]

Is perhaps one of the ESXi nodes in a cluster that is linked to the DSC powered off?

[formz]

One is in maintenance mode. I thought that would have been okay but maybe that's where the error lies.

[LucD]

So do I.
Perhaps add a Write-Host to show which ESXi node the script is connecting to?

[formz]

I will take a look at that, good idea. The script did run and output a valid list, so I'll have to do some more digging as to what is throwing the error. Thanks!

[formz]

That was it. When the host was taken out of MM there were no script errors. Strange.

I'm trying to figure out a way to delete just a specified set of VMs based on naming convention that starts with "tc-gt"

I'm thinking it'll be:

Get-DatastoreCluster -Name 'DataStoreClusterName' | Get-Datastore | Get-VM | Where-Object {$._name -like "tc-gt*"} | Remove-OrphanedData -delete

Is it valid to pass Remove-OrphanedData the names of the VMs like that?

Thanks!

[LucD]

I'm afraid not, the function requires a Datastore as argument

[formz]

Ah ok, thanks for the confirmation.

[formz]

One more (seemingly dumb) question - does the script take into account name changes of the VM itself? IE: Rename the VM and obviously it doesn't update the name on the datastore.

[LucD]

Yes, it should.

The script collects the VMDK filenames from the VM itself, so if they do have another name than 'normal', they will still return the actual name

[formz]

Coming back to this I'm getting the error again. All hosts are connected and seemingly fine.

Exception calling "SearchDatastoreSubFolders" with "2" argument(s): "An error occurred while communicating with the remote host."

At C:\Users\mhenze_sa\Documents\Scripts\Delete Orphan VMDKs.ps1:76 char:9

+         $searchResult = $dsBrowser.SearchDatastoreSubFolders($rootPat ...

+         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

    + CategoryInfo          : NotSpecified: (:) [], MethodInvocationException

    + FullyQualifiedErrorId : VimException

Would it be a Write-Host Get-VMHost within the loop that does the search but outside of the if esleif statements?

IE:

foreach($folder in ($searchResult | where{$_.FolderPath -notmatch “\] \.snapshot/“})){

        Write-host Get-VMHost

          if($fldList.ContainsKey($folder.FolderPath.TrimEnd('/'))){ 

            foreach ($file in $folder.File){ 

[LucD]

As a matter of fact you can't 'see' the ESXi node in this version of the script.

But I suspect you might be using a datastore that is not mapped to all the ESXi nodes (to which it normally should be).

Do you get that same error for all shared datastores?

[formz]

A host not being mapped was my first thought as well so I checked and they are all mapped.

The environment is organized like this:

Cluster: Dev (29 hosts)

Datastore Cluster: DevStorage

Datastores inside the cluster: Dev001, 002, 003, 004, 005, 006, 007, 008

As just a test, I ran it against 001 rather than the entire DSC. That's where I got that error. I ran it against just 002 and it worked as expected and deleted the orphaned vmdks. Both datastores are mapped to the same 29 hosts.

[LucD]

Strange, but I have seen reports of that error before, but was never able to get to the reason, or reproduce the error.

Can you run the following for 2 datastores, the one that produces the error and another one that works ok.

$dsName = '<datastore_name>'

$ds = Get-Datastore -Name $dsName

$ds | Select Name,State,DatastoreBrowserPath,FileSystemVersion,

    @{N='Host#';E={$_.ExtensionData.Host.Count}},

    @{N='Host';E={(Get-View -Id $_.ExtensionData.Host.Key -property Name).Name -join '|'}},

    @{N='Browser';E={$_.ExtensionData.Browser}}

Also, can you check the vpxd log to see if there is any further info available at the moment you get the error?

[formz]

Broken Datastore (I redacted some info)

Name                 : dev001

State                : Available

DatastoreBrowserPath : vmstores:\vcentername.local@443\DevCluster\dev001

FileSystemVersion    :

Host#                : 29

Host                 : host listing

Browser              : HostDatastoreBrowser-datastoreBrowser-datastore-358028

Working Datastore:

Name                 : dev002

State                : Available

DatastoreBrowserPath : vmstores:\vcentername.local@443\DevCluster\dev002

FileSystemVersion    :

Host#                : 29

Host                 : same host listing

Browser              : HostDatastoreBrowser-datastoreBrowser-datastore-358029

Here is the vpdx.log output. I've bolded what I think to be the problem, ascvms034 is a host. But all the other datastores that work are also mapped to that host.

2018-02-27T10:04:48.115-06:00 warning vpxd[10592] [Originator@6876 sub=InvtVmDb opID=HB-host-82797@451276-30907f95] [VpxdInvtVm::SaveGuestToDb] Skipping update of NIC settings for unknown device; key=-1, VM id=349211

2018-02-27T10:04:48.525-06:00 warning vpxd[21364] [Originator@6876 sub=VpxProfiler opID=737b587c] ClientAdapterBase::InvokeOnSoap: (ascvms034.internal, vpxapi.VpxaService.searchDatastore) [SoapRpcTime] took 1800001 ms

2018-02-27T10:04:48.525-06:00 error vpxd[21364] [Originator@6876 sub=Vmomi opID=737b587c] [VpxdClientAdapter] Got vmacore exception: Operation timed out

2018-02-27T10:04:48.531-06:00 error vpxd[21364] [Originator@6876 sub=Vmomi opID=737b587c] [VpxdClientAdapter] Backtrace:

-->

--> [backtrace begin] product: VMware VirtualCenter, version: 6.0.0, build: build-4541947, tag: vpxd

--> backtrace[00] vmacore.dll[0x001BEDFA]

--> backtrace[01] vmacore.dll[0x0005CB7F]

--> backtrace[02] vmacore.dll[0x0005DC5E]

--> backtrace[03] vmacore.dll[0x0001346B]

--> backtrace[04] vmacore.dll[0x00075966]

--> backtrace[05] vmacore.dll[0x000861B3]

--> backtrace[06] vmacore.dll[0x00089F32]

--> backtrace[07] vmomi.dll[0x0005DA9F]

--> backtrace[08] vmomi.dll[0x0005E179]

--> backtrace[09] vpxd.exe[0x0104F2BF]

--> backtrace[10] vpxd.exe[0x00425DA2]

--> backtrace[11] vpxd.exe[0x004288A7]

--> backtrace[12] vpxd.exe[0x0041B862]

--> backtrace[13] vpxd.exe[0x004279FA]

--> backtrace[14] vmomi.dll[0x000815CE]

--> backtrace[15] vpxapi-types.dll[0x0008D24B]

--> backtrace[16] vpxd.exe[0x004063B9]

--> backtrace[17] vpxd.exe[0x00405F49]

--> backtrace[18] types.dll[0x005F310B]

--> backtrace[19] vpxd.exe[0x000762AC]

--> backtrace[20] vmomi.dll[0x0004B0AC]

--> backtrace[21] vpxd.exe[0x000AF962]

--> backtrace[22] vpxd.exe[0x00086C8B]

--> backtrace[23] vpxd.exe[0x0008A784]

--> backtrace[24] vpxd.exe[0x0009AF88]

--> backtrace[25] vpxd.exe[0x000A1D23]

--> backtrace[26] vmacore.dll[0x00153D6E]

--> backtrace[27] vmacore.dll[0x001575AF]

--> backtrace[28] vmacore.dll[0x00158B01]

--> backtrace[29] vmacore.dll[0x0015A835]

--> backtrace[30] vmacore.dll[0x00065F1B]

--> backtrace[31] vmacore.dll[0x00155C90]

--> backtrace[32] vmacore.dll[0x001D2B9B]

--> backtrace[33] MSVCR90.dll[0x00002FDF]

--> backtrace[34] MSVCR90.dll[0x00003080]

--> backtrace[35] kernel32.dll[0x000159CD]

--> backtrace[36] ntdll.dll[0x0002A561]

--> [backtrace end]

-->

2018-02-27T10:04:48.535-06:00 info vpxd[21364] [Originator@6876 sub=vpxLro opID=737b587c] [VpxLRO] -- FINISH session[5275ca08-8730-a648-d890-a30f4b132ee0]52ac7eee-b72d-d3fd-d7df-63e7e6561cec

2018-02-27T10:04:48.535-06:00 info vpxd[21364] [Originator@6876 sub=Default opID=737b587c] [VpxLRO] -- ERROR session[5275ca08-8730-a648-d890-a30f4b132ee0]52ac7eee-b72d-d3fd-d7df-63e7e6561cec -- datastoreBrowser-datastore-358028 -- vim.host.DatastoreBrowser.searchSubFolders: vmodl.fault.HostCommunication:

--> Result:

--> (vmodl.fault.HostCommunication) {

-->    faultCause = (vmodl.MethodFault) null,

-->    msg = ""

--> }

--> Args:

-->

--> Arg datastorePath:

--> "[dev001]"

--> Arg searchSpec:

--> (vim.host.DatastoreBrowser.SearchSpec) {

-->    query = (vim.host.DatastoreBrowser.Query) [

-->       (vim.host.DatastoreBrowser.VmDiskQuery) {

-->          filter = (vim.host.DatastoreBrowser.VmDiskQuery.Filter) null,

-->          details = (vim.host.DatastoreBrowser.VmDiskQuery.Details) {

-->             diskType = true,

-->             capacityKb = true,

-->             hardwareVersion = false,

-->             controllerType = <unset>,

-->             diskExtents = true,

-->             thin = true

-->          }

-->       }

-->    ],

-->    details = (vim.host.DatastoreBrowser.FileInfo.Details) {

-->       fileType = true,

-->       fileSize = true,

-->       modification = false,

-->       fileOwner = false

-->    },

-->    searchCaseInsensitive = <unset>,

-->    sortFoldersFirst = true,

--> }

2018-02-27T10:04:48.648-06:00 info vpxd[225564] [Originator@6876 sub=InvtHostCnx opID=HostSync-host-725482-7766c6c] Not persisting vFlash resource runtime info. If you have hosts configured with vFlash resource, please set vpxd config option: vpxd.PersistVFlashRuntimeInfo to true.

[LucD]

Ok, seems there is a timeout on that first datastore, but I can see no obvious reason why that would happen on that specific datastore.

Does this datastore show high latency on the Performance tab?

Or are there that many folders & files on that datastore?

What I forgot to ask, is this FC, NFS or iSCSI storage?

Also strange the FileSystemVersion on either of the datastores. Could that be due to the PowerCLI version?

I'm using 6.5.4.

[formz]

I just updated my PowerCLI. I was on 5 something. When I re-ran the previous commands for the filesystemversion I got 3.0.

They usage of the datastores, how busy they are, but I'm going to try and run it at an off time and see what happens.

I'll let you know how it goes.

[formz]

Seems that no matter when I run the script it fails on the same couple data stores repeatedly. I'm at a loss at this point.

[LucD]

Since the script is using public API you could consider opening a SR.

But I'm not sure you can open a SR without some kind of Developer Support (or whatever they call it).

You could run the VOMA health check on the datastore, and see if anything is reported.

See KB2036767

[formz]

These are all NFS datastores so VOMA isn't an option. :smileysad:

[LucD]

I see.
Are there any dropped packets on that link to the NFS export?

Is that a multi-NIC connection?

[formz]

Amazing coincidence, I'm about to modify the script for the EXACT same reason.

[LucD]

You can test with this

Get-Datastore -Name SGINSCLSV01_DATASTORE1 |

select Name,Type,@{N='Shared';E={$_.ExtensionData.Summary.MultipleHostAccess}}

[Kumargtk]

Hi Umesh,

This will exclude the Snapshots file right. i want to check for more than 50 servers. If u have any script... please help.

[John Belanger]

Take a look at this post:

Virtually Jason: Deleting Orphaned (AKA Zombie) VMDK Files

I have found it works well for me.

[LucD]

I would be interested to know what the following produces?

That would allow me to further analyse what you suspect might be a flaw in the script.

$vmName = 'VSX'

$vm = Get-VM -Name $vmName

$vm.ExtensionData.LayoutEx.File |

Select Name

[omar13489]

The Output for these lines that give you the whole information about the File Object for VM  :

PowerCLI C:\> $vmName = 'VSX'

PowerCLI C:\> $vm = Get-VM -Name $vmName

PowerCLI C:\> $vm.ExtensionData.LayoutEx.File |

>> Select Name

>>

Name

----

[Local_DataStore_51.19] VSX/VSX.vmx

[Local_DataStore_51.19] VSX/VSX.vmxf

[Local_DataStore_51.19] VSX/VSX.nvram

[Local_DataStore_51.19] VSX/VSX.vmsd

[Local_DataStore_51.19] VSX/VSX.vmdk

[Local_DataStore_51.19] VSX/VSX-flat.vmdk

[Local_DataStore_51.19] VSX/VSX-000001.vmdk

[Local_DataStore_51.19] VSX/VSX-000001-delta.vmdk

[Local_DataStore_51.19] VSX/VSX-000002.vmdk

[Local_DataStore_51.19] VSX/VSX-000002-delta.vmdk

[Local_DataStore_51.19] VSX/VSX-Snapshot1.vmsn

[Local_DataStore_51.19] VSX/VSX-Snapshot2.vmsn

[Local_DataStore_51.19] VSX/vmware-2.log

[Local_DataStore_51.19] VSX/vmware-1.log

[Local_DataStore_51.19] VSX/vmware.log

This means that the File Related to VM, how the script considered it as Orphaned ?

Also the machine is not listed when I check the orphaned machine by :

Get-View -ViewType VirtualMachine -Filter @{'RunTime.ConnectionState'='disconnected|inaccessible|invalid|orphaned'}

[LucD]

Excuse me, but the file that was mentioned by the function that you gave as an example, is in a different folder (root) and has a different name.

[Local_DataStore_51.19] VSXVSX.vmdk

[LucD]

As a matter of fact, you seem to have a bunch of crap files in the root of that datastore.

[Local_DataStore_51.19] VSXVSX.nvram

[Local_DataStore_51.19] VSXVSX.vmdk

[Local_DataStore_51.19] VSXVSX-Snapshot1.vmsn

[Local_DataStore_51.19] VSXVSX-000001.vmdk

[Local_DataStore_51.19] VSXVSX-000002.vmdk

[Local_DataStore_51.19] VSXVSX-Snapshot2.vmsn

[Local_DataStore_51.19] VSXVSX.vmx

Unless you knowingly tried to create a VM in the root folder of the datastore, these all look like orphaned files.

Did you at some point forget to put the slash between the VSX VSX?

[omar13489]

how it is located on different folder ?

The File located under this directory :

/vmfs/volumes/56ee729e-9af0d55f-3ffa-1402ec42a460/VSX/

-rw-------    1 root     root         491 Jan 10 09:31 VSX.vmdk

and it has the same name

but ls -lh show the above line!

and the slash was not shown with the the output i attached, the output came from the script

maybe you thought it is located inside root directory because I used "ls -lh" on the ESX ?

without ls -lh , only ls :

ls /vmfs/volumes/56ee729e-9af0d55f-3ffa-1402ec42a460/VSX/ | egrep VSX.vmdk

VSX.vmdk

[LucD]

Open the File browser in the Web client on that datastore.

[omar13489]

Here is the Datastore Browser from vSphere Client

[LucD]

But what is in the root folder?

[LucD]

Are there besides folders any files in the root?

In any case, can you run the following, and attach the report.txt file?

There must be something special with that VM, I can't reproduce what you are seeing.

And which vSphere and PowerCLI version are you using?

$vmName = 'VSX'

$vm = Get-VM -Name $vmName

$vm.ExtensionData.Layout | Out-File C:\Temp\report.txt

$vm.ExtensionData.Layout.Disk | Out-File C:\Temp\report.txt -Append

$vm.ExtensionData.Layout.Snapshot | Out-File C:\Temp\report.txt -Append

$vm.ExtensionData.LayoutEx.File | Out-File C:\Temp\report.txt -Append

$vm.ExtensionData.LayoutEx.Disk | Out-File C:\Temp\report.txt -Append

[omar13489]

No Files inside the datatstore root directory . all of them folders

after running the code segment, attached the report.txt output

[omar13489]

btw , this issue related to All VMs and not this specific VM

[LucD]

I understand, but let's concentrate on one VM to determine the cause.

The problem is in the missing slash in the filepath

Name         : VSX.vmdk

Folder       : [Local_DataStore_51.19] VSXVSX.vmdk

Size         : 0

CapacityKB   : 31457280

Modification : 1/10/2018 9:31:35 AM

Owner        : root

Thin         : True

Extents      : [Local_DataStore_51.19] VSX/VSX-flat.vmdk

DiskType     : VirtualDiskFlatVer2BackingInfo

HWVersion    : 11

At the moment I have no idea why this would happen.

Which vSphere version and PowerCLI version are you using?

And in which regional settings are you running? What does Get-Culture say?

To eliminate any copy/paste issues, can you check if the attached copy of the functions comes back with the same results?

[omar13489]

Hi LucD,

VSphere Client Version : 6.0.0, Build : 5112508

PowerCLI Version :

PowerCLI C:\> Get-PowerCLIVersion

PowerCLI Version

----------------

VMware vSphere PowerCLI 6.3 Release 1 build 3737840

---------------

PowerCLI C:\> Get-Culture

LCID             Name             DisplayName

----             ----             -----------

1033             en-US            English (United States)

and execute the attached function with the same results

[LucD]

Have you deleted some of your previous replies?
That makes it harder to try and find a solution.

Can you run the attached script, and send me the result (PM will do if there is confidential info in there).

[omar13489]

Hi LucD,

Yes i deleted some replies because it contains sensitive data

Can you check the Output after the test.ps1 script execution

[LucD]

The problem you are seeing definitely comes from the FolderPath.

To check further, can you run the attached script?

[omar13489]

attached the output

can we modify this line :

$key = "$($folder.FolderPath)$(if($folder.FolderPath[-1] -eq ']'){' '})$($file.Path)"

To (Adding the '/' ):

$key = "$($folder.FolderPath)/$(if($folder.FolderPath[-1] -eq ']'){' '})$($file.Path)"

[LucD]

Ok, the issue seems to come from the fact that the FolderPath is not having a / at the end like I see in other environments.

I have no clue what could be causing this.

In the attached version I have adapted the code to intercept that.

Can you give this version a try?

[omar13489]

Hi Lucd,

I think the issue had been solved after manipulating '/' inside FolderPath

The script results shows it find only 40 Orphaned Files

-Message

03/01/2018 13:58:54 Get-VmwOrphan 40 Found orphaned files on Local_DataStore_51.19!

I checked all the founded files and i find that the results are wonderful except that there is a folder called /scratch/log

here is an example :

Name         : sdrsinjector.log

Folder       : [Local_DataStore_51.19] scratch/log/sdrsinjector.log

Size         : 144942

CapacityKB   :

Modification : 2/28/2018 10:24:37 PM

Owner        : root

Thin         :

Extents      :

DiskType     :

HWVersion    :

Is it a System Log File ?

My Second Question :

Can you modify the script to include also the File Deletion Option ?

[LucD]

Looks like someone configured the scratchspace of an ESXi node on that datastore.

The log can be useful, so I wouldn't delete it.

See also KB1033696

On the 2nd question I would suggest you open a new thread for that.

Point to the script in this thread and ask for the delete option.

This thread is becoming way too long :smileylaugh:

[socbizkaia]

Hello LucD,

We are running your script to find orphaned files in our LUNs and it works great! We were able to discover a lot of vmdk files and we verified that all of them were orphaned before deleting them, so there wasn't any "fake" orphaned vmdk. However, I have some questions...

- Is there any special reason why so many orphaned vmdks are generated (problems with vMotion, snaphosts...) ?

- Do you use any Powershell command to pipe the output of your script to delete the files from Powershell? We performed manually "vmkfstools -U file.vmdk" to remove orphaned vmdk files. But we need to do it manually, SSH to the ESXi host and execute the command. How would you do it in Powershell? Any command better than remove-item?

- Your script doesn't provide output for NFS Datastores... Is there any alternative in order to analyze NFS datastores for orphaned files?

Thank you in advance and thank you for providing this useful script!

Christian

[LucD]

Hi Christian,

On your questions:

• There are many possible causes for orphaned vmdk files: vMotion gone wrong, unregistered VMs, removed harddisk from a VM with the delete option,.... There is definitely no single reason for their existance. Each case would have to be investigated individually. But if you find the same cause, you might want to review some scripts and/or procedures
• if you are using the script from Orphaned Files Revisited, you can include NFS datastore by replacing line 67 with

if((($ds.Type -eq “VMFS”) -or ($ds.Type -eq “NFS”)) -and $ds.ExtensionData.Summary.MultipleHostAccess -and $ds.State -eq “Available”) {

• One way to automate the removal of orphaned vmdk files is to use my original post Orphaned Files And Folders – Spring Cleaning. There the function Remove-OrphanedData has a Delete switch. Another option is to use the VimDatastore PSProvider, as I have shown in Re: PowerCLI script for delete vmware-*.logs file​

I hope that helps.

[socbizkaia]

Hello LucD,

thank for your indications!

• We are using the last version of your script published in this forum this year. I am going to modify the 67th line as you suggest to search for orphaned vmdk files in NFS datastores. However, we use the same NFS datastores in two independent vcenters, so my approach would be: if the same file is reported only once, it is not an orphaned file; if the same file is reported twice, it is an orphaned file.
• To remove the files I personally prefer to get a initial list (with this script) and then provide it to a second script (powershell pipe) to delete files. So firtsly we can verify the list. It could be done with PSProvider, however i think that perhaps vmkfstool is a more intelligent approach because not only the specified ".vmdk" file is deleted, but also the related "-delta.vmdk", "-ctk.vmdk"... files. In the tests we performed I never get an "-delta.vmdk", "-ctk.vmdk"... file reported by your script.

Thanks,

Christian

[LucD]

I agree, the vmkfstool command might be a better choice to make sure everything is removed.

I might create a new version of the function to use that command and the Posh-SSH module.

The drawback is that it would need credentials to connect to the ESXi nodes.

The reason you don't see those delta, ctk... files is because they are not reported by the SearchDatastoreSubFolders method.

The VmDiskFileQuery​ filter I use in the script, only returns the "... the virtual disk primary file"

[Sharantyr3]

Hello !

LucD​ please I summon you ! :smileyhappy:

Any way to get this script working on a vsandatastore ?

While checking for storage policy out of date objects, I found out I have orphan files on my vsan.

Running the script returns empty result on vsan, no error.

$outOfDateObjects = Get-SpbmEntityConfiguration | Where-Object {$_.ComplianceStatus -eq 'outOfDate'}
$vmID = $outOfDateObjects[0].Id.Split('/')[0]
Get-VM -id $vmID
$vmID = $outOfDateObjects[1].Id.Split('/')[0]
Get-VM -id $vmID
...

This works for most results, but some

get-vm : 16/10/2020 11:05:19    Get-VM          VM with id 'VirtualMachine-vm-17879' was not found using the specified filter(s).

This lead me to check the TimeOfCheck value returned by spbm and all the objects that are linked to non existant vm ids are pretty old (>1 month last check date)

So I think I have orphan objects on my vsan, but not "orphan" in vsan terms, but orphan like "not linked to any VM".

Can you help me there ? Each $outOfDateObjects does not contain any filename, only non existant vm ID.

So I thought your script could retrieve this for me...

Thank you

[LucD]

Did you check that you are not encountering the issue mentioned in Solved: vSAN statDB object Out of Date compliance |VMware Communities​

[Sharantyr3]

My statsDB object is fine

I found out an interesting fact, most of the out of date hard disks that are linked to non existant VM ID are in fact hard disks of TEMPLATES !

Converting template back to VM make the VM ID valid again !

But it seems over complicated to convert every template, check storage policy, convert back to template.

Anyway, do you think it would be possible to adapt the script to be vsan compatible ?

I think many of my vcenter users may sometimes do "Remove from inventory" instead of "delete from disk" and I will end up with orphaned vmdks.

[FSchleyhahn]

I am trying to use the script to delete the orphaned VMDK files.  I have had luck producing the report that lists all of the orphaned files, but when I run the script with the switch to delete them, nothing happens.  If I attempt to manually delete the orphaned files that appear in the report, it doesn't allow me to.  Is that normal?  Any assistance you can provide will be appreciated as the output shows thousands of orphaned files.  Thank you.

[LucD]

That probably means you don't have the required permissions.
What error message is shown when you try to delete a file?

[FSchleyhahn]

Thank you SO MUCH for your fast reply!  I have been working on this for the past 24 hours and haven't had any luck.  So, here's an excerpt of the Output that I receive:

VHCCFCTXIT_VPDS_1_vol Orphaned VMDK Found: IL084TCTXSF2_1-ctk.vmdk
VHCCFCTXIT_VPDS_1_vol Orphaned VMDK Found: IL084TCTXSF2-ctk.vmdk
VHCCFCTXIT_VPDS_1_vol Orphaned VMDK Found: IL084TCTXDC1-ctk.vmdk
VHCCFCTXIT_VPDS_1_vol Orphaned VMDK Found: IL084TCTXDC1_1-ctk.vmdk
VHCCFCTXIT_VPDS_1_vol Orphaned VMDK Found: IL084TCTXPVS1-ctk.vmdk
VHCCFCTXIT_VPDS_1_vol Orphaned VMDK Found: IL084TCTXPVS1_1-ctk.vmdk
VHCCFCTXIT_VPDS_1_vol Orphaned VMDK Found: IL084TCTXSF1-ctk.vmdk
VHCCFCTXIT_VPDS_1_vol Orphaned VMDK Found: IL084TCTXSF1_1-ctk.vmdk

 

I connect to my vCenter using my account, which is a VMAdmin.

I run the script for ONLY the datastore referenced.

PowerCLI C:\> Connect-VIServer

cmdlet Connect-VIServer at command pipeline position 1
Supply values for the following parameters:
Server[0]: soivct
Server[1]:

Name Port User
---- ---- ----
soivct 443 ILTEST\t.floyd.schleyhahn

PowerCLI C:\> .\Untitled1.ps1

cmdlet Untitled1.ps1 at command pipeline position 1
Supply values for the following parameters:
Datastore[0]: VHCCFCTXIT_VPDS_1_vol
Datastore[1]:
PowerCLI C:\>

 

I'm assuming it runs because nothing happens.  However, when I browse the datastore in my web client, I am still seeing those files.  And if I try to delete one of them (as shown in the attached image), it just says "cannot delete". 

 

 

[FSchleyhahn]

And this is an excerpt of YOUR script.  The ONLY thing I did to it was add the line at 214 (last line).  Is that correct?

if($Delete){

if($folder.FolderPath -eq $rootPath){

$folder.File | %{

If ($PSCmdlet.ShouldProcess(($folder.FolderPath + " " + $_.Path),"Remove VMDK")){

$dsBrowser.DeleteFile($folder.FolderPath + $_.Path)

}

}

}

else{

If ($PSCmdlet.ShouldProcess($folder.FolderPath,"Remove Folder")){

$fileMgr.DeleteDatastoreFile($folder.FolderPath,$dc.MoRef)
Remove-OrphanedData -Datastore $ds -Delete

[LucD]

I'm not sure which version of the script you are using, in fact, I don't have line 214 in my version.

 

In any case, if your are using the script from Orphaned Files And Folders – Spring Cleaning then you should place that code in a .ps1 file.

When you want to run the function, you can add a line at the of that .ps1 file that says

 

 

Remove-OrphanedData -Datastore 'VHCCFCTXIT_VPDS_1_vol'

 

 

Once you are sure that reported VMDK are indeed orphaned, you can add the Delete switch to the call.

 

Remove-OrphanedData -Datastore 'VHCCFCTXIT_VPDS_1_vol' -Delete

 

Perhaps you could attach the .ps1 file to your reply, so I can have a look at the code.

 

[a_p_]

I think that the issue in this case is that CBT file ("...-ctk.vmdk") are not returned by "$_.ExtensionData.LayoutEx.File".

André

[FSchleyhahn]

I preface with saying I am new to PowerShells and scripts, so I apologize in advance if what I am doing or not doing is blatantly obvious.  First, I am running the attached List_Orphan_SOIVCT.ps1 script to generate the Output.txt file so that I can see what datastores have orphaned files.  Next, I run the Delete_Orphan_SOIVCT.ps1 with the expectation that it will "delete" all of the files listed within the Output.txt file.  I am experimenting with only 1 datastore in our test environment before running these in Production.  Attached are the 2 scripts and the Output.txt file.  Perhaps not all of the files listed within the Output.txt file can or should be deleted??

[LucD]

No problem.

The Remove-OrphanedData function does not read a TXT file to delete orphaned data.
First you call the function without the Delete switch (example in attached file).
This will produce a list of VMDK files that the function assumes are orphaned.

Once you are absolutely sure that the files that are reported are indeed orphaned, you call the function again, but now with the Delete switch

Remove-OrphanedData -Datastore 'VHCCFCTXT_VPDS_1_vol' -Delete

The file _ctk files (as A.P. correctly remarked earlier) are generated/used by Changed Block Tracking (CBT) and are definitely not orphaned  VMDK when the VM to which they belong is still registered and not orphaned in the vCenter.

[FSchleyhahn]

Thank you so very much!  I ran your script just as it is and it did give me output to the screen.  So--my next question is, how can I produce a file with this out put and how can I run it for ALL of my datastores within this particular vCenter?  And lastly, after it produces the report, how can I add the -Delete switch to run it on ALL of my datastores.  This is awesome!  Thank you again.

[FSchleyhahn]

I was able to pipe the output of the one datastore to a csv file.  However, I'm still not for sure how to run this for ALL of the datastores.  I tried using an * after  -Datastore, but it's not working.  This is the only line I modified:

Remove-OrphanedData -Datastore * | Export-Csv report.csv -NoTypeInformation -UseCulture

 

So, my last question here would be how to run this report for all of the datastores within the vcenter that I am connected to.  Thanks again for your help.

[LucD]

One way of doing that could be like this

Get-Datastore -PipelineVariable ds |
ForEach-Object -Process {
    Remove-OrphanedData -Datastore $ds
} | Export-Csv report.csv -NoTypeInformation -UseCulture

 

But before you run with the Delete switch, make absolutely sure that the reported VMDK are effectively orphaned.
The VMDK with the -ctk suffix are not all orphaned! 

[FSchleyhahn]

Would this be the correct syntax if I want to include the Delete Switch:

Get-Datastore -PipelineVariable ds | ForEach-Object -Process { Remove-OrphanedData -Datastore $ds -Delete } | Export-Csv report.csv -NoTypeInformation -UseCulture

[LucD]

That is correct

[FSchleyhahn]

Thank you again for all of your help with this.

[FSchleyhahn]

I have run the script to produce the report.csv for ALL of the datastores within my virtual center, but unfortunately, because of the massive volume of datastores, the script keeps timing out.  It produces about 500 lines within the report and that's it.  Is it possible to run this script per cluster and if so, what would the syntax be?  For example, one of the clusters is named "ProdA".  How would I run this script for ONLY the datastores presented to the ProdA Cluster?

 

Get-Datastore -PipelineVariable ds | ForEach-Object -Process { Remove-OrphanedData -Datastore $ds } | Export-Csv report.csv -NoTypeInformation -UseCulture

[LucD]

You could do something like this

Get-Cluster -Name ProdA | Get-VMHost |
Get-Datastore -PipelineVariable ds | ForEach-Object -Process { 
    Remove-OrphanedData -Datastore $ds 
} | Export-Csv report.csv -NoTypeInformation -UseCulture

[FSchleyhahn]

Thank you.  That seems to be working.  What I think is happening is we have 2 datastores that are presented to every host in every cluster in our environment as they contain ISOs and Templates.  The names of these 2 datastores are as follows:

VM_MANAGEMENT_DATASTORE and vm_datastore_mgmt

Using the below statement, is there a way to inject code to EXCLUDE those 2 datastores?

Get-Cluster -Name ProdA | Get-VMHost | Get-Datastore -PipelineVariable ds | ForEach-Object -Process { Remove-OrphanedData -Datastore $ds } | Export-Csv report.csv -NoTypeInformation -UseCulture

[LucD]

You could use a Where-clause.

Get-Cluster -Name ProdA | Get-VMHost |
Get-Datastore -PipelineVariable ds | 
where{'VM_MANAGEMENT_DATASTORE','vm_datastore_mgmt' -notcontains $_.Name} |
ForEach-Object -Process { 
    Remove-OrphanedData -Datastore $ds 
} | Export-Csv report.csv -NoTypeInformation -UseCulture

[harezzebra1]

Hi LucD,

 

I tried below script to find Orphaned VMDK in vSAN using https://www.lucd.info/2016/09/13/orphaned-files-revisited/

and changed line 67 to use type vSAN

if((($ds.Type -eq “VMFS”) -or ($ds.Type -eq “VSAN”)) -and $ds.ExtensionData.Summary.MultipleHostAccess -and $ds.State -eq “Available”) {

It's not giving any output, but I use RVtools I can see lots of Orphaned VMDK files.

Please help.

[LucD]

Did you give the script I attached in Solved: Re: get orphaned vmdk information from vSAN - VMware Technology Network VMTN a try?

[harezzebra1]

Yes and tried running the verbose mode also-

Still getting error

Remove-OrphanedData -Datastore DEV_VSAN -Verbose
VERBOSE: 1/28/2021 2:50:40 PM Get-View Finished execution
VERBOSE: 1/28/2021 2:50:40 PM Get-Datastore Finished execution
VERBOSE: 1/28/2021 2:50:53 PM Get-VM Finished execution
VERBOSE: 1/28/2021 2:50:54 PM Get-Template Finished execution
VERBOSE: 1/28/2021 2:50:54 PM Get-View Finished execution
Exception calling "SearchDatastoreSubFolders" with "2" argument(s): "An error occurred while communicating with the remote host."
At line:72 char:17
+ ... $searchResult = $dsBrowser.SearchDatastoreSubFolders($roo ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [], MethodInvocationException
+ FullyQualifiedErrorId : VimException

[LucD]

Is that with the modified script I linked to in my previous reply?

[harezzebra1]

yes, I used the modified script for verbose output.

[LucD]

The error states "An error occurred while communicating with the remote host.", which has nothing to do with the script.
That seems to be an issue in your environment/setup.

[xsgao]

DANDEROUS!!! I followed this find command and deleted my vmdk files completedly. I wish I can thumb down this post.

[xsgao]

"You can easily find these orphaned vmdk’s via the Service Console / SSH from Putty:

find -iname “*-flat.vmdk” -mtime +7"

DANDEROUS!!! I followed this find command and deleted my vmdk files completely. I wish I can thumb down this post.

 

 

 

[LucD]

The find command by itself will not delete any files, it will only list them.

But you are right, that just lists all VMDK, not only the orphaned ones.
That's where the script I posted comes in, it will check for each VMDK if it is connected to a VM or not.
And my script clearly states that you have to check the list of VMDK it produces, before actually deleting the files.

[Avon]

I  noted that this thread has been quiet for a while. Does the script work with a v7U3 setup?

I have 2 hosts (used for DR) that have their own storage (not vSAN)

I have the script function loaded - login to vCenter OK can see the datastores OK

Running the function gives no output - not even the "not found" message just returns to the PS prompt.

[dauphin77]

 Is there by chance an updated version of this script compatible with vSphere 7.0.3 or maybe some other tool (free or paid) that can produce similar results? In the same boat as Avon where the script won't produce and results.

[LucD]

I'll do some more test.

[LucD]

Did you already try changing line 42 to

        if ($ds.Type -eq "VMFS") {

[dauphin77]

 I changed the following in line 42:

if($ds.Type -eq "VMFS" -and $ds.ExtensionData.Summary.MultipleHostAccess -and $ds.Accessible){

to

if ($ds.Type -eq "VMFS") {

Script does now run but only provides a few lines for information pertaining to 3 of the 4 ESXi hosts in our cluster. (Spreadsheet attached).

I think I read previously that this may not work with vSAN, is that still the case?

 

[LucD]

Why did you change it to

if($ds.GetType().Name -eq "String"){

A Datastore object will never equal a String.
I advised to just leave out the MultiHostAccess test.
Are your datastores not VMFS datastores?

[dauphin77]

 My apologies, I copied the line info for Line 37, not 42. Line 42 in my script is set to:

if ($ds.Type -eq "VMFS") {

I modified my previous post to reflect the correct information in line 42.

[LucD]

I posted a VSAN compatible version in thisanother thread, see Solved: Re: get orphaned vmdk information from vSAN - VMware Technology Network VMTN