Detect unregistered VMs across multiple vCenters

[dbutch1976]

Hello,

With lots of help (mainly from LucD) I have written the script below which will find all unregistered VMs across multiple vCenters. The script works, however, I have a very specific situation where two vCenters are connected to the same datastores. If the VM is online and running on another vCenter other than the 'primary' vCenter STAGE2 because the .vmx file cannot be downloaded. Here are two examples of VMs running on different vCenters with mismatched display names:

Name	DatastoreFullPath	FullName	Result
DC7bVM4_mistmatched.vmx	[IOMEGA] DC7bVM4_mistmatched/DC7bVM4_mistmatched.vmx	vmstores:\vcneo.lebrine.local@443\LEBRINE\IOMEGA\\DC7bVM4_mistmatched\DC7bVM4_mistmatched.vmx	ERROR downloading .vmx to local temp file.
DC7aVM4_Mismatched.vmx	[IOMEGA] DC7aVM4_Mismatched/DC7aVM4_Mismatched.vmx	vmstores:\vcneo.lebrine.local@443\LEBRINE\IOMEGA\\DC7aVM4_Mismatched\DC7aVM4_Mismatched.vmx	ERROR downloading .vmx to local temp file.

In the above example the .vmx files cannot be downloaded because the path in the .fullname column is incorrect, the actual vCenters these VMs reside on are VC7a and VC7b respectively, so in the case of DC7bVM4_mistmatched the correct .fullname would be:

vmstores:\VC7b.lebrine.local@443\LEBRINE\IOMEGA\\DC7bVM4_mistmatched\DC7bVM4_mistmatched.vmx

not

vmstores:\vcneo.lebrine.local@443\LEBRINE\IOMEGA\\DC7bVM4_mistmatched\DC7bVM4_mistmatched.vmx


I am already connected to these vCenters as seen in  $global:DefaultVIServers :
Name                           Port  User                          
----                           ----  ----                          
vcneo.lebrine.local            443   VSPHERE.LOCAL\Administrator   
vc7b.lebrine.local             443   VSPHERE.LOCAL\Administrator   
vc7a.lebrine.local             443   VSPHERE.LOCAL\Administrator  

Can I modify this catch statement so that in the event of an error downloading the .vmx file it cycles through each connected vCenter and try to download the .vmx file substituting name of each connected vcenter in the .fullname variable for all connected vCenters?

} catch {
            $error[0]
            $result = "ERROR downloading .vmx to local temp file."
            $entry.Result = $result
        }

Full script:

####STAGE1 - Persmission required - RO + DS browse###
#Attempt to find a VM matching each .vmx file on the datastore. 
#If no VM can be found using that name add it to the list of suspected unregistered VMs, but also check for the presence of a .lck.
#A .lck file means that the VM is running, and could either either have a different display name, or is running from a different vCenter than the primary.
###NOTE - For faster performance, connect to the largest vCenter last prior to running the script, that vCenter will be the 'primary' for this test.
#($cred = Get-Credential)
#connect-viserver vc7a.lebrine.local -credential $cred
#connect-viserver vc7b.lebrine.local -credential $cred
#connect-viserver vcneo.lebrine.local -credential $cred

Write-Host "$global:DefaultVIServer will be the primary vCenter for this process."
$Datastores = Get-Datastore -Server $global:DefaultVIServer.Name | Where-Object {$_.name -notmatch "local" -and $_.name -notmatch "NFS"}
$unregistered = @()
Write-Host "Starting stage 1 - Identify unregistered VMs."
ForEach ($datastore in $datastores) {
    New-PSDrive -Name TgtDS -Location $datastore -PSProvider VimDatastore -Root '\' | Out-Null
    #get a list of every .vmx file found on the datastore
    $VMXS = Get-ChildItem -Path TgtDS: -Recurse -Filter *.vmx | Where-Object {$_.name -notmatch "vCLS-*" -and $_.name -notmatch "zerto*"}
    Write-Host "Starting $datastore"
    foreach ($VMX in $VMXS) {
        try {
                Get-VM $VMX.name.replace('.vmx','') -ErrorAction:Stop | Out-Null
                } 
            catch {
                $mismatch = Get-ChildItem -Path $VMX.PSParentpath -Filter *.lck | Where-Object {$_.name -contains "$($vmx.name).lck"}
                $unregistered += [PSCustomObject] @{
                    Name = $vmx.Name
                    DatastoreFullPath = $vmx.DatastoreFullPath
                    LastWriteTime = $vmx.LastWriteTime
                    vCNameMismatch = $mismatch.Name 
                    PSParentpath = $VMX.PSParentpath
                    FullName = $VMX.FullName
                    Result = "Stage1 VM not found. The following vCenter(s) were searched: $global:DefaultVIServers - Unregistered VM suspected."
                }
            }
        }
        Write-Host "Done"
        Remove-PSDrive -Name TgtDS
}

$stage1 = $unregistered | select * | Where-Object {$_.name -ne $null -and $_.vCNameMismatch -eq $null}
$lckdetecteds = $unregistered | select * | Where-Object {$_.name -ne $null -and $_.vCNameMismatch -ne $null}
if ($stage1 -ne $null){
    write-host "The following suspected unregistered VMs were found:"
    $($stage1.name.Replace('.vmx',''))
}
if ($lckdetecteds -ne $null) {
    write-host "The following VMs have .lck files associated with them, indicating they are presently running under a different name. Stage 2 will attempt to determine the name."
    $($lckdetecteds.name.Replace('.vmx',''))
}
    elseif ($stage1 -eq $null){
    write-host "There were no unregistered VMs found."
}

$unregistered  
###STAGE2 - Permission required - low level DS operations###
#At this point the unregistered array contains a list of VMs which cannot be found on any connected vCenter. The ones without a .lck file in the same directory as the .vmx are almost certainly unregistered VMs, or they are powered-off and will be assumed to be unregistered.
#The VMs which have a .lck file in the same directory are not found but are running, they will be checked to see if they are running under a different name, or running on another vCenter.

Write-Host "Starting Stage 2 - Attempting to determine the name VMs are running under within vCenter:"
$tempFile = New-TemporaryFile
foreach ($entry in $unregistered) {
    if ($entry.vCNameMismatch -ne $null) {
        Write-Host "Checking to see if $($entry.name.Replace('.vmx','')) is running on $global:DefaultVIServer under a different name."
        try {
            Copy-DatastoreItem -Item $entry.fullname -Destination $tempFile -ErrorAction:Stop | Out-Null
            $VCVMName = Get-VM -Name (Get-Content -Path $tempFile | where { $_ -match 'displayName' }).Split('"')[1]
            $VCname = (Get-Content -Path $tempFile | where { $_ -match 'displayName' }).Split('"')[1]
            $RegisteredVCenter = $VCVMName.Uid.Substring($VCVMName.Uid.IndexOf('@') + 1).Split(":")[0]
            $DSVMName = $entry.name
            $DSpath = $entry.datastorefullpath
            $result = "$VCVMName is POWERED-ON on $RegisteredVCenter. It is named $DSVMName on disk."
            $entry.Result = $result
        } catch {
            $error[0]
            $result = "ERROR downloading .vmx to local temp file."
            $entry.Result = $result
        }
        Write-Host "$result"
    }
}
Remove-Item -Path $tempFile

$unregistered | Export-Csv -Path C:\output\orphanedfiles_20250114_10.csv -NoTypeInformation

 

[Matthew Swint]

You might also look at a script with similar purpose written last year by MITRE in response to an incident they encounted. The script checks for what they refer to as "rogue vms" that may be lurking in the dark within a vSphere environment. https://github.com/center-for-threat-informed-defense/public-resources/tree/master/nerve-incident#rogue-vm-detection-script

[dbutch1976]

Thanks but I'd still like to figure this out on my own as much as possible. Let me know if I can simplify my request:

I have connected to three different vCenters:
$global:DefaultVIServers.name
vcneo.lebrine.local
vc7b.lebrine.local
vc7a.lebrine.local

Each vCenter connects to the same datastore called IOMEGA, I'm trying to find all VMs which are running on the vCenters using a different name, for this I need to download the .vmx file and check the display name. I can do this by using the following command:

Copy-DatastoreItem -Item $entry.fullname -Destination $tempFile -ErrorAction:Stop | Out-Null

Where the the correct $entry.fullname could be any one of the three vcenters, EG vcneo.lebrine.local would be this:
$entry.fullname = vmstores:\vcneo.lebrine.local@443\LEBRINE\IOMEGA\\DC7aVM3_Unregiestered\DC7aVM3_Unregiestered.vmx

I have no way of knowing which vCenter the VM actually resides on, it could be any of them, so I would like to attempt to cycle through each vCenter until I find the correct one (the other two are going to error out).

So logically it would look something like this:

foreach ($vc in $global:DefaultVIServers){
    {
    #attempt the first vCenter
    Copy-DatastoreItem -Item ($vc)$entry.fullname -Destination $tempFile -ErrorAction:Stop | Out-Null
    #If it succeeds, log the result, if not continue trying until one succeeds, if none succeed then log it as an error.
} 

Maybe what I'm attempting to do just isn't possible this why, I don't think a try function can be used to repeatedly attempt to perform an operation using a different variable each time until success. Maybe there's another way to accomplish my end goal?



$vc1 = vcneo.lebrine.local
$vc2 = vc7b.lebrine.local
$vc3 = vc7a.lebrine.local

[LucD]

You can try all VCs one after the other.
It relies on the ErrorVariable to determine if the copy succeeded or not.

And write an error message when all attempts fail.

Something like this

$vcNames = 'vcneo','vc7b','vc7a'

foreach ($entry in $unregistered) {
  $fileVC = $entry.fullname.Split('\')[1].Split('.')[0]
  $vmxFound = $false
  foreach ($vc in $vcNames) {
    $fileName = $entry.fullname.Replace($fileVC, $vc)
    Copy-DatastoreItem -Item $fileName -Destination $tempFile -ErrorAction SilentlyContinue -ErrorVariable copyResult | Out-Null
    if ($copyResult.Count -eq 0) {
      # VMX was copied
      $vmxFound = $true
    }
  }
  if (-not $vmxFound) {
    Write-Error "Could not copy VMX file $($entry.fullname)"
  }
}

------------------------------

Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

------------------------------

[dbutch1976]

Hi LucD,

Having some problems and I'm wondering if I may have found the issue. If the first Copy-DatastoreItem succeeds then the $copyResult.count will be 0, and the loop works great, however if there are subsequent failures it appears that the $copyResult.Count does not reset back to 0, so even if the download is successful attempt it appears that if ($copyResult.Count -eq 0) does not get triggered. I tried putting $copyResult.count = 0 however it says it is a read-only variable. Is this the problem can can this counter get reset prior to each copy attempt?

Thanks for all you do LucD!

[LucD]

You could try leaving (break) the foreach loop when a copy was successful.
There is no need to continue after that.

Something like this

$vcNames = 'vcneo','vc7b','vc7a'

foreach ($entry in $unregistered) {
  $fileVC = $entry.fullname.Split('\')[1].Split('.')[0]
  $vmxFound = $false
  foreach ($vc in $vcNames) {
    $fileName = $entry.fullname.Replace($fileVC, $vc)
    Copy-DatastoreItem -Item $fileName -Destination $tempFile -ErrorAction SilentlyContinue -ErrorVariable copyResult | Out-Null
    if ($copyResult.Count -eq 0) {
      # VMX was copied
      $vmxFound = $true
      break
    }
  }
  if (-not $vmxFound) {
    Write-Error "Could not copy VMX file $($entry.fullname)"
  }
}

------------------------------

Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

------------------------------

[dbutch1976]

Hi LucD,

So I found the issue. Depending on which vCenter was considered "primary" during stage1 of the script the $unregistered.Fullname variable will be different. In Stage2 I'm using this to try different vCenters until I get the right one:

$fileName = $entry.fullname.Replace($fileVC, $vc)

For example:

vmstores:\vcneo.lebrine.local@443\LEBRINE\IOMEGA\\DC7aVM4_Mismatched\DC7aVM4_Mismatched.vmx
becomes:
vmstores:\vc7a.lebrine.local@443\LEBRINE\IOMEGA\\DC7aVM4_Mismatched\DC7aVM4_Mismatched.vmx

The reason the file is failing to download is that the actual correct path is:

vmstores:\vc7a.lebrine.local@443\DC7a\IOMEGA\\DC7aVM4_Mismatched\DC7aVM4_Mismatched.vmx

This is why the only .vmx files which are downloading successfully are from whichever vCenter was "primary" (the most recent vCenter connected to). 

After going on a deeper dive into how the vmstores path is assembled I can now see my issue:

In the examples above LEBRINE and DC7A are datacenter names. I am connected to the correct vCenter then the path is reliably correct, otherwise I don't see how I can reliably get the correct vmstores file path. I could do a .replace to change the datacenter name in the .fullname variable, but that would only work in cases where you only have a single datacenter.

I'm out of ideas on this one, any suggestions?

[LucD]

Can't you save the correct path in Stage 1 in a table, and then use the VM's name as a lookup key to retrieve the correct path?
Are the VM names at least unique?

------------------------------

Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

------------------------------