Maintenance mode and reboot hosts in parallel.

[jonebgood_157]

So I have been using a script for mmode and rebooting a good amount of ESXi hosts when needing to do perform that. I use an input file with ESXi hostsnames and it works fine. My question and problem is trying to figure out how to do them in parallel instead of sequentially. I want to put multiple hosts across different clusters in mmode at once and then reboot. Any ideas on how I can do this?

Here is my code on the normal, sequential one. (some code I have removed, but the main portion is there.)

# Get ESXi hostname from txt file
$ESXiServers = Get-Content $VIHosts | %{Get-VMHost $_}

## Main Part of Code

foreach ($ESXiServer in $ESXiServers) {
RebootESXiHost ($ESXiServer)
}

 
# Mmode and Reboot Host
Function RebootESXiHost ($CurrentServer) {
    # Get ESXi Server name
    $ServerName = $CurrentServer.Name
 
    # Put server in maintenance mode
    Write-Host "** Rebooting $ServerName **"
    Write-Host "Entering Maintenance Mode"
    Set-VMhost $CurrentServer -State maintenance -Evacuate | Out-Null

    # Reboot host
    Write-Host "Rebooting $ServerName"
    Restart-VMHost $CurrentServer -confirm:$false | Out-Null

    # Confirmation Server is offline
    do {
    sleep 15
    $ServerState = (get-vmhost $ServerName).ConnectionState
    }
    while ($ServerState -ne "NotResponding")
    Write-Host "$ServerName is Down"
 
    # Reboot Wait Period
    do {
    sleep 60
    $ServerState = (get-vmhost $ServerName).ConnectionState
    Write-Host "Waiting for $ServerName to Reboot ..."
    }
    while ($ServerState -ne "Maintenance")
    Write-Host "$ServerName is back up"
 
    # Exit maintenance mode
    Write-Host "Exiting Maintenance mode"
    Set-VMhost $CurrentServer -State Connected | Out-Null
    Write-Host "** Reboot Complete **"
    Write-Host ""

[new_bember]

you need to use Start-Job or -parallel option for foreach in case you on pwsh7 and sure iterate over clusters, something like:

$clusters = get-cluster
foreach($cluster in $clusters){
# Start-Job {
  $clusterHosts = get-vmhost
  $hosts2reboot = @()
  foreach($vmhost in $clusterHosts){
    if($yourESXiList.contains($vmhost.Name)){
      $hosts2reboot += $vmhost
    }
  }
  foreach($h in $hosts2reboot){
    RebootESXiHost($h)
  }

# End of start-job
}

[LostInScripting]

Your savior is the -RunAsync Parameter of Set-VMHost. The description tells us the following (looking at the developer portal):

Indicates that the command returns immediately without waiting for the task to complete. In this mode, the output of the cmdlet is a Task object. For more information about the RunAsync parameter, run "help About_RunAsync" in the VMware PowerCLI console.

In theory you would use a foreach loop for running through your hostlist and firing up the evacuation and maintenance mode setting into the background like this:

$global:myMaintTask = @()
foreach ($ESXiServer in $ESXiServers) {
	 Write-Host "Evacuating $($ESXiServer.Name) and entering Maintenance Mode..."
	 $global:myMaintTask += Set-VMhost $ESXiServer.Name -State maintenance -Evacuate -RunAsync
}

This would start the evacuation and maintenance mode setting of all your hosts in parallel. Then you would wait for all Tasks in $global:myMaintTask to finish. I prefer to use global vars here because it makes it easier to debug even if you are filling the var within a function. It will also work with a normal variable like $myMaintTask. Adding some comments this could look like:

# start evacuation and maintenance mode
$global:myMaintTask = @()
foreach ($ESXiServer in $ESXiServers) {
	 Write-Host "Evacuating $($ESXiServer.Name) and entering Maintenance Mode..."
	 $global:myMaintTask += Set-VMhost $ESXiServer.Name -State maintenance -Evacuate -RunAsync
}

# wait until all hosts are in maintenance mode
while (@(get-task -Id $($global:myMaintTask.Id | where {$_ -like "Task-task-*"}) -ErrorAction SilentlyContinue | where {$_.State -eq 'Running' -OR $_.State -eq 'Queued'}).Count -gt 0) {
	Write-Host "Waiting..."
	Start-Sleep -Seconds 10
}

Write-Host "All Tasks have finished!"

Dependent on your infrastructure this may trigger too many vMotions at the same time. Maybe you would want to add a limit for the parallelization. This then could look like this:

$parallelLimit = 3
$global:myMaintTask = @()
foreach ($ESXiServer in $ESXiServers) {
	# are currently more or equal to $parallelLimit tasks running, then wait until the count of tasks falls
	if ($($global:myMaintTask | Measure-Object).count -ge $parallelLimit) {
		# wait until the count of evacuations in parallel fall under $parallelLimit
		while (@(get-task -Id $($global:myMaintTask.Id | where {$_ -like "Task-task-*"}) -ErrorAction SilentlyContinue | where {$_.State -eq 'Running' -OR $_.State -eq 'Queued'}).Count -ge $parallelLimit) {
			# wait
			Start-Sleep -Seconds 10
		}
		Write-Host "Amount of parallel evacuations has fallen under $($parallelLimit)."
	}
	
	# after waiting start the next evacuation
	Write-Host "Evacuating $($ESXiServer.Name) and entering Maintenance Mode..."
	$global:myMaintTask += Set-VMhost $ESXiServer.Name -State maintenance -Evacuate -RunAsync
}

The variable $parallelLimit will steer how many evacuations you trigger in parallel. 

Then I would check if really all hosts on my list have entered the maintenance mode and trigger the reboot only on them.

$hostsInMaint = @()
$hostsFail = @()
foreach ($ESXiServer in $ESXiServers) {
	if ($(Get-VMHost -Name $ESXiServer.Name).ConnectionState -eq 'Maintenance') {
		$hostsInMaint += $ESXiServer.Name
	} else {
		$hostsFail += $ESXiServer.Name
	}
}

The Restart-VMhost command also has the -RunAsync parameter (see documentation). You can then build a reboot block from the above example for evacuation and entering maintenance mode. I would suggest to also include a parallelization limit to not overload your power infrastructure with 50 boots at once.

[tbrock47]

The easy answer is to just use -RunAsync on your Set-VMHost command.

$VMHosts = Get-VMHost (Get-Content $VIHosts | %{Get-VMHost $_})

$VMHosts | Set-VMHost -State Maintenance -Evacuate -RunAsync

RunAsync returns a Task object that you can check on a loop for completion and then issue a mass reboot for them all.

[jonebgood_157]

Thanks for the suggestions; looks like a couple ways I can go about it. I'll do some testing with this and let you know the results. Much appreciated