PowerCli looses connection to vcenter

[dampfross]

I have a PowerCli script to move guests to another datastore. After moving some guests, the script will loose the connection after approx. 10 minutes to the vcenter. Surely, moving lasts just a few minutes - additionaly I have to restart the powershell Session after the break and to change the .csv file. Any ideas solving this lousy problem are welcome

Snippets:
Function Main {
   Get-Module -Name VMware* -ListAvailable | Import-Module
   Set-PowerCLIConfiguration -WebOperationTimeoutSeconds 1800 -Confirm:$false
   $vms = Import-Csv \\verein\sys\admin\PScript\VMware\VCenter01\MoveDatastore.csv -UseCulture
   Connect-VIServer vCenter01

   foreach ($VM in $VMs) {
      Move-VM -VM $VM -Datastore $($VM.NewStore) -DiskStorageFormat 'Thick'
   }
}

 

[LucD]

That is not a lot of information you provide allowing us to help you.

Since your snippet is a function, I assume you call it from somewhere?
Is that running from a PS prompt, an IDE, from within a Scheduled Task?

The $VM variable holds a row from your CSV, but you seem to use that complete row on the VM parameter of the Move-VM cmdlet.
Should that be something like $VM.Name?

Are all the started svMotions completing successfully? See Task & Event in the Web Client.

Did you check the vpxd log for additional information?

Which vSphere, PowerShell and PowerCLI versions are you using?

[dampfross]

Thank you for your reply and help. The function is a ps script, called from PowerShell 7.3.4 with the latest PowerCli 13.1 build 21605386, vSphere 8.0.1 (Standard).

For approx. ten minutes moving datastores runs correctly - mostly the move started - but powershell (or powercli) looses the connection to the vcenter server. The move will be finished correct on the vcenter server.

After loosing the connection, I have to close the Powershell session, delete the moved guests in the .csv file, start powershell again and start the script Move_datastore.ps1. For the next ten minutes the next guests will be moved, one by one.

Additionally I wrote this script by changing the datastore hardware for vsphere 6.0 - to change the disk type and display names - and it was running without any problems the whole night long. Now I use the script to change incorrect disktypes and moving some guest to a new datastore. Not really a main problem for me, but my collegues should use the script without any error.

[LucD]

Is running the svMotion in the background an option?
By just adding the RunAsync switch on the Move-VM cmdlet, the task will be started, but the script will not wait for the completion of the svMotion but immediately continue.

[dampfross]

Yes, we are using svMotion. Though the Servers will get displaynames after moving (Filenames should be short on the datastore) according to our policy, any move have to be closed correclty before the servers will be renamed.

Though I tried to run the script UpdateManger.ps1 - halted before Line 9 for 10 minutes. The script was written 2009 and runs correctly until vsphere 6.7 without problems. Though we updated directly from 6.7 to 8.0 in the last month, it was not necessary to run it. And yes, if there will be updates it needs to run for any host for approx. 20 minutes (for all servers it will be closed after 2 hours)

Attach you the script as text, powershell messages and vpxd.log for the time the script was in the line.

Maybe a bug from vsphere 8.x?

Attachment(s)

vpxd.log.txt   21 KB 1 version

UpdateManager.txt   3 KB 1 version

PowerShell Messages.txt   2 KB 1 version

[LucD]

The error in your PowerCLI session seems to say indeed that the connection to the vCenter is not there.
In the vpxd log I see some strange errors concerning ESXi node esxi07.domain.intern.
Is your vCenter perhaps running (as a VM) on that ESXi node?
Is the interrupt in the connection perhaps occurring when the VCSA VM is svMotioned?

 

[dampfross]

vCenter is running on an another ESXi host (esxi03). The interrupt in the connection occurs in any tested script about approx. 10 minutes. vCenter is not affected by svMotion.

[LucD]

What value is the WebOperationsTimeout set to?

Check with Get-PowerCLIConfiguration.

[dampfross]

For Session and User tie WebOperationstimeOut Seconds are both set to 1800.

[dampfross]

I run the scripts on another machine - Windows Server 2019 instead of 2022 - the connection was lost after 10 minutes too. It seems, the loosing connection problem is a vCenter bug (even upgrade to 8.0.1.00100 did not fixed it.) Maybe a security bug - also the browser sessionto the vCenter closes within a day - also a new behavior.

Will set the vpxd.httpsClientIdleTimeout to -1 (instead of 900). Done and tested - will have not any effects for the script - after exact 10 minutes the connection to the vCenter will be lost.

 

[LucD]

Is there a KB for this bug?

[dampfross]

Sorry, I just assume that it is a bug - or a new feature (wich means sometimes the same).

Additionally - after loosing connection to the vCenter - the PowerShell Console has to be closed before running again the scripts.

Tried to set the "config.license.client.oldServerLsNotificationsSyncSeconds" parameter from 600 to 3600 - but does not have any effect on the scripts.

[FrankWW]

Same issue here after upgrading to vCenter 8.0.1 build 21860503. I do a vmotion in the script, do maintenance on the host and then move the vms back to the host. This worked perfectly fine with V7.0, but with vCenter 8 it fails when putting back the vms. When issuing a connect-viserver <vcenter> in the same powereshell instance the script ran, I get an 'object reference not set to an instance of an object' error. I have to close the powershell window and start a new instance before I can connect to vCenter again.

Looks indeed like a bug to me in vCenter.

 

[LucD]

That same error often indicates an issue in the PowerShell/PowerCLI session.
Looks like there is some cleanup or garbage collection going wrong in PowerCLI.
I don't think this is a vSphere issue or bug

[FrankWW]

Well,  it worked fine last week when we were still at vCenter 7. The issue occurred after updating to vCenter 8. First time it connects fine and I can execute the script, but after several minutes the connection to vCenter is lost and I get the mentioned error.

[FrankWW]

Hi,

the problem seems to be with PowerCLI 13.1. After reverting back to 13.0 the issue is gone and the connection to vcenter stays alive and I don't get the object reference is not set to an instance of an object anymore when using connect-viserver. After upgrading to vCenter 8 we also updated the powercli modules from 13.0 to 13.1. Reverting back to PowerCLI 13.0 while still at vCenter 8.0 the issue doesn't occur. I don't know if using PowerCLi 13.0 with vCenter 8.0 is supported though, but my scripts work fine so far.

Frank.

[LucD]

PowerCLI 13.0 and 13.1 show the same vCenter versions in the Compatibility matrix

 

[FrankWW]

Ok, great. Then we’ll stay at 13.0 for the moment.

[FrankWW]

Hmmm, I was wrong. Even with 13.0 the same issue occurs. After a while the connection is lost and I have to close and restart the powershell session.

[LucD]

Then I suggest opening an SR

[dampfross]

Even you make a clean install with version 12.7, 14 modules seems to be replaced with ones of version 13.x.

After uninstalling all Versions above 12.7, I do a clean install
Install-Module -Name VMware.PowerCLI -RequiredVersion 12.7.0.20091289 -Scope AllUsers

Get-Module -Name VMware.* -ListAvailable | Format-Table -Property Name, Version

Name Version
---- -------
VMware.CloudServices 12.6.0.19606210
VMware.DeployAutomation 8.0.0.21610665
VMware.ImageBuilder 8.0.0.21610262
VMware.PowerCLI 12.7.0.20091289
VMware.PowerCLI.Sdk 12.6.0.19600125
VMware.PowerCLI.Sdk.Types 12.6.0.19600125
VMware.PowerCLI.VCenter 12.6.0.19600125
VMware.PowerCLI.VCenter.Types.ApplianceService 12.6.0.19600125
VMware.PowerCLI.VCenter.Types.CertificateManagement 12.6.0.19600125
VMware.Sdk.Nsx.Policy 4.1.0.21605558
VMware.Sdk.Runtime 1.0.1111.21624264
VMware.Sdk.vSphere 8.0.1111.21624264
VMware.Sdk.vSphere.Appliance 8.0.1111.21624264
VMware.Sdk.vSphere.Appliance.Access 8.0.1111.21624264
VMware.Sdk.vSphere.Appliance.Health 8.0.1111.21624264
VMware.Sdk.vSphere.Appliance.InfraProfile 8.0.1111.21624264
VMware.Sdk.vSphere.Appliance.LocalAccounts 8.0.1111.21624264
VMware.Sdk.vSphere.Appliance.Logging 8.0.1111.21624264
VMware.Sdk.vSphere.Appliance.Networking 8.0.1111.21624264
VMware.Sdk.vSphere.Appliance.Recovery 8.0.1111.21624264
VMware.Sdk.vSphere.Appliance.SupportBundle 8.0.1111.21624264
VMware.Sdk.vSphere.Appliance.System 8.0.1111.21624264
VMware.Sdk.vSphere.Appliance.Update 8.0.1111.21624264
VMware.Sdk.vSphere.Cis 8.0.1111.21624264
VMware.Sdk.vSphere.Cis.Tagging 8.0.1111.21624264
VMware.Sdk.vSphere.Content 8.0.1111.21624264
VMware.Sdk.vSphere.ContentLibrary 8.0.1111.21624264
VMware.Sdk.vSphere.Esx.Hcl 8.0.1111.21624264
VMware.Sdk.vSphere.Esx.Hosts 8.0.1111.21624264
VMware.Sdk.vSphere.Esx.Settings 8.0.1111.21624264
VMware.Sdk.vSphere.VAPI.Metadata 8.0.1111.21624264
VMware.Sdk.vSphere.vCenter 8.0.1111.21624264
VMware.Sdk.vSphere.vCenter.Authentication 8.0.1111.21624264
VMware.Sdk.vSphere.vCenter.Authorization 8.0.1111.21624264
VMware.Sdk.vSphere.vCenter.CertManagement 8.0.1111.21624264
VMware.Sdk.vSphere.vCenter.ConsumptionDomains 8.0.1111.21624264
VMware.Sdk.vSphere.vCenter.Content 8.0.1111.21624264
VMware.Sdk.vSphere.vCenter.Datastore 8.0.1111.21624264
VMware.Sdk.vSphere.vCenter.Deployment 8.0.1111.21624264
VMware.Sdk.vSphere.vCenter.Guest 8.0.1111.21624264
VMware.Sdk.vSphere.vCenter.HVC 8.0.1111.21624264
VMware.Sdk.vSphere.vCenter.Identity 8.0.1111.21624264
VMware.Sdk.vSphere.vCenter.Inventory 8.0.1111.21624264
VMware.Sdk.vSphere.vCenter.ISO 8.0.1111.21624264
VMware.Sdk.vSphere.vCenter.LCM 8.0.1111.21624264
VMware.Sdk.vSphere.vCenter.NamespaceManagement 8.0.1111.21624264
VMware.Sdk.vSphere.vCenter.Namespaces 8.0.1111.21624264
VMware.Sdk.vSphere.vCenter.OVF 8.0.1111.21624264
VMware.Sdk.vSphere.vCenter.Services 8.0.1111.21624264
VMware.Sdk.vSphere.vCenter.Storage 8.0.1111.21624264
VMware.Sdk.vSphere.vCenter.SystemConfig 8.0.1111.21624264
VMware.Sdk.vSphere.vCenter.Tagging 8.0.1111.21624264
VMware.Sdk.vSphere.vCenter.Topology 8.0.1111.21624264
VMware.Sdk.vSphere.vCenter.TrustedInfrastructure 8.0.1111.21624264
VMware.Sdk.vSphere.vCenter.VCHA 8.0.1111.21624264
VMware.Sdk.vSphere.vCenter.Vm 8.0.1111.21624264
VMware.Sdk.vSphere.vCenter.VmTemplate 8.0.1111.21624264
VMware.Sdk.vSphere.vStats 8.0.1111.21624264
VMware.Sdk.vSphereRuntime 8.0.1111.21624264
VMware.Vim 8.1.0.21605554
VMware.VimAutomation.Cis.Core 13.1.0.21605976
VMware.VimAutomation.Cloud 13.1.0.21611174
VMware.VimAutomation.Common 13.1.0.21605386
VMware.VimAutomation.Core 13.1.0.21606170
VMware.VimAutomation.Hcx 13.0.0.20803747
VMware.VimAutomation.HorizonView 13.1.0.21610272
VMware.VimAutomation.License 12.0.0.15939670
VMware.VimAutomation.Nsxt 13.1.0.21606089
VMware.VimAutomation.Sdk 13.1.0.21605170
VMware.VimAutomation.Security 13.1.0.21606510
VMware.VimAutomation.Srm 12.7.0.20091290
VMware.VimAutomation.Storage 13.1.0.21606282
VMware.VimAutomation.StorageUtility 1.6.0.0
VMware.VimAutomation.Vds 13.1.0.21610933
VMware.VimAutomation.Vmc 13.0.0.20797723
VMware.VimAutomation.vROps 13.1.0.21611158
VMware.VimAutomation.WorkloadManagement 12.4.0.18627055
VMware.VumAutomation 12.7.0.20091294

Will test my scripts within the next days.

[FrankWW]

I opened a Service Request with vmware. Let's see what they have to say.

[dampfross]

Test the scripts with PowerCli 12 - no change. Assuming the reason is vCenter 8. Upgrading directly from 6.7 to 8.0, so I have no expierience with 7.x. But all previous powercli up to 12.x run without problems on vCenter 6.7

[FrankWW]

Hi,

VMWare support just confirmed this to be a known issue:

"Object reference not set to an instance of an object" error while reconnecting Connect-VIServer PowerCLI command post ESXi host reboot (93495) (vmware.com)

[LucD]

Nice to know.
Is the 2nd workaround mentioned in that KB working?

[FrankWW]

Haven't tried it yet. Just received this information.

It would mean I have to modify my scripts by retrieving the IP of the host based on it's FQDN and use that one when connecting for the second time after the reboot.

[dampfross]

Thank you for the information.

[FrankWW]

I'm rewriting my scripts, but have an issue. When I connect using the FQDN it connects without asking for credentials (SSO), however, when I use connect-viserver <ip address> it prompts for credentials. Any idea why SSO is not working when connecting through the IP address?

[LucD]

Does the certificate contain the IP address as a SAN

[FrankWW]

No, only the FQDN.

but it doesn't matter, since all our scripts that involve a reboot of the host stopped working due to the issue and the workaround doesn't work.

According to vmware support a fix is scheduled for version in 13.2 which is due in September.

[RBrown3]

I came across this thread yesterday as I was researching this same issue.  In response to the statement from VMware Support that this will be sorted out in 13.2, I can say that I am running 13.2.1 and still have the issue - but not in every circumstance.

 

I was only seeing the issue when I would RDP into a jump server with credentials that also have vCenter permissions.  When connecting to vCenter, I would let the connect-viserver command use the session credentials so that I wouldn't need to pass in credentials.  And when I do this, 10 minutes later the connection to vCenter drops.  But when I would connect from my laptop, where the user I am logged in as doesn't have vCenter permissions, I have to pass in credentials to the connect-viserver command.  And when I do this, the disconnect never happens.  To confirm this, I RDP'd back into the jump server but this time, I passed credentials into the connect-viserver command directly (not using the session credentials) and the disconnect never happens.

 

So, while not a solution, the workaround that I found to this issue is to pass credentials into the connect-viserver command by using "$creds = Get-Credential" and the "-Credential $creds" option in the connect-viserver command.

 

I hope this helps!

-Ron

[FrankWW]

Hi,

in my situation the connection to vcenter was also disconnected after a while, but I could solve it by increasing the disconnect timeout value. For some reason it was set to 5 minutes.

get-powercliconfiguration

set-PowerCLIConfiguration -WebOperationTimeoutSeconds 21600

Franc.

 

[RBrown3]

My timeout was set to 5 minutes, as well.  But my disconnect always happened at 10 minutes, so I wasn't sure it was the setting that was causing the disconnect.  But I can try increasing the timeout as you did and see if it impacts the disconnect time.

[RBrown3]

I set my timeout higher and it still disconnected after 10 minutes.  Maybe our problems are different.

[beefy147]

same experience. increased the timeout in powercli settings but after exactly 10 minutes it bins out

[beefy147]

Thanks Ron. I agree passing the credentials rather than using pass through keeps the session alive.

If you read this how is your SSO configured in your vCenter 8.0

are you using IWA / LDAP or something else?

Thanks

[kippno]

We have exactly the same issue, starting when upgraded to VCSA 8.0.

Some observations:

• Session is dropped exactly after 10 minutes
• Only sessions using pass-through  authentication are affected
• If another session (not using pass-through) for the same user originating anywhere (same powercli host or any other host) is available, the session drop does not occur

Code to test:

 

 

Param (
  [Parameter(Mandatory=$true)]
    [string] $vcenter
)

Connect-VIServer $vcenter

$start = Get-Date
while ($global:DefaultVIServer.IsConnected) {
    $a = Get-VMHost -State Disconnected
    Start-Sleep -Seconds 1
    $end = Get-Date
    $span = New-TimeSpan -Start $start -End $End
    $span.TotalSeconds
}
$end = Get-Date
$span = New-TimeSpan -Start $start -End $end
write-host "---"
write-host "Total runtime: $span"
$end

 

 

[beefy147]

also experiencing this issue using PowerCLI 13.0 and against vCenter 8.0U2a

other than refactoring scripts to not use pass through creds (seriously!?) what can be done here. sounds like its been a bug for a long time

[dampfross]

After opening an additional case by VMware, though even with the latest PowerCli Version 13.2.1-22851661 the connection get lost after 10 minutes, I got now an answer and maybe a helpful advice for a workaround:

Under advanced system settings for vCenter, set

"config.vpxd.authorize.sessionCanOutliveToken" 

and set the value to true.

 

Just checking with an endliess loop script the connection with the vCenter and the connection is still established since 20 minutes. Will let it run over night and give you an update tomorrow.

[dampfross]

Testscript runs for 13 hours without loosing the connection. Seems it´s solved. The vCenter has to be restarted.

[beefy147]

Great news. Thanks for the effort to get this information. I will test it this week on our 8.x instances and report back

[FrankWW]

This solved it for me. No more object reference not found after a reboot of a host when executing an maintenance script.

Thanks!

[burchell99]

Also confirmed as solved. thank you

[Kirt Carson]

I was encountering the same timeout issue, but every 5 minutes.  Without making changes to the system I went with trapping errors and taking action when they occur.  Alternatively you can do a check_vCenters function call ahead of any requests/commands and avoid disconnected errors.  It's much faster than connecting if not disconnected.

Function check_vCenters {$global:DefaultVIServers | select Name, IsConnected | Where-Object IsConnected -eq $false | foreach{Connect-VIServer ($_.Name)}}

This will reconnect timed-out powercli sessions with one or more vCenters that were previously connected.

-------------------------------------------