PowerCLI sessions disconnecting during a script (Seems to only happen after vCenter 8 upgrade)

[Josh Atwell]

We have a larger environment and have a ton of audit and health-check scripts.

Ever since we upgraded our larger vCenter servers to vSphere 8, we're seeing a number of these scripts disconnect and not fully capture info from all hosts/vms.

Is there a new session timeout or max connections/sessions or throttle setting that was introduced in vSphere 8 for PowerCLI/API?

An example is a network audit script that checks for CDP info.  There's a loop that calls the sample code below, if a vCenter has > 150 hosts it seems to just disconnect the session after ~ 150 hosts.  Wrapping the two specific calls below in a try/catch w/ a reconnect works, but, eek.

foreach ($vmhost in $vmhosts){
  $networkSystem = Get-view $vmhost.ConfigManager.NetworkSystem
  foreach($pnic in $networkSystem.NetworkConfig.Pnic){
    $pnicInfo = $networkSystem.QueryNetworkHint($pnic.Device)
    <reporting stuff>
  }
}

This works:

try{$networkSystem = Get-view $vmhost.ConfigManager.NetworkSystem}
  catch{if(!$global:DefaultVIServer[0].IsConnected){connect-viserver $global:DefaultVIServer[0]}; $networkSystem = Get-view $vmhost.ConfigManager.NetworkSystem}
 
try{$pnicInfo = $networkSystem.QueryNetworkHint($pnic.Device)}
    catch{if(!$global:DefaultVIServer[0].IsConnected){connect-viserver $global:DefaultVIServer[0]};$pnicInfo = $networkSystem.QueryNetworkHint($pnic.Device)}

Seems to be easily reproduceable

We're also seeing it on scripts that collect stats during a specific time period for many VMs.

It doesn't seem time based, more load/call based.   We use some VM migrations scripts that run for hours at a time w/o ever disconnecting.

Any idea where to start digging?

No issues w/ vSphere 7 vCenters.

[LucD]

Not that I know of.
I suggest opening an SR.

------------------------------

Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

------------------------------

[amarvarudegt]

Add this advanced settings to the vCenter - config.vpxd.authorize.sessionCanOutliveToken=true

[LucD]

@amarvarudegt do you have a reference to the documentation for this setting?
I can't seem to find it in any docs.

------------------------------

Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

------------------------------

[amarvarudegt]

I was facing the same issue, found it in one of the vmware community threads prior to broadcom migration. I cannot find it again. I added these settings to both my vCenters, havent disconnected ever since.

[LucD]

Strange, that setting is not shown, nor can it be assigned a value.
Not in the Web UI nor with PowerCLI.

Must be a very undocumented setting :-)

------------------------------

Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

------------------------------

[Josh Atwell]

FYI - I set this in a couple environments as a vCenter Advanced Option (=true) and it has cleared up my issue.

config.vpxd.authorize.sessionCanOutliveToken

It definitely makes a difference, it has gotten rid of missing data on all reports for vCenters that it has been applied to. 

LucD - you were not able to apply it?  Any insight?

I still need to open a case to validate/figure out if there is any downside to this setting.

[LucD]

Good for you.

It must be an undocumented feature.
I can't even read it back with a Get-AdvancedSetting cmdlet, that just returns nothing.

------------------------------

Blog: lucd.info  Twitter: @LucD22  Co-author PowerCLI Reference

------------------------------

[Josh Atwell]

I opened a case and they referenced this KB as a known issue under investigation:

https://knowledge.broadcom.com/external/article?articleNumber=317470

It does not list this as a work-around - but lines up.  I would say there's risk of leaving a number of active sessions if leveraging the adv option.