Get-Stat - Slow Performance?

[jwshields]

Hello all,

I've got a question/problem.

At my workplace, we use VSphere/Vcenter Server to manage a few thousand VMs.

I've been working on a PowerShell script to collect resource utilization of VMs using `Get-Stat`

But I'm noticing that Get-Stat is causing a painful amount of delay for the script in general.

It takes about 20 seconds to run one instance of Get-Stat, and in my script, about 60 seconds create a table with what I've scripted.

Is there any alternative command to use to retrieve stats of a VM; or any way that I could speed up the process/better format my commands?

Please see the excerpt from my PS Module and measurements using `Measure-Command` below-

Thank you in advance!

Try{
Write-Host "Retreiving $NodeName information in VSphere..."
Connect-VIServer -Server $VCenter -Credential $AmerSCreds -Force -wa 0 -ErrorAction Stop | Out-Null
$GetStatCPU = Get-Stat -Entity $NodeName -Stat CPU.Usage.Average -IntervalMins 10 -MaxSamples 8 -ErrorAction Stop 
$GetStatMem = Get-Stat -Entity $NodeName -Stat Mem.Usage.Average -IntervalMins 10 -MaxSamples 8 -ErrorAction Stop 
$GetStatNet = Get-Stat -Entity $NodeName -Stat Net.Usage.Average -IntervalMins 10 -MaxSamples 8 -ErrorAction Stop 
Get-VM $NodeName | Select-Object Name,PowerState,NumCPU,MemoryGB,Folder,VMHost -ErrorAction Stop | Format-Table
$VMInfoTable = $GetStatCPU | ForEach {
$TempCPUTable = $_
$match = $GetStatMem | Where {$_.TimeStamp -eq $TempCPUTable.TimeStamp} | Select -Unique
$match1 = $GetStatNet | Where {$_.TimeStamp -eq $TempCPUTable.TimeStamp} | Select -Unique
"" | Select @{n="TimeStamp";e={$TempCPUTable.TimeStamp}}, @{n="CPU Use %";e={$TempCPUTable.Value}}, @{n="Memory %";e={$match.Value}}, @{n="Net Use kbps";e={$match1.Value}}
}
$VMInfoTable|Format-Table|Out-String|% {Write-Host $_}
Disconnect-VIServer -Server $VCenter -Confirm:$False -ErrorAction Stop| Out-Null
}

 C:\Directory> measure-command {^I^I$GetStatCPU = Get-Stat -Entity $NodeName -Stat CPU.Usage.Average -IntervalMins 10 -MaxSamples 8 -ErrorAction Stop
>> ^I^I$GetStatMem = Get-Stat -Entity $NodeName -Stat Mem.Usage.Average -IntervalMins 10 -MaxSamples 8 -ErrorAction Stop
>> ^I^I$GetStatNet = Get-Stat -Entity $NodeName -Stat Net.Usage.Average -IntervalMins 10 -MaxSamples 8 -ErrorAction Stop
>> ^I^IGet-VM $NodeName | Select-Object Name,PowerState,NumCPU,MemoryGB,Folder,VMHost -ErrorAction Stop | Format-Table
>> ^I^I$VMInfoTable = $GetStatCPU | ForEach {
>> ^I^I^I$TempCPUTable = $_
>> ^I^I^I$match = $GetStatMem | Where {$_.TimeStamp -eq $TempCPUTable.TimeStamp} | Select -Unique
>> ^I^I^I$match1 = $GetStatNet | Where {$_.TimeStamp -eq $TempCPUTable.TimeStamp} | Select -Unique
>> ^I^I^I"" | Select @{n="TimeStamp";e={$TempCPUTable.TimeStamp}}, @{n="CPU Use %";e={$TempCPUTable.Value}}, @{n="Memory %";e={$match.Value}}, @{n="Net Use kbps";e={$match1.Value}}
>> ^I^I^I}
>> ^I^I$VMInfoTable|Format-Table|Out-String|% {Write-Host $_}}
TimeStamp           CPU Use % Memory % Net Use kbps
---------           --------- -------- ------------
2017-07-28 01:10:00     21.59    28.05          468
2017-07-28 01:05:00     12.79    20.72          207
2017-07-28 01:00:00     13.85    21.19          278
2017-07-28 00:55:00      18.1    22.05          385
2017-07-28 00:50:00     18.57    25.85          400
2017-07-28 00:45:00      14.7    38.25          233
2017-07-28 00:40:00     12.24    80.39          204
2017-07-28 00:35:00     83.74    33.05           55
Days              : 0
Hours             : 0
Minutes           : 1
Seconds           : 7
Milliseconds      : 394
Ticks             : 673940985
TotalDays         : 0.000780024288194444
TotalHours        : 0.0187205829166667
TotalMinutes      : 1.123234975
TotalSeconds      : 67.3940985
TotalMilliseconds : 67394.0985
 C:\Directory> measure-command {^I^I$GetStatCPU = Get-Stat -Entity $NodeName -Stat CPU.Usage.Average -IntervalMins 10 -MaxSamples 8 -ErrorAction Stop
>> ^I^I$GetStatMem = Get-Stat -Entity $NodeName -Stat Mem.Usage.Average -IntervalMins 10 -MaxSamples 8 -ErrorAction Stop
>> ^I^I$GetStatNet = Get-Stat -Entity $NodeName -Stat Net.Usage.Average -IntervalMins 10 -MaxSamples 8 -ErrorAction Stop
>> ^I^IGet-VM $NodeName | Select-Object Name,PowerState,NumCPU,MemoryGB,Folder,VMHost -ErrorAction Stop | Format-Table
>> ^I^I$VMInfoTable = $GetStatCPU | ForEach {
>> ^I^I^I$TempCPUTable = $_
>> ^I^I^I$match = $GetStatMem | Where {$_.TimeStamp -eq $TempCPUTable.TimeStamp} | Select -Unique
>> ^I^I^I$match1 = $GetStatNet | Where {$_.TimeStamp -eq $TempCPUTable.TimeStamp} | Select -Unique
>> ^I^I^I"" | Select @{n="TimeStamp";e={$TempCPUTable.TimeStamp}}, @{n="CPU Use %";e={$TempCPUTable.Value}}, @{n="Memory %";e={$match.Value}}, @{n="Net Use kbps";e={$match1.Value}}
>> ^I^I^I}
>> ^I^I$VMInfoTable|Format-Table|Out-String|% {Write-Host $_}}
TimeStamp           CPU Use % Memory % Net Use kbps
---------           --------- -------- ------------
2017-07-28 01:10:00     21.59    28.05          468
2017-07-28 01:05:00     12.79    20.72          207
2017-07-28 01:00:00     13.85    21.19          278
2017-07-28 00:55:00      18.1    22.05          385
2017-07-28 00:50:00     18.57    25.85          400
2017-07-28 00:45:00      14.7    38.25          233
2017-07-28 00:40:00     12.24    80.39          204
2017-07-28 00:35:00     83.74    33.05           55
Days              : 0
Hours             : 0
Minutes           : 1
Seconds           : 5
Milliseconds      : 509
Ticks             : 655098115
TotalDays         : 0.00075821541087963
TotalHours        : 0.0181971698611111
TotalMinutes      : 1.09183019166667
TotalSeconds      : 65.5098115
TotalMilliseconds : 65509.8115
 C:\Directory> measure-command {Get-Stat -Entity $NodeName -Stat CPU.Usage.Average -IntervalMins 10 -MaxSamples 8 -ErrorAction Stop}
Days              : 0
Hours             : 0
Minutes           : 0
Seconds           : 19
Milliseconds      : 514
Ticks             : 195141443
TotalDays         : 0.00022585815162037
TotalHours        : 0.00542059563888889
TotalMinutes      : 0.325235738333333
TotalSeconds      : 19.5141443
TotalMilliseconds : 19514.1443
 C:\Directory> measure-command {Get-Stat -Entity $NodeName -Stat CPU.Usage.Average -IntervalMins 10 -MaxSamples 8 -ErrorAction Stop}
Days              : 0
Hours             : 0
Minutes           : 0
Seconds           : 18
Milliseconds      : 527
Ticks             : 185270404
TotalDays         : 0.000214433337962963
TotalHours        : 0.00514640011111111
TotalMinutes      : 0.308784006666667
TotalSeconds      : 18.5270404
TotalMilliseconds : 18527.0404
 C:\Directory>

[LucD]

Try like this.

It brings the number of Get-Stat calls down to 1.

Try{ 

    Write-Host "Retreiving $NodeName information in VSphere..." 

    Connect-VIServer -Server $VCenter -Credential $AmerSCreds -Force -wa 0 -ErrorAction Stop | Out-Null 

    Get-Stat -Entity (Get-VM -Name $NodeName) -Stat CPU.Usage.Average,Mem.Usage.Average,Net.Usage.Average -IntervalMins 10 -MaxSamples 8 -ErrorAction Stop |

    Group-Object -Property TimeStamp | %{

        New-Object PSObject -Property @{

            Timestamp = $_.Name

            'CPU Use %' = $_.Group | where{$_.MetricId -eq 'CPU.Usage.Average'} | select -ExpandProperty Value

            'Memory %' = $_.Group | where{$_.MetricId -eq 'Mem.Usage.Average'} | select -ExpandProperty Value

            'Net Use kbps' = $_.Group | where{$_.MetricId -eq 'Net.Usage.Average'} | select -ExpandProperty Value

        }

   

    Disconnect-VIServer -Server $VCenter -Confirm:$False -ErrorAction Stop| Out-Null 

}

If it still needs to go faster, we would need to call the API methods directly, but that would require more code.

A question, are you actually doing these reports VM per VM.

It would be a lot faster if you could call Get-Stat for all VMs in one call, and then use the Group-Object cmdlet to produce reports per VM.

But I would need to see more iof your code for that.

PS: the Connect-VIServer for each VM will also produce a lot of overhead.
Perhaps better to do that once at the beginning of the script?

[jwshields]

Hi LucD,

Thank you for the quick reply.

I've been at work and playing around with what you suggested.

I hadn't realized you could consolidate the Get-Stat request down into one command!

That really saves me a bunch.

Thank you!

I think 20 seconds is better than 60, that fulfills my requirements. But that said, is there any documentation for the API methods that you mentioned?

The script itself is a sort of "multi-tool" and we use it across a few environments with different VCenters; and the workload that'll be done with the script itself isn't terribly voluminous.

So the overhead from doing a connect/disconnect, and doing each VM by itself is a negligible cost. This isn't for a massive report or anything of the sort, so I have no real need to get ALL VMs at once.

I took the code you gave and tweaked it a bit to make a "friendlier" table.

Thanks again!

$VMInfoRawTable = (Get-Stat -Entity $NodeName -Stat CPU.Usage.Average,Mem.Usage.Average,Net.Usage.Average -IntervalMins 10 -MaxSamples 8 -ErrorAction Stop)
$CPUUseValue = $VMInfoRawTable | Where {$_.MetricID -eq 'CPU.Usage.Average'} | Select TimeStamp,Value
$MemUseValue = $VMInfoRawTable | Where {$_.MetricID -eq 'Mem.Usage.Average'} | Select TimeStamp,Value
$NetUseValue = $VMInfoRawTable | Where {$_.MetricID -eq 'Net.Usage.Average'} | Select TimeStamp,Value
$VMInfoTable = $CPUUseValue | ForEach {
$TempCPUTable = $_
$match1 = $MemUseValue | Where {$_.TimeStamp -eq $TempCPUTable.TimeStamp}
$match2 = $NetUseValue | Where {$_.TimeStamp -eq $TempCPUTable.TimeStamp}
"" | Select @{n="TimeStamp";e={$TempCPUTable.TimeStamp}}, @{n="CPU Use %";e={$TempCPUTable.Value}}, @{n="Mem Use %";e={$match1.Value}}, @{n="Net Use kbps";e={$match2.Value}}
}
Get-VM $NodeName | Select-Object Name,PowerState,NumCPU,MemoryGB,Folder,VMHost -ErrorAction Stop | Format-Table
$VMInfoTable | Format-Table | Out-String | % {Write-Host $_}

[LucD]

The API methods that deal with Performance data are all defined on the PerformanceManager.

I did a post on a function (Get-Stat2) that used these API methods to retrieve performance data.

Have a look at Datastore usage statistics

That post is a bit dated, so the function possibly needs some reworking to handle the latest versions.