PowerCLI

 View Only
Expand all | Collapse all

total consumption on cluster per vm

  • 1.  total consumption on cluster per vm

    Posted Mar 05, 2009 04:42 PM

    My environment consists of a virtual center with one cluster of 3 esx boxes. We also use VMotion to move around the 3 servers. Is it possible to get using powershell how much a single vm consumes of total resources on the vmware cluster and shoot that out into a csv? I ask for a single VM but really, I am looking in the csv to list all VM's and how much total resources (like mem.consumed.average

    mem.sysUsage.average , and cpu.usage.average) they consume on the cluster.

    Any help would be greatly appreciated!



  • 2.  RE: total consumption on cluster per vm
    Best Answer

    Posted Mar 05, 2009 09:09 PM

    Let's tackle this step by step.

    1) Statistical data for VMs that can have been vmotioned

    There is a problem when you ask for -Realtime statistics.

    The reason being that these data are kept on the ESX server, not on the VC.

    The Get-Stat cmdlet only returns data for the ESX host on which the VM is currently running.

    So if we want to see statistical data for VMs that have been vmotioned it is advised to go for one of the historical intervals !

    Note that this is not a VITK problem, the QueryPerf method in the SDK displays the same behavior.

    2) How to calculate the usage % against the cluster ?

    I don't think the script should ask for percentages for the statistics (ex. cpu.usage.average).

    In this case it's better to go for statistics that are expressed in a value for which we also have a total value on the cluster level.

    What this means is the script should measure for example CPU usage in MHz (cpu.usagemhz.average).

    3) What do we take as the available resources on the cluster ?

    On a cluster we can do a straightforward summation of the available resources on each of the clusternodes.

    For example, a cluster with 3 nodes where each node has a quadcore of 2.669 GHz would have 3 x 4 x 2.669 GHz of CPU resources available.

    But is that what we want to measure the VM consumption against ?

    On an ESX server (a clusternode) only part of the total available resources on the server is allocated to the guests.

    Part of the available resources goes to the "system".

    Taking the previous points into account the following script, in my opinion, expresses the CPU and memory usage of each VM as a percentage of the available cluster resources.

    $report = @()
    
    $cluster = Get-Cluster <clsutre-name> | Get-View
    $clusterCPU = $cluster.Summary.EffectiveCpu
    $clusterMem = $cluster.Summary.EffectiveMemory * 1Mb
    
    $from = [Datetime]"03/05/2009 00:00"
    $to = [Datetime]"03/05/2009 23:59"
    
    Get-VM | % {
    	$cpu = $_ | Get-Stat -Stat cpu.usagemhz.average -IntervalMins 5 -Start $from -Finish $to
    	$mem = $_ | Get-Stat -Stat mem.consumed.average -IntervalMins 5 -Start $from -Finish $to
    	for($i=0; $i -lt $cpu.Count; $i++){
    		$row = "" | select VM, Timestamp, CPUperc, Memperc
    		$row.VM = $cpu[$i].Entity.Name
    		$row.Timestamp = $cpu[$i].Timestamp
    		$row.CPUperc = "{0:N2}" -f ($cpu[$i].Value / $clusterCPU * 100)
    		$row.Memperc = "{0:N4}" -f ($mem[$i].Value / $clusterMem * 100)
    		$report += $row
    	}
    }
    $report
    



  • 3.  RE: total consumption on cluster per vm

    Posted Mar 05, 2009 10:45 PM

    Thanks LucD I think this may really be what the exec's are looking for :smileyhappy: Simple questions...how can I use the Export-Csv cmdlet to export this out into csv format? I tried it..but I believe I put it in the wrong place.



  • 4.  RE: total consumption on cluster per vm

    Posted Mar 06, 2009 02:34 AM

    Never mind, I got it!

    $report | Export-Csv "C:\ClusterStats.csv" -noTypeInformation

    Thanks LucD...this looks (I sure hope) like what they are looking for!



  • 5.  RE: total consumption on cluster per vm

    Posted Mar 06, 2009 03:43 PM

    I was trying to understand this:

    $cpu = $_ | Get-Stat -Stat cpu.usagemhz.average -IntervalMins 5 -Start $from -Finish $to

    IntervalMins is over that time correct? I changed it to 15 from 5 and it still showed it as it was 5...am I misssing something?



  • 6.  RE: total consumption on cluster per vm

    Posted Mar 06, 2009 04:42 PM

    Yes, that's correct.

    But you have to understand the principal of the Historical Intervals.

    There are 4 of these and by default they run over 5, 30, 120 and 1440 minutes.

    If you select a value in between the Get-Stat cmdlet calculates the nearest interval.



  • 7.  RE: total consumption on cluster per vm

    Posted Mar 06, 2009 05:03 PM

    Ahh thanks ...I did not know of that historical interval value. Curious..is it easy to chart this?



  • 8.  RE: total consumption on cluster per vm

    Posted Mar 06, 2009 06:54 PM

    You can see these intervals in the VC Client.

    In the VIC go to entry.

    If by "chart" you mean at which -Intervalmins value Get-Stat would switch to another interval that should be easily automated.

    Something like this perhaps:

    $esx = Get-VMHost <ESX-server>
    for($i=1; $i -lt 1450; $i++){
      $stats = $esx | Get-Stat -Stat cpu.usage.average -MaxSamples 2 -IntervalMins $i
      $delta = $stats[0].Timestamp - $stats[1].Timestamp
      Write-Host $i $delta.Minutes
    }
    

    You should see something like this

    Note that for the 1 and 2 interval the statistical data interval is 20 seconds, which corresponds with the 5th interval, the Realtime interval.

    1 0
    2 0
    3 5
    4 5
    5 5
    6 5
    7 5
    8 5
    9 5
    10 5
    11 5
    12 5
    13 5
    14 5
    15 5
    16 5
    17 5
    18 30
    19 30
    20 30
    21 30
    22 30
    23 30
    24 30
    25 30
    ....
    



  • 9.  RE: total consumption on cluster per vm

    Posted Mar 06, 2009 06:56 PM

    I actually am curious if instead of exporting to excel, i can chart this using OWC or something. Since I need to show this to execs..pictures are always easier :smileywink:



  • 10.  RE: total consumption on cluster per vm

    Posted Mar 06, 2009 07:24 PM

    I see.

    Actually I can imagine several ways of charting this.

    1) automate (from with the PS script) the creation of graphs in Excel

    2) use PowerGadgets, a PS snapin ($) that offers a number of good cmdlets for creating graphs

    3) use PowerBoots, a library that allows a PS script to call all kind of graphical functions.

    In fact I have been playing with PowerBoots recently and I'm quit impressed by this library.



  • 11.  RE: total consumption on cluster per vm

    Posted Mar 06, 2009 07:27 PM

    Brilliant! I will take a look at powerboots! Thanks!



  • 12.  RE: total consumption on cluster per vm

    Posted Mar 17, 2009 05:35 PM

    LucD, I was going through this again and had a few simple questions...wonder what your thoughts are:

    1. You used:

    $clusterCPU = $cluster.Summary.EffectiveCpu

    $clusterMem = $cluster.Summary.EffectiveMemory * 1Mb

    Why the above over:

    Summary.TotalCpu

    Summary.TotalMemory

    Side note, what is the logic behind the 1 mb?

    2. You used:

    Get-Stat -Stat mem.consumed.average

    Why the above over:

    mem.usage.average

    Just trying to wrap my head around a few things here :smileyhappy:



  • 13.  RE: total consumption on cluster per vm

    Posted Mar 17, 2009 07:05 PM

    1. I used

    $clusterCPU = $cluster.Summary.EffectiveCpu

    $clusterMem = $cluster.Summary.EffectiveMemory * 1Mb

    because those are the memory and cpu resources that are available to run virtual machines.

    The properties totalCPU and totalMemory are all the memory and cpu resources that are available.

    The difference between the two is that the Effective properties do not include the overhead from for example the console.

    The division by 1Mb is to get the value in Gb.

    See the ComputeResourceSummary object for more info on these properties.

    2. In the Programming Guide, appendix A both metrics are described as follows:

    • usage : memory currently in use. This is active memory as a percentage of total available memory.

    • consumed : amount of host memory consumed by the virtual machine for guest memory

    The primary reason for choosing "consumed" is that is expressed in Kb, while "usage" is expressed in percentage.

    The secondary reason is that "usage" includes all memory used by the virtual machine while "consumed" expresses what is used for the virtual machine memory.



  • 14.  RE: total consumption on cluster per vm

    Posted Mar 17, 2009 07:45 PM

    Thanks! Once again you have cleared this up for me :smileyhappy:



  • 15.  RE: total consumption on cluster per vm

    Posted Mar 20, 2009 03:59 PM

    Is it possible to get DiskIO in this? I am not sure if that even makes sense across the cluster. I was also trying to see if I can get latency as well...I know I can do that through ESXTop...



  • 16.  RE: total consumption on cluster per vm

    Posted Mar 24, 2009 07:53 PM

    Luc, any chance you have a sample of the output from this script?

    rajesh singh



  • 17.  RE: total consumption on cluster per vm

    Posted Mar 24, 2009 08:25 PM

    Sure, this is from a small test cluster we use.

    The name of the VM is erased.

    I attached the file since the Insert Image button gives a system error



  • 18.  RE: total consumption on cluster per vm

    Posted Mar 25, 2009 01:11 PM

    After some discussion with LucD, it seems that a bug was found in the original script. I am posting here the correct script which LucD has fixed as well as some details for anyone who may be interested:

    clusterName = <ClusterName>
    $report = @()
    
    $cluster = Get-Cluster -Name $clusterName | Get-View
    $clusterCPU = $cluster.Summary.EffectiveCpu
    $clusterMem = $cluster.Summary.EffectiveMemory # In Mb
    
    $from = [Datetime]"03/24/2009 00:00"
    $to = [Datetime]"03/24/2009 23:59"
    
    Get-Cluster -Name $clusterName | Get-VM | % {
    # IntervalMins can be some other numbers as well...see earlier comments in this post
    $cpu = $_ | Get-Stat -Stat cpu.usagemhz.average -IntervalMins 120 -Start $from -Finish $to  
    $mem = $_ | Get-Stat -Stat mem.consumed.average -IntervalMins 120 -Start $from -Finish $to   # In Kb
    
    for($i=0; $i -lt $mem.Count; $i++){
        $row = "" | select VM, Timestamp, CPUperc, Memperc
        $row.VM = $cpu[$i].Entity.Name
        $row.Timestamp = $cpu[$i].Timestamp
        $row.CPUperc = "{0:N2}" -f ($cpu[$i].Value / $clusterCPU * 100)
        $row.Memperc = "{0:N4}" -f ($mem[$i].Value / $clusterMem * 100 / 1Kb)
        $report += $row
        }
    }
    
    # I export to CSV but it can be graphed as well
    $report | Export-Csv "D:\ClusterStats.csv" -noTypeInformation
    
    

    Output of this would look something like:

    VM Timestamp CPUperc Memperc

    MyVM 3/24/2009 23:00 7.36 3.9338

    MyVM 3/24/2009 21:00 6.25 3.4461

    MyVM 3/24/2009 19:00 0.57 2.876

    MyVM 3/24/2009 17:00 3.51 3.5603

    MyVM 3/24/2009 15:00 7.4 3.8084

    MyVM 3/24/2009 13:00 7.25 3.5009

    MyVM 3/24/2009 11:00 0.6 2.8594

    This can be read as 7.36% is the percentage CPU used by the guest calculated against the effective CPU and 3.9338% is the percentage Memory used by the guest against the effective Memory. Both of these values are across the entire cluster.



  • 19.  RE: total consumption on cluster per vm

    Posted Mar 25, 2009 11:31 PM

    Guys, I've been trying to run this script with no luck. It writes out the CSV but with zero data. I am explicitly passing the cluster-name. I wonder if I need to be running VI Toolkit 1.5?

    Also, i sthere a particualr reason why you don't have this script query all of the availa clusters as opposed to requiring a specific cluster?

    Thanks,



  • 20.  RE: total consumption on cluster per vm

    Posted Mar 26, 2009 06:10 AM

    The Get-Stat cmdlet had some problems in VITK v1 but has been much improved in VITK v1.5.

    So yes, you better use VITK v1.5.

    To run the script for all clusters in your environment requires a minor update.

    This should do the trick.

    $report = @()
    
    $cluster = Get-Cluster -Name $clusterName | Get-View
    $clusterCPU = $cluster.Summary.EffectiveCpu
    $clusterMem = $cluster.Summary.EffectiveMemory # In Mb
    
    $from = [Datetime]"03/24/2009 00:00"
    $to = [Datetime]"03/24/2009 23:59"
    
    Get-Cluster | % {
        $clusterName = $_.Name
    	$_ | Get-VM | % {
    		# IntervalMins can be some other numbers as well...see earlier comments in this post
    		$cpu = $_ | Get-Stat -Stat cpu.usagemhz.average -IntervalMins 120 -Start $from -Finish $to 
    		$mem = $_ | Get-Stat -Stat mem.consumed.average -IntervalMins 120 -Start $from -Finish $to # In Kb
    
    		for($i = 0; $i -lt $mem.Count; $i++){
    			$row = "" | select Cluster, VM, Timestamp, CPUperc, Memperc
    			$row.Cluster = $clusterName
    			$row.VM = $cpu[$i].Entity.Name
    			$row.Timestamp = $cpu[$i].Timestamp
    			$row.CPUperc = "{0:N2}" -f ($cpu[$i].Value / $clusterCPU * 100)
    			$row.Memperc = "{0:N4}" -f ($mem[$i].Value / $clusterMem * 100 / 1Kb)
    			$report += $row
    		}
    	}
    }
    
    # I export to CSV but it can be graphed as well
    $report | Export-Csv "D:\ClusterStats.csv" -noTypeInformation
    



  • 21.  RE: total consumption on cluster per vm

    Posted Mar 26, 2009 07:22 PM

    I wanted to get this data into sql server daily instead of a CSV so I updated the code as below (which should work...have not tested it yet). I was planning on scheduling this to run (which has been explained on this forum). My question is how to update the from/to correctly

    $cpu = $_ | Get-Stat -Stat cpu.usagemhz.average -IntervalMins 120 -Start $from -Finish $to

    $mem = $_ | Get-Stat -Stat mem.consumed.average -IntervalMins 120 -Start $from -Finish $to # In Kb

    should it just be

    $cpu = $_ | Get-Stat -Stat cpu.usagemhz.average -IntervalMins 120

    $mem = $_ | Get-Stat -Stat mem.consumed.average -IntervalMins 120

    I get the feeling that this is not the right way...

      $report = @()
    
      $cluster = Get-Cluster -Name $clusterName | Get-View
      $clusterCPU = $cluster.Summary.EffectiveCpu
      $clusterMem = $cluster.Summary.EffectiveMemory # In Mb
    
      $from = [Datetime]"03/24/2009 00:00"
      $to = [Datetime]"03/24/2009 23:59"
    
      Get-Cluster | % {
        $clusterName = $_.Name
        $_ | Get-VM | % {
    		# IntervalMins can be some other numbers as well...see earlier comments in this post
    		$cpu = $_ | Get-Stat -Stat cpu.usagemhz.average -IntervalMins 120 -Start $from -Finish $to 
    		$mem = $_ | Get-Stat -Stat mem.consumed.average -IntervalMins 120 -Start $from -Finish $to # In Kb
    
    		
        for($i=0; $i -lt $mem.Count; $i++){
            $row = "" | select VM, Timestamp, CPUperc, Memperc
            $row.VM = $cpu[$i].Entity.Name
            $row.Timestamp = $cpu[$i].Timestamp
            $row.CPUperc = "{0:N2}" -f ($cpu[$i].Value / $clusterCPU * 100)
            $row.Memperc = "{0:N4}" -f ($mem[$i].Value / $clusterMem * 100 / 1Kb)
       
            $conn = New-Object System.Data.SqlClient.SqlConnection("Data Source=SQL1; Initial Catalog=Test; Integrated Security=SSPI")
            $conn.Open()
    
            $cmd = $conn.CreateCommand()
            $cmd.CommandText ="INSERT Table1 VALUES (@vm, @timestamp, @CPUPerc, @Memperc)"
       
            $vmParam = $cmd.CreateParameter()
            $vmParam.Value = $row.VM
            $cmd.Parameters.Add($vmParam)
    
            $timestampParam = $cmd.CreateParameter()
            $timestampParam.Value = $row.Timestamp
            $cmd.Parameters.Add($timestampParam)
       
            $cpuParam = $cmd.CreateParameter()
            $cpuParam.Value = $row.CPUperc
            $cmd.Parameters.Add($cpuParam)   
       
            $memParam = $cmd.CreateParameter()
            $memParam.Value = $row.Memperc
            $cmd.Parameters.Add($memParam)
       
            #execute query
            $cmd.ExecuteNonQuery()
            
            #close connection
            $conn.Close()
    		}
    	}
    }
    
    



  • 22.  RE: total consumption on cluster per vm

    Posted Mar 26, 2009 08:14 PM

    Assume you schedule this at 03:00 in the morning and that you want to run this for the previous day, then you could do

    $from = (get-date).AddDays(-1).Date
    $to = $from.AddDays(1).AddMinutes(-1)
    



  • 23.  RE: total consumption on cluster per vm

    Posted Mar 26, 2009 08:16 PM

    Much better idea...thanks!



  • 24.  RE: total consumption on cluster per vm

    Posted Mar 27, 2009 12:14 AM

    I just ran this and thought I'd share my output as well. I scrubbed my data so it's ready for the masses.



  • 25.  RE: total consumption on cluster per vm

    Posted Mar 27, 2009 12:34 AM

    This must be a powerful cluster seen the low percentages you get :smileywink:



  • 26.  RE: total consumption on cluster per vm

    Posted Mar 27, 2009 03:11 AM

    Indeed Luc. To put it in perspective, the script took about 40 mins to run and output ~4700 lines to CSV across 5 of my clusters with ~600VMs. All consist of DL 585's with plenty of RAM. This script has proved itself quite useful. I sort on the Memperc & CPUperc to give me an idea of which VMs are eating resources. It's one of many tools in the arsenal.

    Let me ask you, each machine is sampled at 2hr intervals from 04.00 to 22.00. So the percentage columns represent the CPU & Mem consumption at that time. How does this account for ESX Host overhead? I know you discusse this at the beginning of this thread but I am not clear on this.

    I also wonder about thresholds. At what point does a value make me look twice and think...hmmm, this VM might be resource contrained or otherwise in trouble. Part of that answer is user experience too, but just want to see what you think. You commented that my perc values are low...what have you seen on the higher side?

    Thanks,



  • 27.  RE: total consumption on cluster per vm

    Posted Mar 27, 2009 06:53 AM

    Jindijee, I think the percentage shows the average CPU & Mem consumption over 2 hour intervals not the consumption at 02:00, 04:00....

    For the interpretation of performance statistics I think there are people in the Performance community who are better much placed to answer your questions.

    But I'm not a firm believer of fixed thresholds and general rules of thumb.

    Each number has to be analysed with the type and function of the VM in the back of your mind.

    For example, a VM that hosts a CPU intensive application can, in my experience, happily use more than 50% of the available vCPU(s).

    To determine if the application has problems you would need to look at the performance figures inside the OS on the VM as well.

    As a general rule, I would also use the CPU wait time of a VM.

    That can probable indicate better if a VM has a CPU problem than the average CPU usage.

    Same goes for the IO, have a look at the queue lengths.

    In my opinion, performance analysis is an art and requires a good understanding of the environment (ESX hosts, SAN infrastructure, VMs, applications on the VMs...), a lot of experience and a huge amount of common sense :smileywink:



  • 28.  RE: total consumption on cluster per vm

    Posted Mar 27, 2009 01:34 PM

    I have to agree with LucD here...as I was trying to do the same thing and that was the original point of this script (in some ways). Threshold for performance varies and it really is an art and depends on the application (we have some that are not as resource intensive as others).

    LucD you mentioned CPU wait time and IO...but that is per VM not across cluster right? Also, what is the reason behind CPU wait time?



  • 29.  RE: total consumption on cluster per vm

    Posted Mar 27, 2009 03:06 PM

    Yes, CPU wait time (the cpu.wait.summation metric) is per VM.

    Afaik, CPU wait expresses, in milliseconds, the amount of time the CPU is idle because the task(s) is/are waiting for something else (memory page in, IO completion...) to finish.

    This most of the time indicates a busy resource (memory, disk...)

    For IO you could look at how long the queue with IO commands becomes.

    The disk.queueLatency.average metric, for host systems, could be a good indication.

    If this goes up it could mean that your IO system can't follow.

    You can learn a lot by looking at esxtop, for ESX hosts, or top, for most Linux guests, or Performance Manager, for most Windows guests.

    VMware regularly publishes technical papers on performance related issues.

    And have a look at the threads in the excellent Performance community.