IOPS report

[cshells]

We currently just bought some new storage and I am looking for the best way to get IOPS reports from our virtual machines. More specifically I want to get the IOPS for the last quarter within the time frame of 7am-6pm. We are currently on vSphere 4.1. A good majority are View desktops, so thats why I want to look more specifically at business hours. Any ideas? Thanks!

[Daniel Pfuhl]

You could do that with vCOps.

[cshells]

Currently we dont have vCOps. So I was hoping there were some other options. Thanks for the response though!

[dchrist2]

How exactly do you do that in vCops?  The tool is powerful but the gui needs work.....hard to find what you need.

How would I find total/average IOPS per cluster over a specific time frame?

[Daniel Pfuhl]

There are several ways to get a view on the data.

One might be:

- in the vsphere gui

- select the object you want to report on (on the left site of the gui)

- go to the analysis tab

- set the focus to storage

- see which heat maps are available to suit you needs

- there should be a few which might help you out

- build you own heat map, if you need other corelations

- heatmaps do not give you a historical view

for historical views you might:

- go to the vsphere gui

- select the object you want to report on (on the left site of the gui)

- select the planning tab

- select summary

- see which storage related metrics over time you can view

or

- go to the vsphere gui

- select the object you want to report on (on the left site of the gui)

- select the operations tab

- select the all metrics button

- choose from the whole range of metrics (you need to double click on the metric to let ot appear on the right site of the screen in a diagram)

- you also can corelate/combine metrics in this metric charts view

HTH

daniel

[mcowger]

If you aren't already running a performance reporting tool, you wont be able to gather this.  This information isn't kept for that long in vCenter's DB - this level of data is usually only kept for about 24 hrs by defualt.

[MKguy]

Here is a quick and (really) dirty powershell way providing the average and maximum read and write IOPS over a custom period of days:

$daysback = -30
$vms = Get-VM
write-host ( "{0,-40}`t{1,8}`t{2,8}`t{3,8}`t{4,8}" -f "VM", "W IOPS avg", "R IOPS avg", "W IOPS max", "R IOPS max" ) `
; $vms | sort | % {
  $wval = (((Get-Stat $_ -stat "datastore.numberWriteAveraged.average" -Start (Get-Date).adddays($daysback) -Finish (Get-Date) ) | select -expandproperty Value)  | measure -average -max);
  $rval = (((Get-Stat $_ -stat "datastore.numberReadAveraged.average" -Start (Get-Date).adddays($daysback) -Finish (Get-Date) ) | select -expandproperty Value)  | measure -average -max);
  write-host ( "{0,-40}`t{1,8:N2}`t{2,8:N2}`t{3,8:N2}`t{4,8:N2}" -f $_.Name, $wval.average, $rval.average, $wval.maximum, $rval.maximum )
}

Output is like:

VM                                         W IOPS avg    R IOPS avg  W IOPS max  R IOPS max
VM1                                                1,96            0,67            7,00           31,00
VM2                                                0,46            0,53            7,00           17,00
VM3                                                7,57            7,66           52,00           73,00
VM4                                                3,47            0,00             4,00            0,00
[...]

Be aware that historic performance data is rolled up in the vCenter DB. So if you query data from the last 30 days like in this example, the values are made up of 2 hour averages rolled up in the DB. Especially the peak values may look a little low then. Go with -1 (dailly, 1 minute averages) or -7 (weekly, 30 minute averages) in that case and run it periodically.

Here is also a version that will only evaluate values with a timestamp between 07:00 - 18:30:

$daysback = -30
$vms = Get-VM
write-host ( "{0,-40}`t{1,8}`t{2,8}`t{3,8}`t{4,8}" -f "VM", "W IOPS avg", "R IOPS avg", "W IOPS max", "R IOPS max" ) `
; $vms | sort | % {
  $wval = (((Get-Stat $_ -stat "datastore.numberWriteAveraged.average" -Start (Get-Date).adddays($daysback) -Finish (Get-Date) ) | ? {$_.Timestamp.ToString() -match ' (0[7-9]|1[0-8]):'} | select -expandproperty Value)  | measure -average -max);
  $rval = (((Get-Stat $_ -stat "datastore.numberReadAveraged.average" -Start (Get-Date).adddays($daysback) -Finish (Get-Date) ) | ? {$_.Timestamp.ToString() -match ' (0[7-9]|1[0-8]):'} | select -expandproperty Value)  | measure -average -max);
  write-host ( "{0,-40}`t{1,8:N2}`t{2,8:N2}`t{3,8:N2}`t{4,8:N2}" -f $_.Name, $wval.average, $rval.average, $wval.maximum, $rval.maximum )
}

[cshells]

MKguy,

I apologize for such a late response. Had some surgery and was out for awhile. Thanks for the powershell script. I will try this out. I think the second script is exactly what I am looking for. Do you know how far back you can query?

[cshells]

MKguy,

So I ran this script today and I am getting an error. The metric counter "datastore.numberwriteaverage.average" doesn't exist for entity.

Do different versions of vSphere store this statistic different?

[MKguy]

Oh right, seems like you've ran into the same problem as I was after upgrading from 4.1 to 5.0. VMware removed the per-VM IOPS counters from the default vCenter statistics collection levels, so your vCenter database does not store any historical data about them anymore.

You can change this behaviour as described in this post:

http://alpacapowered.wordpress.com/2012/12/17/control-vcenter-performance-counter-collection-and-get-back-vm-iops-statistics/

Unfortunately for you, this means you don't have any access to past data on a per-VM basis. You would need to either start from scratch after making the changes above and wait until you get some amount of samples you're comfortable with, or refer to host-level total counters.

The latter could be quite skewed and inaccurate however, due to non-VM generated IO like StoragevMotions, cloning operations or things like VAAI.

[cshells]

Actually, we are a 4.1 environment. So I am not sure why it is giving me that error. I have tried to pull numberwrite, numberread, and I get the same error with everything.So I can see these stats on the performance tab for my VMs in real-time, but does it keep historical data for this?

[MKguy]

If in the vSphere Client you only see real-time metrics of the last hour for any given counter, then no, vCenter does not store these metrics.

I'm not sure if VMware incorporated that into a later 4.1 Update too. First, check the statistics collection level in the vSphere Client under Administration->vCenter Server Settings->Statistics

Download the LevelMappingUtility.zip file from http://kb.vmware.com/kb/2009532

Connect to your vCenter in PowerCLI and execute the following:

Import-Module D:\VMware.VimAutomation.PowerCliExtensions.CounterLevelMapping.psm1

Get-PxCounterLevelMapping | ? {$_.Name -match "datastore.number(Read|Write)Averaged.average" }

Post the output. The PerDeviceLevel number for these metrics is probably higher than the statistics collection level specified in the vCenter settings, meaning historical data won't be stored at all.

[cshells]

Quick question, importing this module doesn't do anything to vCenter correct? I see in the KB you can make changes, but I assume if you don't configure it, it won't have any affect by importing it into vCenter.

[mcowger]

Correct - its just importing to your Powershell instance.

[cshells]

Thanks guys!

I ran that command and here is the output.

[MKguy]

And what about the vCenter Collection level I asked for? We need to compare both values to get a picture.

If it is at 1, either raise it to 2 so the above metrics are included or use what's described in the post about adjusting the PerDeviceLevel to 1 for these metrics.

Regardless of that you will only start collecting historical IOPS data per VM from that point onwards.

[cshells]

MKguy,

Sorry about that. I didn't even realize I didn't post that. So the Collection level is set to 1. So if I raise the Collection level in vCenter to 2 I assume the only affects would be the database increasing in size? Then we would start collecting historical data from the point we boosted to level 2. I appreciate all the help with this.

[cshells]

Ok. So I think I got an understanding of this. I can use the database size calculator to estimate the changed size. Then I could change the interval for 1 day, 1 week, and 1 month to two. All data after one day is rolled up into one week, one week into one month, and then purged after one month. So then I could look at monthly statistical data. I guess the only part I didn't totally understand was the different levels and what they mean (1,2,3,4). Or how the number I pulled from the PowerCLI relates to the number in vCenter.

[MKguy]

So then I could look at monthly statistical data. I guess the only part I didn't totally understand was the different levels and what they mean (1,2,3,4). Or how the number I pulled from the PowerCLI relates to the number in vCenter.

It basically works like this:

You have a whole bunch of different performance metrics on every vCenter object, in the VM-context you have for example "datastore.numberWriteAveraged.average" for Write IOPS or "cpu.usagemhz.average" for Mhz CPU usage and many, many more.

All of these performance metrics are tied to a certain statistics collection level controlled by vCenter. In your case "datastore.numberWriteAveraged.average" belongs to level 2 and "cpu.usagemhz.average" (will most certainly) belongs to level 1.

Your vCenter is configured to store metrics of statistics level 1 only (as you have confirmed in the vCenter Server Settings window), meaning it won't rollup and store any historical data of metrics belonging to higher statistics levels such as "datastore.numberWriteAveraged.average" which is at level 2.

Now you can either globally increase the statistics collection level to 2, which will result in a lot more historical performance metrics being collected (including the desired "datastore.numberWriteAveraged.average") and thus increasing the size of the vCenter DB by probably a lot more than what would be necessary, or you can configure just selected metrics to belong to statistics level 1 instead in order to avoid collecting too much useless data.

The first option is a bit easier as it's just a couple of clicks, the latter option is described here:

http://alpacapowered.wordpress.com/2012/12/17/control-vcenter-performance-counter-collection-and-get-back-vm-iops-statistics/

http://kb.vmware.com/kb/2009532

[cshells]

Perfect! Thanks MKguy, that all makes sense. I will take a look at the links and get that set up.