My IOPS are too good (using IOMeter), any ideas why?

[admin]

Hey All,

I am trying to find the flaw in my testing setup.  I have been running many IOMeter tests, and I keep getting results that are too good to be true.

I am getting 1650 IOPS from a 6-disk RAID10 LUN on an HP p2000 array (8Gb FC).  This seems like way too many.  With an assumption of 145 IOPS/Drive, I should have a max of 870 before the RAID penalty.

Thanks,
Drew

• BL 460G6
  
  • (2 sockets, 6 cores per socket, Nehalem)
    
  • 144GB memory
    
  • Local disks are 15K SAS
    
• p2000 Array
  
  • 8 Gb FC
    
  • 6 x 300 GB 10K RPM SAS drives (RAID 10)
    
  • Dual controllers, cache is 2GB/2GB
    
  • 64K Chunk Size
    
• ESXi 4.0 u1
  
  • VMFS volume has 2MB block size
    
• VM is Vista SP1 (have not been able to find out how to turn off disk caching - box is grayed out in disk properties)
  
  • 1 vCPU, 1024MB RAM
    
  • Using a second hard drive as an unformatted physical disk
    
  • 4 Workers, 3 @ 80,000,000 sectors, 1 @ 30,000,000 sectors (all with 32 queue depth)
    
  • 4KB transfer size
    
  • 100% Random, 70% read 30% write
    
  • 32 Outstanding Transfer
    
• Test Connection Rate (not checked)
  

[admin]

I ran the test a few more times and am still getting 1650 IOPS.  Crazy.

According to what I can find, my IOPS per drive should be 143 1/(seek time 4ms+latency 3ms).

Given that there are 6 drives in this LUN, the aggregate latency should be 858 (before RAID).

Drew

[mcowger]

You are experiencing the effects of caching, my friend, both within the VM and at the P2000 array level.

[Josh26]

Hi,

"Theory" around expected IOPS almost never pans out.

As stated, you are seeing the effects of caching. You are also seeing the high performance of FC and HP's RAID platform.

Edit: I find it interesting on these platforms is when you rebuild the array with RAID6 and find it's marginally slower than RAID10. Every theorist and his dog will tell you it has to be unusably slow. They will be wrong.

[admin]

That's what I was thinking too.  I tried to turn off caching in the OS, but the option was greyed out.  I looked everyhere I could think of in ESX to disable ot, but no luck.  I haven't checked the p2000 yet to see if I can disable the cache.

Any ideas how I could disable the caching in the VM?  I can understand why I should leave the cache enabled on the p2000.

thanks,

Drew

[mcowger]

ESX itself doesn't have a cache, which is why you cant find it there.  Dunno how to do it in windows.

[JRink]

When your run IOMeter, you need to make sure the test file it creates on your VM is larger than the total size of your array cache.  Check the size of that file and let us know how big it is...

[admin]

Jrink,

Thanks - I am using 80,000,000 sectors, which should be about 40GB.  I have dual controllers, each with 2GB of cache.

So my file far exceeds the caching capability of the array.

Thanks,

Drew

[rickardnobel]

Drew wrote:

I ran the test a few more times and am still getting 1650 IOPS.  Crazy.

According to what I can find, my IOPS per drive should be 143 1/(seek time 4ms+latency 3ms).

The IOPS formula above, is it known what IO size it is for?

[admin]

No, there wasn't an input for IO size.  Just operations per second based on seek time and latency.  If there is a more accurate formula, please let me know!

[rickardnobel]

Drew wrote:

No, there wasn't an input for IO size.  Just operations per second based on seek time and latency.  If there is a more accurate formula, please let me know!

I was just thinking that two things might factor in:

Small IO size would in general mean more possible IOPS per second (perhaps than expected by the formula)

The 40 GB test file is spread over the six RAID 10 disks you have, which might mean that each disk has something like 13 GB each. Since the disk is a 300 GB disk then the vendor specified average seek time is probably over "larger areas" of the disk. Your iometer file is much smaller and the disk arms have to move a shorter distance, which could mean a much lower seek time.