CPU %READY Question

[GAPCABIV]

Howdy folks and happy Friday to all.  I have a quick question regarding CPU Ready.  I have read and been told that when a VM hits a CPU Ready of 6% that I should keep an eye on things and at 10% there is an issue that really should be resolved ASAP.

vSphere 5.1 Cluster running Windows VMs (2000-2012)

My question though is do those numbers apply per vCPU or should that be total % Ready regardless of number of vCPUs?

For instance, when I run ESXTOP for 30 minutes in batch mode and look at the results let's say I see a 4 vCPU VM with the following:

Group CPU %Ready = 15.32

vCPU0 %Ready = 3.75

vCPU1 %Ready = 3.69

vCPU2 %Ready = 3.72

vCPU3 %Ready = 3.98

Should I be concerned that the total is over 10% or does everything appear cool because each individual vCPU is well below the 6% that I should start being concerned about?

Thanks

[weinstein5]

Really the ultimate question is what is the end user experience because if the VM is hosting a non CPU intensive application it can tolerate a higher CPU ready -

[GAPCABIV]

Thank you David for the reply.  It is the end user experience that prompted my investigation.  We recently started using MS SCOM to monitor all of our servers (VMs included) and we have noticed that every 5min 20sec that CPU Ready values spike across all VMs particularly when healthcheck.exe runs a vbscript called scompercentagecputimecounter.vbs.  You can see the pattern in the attached image and this is consistent across all 7 hosts in my vSphere cluster.

I am just trying to figure which VMs are most impacted and am curious if the 6% and 10% thresholds should essentially be doubled to 12%/20% for 2 vCPU VMs or quadroupled to 24%/40% for 4vCPU VMs as long as no single vCPU exceeds the 6/10% number in those SMP VMs

.

[GAPCABIV]

When troubleshooting an issue like high CPU % Ready values, what other metrics should I be looking at either through vsphere/esxtop or directly in the guest OS?  I don't see high CPU usage in either vSphere or the guest OS.  Even Processor Queue Length, when viewed through perfmon, seems fairly normal during the high ready spikes.  What else do you suppose I should be looking at?  Specifically in the Windows OS, what perfmon counter relates to vSphere CPU % Ready?  I would think System > Processor Queue Length but that doesn't seem to have much relationship.

[jrmunday]

Should I be concerned that the total is over 10% or does everything appear cool because each individual vCPU is well below the 6% that I should start being concerned about?

Yes, this is the correct approach as the maximum %RDY time for a 4x vCPU VM is 400% .. so your total divided by the number of vCPU's will give you the average - which is below the the recommended guidelines. Depending on your CPU model (AMD or Intel) hyper-threading should be considered as well in this calculation.

I think you are looking at the correct metrics, %RDY time (or latency) from a hypervisor perspective and CPU ulilisation in relation to processor queue length from an OS perspective (make sure you look at MAX, and not only AVG).

One additional counter to consider is the co-scheduling (Co-stop) summation in milliseconds where a VM is ready to run but unable to due to co-scheduling constraints.

Cheers,

Jon

[GAPCABIV]

Thanks Jon.  That is very helpful.  With regards to co-stop numbers they are consistently 0 and even during these ready spikes they rarely exceed 1.0.  Typical values during the spikes are 0.1 - 0.5 however, I have seen co-stop just as high as 11.07.  Again, that is a 4 vCPU VM.  Duncap Epping's ESXTOP blog, and others I have read would seem to indicate that a co-stop value above 3 should be a concern.  Can you tell me if the 11.07 should be divided by 4 (4 vCPUs) and therefore the co-stop value per CPU is below 3 and should not be a concern or is any value above 3 regardless of number of vCPUs an indication that there is a problem and perhaps I should bust that particular VM down to 2 vCPUs?

Just to let you know as well, I am speaking here of my vCenter VM, which was built to VMware's specs based on the size of our environment and therefore it has 4 vCPUs as that is what all the documentation from VMware says it needed.

[raffic_ncc]

Thanks for your valuable answer....