Number of VMs per datastore

[slctaylor]

On ESXi 6.7 and 7, is there a guideline for how many VMs per datastore? I've tried to search, but I am not find anything very current.  I know the old recommendations from like 10+ years ago was about 15-25 depending on activity due to things like SCSI locks, queue depth and reservations.  That's what I've been still going by, and usually try not to exceed 25.   I've had some with 30-35 for a little while until I could rebalance, and they have been OK.   

Problem is, that we have hundreds of VMs, so spreading them out amongst dozens of datastores to keep the counts at 25 does become a bit of administration overhead and often makes for space silos.  Maybe you will need space, but the only datastores that have a bunch of free space already have a lot of small VMs on them, for example.  I'm getting some pushback from some on my team who think we should just make a few huge datastores and fill them with 100s of VMs each, which seems like a bad idea to me, but maybe my training is just outdated. 

Just wondering if there is an official recommendation or what other people are doing.   

We are running two different SAN, one on 16GB FC and another on 10Gb iSCSI, both are very fast storage.   

Thanks in advance.  

[continuum]

Disclaimer: this is a biased point of view - my daily job is to recover VMs from corrupt datastores.

Questions you have to answer first:
1. do you replicate your datastores regularly - does this really work ???
2. do you backup your VMs daily - does this really work ???

Assume one of your datastores fails - can you be back in production over the weekend ? - or do you rather need a whole week ?
In my point of view the number of VMs is less relevant than the size of your datastores.
A datastore that is so large that replacing / rebuilding takes a week is IMHO a bad idea - a really bad idea.

All parameters you mentioned like SCSI locks, queue depth and reservations may cause minor complaints.
If a recovery takes a week or longer it can result in a business desaster !

My recommendation: a datastore that you cant replace / rebuild over the weekend is too large.

Ulli

[MikeDawson]

really hoping to get more answers on this question as I'm in the exact same boat as @slctaylor

is the "20 VMs per Datastore" rule of thumb outdated?  if so, why or how?

[Pawel Koc]

There is no simple answer: "How many VMs should I run on Datastore?" because it depends on many factors.
First of all, let's take a look at maximums (vSphere 8):

Powered on virtual machines per VMFS volume: 1024
As you can see, it is far from "20 VMs per Datastore".

But which factors have an impact on VM count on DS? We must take a look at the documentation and basic best practices:

https://www.vmware.com/docs/vsphere-esxi-vcenter-server-80-performance-best-practices

Take a look at "General ESXi Storage Recommendations."

So, how many VMs?

• It depends on the KPI and SLA that you want to meet. If your workload does not require good performance and latency may be high, you can put more VMs. Do you need more IOPS per VM? You will probably put fewer VMs there.
• Place VMs on Datastore and monitor IOPS, Latency, Queues, etc.
• If you know earlier what performance VMs need, calculate how many VMs you can place on DS.
• One VMFS performance is not equal to others VMFS. You can have VMFS on a high-end storage array, but someone can use HDD drives. That's why you cannot say 20VMs per DS.

------------------------------
-------------------------
www.blanketvm.com
-------------------------
VCP-VCF, VCP-DCV, VCP-CMA, VCAP-DCV, VCAP-CMA, vROPS|vSAN|VCF Deployment Specialist
-------------------------
------------------------------

[MikeDawson]

Thank you for the reply, however that really doesn't answer what I'm looking for, perhaps I didn't ask it correctly

I'm looking for the actual technical reasons why putting fewer VMs on a datastore would (or wouldn't) increase their storage performance. 

For example, does a VM issuing a write to the datastore cause a lock for the entire datastore during the write operation, affecting all VMs on it? 

if so, is that the case for both Fibre Channel and iSCSI? or does it vary per SAN provider?  If so, what makes it vary? or is that outdated information in this day and age?

what other parameters, settings, and configurations would make fewer VMs per datastore be more performant?  Taking this theory to its logical conclusion, if you had 100 VMs, would creating 100 datastores make their storage performance faster?  If so, why?

This is the type of info I'm looking for 

[Pawel Koc]

I'm looking for the actual technical reasons why putting fewer VMs on a datastore would (or wouldn't) increase their storage performance. 

Let's say that you have 100 VMs.

Each will generate 200 IOPS.

You have Datastore that can reach 6000 IOPS.

What happen if you run 30 VMs on DS? All will generate 6000 IOPS.

But if you run 100 they will need 20000 IOPS but only 6000 is available.

They will not work as you expect. Latency will increase etc.

------------------------------
-------------------------
www.blanketvm.com
-------------------------
VCP-VCF, VCP-DCV, VCP-CMA, VCAP-DCV, VCAP-CMA, vROPS|vSAN|VCF Deployment Specialist
-------------------------
------------------------------