All flash vSAN performance expectations?

[BB9193]

We just deployed an all flash vSAN cluster comprised of 4 Dell R640 ready nodes.  Each node is comprised of:

2 Intel Xeon Gold 6246 @ 3.30 GHz
382 GB RAM
1 Intel Optane P4800x for cache
4 NVMe PM1725B for capacity
1 disk group per node

The vSAN traffic is running over a 25 GB core.  Dedup and compression is disabled, as is encryption.  We're using 6.7 U3.  All firmware and drivers up to date.  Storage policy is R1 FTT1.

I've deployed HCIBench and am currently running test workloads with it.  The datastore is empty except for the HCIBench VM's.  The Easy Run workload of 4K/70% Read/100% Random produced the following results:

I/O per Second: 189042.27 IO/S
Throughput: 738.00 MB/s
Read Latency: 1.48 ms
Write Latency: 1.15 ms
95th Percentile Read Latency: 3.00 ms
95th Percentile Write Latency: 2.00 ms

What should I be shooting for with regard to HCIBench results to be able to verify all is well and I can begin moving my production workload into vSAN?  I'm currently testing the other 3 Easy Run workloads and can post any of those results if needed.

[srodenburg]

I don't have HCIbench numbers for you but my lab has 12G dual-ported SAS SSD's (highest vSAN HCL performance category "F") and high-end Enterprise SATA SSD's as capacity devices. Everything I do just flies. Super zippy. Cloning a 100gig VM -> BAM! done. Working with Servers and doing heavy stuff, it goes like a bat out of hell. Your flash hardware is even faster so you can only expect goodness. The flash-devices and the CPU's in my Lab are fast enough to consistently max out the 10gig links between nodes when I really hammer it.

And I use "compression only" in vSAN 7 U1 and the difference between "no compression" or "with compression" is measurable, but as a human, I don't feel the difference. My fat SQL queries are only fractionally slower with compression turned on, it's almost statistically irrelevant (error margin). With Deduplication+Compression active I noticed a loss in "snappiness" and responsiveness. But "compression only", almost nothing, you could fool me with a placebo.

Honestly, don't get a hard-on about benchmark numbers too much. If it goes like a rocket, it's fast. And vSAN all-flash with proper hardware like you have, goes like a rocket. Trust me.

What can ruin the party though is using crappy switches for vSAN traffic. I've seen people use fat servers connected to cheap-skate switches with small per-port buffers (which saturate quickly) and weak packet-forwarding performance in general and then all your super fast flash storage is slowed down by relatively slow inter-node traffic. Under stress, this aggravates quickly as the switches just can't cope.
It makes a difference for latency if the vSAN vmkernel ports of two nodes have a 0.6ms rtt or a 0.2ms rtt between the two nodes. Rule of thumb:  switches that "think too much" or are simply not very fast (cheap crap), tend to introduce a latency not-befitting the super duper NVMe flash-devices inside the nodes.

[ManuelDB]

From my experience, on hcibench you can expect around these results per node with 2DG per node (on 100% read 100% random 4k):

NVME Cache: 150-170Kiops

SAS Cache: 110-130Kiops

SATA Cache: 60-70Kiops

I consider only the cache because if you run the default test all will be placed on cache, and it's there that you find eventually the bottlenecks.

With 1DG per node just divide them by 2. On Optane I think that 100% read will be just around the NVME performance listed (Optane shines on writes and low latencies, on reads are not much better than NVME)

So for your configuration with 1DG per node I'll expect about 350-400K iops on 100% read 100% random 4K

What are your resutls?

[BB9193]

I posted the results for the 4K/70% Read/100% Random workload in my original post above.  The 256K/0% Read/0% Random workload however has me a little concerned:

Number of VMs: 8
I/O per Second: 11854.95 IO/S
Throughput: 2963.00 MB/s
Read Latency: 0.00 ms
Write Latency: 6.16 ms
95th Percentile Read Latency: 0.00 ms
95th Percentile Write Latency: 12.00 ms

Although the more research I'm doing maybe that's just due to the block size?

Here are the results for 4K/100% Read/100% Random:

Number of VMs: 8
I/O per Second: 330801.05 IO/S
Throughput: 1292.00 MB/s
Read Latency: 0.82 ms
Write Latency: 0.00 ms
95th Percentile Read Latency: 1.00 ms
95th Percentile Write Latency: 0.00 ms

[BB9193]

So I posted a response here a couple of days ago but its gone now, not sure what happened.  I'll post it again with additional info.

My 4K 100% Read 100% Random results are:

Number of VMs: 8
I/O per Second: 330801.05 IO/S
Throughput: 1292.00 MB/s
Read Latency: 0.82 ms
Write Latency: 0.00 ms
95th Percentile Read Latency: 1.00 ms
95th Percentile Write Latency: 0.00 ms

I'm good with these and its what I would expect.  The write side however is much lower than I was anticipating.  Here are my 4K 100% Write 100% Random results:

Number of VMs: 8
I/O per Second: 104066.28 IO/S
Throughput: 406.00 MB/s
Read Latency: 0.00 ms
Write Latency: 2.63 ms
95th Percentile Read Latency: 0.00 ms
95th Percentile Write Latency: 8.00 ms

VMware has initially said this is to be expected due to the redundancy of vSAN.  It doesn't get better than Optane for the cache tier, so I'm confused by this.  We've pushed back on VMware for further verification.

Do these write results look optimal?

[ManuelDB]

Have you tryed with more vms? I think at least 4 per host, so 16 in total, with 4 core each (you have a lot of cores), and 8vmdk each (this is the default number)

In the Proactive Test on network test, do you get 10Gbps?

One thing that you can do for test if network is the bottleneck, is this:

- Create a VM with 1 disk with FTT0
- Check the position of the vmdk (VM->monitor->VSAN disk placement
- Test vmotioning the VM with the VM with disk no the same host (best performance expected), and VM on every other host in order to test performance of network
- For testing also a CrystalDiskMark will be enough. On the sequential read, you must get the full performance on the same host and the performance maxed out by 25Gbps link on the others. Can you share that results? Both read and write

[BB9193]

The number and size of VM's I'm running is the recommendation by HCI Bench based on my configuration.  The Proactive test shows the full 10 GB, but I think this only tests the VM Network, not actual vSAN traffic.  Our vSAN traffic has 25 GB dedicated to it.

I created a new policy with FTT0, but I'm not following how to vMotion the actual disk as there is only one datastore.

[srodenburg]

"but I'm not following how to vMotion the actual disk as there is only one datastore."

He is not talking about storage vMotion. He means "find out where the single disk-component" (with FTT=0, there is only one data-component as it's not mirrored) and do a normal vMotion of the VM to that host. That way, the physical disk is the same host as where the VM is running so the network is out of the equation.

[BB9193]

Gotcha.  I've vMotioned the VM to the same host where its hard disk resides.  The VM Home and VM Swap components are still on other hosts.

Now how am I supposed to test this, with CrystalDiskMark?

[BB9193]

CrystalDiskMark results.

VM on same host as disk:

[Read]
SEQ 1MiB (Q= 8, T= 1): 2220.505 MB/s [ 2117.6 IOPS] < 3775.22 us>
SEQ 128KiB (Q= 32, T= 1): 2430.383 MB/s [ 18542.4 IOPS] < 1724.83 us>
RND 4KiB (Q= 32, T=16): 472.854 MB/s [ 115442.9 IOPS] < 4430.71 us>
RND 4KiB (Q= 1, T= 1): 59.269 MB/s [ 14470.0 IOPS] < 68.94 us>

[Write]
SEQ 1MiB (Q= 8, T= 1): 2158.187 MB/s [ 2058.2 IOPS] < 3878.06 us>
SEQ 128KiB (Q= 32, T= 1): 1902.626 MB/s [ 14515.9 IOPS] < 2200.66 us>
RND 4KiB (Q= 32, T=16): 378.292 MB/s [ 92356.4 IOPS] < 5474.68 us>
RND 4KiB (Q= 1, T= 1): 11.953 MB/s [ 2918.2 IOPS] < 342.32 us>

VM on different host than disk:

[Read]
SEQ 1MiB (Q= 8, T= 1): 1934.667 MB/s [ 1845.0 IOPS] < 4332.42 us>
SEQ 128KiB (Q= 32, T= 1): 1849.289 MB/s [ 14109.0 IOPS] < 2266.81 us>
RND 4KiB (Q= 32, T=16): 418.366 MB/s [ 102140.1 IOPS] < 5007.07 us>
RND 4KiB (Q= 1, T= 1): 36.948 MB/s [ 9020.5 IOPS] < 110.67 us>

[Write]
SEQ 1MiB (Q= 8, T= 1): 2148.930 MB/s [ 2049.4 IOPS] < 3894.09 us>
SEQ 128KiB (Q= 32, T= 1): 2083.816 MB/s [ 15898.3 IOPS] < 2009.56 us>
RND 4KiB (Q= 32, T=16): 336.430 MB/s [ 82136.2 IOPS] < 6192.49 us>
RND 4KiB (Q= 1, T= 1): 10.387 MB/s [ 2535.9 IOPS] < 393.97 us>

 

[ManuelDB]

Not bad. For example, in a cluster where I'm working right now with Inel P4610 NVME as cache I get around 2200MB/s read and 1200MB/s write on same host, and 1900/1100 on other hosts.

Considering that 25Gbps can achieve max of 3GB/s and that some overhead is expected, I think that my results and your results are compatible with the installed hardware (you have much stronger writes) and that networking is not an issue/bottleneck (you have also only 1 DG for host)

As you can see, anyway, you are acheaving 100k iops 4k on random Write with a single host, so from a cluster perspective, I'm expecting 350K iops with FTT0 and obviusly a little less than 180K with FTT1 (add also the checksum penalty and you will get 150K iops I think).

With your test on HCI Bench you get only 100K (with FTT1?), so something is not working as expected.

Have you noticed congestion or network packet drop during the test? (You must run VSAN Observer to analyze well packet drops, the charts on VCenter are totally wrong and show no packet drops when it is happening)

[ManuelDB]

Ah, another information: what switches are you using?

[BB9193]

We're running a pair of Dell S5212F-ON switches dedicated for vSAN traffic.

[ManuelDB]

and what about congestion/packets drop on VSAN nics?

You must check them with VSAN Observer

http://www.vmwarearena.com/how-to-use-vsan-observer/

 

[BB9193]

UPDATE: so after 2.5 months of troubleshooting with Dell and VMware, VMware has finally come back to say there is a vSAN bug limiting vmdk throughput to 40 MB/s and causing the latency spikes we're seeing.  They could give no ETA on a fix or even if a fix was forthcoming.  Their only recommendation was to break up all my vmdk's to no more than 50 GB each, which is not feasible.  A 2 TB file server would require over 40(!) vmdk's.

I can't tell you how disappointing this is and we're starting to doubt vSAN as a viable option for our environment.

[ManuelDB]

Wow, that's an unexpected findings, but can be useful also for a cluster that I'm following.

Can you give me the SR for reference? So I can ask if I'm subject to the same issue, considering that my configuration is pretty similar to yours.

In my case all benchmarks seems fine, but I get some strange latency spikes... Thanks!

 

Anyway, consider that bugs are present in everything. I know that starting with a bug is annoying, but it could be VSAN as any other piece of Hardware/Software.

Recently a customer has an issue with a phisical storage that corrupt VMDK every times it do a storage vmotion for example...

[ManuelDB]

another thing: they say to break up vmdk to 50GB each, but can you obtain this result using VSAN policy and number of stripes? or changin VSAN.ClomMaxComponentSizeGB to 50GB until the fix?

Or is it an issue of the size of the VMDK (so not related of the size of object)?

 

This can be a complete gamechanger in the impact of this bug. If we (I put myself also inside this issue!) can solve it using VSAN.ClomMaxComponentSizeGB I think it's not a great issue for small cluster, and we can wait the solution.

Elsewere, if the solution is to user more VMDK I'm with you that this can't be a solution. How do you split a Database of 400GB??

[ManuelDB]

I have just found that VSAN.ClomMaxComponentSizeGB can't be set lower that 180GB, but the combination of this + stripe could help if it's related to component size and not vmdk size...

[BB9193]

I don't have the SR for the VMware side as VMware was engaged by Dell through my Dell ticket number.  I'll see if I can get that from Dell.

According to VMware there is no known workaround and my understanding is it can happen to any size vmdk.  The issue affects the way the vSAN side handles an IOPS burst to a single vmdk at a time.  Their logic is that breaking up the vmdk's will distribute the IO's across multiple vmdk's, which makes sense, but its just not feasible.

I understand that bugs happen and that's just part of the deal, but the frustrating part for us is that we started reporting problems literally as soon as the cluster was deployed and its taken 2.5 months just for us to get to this point.  We're now in our business critical season so even if I wanted to break up a vmdk its too late.  We're going to have to run this cluster as is for the next few months.

What I don't understand is what the trigger is for this bug?  I've asked but thus far have not heard back.  VMware says only a very small number of vSAN customers are affected, but there is nothing special about our cluster as it consists of Dell Ready Nodes and was configured by the Dell ProDeploy team.  So I don't understand why we would be affected while the majority of customers are not.

[ManuelDB]

My doubt is that I'm in a similar situation.

Have done a POC with 2 servers and all was blazing fast on 6.7U3

Now, exactly the same hardware but with 5 servers, go slow on 7.0U1 (I haven't tryed 6.7U3 on this configuration unfortunately).

We have digged a lot and given the fault to the switches, but it could also be that bug. I have just asked to the support (I have an open case for performances, that are wonderful on HCIBench (small VMDKs) but not as good in production because of latency spikes)

[BB9193]

VMware has reviewed our cluster end to end and while originally they suspected there was a networking problem they eventually identified the bug.

[TomIvone]

  Do you have a service request or bug number we can reference on the Dell or VMware side, I believe we might be having the same issue with our Dell all flash ready node *3 VSAN environment.

The environment is everything we expect it to be and more in terms of performance, except for single VM specific burst IO requirements.

VMware cannot find any issues and have confirmed the performance is what is expected, but when testing with HammerDB high burst IO on a single VM our performance is terrible.

From our testing on a single host with the same spec vs 3 node VSAN we are seeing a drop of minimum 70% TPM performance and in some cases higher.

While we expect a performance drop in performance single host vs HCI we do not believe the burst IO performance drops we are seeing are expected.

Test we are running

HammerDB - SQL TPC-C (Cloned VM's so identical)

1 Warehouse, 10 Users

Single host - 335000 TPM

VSAN - 95000 TPM

[BB9193]

The Dell SR is 1044143886 and I believe the VMware SR is 20175210611.  I also recommend running HCI Bench to get full benchmarks for your entire cluster.  

Watch the performance graphs per VM during normal day to day operations.  What we see are random latency spikes throughout the day on all VM's running on vSAN.  They typically range anywhere from 5 ms to 30 ms.  We actually saw a spike of 130 ms recently.

This latency issue was totally unexpected for us as we anticipated all flash with Optane to have extremely low latency.  Dell was caught off guard as well.

[BB9193]

 You guys make any progress on this?

[TomIvone]

We were unable to 100% prove this was our issue and we determined that the workload requirement was unsuitable to VSAN, test within a Dell lab using Optane based storage were unable to get the performance we require in a single VM use case.

At this stage we are still working through if we keep VSAN as it meets 99% of our requirements and use a single host + replication for the 1%, other option would be to ditch VSAN and go back to the SAN + Raid10 model.

[BB9193]

 What size was the vmdk on the test VM?  vSAN has throughput limitations per vmdk.

Supposedly they have made decent performance improvements in the later versions of 7.x.  I'm hoping there will be a business stable release of 7.x ready for later this year.

[TomIvone]

120GB and 100GB.

My requirement for one VM in this environments is our main issue.

It must be able to using HammerDB and MS SQL reach 200,000 TPM using the following.

Number of warehouses: 1

Virtual users to build schema: 1

Virtual users: 10

[BB9193]

 We are also disappointed with our SQL performance in vSAN.

[Sharantyr3]

Hello

That explains why I can't reproduce huge latency spike I see on my all flash vsan cluster using ioanalyzer, I use small disks on these appliances and get good results...

But on the other hand, production VMs running sustained write stream show very poor performances :

(this is a pftt1 sftt 1 raid 1 VM)

On my side I got no success with vmware support (I got tired to run tests and debug on our production environment while the issue is undeniable).

My personal feeling is vsan is not working well with big io size (>256KB) but your issue is with 4KB io size...

Following this thread for news.

 

I would be happy to bench and share results here to compare performances and help, if you tell me which benchs you'd like to run.

Our current setup is stretched (I can run tests on non stretched storage policy) 7+7 all flash, each node has 3 DGs (6+1, 5+1, 5+1), 25GBps ethernet, dell switchs S5248F-ON

[BB9193]

My HCI Bench test results and parameters are posted early on in this thread if you want to try any of those for comparison.

I'm told this issue(s) will possibly be addressed in 7.0 U3.

[Sharantyr3]

Can you confirm me the bench target that you want me to test ?

4K 100% Write 100% Random - 7 workers each one on different ESXi

pftt0 sftt1 - raid 1 (non stretched)

 

Here are my results on ioanalyzer :

Write latency about 5ms

 

Same test but 100% read instead of 100% write :

Read latency about 0.5ms

 

Please note, unlike you, I have no nvme, only regular sas ssds, write intensive for cache.

My ioanalyzer is pretty old and doesnt generate graphs or latencies anymore, I can setup HCIBench if needed.

[BB9193]

 Yes, you would need to use HCI Bench.  Here are some of my results from previous runs:

My 4K 100% Read 100% Random results are:

Number of VMs: 8
I/O per Second: 330801.05 IO/S
Throughput: 1292.00 MB/s
Read Latency: 0.82 ms
Write Latency: 0.00 ms
95th Percentile Read Latency: 1.00 ms
95th Percentile Write Latency: 0.00 ms

Here are my 4K 100% Write 100% Random results:

Number of VMs: 8
I/O per Second: 104066.28 IO/S
Throughput: 406.00 MB/s
Read Latency: 0.00 ms
Write Latency: 2.63 ms
95th Percentile Read Latency: 0.00 ms
95th Percentile Write Latency: 8.00 ms

[Sharantyr3]

Hello,

Tried hcibench but the pressure on our vsan cluster was too high, first time I see "congestions" counter raising up. I had to abort testing when vsan performance service was hung and alarms about host communication problem with vcenter were happening.

Also you didn't mention which hcibench u did run (easy run or custom?). I choose custom as I don't like "auto" things run on their own but maybe 7 VMs with each one 4 disks was too much.

How many VMs per ESXi, and how many disks of which size per test VM did you run ?

Also what "Working-Set Percentage" did you chose ?

 

I can run new tests on off hours.

 

 

edit, bench with reduced load on vsan (by reducing number of vmdk per vm to 1) :

 

Case Name	Job Name	Number of VMs	Number of VMs Finished Early	IOPS	Throughput(MB)	Read Latency(ms)	Write Latency(ms)	Read 95tile Latency(ms)	Write 95tile Latency(ms)	Blocksize	Read Percentage	Total Outstanding IO	Physical CPU Usage	Physical Memory Usage	vSAN CPU Usage
fio-1vmdk-100ws-4k-0rdpct-100randompct-2threads-1617721835	job0	7	0	12393,06	48	0	1,13	0	1	4K	0%	14	0.0%	44.93%	0.0%
fio-1vmdk-100ws-4k-100rdpct-100randompct-2threads-1617722455	job0	7	0	38420,11	150	0,37	0	0	0	4K	100%	14	N%	N%
fio-1vmdk-100ws-512k-0rdpct-100randompct-2threads-1617723057	job0	7	0	4067,45	2033	0	3,65	0	4	512K	0%	14	0.0%	44.93%	0.0%
fio-1vmdk-100ws-512k-100rdpct-100randompct-2threads-1617723569	job0	7	0	8413,18	4206	1,73	0	3	0	512K	100%	14	0.0%	44.93%	0.0%

working set % : 100

1 VM / ESXi, 1 disk 10GB 2 threads / VM

7 VMs total, raid 1

 

Weird, even reducing load on vsan, the vsan performance service seems to get hammered by hcibench  and gets unresponsive while running benchs (no more performance graphs on vcenter for vsan tab).

 

I need your exact tests specifications to run the same here for comparison

 

 

edit 2

 

Case Name	Job Name	Number of VMs	Number of VMs Finished Early	IOPS	Throughput(MB)	Read Latency(ms)	Write Latency(ms)	Read 95tile Latency(ms)	Write 95tile Latency(ms)	Blocksize	Read Percentage	Total Outstanding IO	Physical CPU Usage	Physical Memory Usage	vSAN CPU Usage
fio-4vmdk-100ws-4k-0rdpct-100randompct-2threads-1617737533	job0	7	0	49954,22	195	0	1,12	0	1	4K	0%	56	N%	N%
fio-4vmdk-100ws-4k-100rdpct-100randompct-2threads-1617738028	job0	7	0	146658,57	572	0,38	0	0	0	4K	100%	56	15.69%	45.0%	1.61%
fio-4vmdk-100ws-512k-0rdpct-100randompct-2threads-1617739020	job0	7	0	10684,99	5342	0	5,34	0	9	512K	0%	56	N%	N%
fio-4vmdk-100ws-512k-100rdpct-100randompct-2threads-1617739833	job0	7	0	26186,92	13093	2,16	0	3	0	512K	100%	56	0.0%	45.0%	0.0%

 

working set % : 100

1 VM / ESXi, 4 disks 100GB 2 threads / VM

7 VMs total, raid 1

 

 

Looking my numbers I find the difference % ratio between read and writes iops quite the same as you (you got faster but less cache disks)

 

What numbers would you be expecting (pre-sale vendor said) ? 

 

[akaddour]

I too am finding some unexpectedly "average" performance on a 5-node all-flash vSAN cluster I'm working on for a client (vCenter 6.7), vSAN on-disk format version = 10.

The environment private cloud so I don't have full visibility of the back end network, however I do see that the hardware specs of the host servers are very good and modern.

I use a combination of tools to assess performance. HCIbench for cluster level performance, but for 'real-word' performance test I use simple SSD read/write utilities on a Windows VM guest. I won't go into the exact numbers, but basically the VM guest performance on vSAN performs worse than it does on VMware workstation, installed on my laptop with a single consumer grade SSD. The performance is slightly better than my testing lab server, which has 8x Samsung Evo 850's in RAID10 on an old LSI MegaRAID 9261-8i controller.

I have a case open with VMware and the private cloud vendor, so we are trying to work out what is causing the poor write performance and high latency spikes.

[ManuelDB]

For me it was because on VSAN nics I was not using lacp and the 2 ports was configured as active/active and not active/passive and Aruba switch was awful as intraswitch performance.

[Sharantyr3]

Same here, and I tell you why : benchmark use small to average io size : good perfs on vsan.

When you use file copy or any real world use case, like database dump, etc, you may get poor performances (by poor I mean not what you would expect from SSDs). I write "may" because it depends on the OS, filesystem etc, but in the end, if your io size are > 1MB, performances are poor.

Everyone here will just say to you, "file copy is not a real benchmark" end of discussion.

But as you, I think it may not be a benchmark, but it's real world use case, and performances are not ok.

And I can't find any bottleneck in the chain, neither vmware support could, so I gave up on my support request.

 

Overall, servers works just fine because 90% of iops are small io. Just don't be surprised that when you do a filecopy you get poor performances, it's by design.

[BB9193]

We had an escalation ticket open about this for months and support eventually confirmed that with vSAN 6.7 there are low throughput limits per vmdk.  They actually suggested we break up all vmdk's to no more than 50 GB each to get around these limits, which obviously is ridiculous.

If you have tickets open, ask support about the vmdk throughput limit as I don't recall the specifics.

[kastlr]

Hi,

using a single file copy job on any shared storage array will always result in lower performance.

This is by design, as any shared storage is designed to get accessed by multiple hosts (different OS, different applications, different IO profiles, multiple threads).

So IF your used case requires high single thread IO performance it would require tuning on the VM side.

Here're some recommendations to increase windows single thread IO performance.

• disable Windows Write cache
• allign NTFS file system
• if possible, configure guest OS to use a maximum IO size of 64kb
• use multiple PVSCSI Controller per VM (up to 4)
• create multiple vmdks and use a SPBM with proper number of stripes
• spread those vmdks evenly across the PVSCSI controllers
• use windows dynamic disk to create a striped volume out of those vmdks

This should increase performance of those single thread applications.

Just a side note.

Windows explorer file copy tasks uses different IO Sizes for read and writes, but both are larger than 64KB per IO.

vSAN is optimized for 64kb IOs.

vSAN has to split larger IOs into smaller chuncks, and such activity would increase the latency of the IO.

And when using only a single/few vmdks with default SPBM stripe setting of 1 you might end working with a single cache device instead of spreading the load on multiple cache devices.

 

But I would be really interested in a reference/kb article regarding vmdk throughput limit.

[Sharantyr3]

Hello,

What you are writing is partialy right. Have you ever tested file copy on low end array ?

I get more throughput using a 60 magnetic disks in sequential read / write than in a full flash vsan cluster.

But it's as you said, the vsan is here to serve many "customers" not only my benchmark, whereas the low end array will go all in for me.

 

I think the main problem here is to accept the fact that, in a vsan cluster, you may get less performances on one job with hundred of flash disks working than you would get on a single direct attachment flash disk (or even local raid 5 raid card with 3 flash disks). It's disappointing but it's by design. 

And it's good to know it - one VM can't compromise the whole vsan storage cluster.

 

What's the most frustrating I think is no bottleneck is seen in the chain.

Check data disks : latency ok

Check cache disks : latency ok and not filled up

Check network cards : way below full bandwitdh

Check switchs : underutilized

Check vsan : no bottleneck in graphs

 

Also some graphs doesn't make sense to me, and that is frustrating too.

But I believe the global vsan graphs are not accurate : it's related to our discussion. The vsan graphs show an average performance of VMs doing IOps.

And if at a specific point of time, only one VM is doing high IOPS : this VM will make the graphs "averages" go crazy.

Typical exemple : the backups.

Look at this (I removed writes as they are not spiking and it's easier to read and get my point) : 

 

We see huge spikes in read latency while iops don't raise, but throughput do raise. So it's the sign of increasing IO size incoming on the vsan.

It's typical of backup jobs, reading big chunks.

You can see later an iops read spike while latency don't increase this much.

 

At first I thought there was a problem of latency in my vsan, but in fact there is no problem : just one VM doing a lot of stuff and result in "false average" latency graph.

I believe that, if during this backup time, I had many VMs doing "normal" iops (at least 3 or 4 more than the backup job), my graphs would show "normal" latency, not such spikes.

I don't know how this could be fixed.

[kastlr]

Hi,

even a "low end array" has specialized storage controllers with (more or less) DRAM acting as cache.
vSAN uses SSDs as "cache devices", so there's by design a big difference.
That's why it's required to choose a design which will cover your needs.

What's the SPBM you're using for those special used case?
And what kind of capacity SSDs do you use, NVMe, SAS or SATA?

Even if a vmdk is build out of multiple components (limited to 255 GB/component) this doesn't automatically mean that the data is striped.

So if you run a file copy job on your vmdk with the default SPBM setting (Stripe=1) your sequential reads usually will end on a single disk.
Windows explorer doesn't send multiple read IOs, therefor you won't see bottlenecks on the device side.

Instead it will look like this

• large Read IO raised by VM
• vSAN checks if data is on cache disk or not (Cache Hit or Miss)
• vSAN has to split it into smaller chunk
• vSAN will send those smaller Read IO requests to a/multiple disks (when Stripe is set to 1 typically to a single disk)
• disk will process the Read IO and returns data
• vSAN combines the smaller chunks
• vSAN returns the data requested by the VM
• VM sends the next large Read IO

Windows Explorer file copy read operations are like a sequential read with no OIOs, not the best workload for vSAN.

Backup operators doesn't care about latency, all they're asking for is high throughput.

The vSAN graphs aggregates data (as many other monitoring tools), this could end in a scenario which you described.
But that's normal, and as long as your users doesn't raise concerns about performance it's fine.

It's like on the German autobahn, the average speed is 120 km/h, but at 8am it could be less than 40 km/h while at 3am you might drive as fast as your car could run.

Do you have an actual performance problem or are you only concerned about the numbers reported by the vSAN graphs?

[srodenburg]

Install a free appliance called "SexiGraf" and configure it to talk to your vCenter. After an hour or so, you can select dashboards for the various vSAN types and find out on what latency on which level is affecting you. Disk, Client or Owner.  Very good tool for this sort of thing and easier to use than VMware's own IOINSIGHT. It also runs all the time so you can look at historical data.

QuickStart – SexiGraf

Everything you need to download and all Infos are on this Quickstart page.

[Neverland5]

I've been following a issue like this with a customer for awhile, and I'm boiling it down to a few things:

Raid Stripe Width

Raid level

Drive throughput

Network bandwidth

 

My best guess is a stripe of 1 and raid 1, leaving one copy on a disk group and the other copy across a 10gbit link (to another server or the other side of a stretched cluster) You're only going to see 22Gbit/S at the MAX. That's something like 3GB/s, AKA 90,000 32Kbit IOPS.

If you'd ever even see the full 3GB of read IOPS since drives aren't usually as fast as the full throughput.

 

I'd increase raid stripe width.

Side note: I have heard that link teaming doesn't aggregate across the link either, you only get one of the two links, and you can't team vSAN it will only ever use one link.