High-Capacity VSAN 6 Nodes

[jwhitehv]

In looking at the current VSAN 6 HCL, there are no high capacity SATA drives. Actually, no SATA drives at all for VSAN 6. Just wondering how others are putting high-capacity VSAN nodes together?

Is this a data error? Or a problem with SATA and VSAN 6? Or just a backlog in the qualification lab?

Without 4TB or 6TB drive options, the possible density per node plummets. Standard 2U servers generally come in either 12 x 3.5" or 24 x 2.5" drives. 12 x 4TB drives in VSAN 5.5 configs would yield 48TB of raw storage. 12 x 6TB drives gets 72TB raw. At least the 4TB drives are available in NL-SAS configs from some vendors and OEMs, but not all. Anyone doing this at all? Dell FX2 could get a full 35 drives, but the 2TB 2.5" SAS drives aren't on the HCL, so no 70TB 2.5" configuration.

Also, there's no FusionIO products on the list.

Maybe they're just on there as OEM parts? Actually, I double-checked and there are no PCIe cards on the VSAN 6 HCL.

How are other people doing high-capacity nodes?

Is 12 x 6TB per 2U server even possible with parts off the HCL?

[jonretting]

I could have sworn there were some SATA Dell disks listed on their, or I am probably thinking Seagate Constellation SAS.  From my experience as long as you have an HBA with decent QD (min LSI 2308), and over the top Enterprise PCIe SSD NVME, 6TB SATA 7200 disk have worked out just fine. Had flying success with high-end PCIe and 4x 6TB SATA per host. Works great. However 4TB SAS really is the way togo. IMHO

Clarification would great pertaining to you finding.

Cheers

[jwhitehv]

as long as you have an HBA with decent QD (min LSI 2308), and over the top Enterprise PCIe SSD NVME, 6TB SATA 7200 disk have worked out just fine

Except if the components aren't on the HCL, the user can be denied support. That's not a great place to be.

[jonretting]

Correct -- It all depends on the risk assessment. The cutting edge is always not without risk. However vSphere by definition removes a great deal of risk. Somethings not working shift things around and rebuild it. Being locked down to physical hardware and networking is the real risk. In the case where I deployed that non HCL 6TB cluster, it was a very established high capacity replicated vSphere environment. In the end it was their acknowledged risk, as I do not condone SATA in any form. This statement does not however make your assessment of the current HCL list any less important. Best, -Jon

[zdickinson]

I have previously asked in forums about the lack of any PCIe cards on the HCL, I believe the answer was (as you suggested) a back log in QA.

As for the lack of large capacity SATA drives...  I would agree with retting, SAS is the way to go.  And I'm curious about your use case.  If you take 12 x 4 TB drives = 48 TB per node x 64 nodes (Max in a vSAN 6 cluster) = 3,072 TB.  I mean wow!  What are you scaling to 3 Petabytes?

Thank you, Zach.

[Bleeder]

The funny thing is that VMware already published a whitepaper (Virtual SAN 6.0 Performance: Scalability and Best Practices) several months ago where they used Intel P3700 PCIe SSDs.. but they're still not HCL'd.  Customers have been waiting over a year for those things to show up on the HCL..

[jwhitehv]

The funny thing is that VMware already published a whitepaper (Virtual SAN 6.0 Performance: Scalability and Best Practices) several months ago where they used Intel P3700 PCIe SSDs.. but they're still not HCL'd.  Customers have been waiting over a year for those things to show up on the HCL.

That is really frustrating. Scalability needs to be supported.

[jwhitehv]

I'm curious about your use case.  If you take 12 x 4 TB drives = 48 TB per node x 64 nodes (Max in a vSAN 6 cluster) = 3,072 TB.  I mean wow!  What are you scaling to 3 Petabytes?

One doesn't need to scale to 64 nodes to want high storage density.Any time someone asks about VSAN, storage density per socket becomes an issue. That's $2.5K per socket and some customers aren't willing to drop down to a single socket per host, meaning $5K/host in VSAN licensing. Since external storage expansion doesn't seem to be supported despite GA announcements, VSAN starts looking like a capacity license.

2U 24 disk SFF host:
3 disk groups of 7 HDD + 1 SDD

The largest SFF HDD on the HCL is 1.2TB, so

24 * 1.2TB = 28.8TB/host

Using 6TB LFF drives:

12 * 6TB = 72TB/host

That's 2.5x the storage per VSAN socket license.

With Dell's FX2 architecture, you can get three FD322 storage modules with 5 x 7 SFF drive disk groups.
35 * 1.2TB = 42TB

6TB drives are still 1.7x the storage per VSAN socket license.

If you're willing to go larger than 2U, the HP SL4540 will scale to 60 LFF drives, so you can reach the 35 drive/host VSAN limit. The Cisco C160 scales to 60 LFF drives, but it's controller isn't on the VSAN HCL.

HP seems to have the best capability to scale VSAN, with the SL4540, 35 LFF drives, and a controller and 6TB SAS drives on the HCL. And even then, the largest HP SDD on the VSAN HCL is 1.6TB, when you should have 2.1TB per 42TB disk group.

I smell a blog post coming on.

[Bleeder]

I think you hit the nail on the head with the comment about VSAN socket licenses.. Think about it.

[jonretting]

Blog post indeed.

[zdickinson]

Let's take this example:

Using 6TB LFF drives:

12 * 6TB = 72TB/host

Let's say it's a minimum 3 node cluster, that's 216 TB before overhead.  It work load can consume that much space, but still perform well with only 6 sockets in the cluster?  It seems like with disk that dense, you would run out of compute power before you could fill it up.  Is it a few VMs that are massive?  A lot of little VMs that don't consume much CPU?  Something else I'm missing?  Thank you, Zach.

[Bleeder]

Does NexentaConnect, SoftNAS, etc answer that question?

[jonretting]

Correct me if im wrong but Netextenda just utalizes a VM with VSAN attached vmdks, and then offered up as SAN/iSCSI/NFS/CIFS. Along the same notion VVols would be a separate solution from VSAN, more akin to DAS with the benefits of VSAN like Virtual Volumes, performance tiers, etc. Or so is my current understanding  Best, -Jon

[Bleeder]

Correct, using 3rd-party solutions that turn VSAN into a NAS.  Not much compute needed.  Just lots of space and IO :smileyhappy:

[jonretting]

It does in my minds eye add significant complexity, and once again locked into hardware types. Also how many Vvol system failures can you tolerate... In essence it seems like inverted convergence, where we took what we learned from VSAN convergence, inverted the principals and applied it to the physical SAN. Thanks, -Jon

[jwhitehv]

Does NexentaConnect, SoftNAS, etc answer that question?

Answer what question? Definitely not, "How are other users scaling VSAN?"

[zdickinson]

I think he was answering my question about small compute power with a large amount of storage.  And yes, I can see that if you used something like Nexenta to serve up large amounts of storage you would need a small amount of compute.

72 cores in a 3 node cluster seems small to me.  I guess you are correct, it depends on the environment.

Back to your initial question of whether or not anyone had used vSAN at high density...  I don't have a direct answer, but Duncan has a had a small series of large scale deployments of vSAN.  Not sure of the density.  Thank you, Zach.

http://www.yellow-bricks.com/2015/07/06/virtual-san-is-breaking-down-silos-for-united-utilities/

Virtual SAN enabling PeaSoup to simplify cloud

[jwhitehv]

Let's say it's a minimum 3 node cluster, that's 216 TB before overhead.  It work load can consume that much space, but still perform well with only 6 sockets in the cluster?  It seems like with disk that dense, you would run out of compute power before you could fill it up.  Is it a few VMs that are massive?  A lot of little VMs that don't consume much CPU?  Something else I'm missing?

Not sure what you mean. Environments don't have a uniform compute to storage ratio and adjusting the core count per socket from 4-12 would be my first choice of scaling up compute requirements. 6 sockets can represent 72 cores. This client has a ton of data to store, but no analytics to run against it. One needs to weigh the cost of high-end CPUs against the cost of additional VSAN socket costs.    

[elerium]

The VSAN HCL is pretty slow to update. Since release of 6.0 i don't think any NVME cards or SATA disks have made it into HCL yet. Of all the parts on the HCL, raid controller is most important followed by SSD. I'm not sure how important magnetic disks are to the HCL so far. The generalized thing I keep reading is that if you don't use enterprise/quality parts or try to cheap out with consumer level parts, things will suck and data will be lost. In particular, SSD drives will need power loss protection capability which isn't found on many consumer level drives.

I'm willing to take some risks since my cluster is used for development purposes, so I've built my VSAN6 cluster with magnetic drives and SSD not on HCL. I'm using WD RE4 4TB SAS, and Intel P3700 1.6TB for SSD on Dell R730xd which has 16 3.5" bays. I end up with 56TB raw or 28TB usable (ftt=1) storage for 2U of rackspace. Seeing that other users have reported great success with the P3700 and both Intel/VMWare have released a performance whitepaper with the P3700 in use, I decided to go with this SSD. The only grief I've had is actually with the Dell H730 raid controller and that was when it was on the HCL firmware version and on HBA mode (both recommended on HCL). I told the VMWare support engineer that I flashed to a newer non-HCL firmware from Dell to fix PSOD/controller resets and change from HBA to RAID0 and it's been golden since. Support didn't give me any kind of grief at all for not following HCL.

In regards to VMWare not supporting you when you aren't using HCL parts, I have not found this to be the case. I had 2 separate tickets open, support checked if everything was on HCL and warned me that data loss could occur from not following HCL recommendations and asked if I understood. Then proceeded to assist with my tickets.

In regards to licensing, I don't like how nodes that don't contribute to storage but are in the same cluster still require VSAN licensing, otherwise even with socket license pricing, it's an amazing performance for value product.

[jwhitehv]

In regards to VMWare not supporting you when you aren't using HCL parts, I have not found this to be the case. I had 2 separate tickets open, support checked if everything was on HCL and warned me that data loss could occur from not following HCL recommendations and asked if I understood. Then proceeded to assist with my tickets.

That's helpful to know from a customer point of view. I'm approaching from a consultant's point of view, following the various design whitepapers from VMware which all hammer the HCL. On a conceptual basis, I can understand that the underlying point might be to use enterprise grade SSDs, controllers with stable drivers and a certain minimum queue depth, etc. But that's not what's happening. Effectively, the VSAN HCL is a list of parts that partners and consultants are limited to recommending from. A customer might make a decision to allow a deviation from it, but I can't imagine making a recommendation of parts not on the HCL, knowing that VMware is only going to give "best effort" support.

[elerium]

I absolutely agree, if I were consulting for customers, I'd stick to VSAN HCL only as that is stressed very much by VMWare as best practice there is risk from deviating from it. The current HCL does greatly limit in terms of price/capacity (no large SATA) and performance (no NVME drives).

It does look like they are slowly updating HCL, the WD RE4 4TB SAS was originally not in HCL when I implemented, I checked today and it's been qualified! :smileygrin:

[jwhitehv]

There are some 6TB and 4TB SAS drives on the HCL:

HGST drives and only HP as an OEM. So if building on anything other than HP, users can't get OEM support on their drives.

Better, as users can get 4TB drives from four OEMs (I'm not counting IBM since the sale of System X to Lenovo). Cisco is noticeably absent from both lists.

[jonretting]

This is kinda nit picky. You would have the only listed options of HGST or OEM HP. That isn't to say your cluster wouldn't be supported by vmware, if you went with the Dell edition. I think the issue comes down to a frequent mantra of limited HCL testing capabilities, or something to that tune. Granted the discussion would no longer be technical, but might it be wiser a subject. As previous users mention the lack of NVME PCIe and many other things. Is the HCL a lofty goal, but under serviced? Thinking aloud. -Jon

[jwhitehv]

This is kinda nit picky. You would have the only listed options of HGST or OEM HP. That isn't to say your cluster wouldn't be supported by vmware, if you went with the Dell edition. I think the issue comes down to a frequent mantra of limited HCL testing capabilities, or something to that tune.

Wait, what? You're suggesting that VMware support is obligated to fully support systems not on the HCL?

[jonretting]

No -- I'm suggesting they are not necessarily mutually inclusive. In my opinion. Moreover that the HCL is under serviced. Thanks, -Jon

[jwhitehv]

No -- I'm suggesting they are not necessarily mutually inclusive. In my opinion. Moreover that the HCL is under serviced.

Well, I agree that product seems to be slow getting to the HCL. But again, from a consultant or partner point of view, it's difficult to recommend product not on the HCL hoping to get support instead of with product on it, knowing the customer will get support.

[David Boone]

Not sure what the discrepancy is.  I'm seeing both high capacity drives and PCI-e SSDs for 6.0.  The Fusion-IOs show up under SanDisk.

[Bleeder]

Not sure what the discrepancy is.  I'm seeing both high capacity drives and PCI-e SSDs for 6.0.  The Fusion-IOs show up under SanDisk.

Hmm, did those get HCL'd for 6.0 recently?  I remember checking the SX300-1600 earlier in the month, and it was only HCL'd for 5.5.

[jwhitehv]

Not sure what the discrepancy is.  I'm seeing both high capacity drives and PCI-e SSDs for 6.0.  The Fusion-IOs show up under SanDisk.

Yes, there are now PCIe drives certified for VSAN 6! These have literally appeared today! The Intel 3700 family on the scaling guide is still not there though.

    

Model	Device Type	Part Number	Capacity
Fusion ioMemory PX600-1000	PCI-E	SDFACAMOP-1T00-SF1	1000
Fusion ioMemory PX600-1300	PCI-E	SDFACAMOP-1T30-SF1	1300
Fusion ioMemory PX600-2600	PCI-E	SDFACAMOP-2T60-SF1	2600
Fusion ioMemory PX600-5200	PCI-E	SDFACCMOP-5T20-SF1	5200
Fusion ioMemory SX300-1300	PCI-E	SDFACAMOS-1T30-SF1	1250
Fusion ioMemory SX300-1600	PCI-E	SDFACAMOS-1T60-SF1	1600
Fusion ioMemory SX300-3200	PCI-E	SDFACAMOS-3T20-SF1	3200
Fusion ioMemory SX300-6400	PCI-E	SDFACAMOS-6T40-SF1	6400

[Bleeder]

Well, better late than never?

I'm a bit disappointed that there isn't a better way to be notified of updates to the VSAN HCL.  It looks like I'll have to check for updates manually.  I had been relying on RSS updates to the Virtual Blocks blog.

[jonretting]

Yup checking sites for updates is annoying, and even worse when they change something element/structurally. Typically if i have to check something on the web multiple times in a couple days, it is worth automating the process with a little bit shell/curl/smtp/git/cron. That all goes to shit however when i get an email containing "<div clas"  differs from the previous attempt. It seems the problem is lack of any standardization across any company/downloads. Typically you have to fumble along their ill conceived frontends, and gets even more aggravating when you have to asynchronously post form data. For the first time this year google api's was unreachable in SOCAL, even workarounds were hard due to SSL. A lot more time is usually spent just locating the item you know you need, have already worked out the fix, but for some blast-famous reason downloads.company.com is down. I would hate to be the sysadmin taking down the public downloads, since it's something that already eats away at my soul. Hehe Cheers, -Jon

[jwhitehv]

I've been writing up my experiences in this blog series: High Capacity VSAN Nodes