Need help setting up VSAN. Zero capacity vsanDatastore.

[dutsnekcirf]

I'm trying to setup a new datacenter complete with a vSAN setup. 

I have three hosts.  Each one has one 120GB SSD (intended to be used for the cache tier) and one 500GB SSD (intended to be used for the capacity tier).  Each host has 8 copper 1Gbps Network Interfaces. 

I've installed ESXI server 6.5 on each host onto the 500GB SSD.  ESXI appears to take the entire hard drive and create a small system partition for the OS and then uses the remaining free space for a regular datastore. 

Next I installed vCenter server onto one of the hosts and am now accessing my environtment through the vSphere Web Client. 

Next I created a Data Center and a Cluster.  I've added all three of my hosts to the cluster.  Next I added a vmkernel adapter to each host and enabled vSAN and vMotion traffic, IP'd them, and connected them together so they can talk over the vSAN network. 

Finally I selected my cluster and turned on Virtual SAN.  Initially I had it in Automatic mode and it created a vsanDatastore on each host with 0b of capacity.  So I switched it to manual but I can't get it to see my 500GB SSDs to be used as my capacity tier.  It only shows my 120GB SSDs to be used for the caching tier. My 500GB SSDs have datastores that fill up the remaining free space on the drive.  

In an attempt to free up space to be used for the capacity tier I deleted the datastores that were initially created when I installed ESXI server and then recreated them in a smaller size (26GBs).  But even with unallocated space on my 500GB SSDs, Virtual SAN doesn't seem to see the drives to be used for capacity in the vSAN Datastore. 

Is it not possible to use the 500GB SSDs for both the ESXI system partition for each host and remaining space to be used in the vSAN Datastore? 

Should I actually have three hard drives installed in each host? one for the ESXI OS, one for Caching, and one for Capacity? 

Is the fact that they're all SSDs an issue? I've read that this could be considered an "all flash" configuration and if so should I be using 10Gbps NICs?  Since I don't have any 10Gbps NICs (nor do I have the network infrastructure to support them) do I need to replace the 500GB SSDs with magnetic spinning HDDs instead?  I'm more willing to do that than install 10Gbps NICs. 

[TheBobkin]

Unfortunately vSAN requires unformatted drives to be used for vSAN (either for cache-tier or capacity-tier).

"Problem

When you attempt to configure either Virtual SAN or virtual flash resource, a local SSD disk does not appear on the list of disks to be used.

Cause

This problem might occur when a local SSD intended for use with either feature has been already formatted with VMFS. Neither Virtual SAN nor virtual flash can share the SSD disk with VMFS or any other file system."

https://pubs.vmware.com/vsphere-55/index.jsp?topic=%2Fcom.vmware.vsphere.troubleshooting.doc%2FGUID-A17F5D39-C1F3-47B1-9645-5C247412F370.html

(5.5 Doc but the it is the same in 6..0 & 6.5)

You can verify that there is a partition on the device by connecting to the host via SSH (with PuTTy or similar):

# cd/dev/disks

# ls -lah

You will see drives with partitions like this:

naa.6000eb603c54216e7c613ac3a221ee2c

naa.6000eb603c54216e7c613ac3a221ee2c:1   <<<partition

vSAN-formatted disks will have 2 partitions.

So yes you will require another location for ESXi installation, Do your servers have internal SD card slots that could be used for this function?

Booting over network-attached storage is another option.

Or as you said, an additional smaller driver attached locally.

[dutsnekcirf]

So I went and bought a 64GB thumbdrive and reinstalled ESXI to the thumbdrive.  I noticed that it did not create a datastore on the thumb drive into which I can install the vCenter Server Appliance.  It also won't allow me to manually create a datastore on the thumbdrive.  Is it not possible to create a datastore on a thumb drive?  Can I fool ESXI into thinking that the thumbdrive is in fact "persistent" storage?  The thumbdrive would actually be pretty persistent since I plugged it into a USB port inside the chassis of the host.  No one will pull this thing out unless they crack open the machine.

[admin]

you cant make vmfs datastore on usb stick where esxi installed

thats why SATAdom is recommended.

if you have additional usb stick. yes you can

http://www.virten.net/2015/10/usb-devices-as-vmfs-datastore-in-vsphere-esxi-6-0/

[TheBobkin]

I wouldn't advise installing or running a vCSA off a thumb-drive as these type of devices are generally not intended or capable of any reasonable amount of IOPS, and as such I don't think your vCSA would function very well nor be reliable.

The reason why install of an ESXi host on a thumbdrive will work (while not recommended nor necessarily durable) is that this data is mostly only accessed at boot, vCSA is a different beast entirely and should be treated more like a VM.

Why not put vCSA on the vsandatastore? (after you have formatted the capacity drives and added them to the disk-groups)

If you have some SAN-storage attached and available you could install the  vCSA there also, this can be of benefit if there are issues with the vSAN cluster (as if the vCSA resides on vsandatastore and there is a problem with vSAN then you may not be able to use vSphere Client (to vCenter) or the Web Client).

Bob