Cannot use VSAN at all, can't even create a folder

[dbutch1976]

Hello,

I am running a lab with a 3 node vSAN environment, vsphere8 across the board with the most recent builds.

I recently recreated my VSAN however I am unable to do anything at all with it, cannot create VMs on it, can't migrate anything onto it, I can't even create a new folder on it. I get the following error:

Cannot complete file creation operation. There are currently 1 usable fault domains. The operation requires 2 more usable fault domains. Failed to create object.

I'm just learning about VSAN, but from what I've read I can't see anything wrong with my setup regarding fault domains, although I have played with creating additional domains and it doesn't seem to make a difference. Since I can't do literally anything using this storage I think this error may be a red herring. My fault domain config is below, is there anything wrong?

 

 

[TheBobkin]

, by default (e.g. without configuring Fault Domains) each node is a Fault Domain - there is no need to configure this unless it is a stretched cluster or perhaps some form of rack-awareness configuration.

 

You are likely getting this error message because either 1. The nodes cannot communicate with each other or 2. You only have Disk-Group(s) configured on 1 node.

 

Check Skyline Health, it will tell you if 1. is the case.

[dbutch1976]

You are correct:

 

But I don't understand what the communication issue could be. All the IP addresses are correct and responding on the network, everything is pingable, I can't really figure out what to check. Here is an example of the config on one of the hosts:

 

Since it's a lab I'm not using dedicated VMkernels for the VSAN traffic, could that be part of the problem?

[TheBobkin]

, Maybe the node unicastagent lists didn't get populated due to order of how you configured things here - check via SSH on the nodes:

# esxcli vsan cluster unicastagent list

 

If this is blank or incomplete (should contain the entries of the 2 other nodes on each) then move the hosts out of cluster to DataCenter and back into the cluster to force vCenter to populate the unicast agent lists.

[TheBobkin]

 FYI - The multihoming of traffic types you mentioned - this is not advisable as thing like burst of vMotion or vSAN traffic might inundate the link causing issues for Management network but it wouldn't cause the issue you have here.

 

If it isn't a unicastagent issue and if you are checking vmkping properly then most likely it is a port issue (UDP 12321 and/or TCP 2233 specifically).

[dbutch1976]

Thanks for your help, I definitely think we're on the right track. 

From what I can tell the unicastagent list appears to be blank. I remove one of the hosts from the cluster and moved it back in, I could see tasks being kicked off to update VSAN configuration which completed, however aftere placing the host back into the cluster and re-running esxcli vsan cluster unicastagent list it still appears to be blank. vmkping looks fine, but unless my commands to check the ports are wrong then they either aren't responding or I'm just not doing it correctly, I'm using:
nc -z 192.168.0.154 2233
nc -z 192.168.0.154 12321

Any suggestions on what to check next?

 

 

 

[TheBobkin]

 

Was cluster shutdown wizard run at some point here?

If nodes are not getting unicastagent entries pushed to them then maybe they are set to ignore these updates - this can be checked with the below and both of these set to default values :

# esxcfg-advcfg -g /VSAN/DOMPauseAllCCPs
# esxcfg-advcfg -g /VSAN/IgnoreClusterMemberListUpdates

# esxcfg-advcfg -s 0 /VSAN/DOMPauseAllCCPs
# esxcfg-advcfg -s 0 /VSAN/IgnoreClusterMemberListUpdates

 

After setting these to 0, try move out of cluster then back in again, these lists can also be manually populated but getting vCenter to do this is preferable.

[dbutch1976]

In answer to your question, yes, at some point in the troubleshooting I believe I have attempted to use the cluster shutdown wizard which (if memory serves) did not complete successfully. Since then I have tried deleting and recreating the cluster.

I ran :

# esxcfg-advcfg -g /VSAN/DOMPauseAllCCPs
# esxcfg-advcfg -g /VSAN/IgnoreClusterMemberListUpdates

Both returned a result of 1, so I ran the other two commands you referenced:

# esxcfg-advcfg -s 0 /VSAN/DOMPauseAllCCPs
# esxcfg-advcfg -s 0 /VSAN/IgnoreClusterMemberListUpdates

Placed each host in maintenance mode, moved it out of the cluster, moved it back in and exited maintenance mode. I completed these steps one host at a time. 

Now when I run esxcli vsan cluster unicastagent list The list is now populated, I can create folders, and I am able to migrate workloads onto the vSAN! Huge thanks TheBobkin!

[TheBobkin]

Happy to help !

[dbutch1976]

Hi TheBobkin,

Once again I'm in the same situation with my VSAN where I cannot even create a folder. It's a bit different this time, when I run the commands below they both return 0 (previously they returned 1)

Just like previously I placed each host in maintenance, moved it out of the cluster, then moved it back in. Unfortunately this time around it didn't fix the issue.

This issue was caused when I reinstalled ESXi on all three nodes. During the reinstall I also erased all the cache and capacity tiers on the VSAN. Any suggestions on how to force the member list updates other than moving the hosts in and out of the cluster?

 

 

 

[TheBobkin]

 What was the order of enabling vSAN on the cluster, moving the nodes into the cluster, configuring the vmk for vsan-traffic?

 

I have found oftentimes that if you do it in the order of the above 3 steps that it doesn't populate the unicastagent lists on the nodes, better to configure the vsan-vmk first, before moving the nodes into the cluster.

 

Your unicastagents are indeed blank here, confirm vSAN was enabled on the nodes (esxcli vsan cluster get) and you can generally get these to populate by untagging vsan-traffic and re-tagging it on a node, that or move each node to Datacenter level and back into cluster (is there a particular reason you don't want to/can't do that?).

[dbutch1976]

Yep, adding a vmk adapter with VSAN tagged appears to have been the fix. Thanks again!

[dbutch1976]

Sorry to keep resurrecting this old thread, but I think I've experienced just about every problem under the sun in my VSAN and you've been on point with every issue, hoping I can get your help one more time.

I had an issue on the iSCSI SAN in my lab (nothing to do with VSAN). This had a knock on affect on my VSAN because all three nodes were boot from SAN and they were using the iSCSI SAN. So I didn't see any option other than to bring all three nodes down at the same time, (this is a 3-node vSAN cluster). Essentially all three nodes, the vCenter, everything in the environment had to go down.

Now that the initial issue has been resolved I have brought everything back online. The VSAN is visible, (I've run esxcli vsan cluster unicastagent list on each host). Everything looks good (example below).

 

 

As you can see all VMs are inaccessible. Hosts are out of maintenance mode, I've rebooted each one, one at a time, dozens of rescans. Is VSAN dependent on the vCenter? If that's the case I may have cut off the branch I was standing on. Any suggestions?

 

[dbutch1976]

Fixed it myself this time around.... I'm starting to get good with this VSAN troubleshooting!

Found a helpful article here:
vSAN recovery after vCenter failure - VMware Technology Network VMTN

After running esxcli vsan cluster get I noticed that 2/3 hosts were showing maintenance mode here even though they weren't showing that they were in maintenance mode directly on the host. This makes sense because 2/3 hosts were not going into maintenance mode properly when I attempted to shut them down and in the end I had no choice but to power them off. Looks like they went into a state where maintenance mode was on for VSAN but didn't show on anywhere else. The fix, simply place the host in maintenance mode and take it out again, boom everything fixed.