Suspected Stale Skyline Health - vSAN Object Health - Inaccessible

[Edd Watton]

Following a disk group re-creation (needed after enabling Support for large cache tier capacity (>600GB, up to 1.6TB) in vSAN 8.0), we have an alert in vSAN Skyline Health for "vSAN Object Health" stating there is one object that is Inaccessible. However, when you click VIEW DETAILS and it takes you to the Virtual Object page, all objects are healthy.

I suspect the original error this has something to do with vSAN File Services. VSFS flagged up in the health check and it redeployed a new VM onto the host. (vSAN File services uses a policy that stores its VM data on a single host)

I have run vsan.check_state -r . from a rvc and it does not return any Inaccessible objects.

Step 1: Check for inaccessible vSAN objects
Step 1b: Check for inaccessible vSAN objects, again

As both vsan.check_state and the Virtual Object page do not show any inaccessible objects, I think the Skyline Health check is somehow processing stale data.

Does anyone have any ideas how to troubleshoot this?

Many Thanks

[Edd Watton]

I have found the Inaccessible object but not worked out how to remove it.

Using esxcli vsan debug object list --health=inaccessible shows one object.

This object as suspected belonged to a vSAN File Services VM.

This is confirmed as the spbmProfileNmae is FSVM_Profile_DO_NOT_MODIFY and the policy shows as hostFailuresToTolerate of 0. It is also worth noting the Size and Used are both 0.00GB

I have tried /usr/lib/vmware/osfs/bin/objtool delete -u [object UUID from the above command] -f but it returns, 'No such file or directory' 

So at this point, I can see why the vSAN health checks are flagging one object as inaccessible but I cannot remove the reference to it. Any ideas?

[Edd Watton]

For anyone else who stubbles across this post. I managed to remove this record using the following command from an ESXi host

cmmds-tool delete -u  [object UUID from 'esxcli vsan debug object list --health=inaccessible']

Prior to removing it, I used the command below to validate I had the correct object

cmmds-tool find -t DOM_OBJECT -u  [object UUID from 'esxcli vsan debug object list --health=inaccessible']

[TheBobkin]

Hello Edd

"I have tried objtool delete -u [object UUID from the above command] -f but it returns, 'No such file or directory' "
If that is the exact command you used then that isn't expected to work as that is not the full dir path for objtool (it is /usr/lib/vmware/osfs/bin/objtool).

"cmmds-tool delete -u  [object UUID from 'esxcli vsan debug object list --health=inaccessible']"
I really wouldn't advise ever deleting CMMDS entries unless there is literally no other choice (only scenario I can recall is if there is an object with partial delete state and no longer has a DOM Owner so no-one can process objtool delete), objtool delete would likely have worked fine here, that being said, always be super-careful and 100% sure when using that either as there is no undelete option.

[Edd Watton]

Hi @TheBobkin

For clarity I did abbreviate objtool and not provide the full path. I was using "/usr/lib/vmware/osfs/bin/cmmds-tool delete -u  [object UUID from 'esxcli vsan debug object list --health=inaccessible']". I have updated my previous posts to reflect the full path.

Thanks for highlighting the danger of cmmds-tool delete. I believe we were in the state you describe.

Thanks