vSAN Expert Challenge # 2 - Update 1

[Pkiertzner]

Dear vSAN Expert

How to?

vSAN cluster failure. vsanDatastore recreated. All vmnamespaces gone. Only vmdk objects left, but unable to browse for them.

They all list as healty with "esxcli vsan debug vmdk list" and from RVC /localhost/Datacenter/computers> vsan.obj_status_report 0 -t they are also listed but as "Unassociated objects"

Path i vmdk object changed to new vsanDatastore uuid with "/usr/lib/vmware/osfs/bin/objtool setAttr -d /vmfs/volume/vsan...../ -u vmdk object uuid. But still unable to browse for the vmdk file.

That got me to think. What does  "/usr/lib/vmware/osfs/bin/objtool open" ? Can I open the vmdk object and then copy out the vmdk file and remount the vmdk file to a newly created VM?

Are there another tool to extract the vmdk from the vmdk object?

Can I recreate the lost vmnamespace with a specific uuid to match the group uuid from the vmdk object?

Do you accept the challenge?

Update 1:

With help from TheBobkin I managede to recreate/rejoin the original vSAN cluster. But the vmdk objects/files does still not show in the vsanDatastore..

[johandijkstra]

Difficult...

Sorry I really want to help, but this seems difficult.

When you try to explore the vSAN Datastore with WinSCP or FastSCP, are you able to see all the VMDK's?

[Pkiertzner]

Hi

I can't list from esx. I can't browse from vCenter. I can't from WinSCP etc.

I think the problem is because of the missing vmnamespace object. The vmnamespace object uuid is listed as group uuid and datapath in the vmdk object as you can below in red. I can change the datapath but that does not help. I haven't found a way to change the group uuid of the vmdk object or create a new vmnamespace with a specific uuid.

From RVC.

+-----------------------------------------+---------+---------------------------+

| VM/Object                               | objects | num healthy / total comps |

+-----------------------------------------+---------+---------------------------+

| Unassociated objects                    |         |                           |

|    4a59ad58-8859-bd11-31cc-00249b1aced2 |         | 3/3                       |

|    41d7ac58-5aa6-0413-c7fb-00249b1aced2 |         | 3/3                       |

|    4696a557-0611-0028-fd8a-b8aeedec579d |         | 1/1                       |

|    8dcac257-fcf9-5429-6bc5-b8aeedec5878 |         | 3/3                       |

|    fc111759-04cc-9c61-ba99-00249b1aced1 |         | 3/3                       |

|    bdfd5758-f60c-3c7e-3ddf-b8aeedec5878 |         | 3/3                       |

|    218b6658-52bb-1f9d-4b2e-b8aeedec43bf |         | 3/3                       |

|    fd111759-a22e-43a9-e637-00249b1aced1 |         | 3/3                       |

|   f0706b58-3898-8ac9-0130-00249b1aced0 |         | 3/3                       |

|    e4611859-e2de-38cd-8c58-00249b1aced2 |         | 3/3                       |

|    73927658-f45f-f7d6-de14-00249b1aced0 |         | 3/3                       |

+-----------------------------------------+---------+---------------------------+

All vdisk / vmdk object are healthy but listed as Unassociated objects.

One specific object. All components are healthy:

/localhost/Datacenter/computers/K-Street/hosts> vsan.object_info 0 f0706b58-3898-8ac9-0130-00249b1aced0

DOM Object: f0706b58-3898-8ac9-0130-00249b1aced0 (v5, owner: unknown, proxy owner: None, policy: cacheReservation = 0, hostFailuresToTolerate = 1, spbmProfilt Storage Policy, SCSN = 535, forceProvisioning = 0, spbmProfileGenerationNumber = 4, proportionalCapacity = 0, CSN = 528, objectVersion = 5, stripeWidth = 1

  RAID_1

    Component: cc071059-e2cd-fed3-82e1-b8aeedec43bf (state: ACTIVE (5), host: 57a5723b-0faa-b572-7bf7-b8aeedec43bf, md: 5287a626-2038-acdc-0314-ed299e7521ef,

                                                     votes: 1, usage: 19.1 GB, proxy component: false)

    Component: 615b1859-c27f-2ea3-0a9f-00249b1aced0 (state: ACTIVE (5), host: 192.168.254.203, md: t10.ATA_____Samsung_SSD_850_EVO_250GB_______________S2R6NXH536089Z_____,

                                                     votes: 1, usage: 19.1 GB, proxy component: false)

  Witness: d9631859-7e2c-e993-725a-00249b1aced0 (state: ACTIVE (5), host: 57a5701e-edc5-0732-3db1-b8aeedec579d, md: 52fca517-c33b-af7b-babb-6ce82bea783b, ssd

                                                 votes: 1, usage: 0.0 GB, proxy component: false)

  Extended attributes:

    Address space: 42949672960B (40.00 GB)

    Object class: vdisk

    Object path: /vmfs/volumes/vsan:522c45550f08253c-da575dbd0b111b51/ec706b58-98e7-a27e-2079-00249b1aced0/DC02.vmdk

    Object capabilities: NONE

[TheBobkin]

Hello,

You can create a namespace Object using osfs-mkdir but this will likely be easier:

Try this with a least-important VM that you know the disk sizes for (and which disk is which).

Make a dummy VM with the same number of disks (of the EXACT same size) as the original disks.

Get the UUID of the namespace of this dummy VM.

Edit the Directory path of the lost VMs vmdk Objects so that it uses this UUID using objtool setAttr e.g. /vmfs/volumes/vsandatastoreUUID/DummyVMNamespaceUUID/DummyVM.vmdk

Edit the Dummy VM vmdk descriptors using vi, replace the Extent Description Disk Object UUID vsan://xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx with the Lost VMs disk Object UUID.

Bob

-o- If you found this comment useful please click the 'Helpful' button and/or select as 'Answer' if you consider it so, please ask follow-up questions if you have any -o-

[Pkiertzner]

Hi Bob

THANKS. You are a true vSAN Expert.

If you ever come to Denmark, please let me know. Have a fantastic weekend and once again THANKS.