Backup & Recovery

 View Only
  • 1.  Datastore is missing after reboot

    Posted Oct 06, 2020 01:02 PM

    Hi all,

    I built a new ESXi 6.5 with RAID6 local storage and configured host cache.  I migrated VMs from old hardware server to this new built. The ESXi was frozen and responded.  I rebooted the ESXi host.  Afterward, the datastore is missing.  The Host cache - datastore was full.  My vCenter is also running on this new ESXi host.

    I also attached VMFS-Header-dump file.

    Please help me to restore the VMs.  It would be sincerely appreciated.

    Missing datastore01-raid6

    vmkernel.log

    _______________

    2020-10-05T22:50:24.112Z cpu24:67260)Activating Jumpstart plugin rhttpproxy.

    2020-10-05T22:50:24.349Z cpu0:67680)Res3: 2552: datastore01-raid6: Inconsistency detected for type 5 resource file metadata signature 0, version 0

    2020-10-05T22:50:24.349Z cpu0:67680)Vol3: 3155: Failed to get object 28 type 1 uuid 5f7aec3f-bcd9772c-864b-f40270e0d568 FD 0 gen 0 :No filesystem on the device

    2020-10-05T22:50:24.349Z cpu0:67680)WARNING: Fil3: 1389: Failed to reserve volume f532 28 1 5f7aec3f bcd9772c 2f4864b 68d5e070 0 0 0 0 0 0 0

    2020-10-05T22:50:24.349Z cpu0:67680)Vol3: 3155: Failed to get object 28 type 2 uuid 5f7aec3f-bcd9772c-864b-f40270e0d568 FD 4 gen 1 :No filesystem on the device

    2020-10-05T22:50:24.354Z cpu0:67680)Res3: 2552: datastore01-raid6: Inconsistency detected for type 5 resource file metadata signature 0, version 0

    2020-10-05T22:50:24.354Z cpu0:67680)Vol3: 3155: Failed to get object 28 type 1 uuid 5f7aec3f-bcd9772c-864b-f40270e0d568 FD 0 gen 0 :No filesystem on the device

    2020-10-05T22:50:24.354Z cpu0:67680)WARNING: Fil3: 1389: Failed to reserve volume f532 28 1 5f7aec3f bcd9772c 2f4864b 68d5e070 0 0 0 0 0 0 0

    2020-10-05T22:50:24.354Z cpu0:67680)Vol3: 3155: Failed to get object 28 type 2 uuid 5f7aec3f-bcd9772c-864b-f40270e0d568 FD 4 gen 1 :No filesystem on the device

    2020-10-05T22:50:24.359Z cpu0:67680)Res3: 2552: datastore01-raid6: Inconsistency detected for type 5 resource file metadata signature 0, version 0

    2020-10-05T22:50:24.359Z cpu0:67680)Vol3: 3155: Failed to get object 28 type 1 uuid 5f7aec3f-bcd9772c-864b-f40270e0d568 FD 0 gen 0 :No filesystem on the device

    :

    [root@nyesxi65:~] esxcfg-scsidevs -m

    VmFileSystem: SlowRefresh() failed: Cannot open volume: /vmfs/volumes/5f7aec3f-bcd9772c-864b-f40270e0d568

    naa.62cea7f066bb9e00270d963cf8ff3fcf:3                           /vmfs/devices/disks/naa.62cea7f066bb9e00270d963cf8ff3fcf:3 5f7ae56d-fc1a455c-1fc1-f40270e0d568  0  datastore03-donot-used

    naa.62cea7f066bb9e00270dacb6fee7b3a8:1                           /vmfs/devices/disks/naa.62cea7f066bb9e00270dacb6fee7b3a8:1 5f7af189-6c4748c6-fcff-f40270e0d568  0  datastore02-host-cache

    naa.62cea7f066bb9e00270d965e2b5c4e51:1                           /vmfs/devices/disks/naa.62cea7f066bb9e00270d965e2b5c4e51:1 5f7aec3f-bcd9772c-864b-f40270e0d568  0  datastore01-raid6

    [root@nyesxi65:~] voma -m vmfs -d  /vmfs/devices/disks/naa.62cea7f066bb9e00270d965e2b5c4e51:1

    Checking if device is actively used by other hosts

    Scanning for VMFS-6 host activity (4096 bytes/HB, 1024 HBs).

    Running VMFS Checker version 2.1 in default mode

    Initializing LVM metadata, Basic Checks will be done

    Phase 1: Checking VMFS header and resource files

       Detected VMFS-6 file system (labeled:'datastore01-raid6') with UUID:5f7aec3f-bcd9772c-864b-f40270e0d568, Version 6:81

    ON-DISK ERROR: Unexpected 'resourcesPerCluster' 0, expected 32

    ON-DISK ERROR: Unexpected 'clustersPerGroup' 0, expected 8

    ON-DISK ERROR: Unexpected 'clusterGroupOffset' 0, expected 65536

    ON-DISK ERROR: Unexpected 'resourceSize' 0, expected 65536

    ON-DISK ERROR: Unexpected 'clusterGroupSize' 0, expected 16842752

    ON-DISK ERROR: Unexpected 'bitsPerResource' 0, expected 1

    ON-DISK ERROR: Unexpected 'version' 0, expected 1

    ON-DISK ERROR: Unexpected 'signature' 0, expected 1919315300

    ON-DISK ERROR: Unexpected 'numAffinityInfoPerRC' 0, expected 32

    ON-DISK ERROR: Unexpected 'numAffinityInfoPerRsrc' 0, expected 1

    ON-DISK ERROR: Unexpected 'clustersPerGroup' 0, expected 8

    ON-DISK ERROR: Unexpected 'clusterGroupOffset' 0, expected 65536

    ON-DISK ERROR: Unexpected 'resourceSize' 0, expected 65536

    ON-DISK ERROR: Unexpected 'clusterGroupSize' 0, expected 65536

    ON-DISK ERROR: Unexpected 'bitsPerResource' 0, expected 1

    ON-DISK ERROR: Unexpected 'version' 0, expected 1

    ON-DISK ERROR: Unexpected 'signature' 0, expected 1919315300

    ON-DISK ERROR: Unexpected 'numAffinityInfoPerRsrc' 0, expected 1

    ON-DISK ERROR: <FD c0 r4>: invalid address PB2 cow 0 cnum 0 rnum 0

    ON-DISK ERROR: Corruption too severe in resource file [SB]

             ERROR: Failed to check sbc.sf.

       VOMA failed to check device : Invalid address

    Total Errors Found:           20

       Kindly Consult VMware Support for further assistance

    Best regards to all



  • 2.  RE: Datastore is missing after reboot

    Broadcom Employee
    Posted Oct 06, 2020 03:50 PM

    Hi,

    VOMA is reporting on disk errors and it does not look good. Should consider raising a ticket with VMware so they can collect datastore dumps and attempt to fix the errors.



  • 3.  RE: Datastore is missing after reboot

    Posted Oct 06, 2020 07:57 PM

    Thank you!



  • 4.  RE: Datastore is missing after reboot

    Posted Oct 06, 2020 05:17 PM

    Hi

    this does not look good - not at all.

    It actually looks like a raid rebuild gone wrong: vmdk-descriptorfiles appear in one piece but larger files like vmx-files for example are already broken into fragments - which should not happen with a healthy volume.

    Results are quite strange - the heartbeat section (4 mbs) for example looks like it appears in one piece ....

    Apparently your vmdk files are not thin provisioned - so you have a better than zero chance to eventually carve out flat.vmdks.

    This vmdk-files still have named entries :

    itcweb01-flat.vmdk

    itcweb01_1-flat.vmdk

    nyvcsa65-flat.vmdk

    nyvcsa65_1-flat.vmdk

    nyvcsa65_2-flat.vmdk

    nyvcsa65_3-flat.vmdk

    nyvcsa65_4-flat.vmdk

    nyvcsa65_5-flat.vmdk

    nyvcsa65_6-flat.vmdk

    nyvcsa65_7-flat.vmdk

    nyvcsa65_8-flat.vmdk

    nyvcsa65_9-flat.vmdk

    nyvcsa65_10-flat.vmdk

    nyvcsa65_11-flat.vmdk

    nyvcsa65_7-flat.vmdk

    nyvcsa65_8-flat.vmdk

    nyvcsa65_9-flat.vmdk

    nyvcsa65_10-flat.vmdk

    nyvcsa65_11-flat.vmdk

    zzz_ArgosPBX-flat.vmdk

    AGMail01-flat.vmdk

    SqlServer-flat.vmdk

    SqlServer_1-flat.vmdk

    Usually I would consider those vmdks as recoverable - but I dont think that applies to this case - as even comparably small files like vmx-files are incomplete.

    I assume that your best chance is an attempt to carve out flat.vmdks

    Ulli



  • 5.  RE: Datastore is missing after reboot

    Posted Oct 06, 2020 05:36 PM

    Hi Ulli,

    Can you help to see if I can restore any VM on this?  I sent you chat via Skype.

    Thank you!