Storage oddities - Corrupted Partition Table resulting in Vanishing VMFS5 and VMFS6 Datastores - Why?

[kpcongdon]

I've had something odd occur twice now in our environment, each occurrence in a different datacenter.  I'm a bit stumped, and a little concerned that I can't determine the root cause for this.

What has happened is:

1.  Datastore on FC attached EMC VMAX250f LUN becomes unmounted from all attached hosts

2.  Datastore cannot be re-mounted

3.  Rescan of storage results in the Datastore being removed entirely from the list of Datastores.  The underlying LUN for this datastore is now listed as available for creation of a new Datastore, and vSphere does not recognize that there is any existing Datastores on the LUN.

4.  VOMA recognizes that there is a VMFS file system detected on the lun, and shows 0 errors

5.  "partedUtil getptbl" shows that there are no partitions defined on the LUN

6.  Following KB2046610, I was able to recover both Datastores by re-creating it's entry in the Partition Table, and I'm then able to remount the LUN on all the hosts

So, this has happened twice now, each in a different datacenter and thus on different storage frames.  The commonality is however, that these LUNS were both provisioned on the same model of storage frame, an EMC VMAX250F.  These frames are roughly 6 months old and are the first of this model placed into service in our datacenters.

In datacenter 1, our vsphere hosts are ESXi 6.5U1 with vCenter 6.5U1.  In datacenter 2, our hosts are ESXi 6.0d with vCenter 6.5U1.

I'm at a loss for trying to determine why this may be happening.  Is there anything configuration wise on the SAN side that would possibly explain this issue?

[Rahul Parmar]

Below are the common reasons for partition table corruptions, Ensure you check on the below parameters If it could be one of the reason:  

1)VMFS LUNs presented to non-esxi hosts like Windows OS, Solaris, and Linux hosts.

2)User error, Vmware has introduced some mechanisms to prevent such corruption, except for hardware/firmware issues

3)A Less common cause is Storage array rebuilding the RAID set after losing one of the backing disks.

[kpcongdon]

Thanks for the feedback.

VMWare support did come across these entries in the vmkernel log for the lun in question, on the day the issue was noticed.  I'm waiting for feedback from my storage admin on what the cause might be for this.  Any thoughts on these entries?

vmkernel: the error "Device expanded" has been found 152 times, latest occurrence (display 3 only):
2017-10-24T16:23:48.038Z cpu20:90123)LVM: 4014: [naa.60000970000197800628533030333238:1] Device expanded (actual size 1048575199 blocks, stored size 1048573952 blocks)
2017-10-24T16:23:48.055Z cpu20:90123)LVM: 4014: [naa.60000970000197800628533030333238:1] Device expanded (actual size 1048575199 blocks, stored size 1048573952 blocks)
2017-10-24T16:23:48.057Z cpu20:90123)LVM: 4014: [naa.60000970000197800628533030333238:1] Device expanded (actual size 1048575199 blocks, stored size 1048573952 blocks)

[kpcongdon]

I found the cause of the 'Device Expanded' events being logged.  Turns out the offset I used when recreating the partition table needed to be adjusted as per KB2046610 .  Once corrected, this error went away.