How do I force a mount of a VMFS volume ?

[continuum]

I have a Raid 10 volume with a 700 Gb vmfs-volume and cant mount it with Viclient.

How can I mount it from cli ?

[vCloud9]

Here is link to the community thread http://communities.vmware.com/message/1838111#1838111 which discusses how to force mount the volume through PowerCLI. Hope this is what you are looking for.

-vCloud9

[continuum]

If I read that script correctly it either needs  a VMFS-uuid or a VMFS´-label.

I have none of that
The only way I can address that volume is by using either
/dev/disks/naa.6001e4f0107c5800178e9061850eeba6:1
or vml.02000000006001e4f0107c5800178e9061850eeba6504552432035:1

[vCloud9]

Did you tried getting the UUID by running the following command from the ESXi console ls -lah /vmfs/volumes

This article talks indepth about managing VMFS volumes http://www.harleystagner.com/vcdx/vcdx-enterprise-admin-notes-manage-vmfs-filesystems-using-command-line-tools.php

-vCloud9

[continuum]

yes - unless there is some super-hidden command to read the UUID or label of the vmfs-volume I have to assume that ESXi trashed that info.

I am ready to inject that details with dd or a hammer if I only knew how

[rickardnobel]

Ulli Hankeln wrote:

I have a Raid 10 volume with a 700 Gb vmfs-volume and cant mount it with Viclient.

For me the "Add storage" means = Create New VMFS on empty disk and not add an already existing VMFS partition. You have of course done a Rescan to see if the VMFS partition is not already visible?

[continuum]

Hi Rick
I only posted that screenshot to show that ESXi "somehow" accepts the volume as VMFS.
But that is about as far as I can get.

I dumped the first Gb into a textfile and read the strings - so I know that the info I would expect in the .*.sf files in the root of a VMFS volume seems to be there.
partedUtil also says that this is a VMFS volume.

I guess what I need now is a way to write a UUID or VMFS-label to the volume

[a_p_]

Did you already run

esxcli storage vmfs snapshot list

see http://kb.vmware.com/kb/1011387

You may also run a rescan and check for errors/warnings in the vmkernel log.

André

[continuum]

yes - that command does not list anything

checking the vmkernel log also gives no hints

[a_p_]

One more other thought on this. Is there a chance that the datastore name of the unmounted VMFS volume is the same as one of the already mounted datstores (e.g.  a default name)? Maybe it's possible to temporarily rename the existing datastores to "free up" the names!?

André

[continuum]

let me sum up what I tried so fars

I created a dump of the whole LUN a la dd if=/dev/disks/blabla of=/vmfs/volumes/free-datastore/diskdump-flat.vmdk

Then I copied that flat.vmdk to another ESXi , created a descriptorfile for it and use it inside a VM
Now I created a snapshot and boot the VM with a ESXi stateless LiveCD.

That means I can rule out all misconfigurations a la dublicate VMFS-uuids or VMFS-labels.

With that scenario I then tried
- rebuild the GPT-partitiontable assuming a start offset of 2048
- rebuild the GPT-partitiontable assuming a start offset of 128
- rebuild a MBR-partitiontable assuming a start offset of 2048

- read the volume with latest vmfs-tools version 2.5 - compiled from source so that I get VMFS 5 support - result is "invalid magic number"

- I created a new vmfs-volume using exactly the same disklayout and dumped the first 1 MB of the working volume to the bad volume

I tried all tricks that would apply if the volumes was treated as a snapshot - but those tricks dont work
ESXi does not detect it as a snapshot

I was able to extract vmdk descriptorfiles and vmx-files and vmware.logs and other stuff that you would expect to be inside the VMFS- headers
I used scalpel to carve out user data from inside the vmdks

I asked for help at the homepage of the vmfs-tools - see http://glandium.org/blog/?p=2539&cpage=1#comment-149313

If you have further suggestions even bizar ones like using - please suggest them

[Mauro Bonder]

Hi,

I don´t know if this will works for you, but....... (using esxcfg-volume -m

Command line

You need to use the esxcfg-volume command. It can be used in this way:

• Execute this command to list the volumes that are detected as snapshots/replicas:
  
  # esxcfg-volume -l
  
  The output appears similar to:
  
  VMFS3 UUID/label: 49d22e2e-996a0dea-b555-001f2960aed8/VMFS_1
  Can mount: Yes
  Can resignature: Yes
  Extent name: naa.60a98000503349394f3450667a744245:1 range: 0 - 97023 (MB)
  
  Here the Datastore UUID is 49d22e2e-996a0dea-b555-001f2960aed8 and its last label is VMFS_1.
  
  
  
• To mount the volume without performing a resignaturing of that volume (this volume is unmounted when the ESX host is rebooted), run this command:
  
  # esxcfg-volume -m <VMFS UUID|label>
  
  For example:
  
  # esxcfg-volume -m "VMFS_1"
  # esxcfg-volume -m "49d22e2e-996a0dea-b555-001f2960aed8"
  
  
  
• To mount the volume without performing a resignaturing of that volume (this volume is mounted when the ESX host is rebooted), run this command:
  
  # esxcfg-volume -M <VMFS UUID|label>
  
  For example:
  
  # esxcfg-volume -M "VMFS_1"
  # esxcfg-volume -M "49d22e2e-996a0dea-b555-001f2960aed8"
  

[RobertNikogosia]

Thank you. This worked for me!

[lbomb]

This saved my rear end thank you very much!!

[a_p_]

Well, I can't promise you anything, but if you can provide the first as well as the the last 34 data blocks (512 bytes/block) of the disk, I would be interested in taking a look at the partition table entires.

André

[continuum]

Andre - unfortunately the VMFS-volume is onlz 700Gb while the whole RAID-volume itself is 930 or something like that.

# partedUtil getptbl /dev/disks/naa.6001e4f0107c5800178e9061850eeba6
gpt
121795 255 63 1956642816
1 2048 1468003634 AA31E02A400F11DB9590000C2911D1B8 vmfs 0

thats the output of partedUtil against the original RAID

which parts do you want to see ?

[a_p_]

Thinking of it and looking at the output of the partedUtil command, I think it's more an issue inside the VMFS partition than with the GPT partition table itself. Now since you were asking for even bizarre suggestions, I do have one. With the snapshot you have this should be easy to revert if it does not help.

What happens if you use the pertedUtil command to resize the partition and then run the vmkfstools command to grow the VMFS datastore?

André

[continuum]

good idea - it will probably fail but then the error-messages of vmkfstools may be useful.
I will try that righ now

[continuum]

Hi Andre

~ # vmkfstools --growfs /vmfs/devices/disks/vml.0000000000766d686261313a303a30:1 /vmfs/devices/disks/vml.0000000000766d686261313a303a30:1
Failed to get info from head device path /dev/disks/mpx.vmhba1:C0:T0:L0:1.
Error: No such file or directory

that confirms my theory : the partitiontable is fine - the.sf files are fine - just something in the begining of the VMFS-volume is either wiped or seriously corrupted

by the way - I just tried the Diskinternals VMFS Recovery against this raidset and it does not even try to recover or scan on the VMFS-level.
It directly goes into recover-lost-files mode

... the search goes on :smileywink:

[continuum]

Made some progress - learned how to find out the original Volume-label and the original Volume-uuid with a hexeditor
and the commands to use to mount or unmount them.

Dead Volume-label: "OS_Datastore1"
Dead Volume-uuid: "500021fb-fd996b18-cfb0-000c29f74932"

for comaprison
Working Volume-label: "small (1)"
Working Volume-uuid: "4f2198d2-9842a778-0fd7-00237dfdbdaa"

using the label: works and results are visible immediatly
~ # esxcli storage filesystem unmount --volume-label="small (1)"
~ # esxcli storage filesystem mount --volume-label="small (1)"

Using the uuid : works and results are visible immediatly
~ # esxcli storage filesystem unmount --volume-uuid="4f2198d2-9842a778-0fd7-00237dfdbdaa"
~ # esxcli storage filesystem mount --volume-uuid="4f2198d2-9842a778-0fd7-00237dfdbdaa"

Now the same commands against the dead volume :
~ # esxcli storage filesystem mount --volume-label="OS_Datastore1"
No volume with label 'OS_Datastore1' was found
~ # esxcli storage filesystem mount --volume-uuid="500021fb-fd996b18-cfb0-000c29f74932"
No volume with uuid '500021fb-fd996b18-cfb0-000c29f74932' was found

[mertez]

continuum: Did you maybe find a solution for mounting a vmfs volume and datastores?

[continuum]

Hi
I have not found a single good solution for the problem : "ESXi does not mount an existing datastore"

In my view the mentioned problem comes in several flavours / variants which depends on the level of corruption.

1. the volume is healthy but smells suspicious: use an ESXi that has never seen datastores before (ESXi-LiveCD or newly installed USB-stick) - reading the datastore once with unspoiled ESXi may be enough to fix the "bad smell"

2. the volume is healthy but it is detected as snapshot-LUN: either force mount or resignature

3. the partitiontable is bad: fix the table with partedUtil

4. the volume is healthy but ESXi decides not to display the content: sometimes using a standalone ESXi not connected to a vCenter works after several reboots and setting new signatures or reformatting.
- if not - or if you are in a hurry use vmfs-fuse from linux

5. the volume has healthy data but has  slightly damaged VMFS-metadata: use vmfs-fuse - ESXi itself no longer helps

6. the volume has healthy data but severely damaged VMFS-metadata and vmfs-fuse can not read it with "invalid magic number" or "missing magic number" : use UFS-explorer

7. the volume has healthy data but the VMFS-metadata is partly or completely wiped blank: call me - I can extract thick provisioned vmdks in a completely manual way

8. VMFS-metadata AND the actual data is partly corrupted like typical after a failed Raid5 rebuild: call Ontrack if the data is worth it

Whatever you do - watch out: ESXi can become its own worst enemy - it may decide to use a disk that looks empty for scratch space - this can seriously damage a VMFS volume before you even get the chance to repair it

Back to the question: how to mount a VMFS-volume from cli ?

To my best knowledge only case 2 and 3 are documented and the documented procedures work as expected.
Case 1 and 4 are as far as I know not documented in any way but sometimes the problems can be fixed with esxi-commandline tools if you are stubborn and use vooddoo

Fixing slightly corrupted VMFS-metadata is as far as I know not documented at all