Could not power on VM : Invalid metadata - storage

[Ma5ek]

Hi,

I am new at this forum and I have problems starting VM. My configuration of esx 4.1 is :

- two host in cluster ( esx1 and esx2 ) HA

- datastore is on DELL MD storage conected via ISCSII

When I try to start VM on esx1 I get this error:

Failed to power on VM

Could not power on VM : Invalid metadata.

If I migrate host to esx2 the VM power ON without any errors. When the VM is on I can migrate host back to esx1 and it works normaly until first shutdown. Then I have to repeat the procedure.

Can you help me to resolve this problem ?

Thanks,

Alan

[prabuvmware]

Can you paste the VMkernel logs?

[a_p_]

Welcome to the Community,

please see whether http://kb.vmware.com/kb/1030719 can help.

André

[admin]

Where is the swap file for VM getting created . is it on same datastore as VM directory or another one

[Ma5ek]

Hi,

I have checked the settings and the swap files are in the same directory as the VM. I will attach log files - vmware-7 is log file from VM and syslog are logs from esx1. If this are not logs you nedd please tell me.

Thanks,

Alan

[admin]

Is this is the only VM from that datastore facing this issue? Or multiple VMs? Also try to create a file inside the VM folder and see if that allow you to create files.

If this is a Metadata corruption then you have to contact VMware Support to fix the metadata corruption.

[Ma5ek]

Hi,

This error is on any VM on host esx1. If I migrate VM to different host ( in my case esx2) the VM strats normaly. I can also create or delete files in side VM folder.

Alan

[julienvarela]

Hi,

Are you able to browse your datastore and create a file from your esx1? 

Then try this, power on your VM on esx2, and then , migrate this VM using vMotion to esx1, is this works?

Regards,

Julien.

[Ma5ek]

Hi,

Yes I can browse the store and I can create files on esx1, esx2 - localstore and on Dell MD Storage store. If I power VM on ESX2 and migrate to ESX1 then it works - until I have to restart VM on ESX1.

Regards,

Alan

[julienvarela]

Hi,

I see this on your log files

"[2013-07-17 13:43:17.723 F5670B90 info 'ha-eventmgr'] Event 2890 : At least one corrupt resource metadata region was detected on volume 4d1c7cab-1c64ceea-1298-0010188b4918 (VM_Store). Other regions of the volume might be damaged too."

Your volume is corrupted.

assuming to this, you need to delete the datastore, delete the Lun from your storage area. Then, create a new lun and re-create your datastore.

Just before, so can you move your vm on another datastore and try to make the same operation?

Regards,

Julien.

[Ma5ek]

Hi,

Thank you for your advice. I will try it tommorow, because I am at home wright now. Can you tell me or do you have any clue why is it working on esx2 but not on esx1 where the VM_store is on DELL MD 3000 storage, so both ESX1 and ESX2 share the same datastore ?

Regards

Alan

[julienvarela]

Hi,

Good point, but i haven't the logical answer :smileyhappy:.But i will take a look.

Ok no problem lets try and tell me if it is ok.

Regards,

Julien.

[Ma5ek]

Hi,

Thank you for you time. I will try tommorow and report.

Regards.

Alan

[admin]

As I said before contact VMware support if you have valid agreement. Certain Metadata corruption can be fixed by VMware. VMware help you to take the metadata dump and they will check it whether they can fix the metadata. If they are saying that not fixable then you can go ahead and format the LUN. Make sure you have valid backup of your VMs

[Ma5ek]

Hi,

I have just test and move one of VM from MD3000 storage to localstore of ESX1 and the VM turn it on normaly. I can not migrate the machine to esx2, because local store of ESX1 is not shared with ESX2. I have also noticed that ESX 4.1 in cluster is not updated ( basic instalation without updates ). So my actions will be :

1. Fully update ESX 4.1 cluster

2. If this does not work I will contact VMware support as AT10 said

3. If they can not help me then the last option format the VM store and delete LUN from MD3000 Store

But I will also need your help. How can I update cluster ? I have allready installed update manager on vsphare. I will install only updates for ESXi ( without Linux or Windows VM updates ). What is the procedure for updating ESXi 4.1 cluster ?

Regards, Alan

[julienvarela]

Hi,

You can find here a documentation for upgrade your ESXi. http://www.vmware.com/pdf/vsphere4/r41/vsp_41_upgrade_guide.pdf

But major step are :

- 1 Check if your host are compatible 4.1 using HCL. VMware Compatibility Guide: System Search

- 2 Upgrade your vCenter to vCenter 4.1 

- 3 Put one host to maintenance mode.

- 4 Upgrade this host using update manager or using update ISO.  VMware vSphere 4: Private Cloud Computing, Server and Data Center Virtualization

- 5 Repeat 3-4 for all your host.

regards,

Julien.

[Ma5ek]

Hi,

I have ESX 4.1 on both hosts ( 4.1.0., 260247 ) and vspahare client/server version 4.1.0 build 258902. I just need to instal critical patches for esx 4.1.0

Could you help me with the procedure ?

Regards

Alan

[julienvarela]

Hi,

So you need to have vCenter update manager installed.

• Then go to update manager administration and create a new baseline , type a name and select host patch :

• Select dynamic patch option and select product as needed. (Here in Severity , select critical, if you need only critical patch).

• Then leave by default and click on finish.
• After you need to attach your baseline to your cluster. So select your cluster, go to update manager tab and click on "Attach" and select your baseline previously created.-
• Right click on your cluster and click on "Scan for update".
• Select a host, put it in maintenance mode. Then right click and remediate.

Et voila :smileywink:

Regards,

Julien.

[Ma5ek]

Hi,

Thank you - you are the man :smileyhappy: .

I will report - probably I will do the work over the weekend.

Thanks again,

Regard

Alan

[Ma5ek]

Hi everyone,

I have updated ESXi 4.1 with latest updates. The problem still remains. But I have noticed, that problem remain only on certain VM. I will try to contact Vmware, but I do not have support anymore. By your experiance, will they help me anywhey ?

Regards, Alan

[julienvarela]

Hi,

If you don't have an active support contract attach to your customer account you Will not be able to open a case.

Regards,

Julien

[Ma5ek]

Hi,

My problem still persist. So I have installed extra Buffalo NAS with ISCSI connections. My configuration is now :

ESX1 server with local store

ESX2 server with local store

Both servers are HA ( cluster )

DELL MD3000 storage system connected via ISCSI

Buffalo NAS connected vi ISCSI.

ESX1 and ESX2 can view both storage systems.

I tested and move few VM to new NAS storage Buffalo. Everything was working perfectly ( shutdow, migrating from one host to other, turn on /off .... ). So I think there is problem in old store on DELL MD3000. Then I test copying VM from old store ( DELL ) to local store of ESX1. The procedure was OK. But when I try to copy VM back to old store ( DELL MD3000 ) error accured :

Error caused by file [esx1-Localstorage] hist/hist_analyzer.vmdk

Task : copy file

For other files the copy procedure was sucesfully, except for vmdk file ( cca. 41 Gb ). And I have enought space on store. So I have to copy file to new store ( buffalo ). This procedure was OK.

I run the same procedure from ESX2 and it was OK. I can copy from old store to local store of ESX2 and via verca.

I have also find the error log ( the store with problems is named VM_store - there are saved VM ) :

At least one corrupt resource metadata region was detected on volume 4d1c7cab-1c64ceea-1298-0010188b4918 (VM_Store). Other regions of the volume might be damaged too.

error

9.8.2013 13:40:15

esx2.xxxx.local

The same error is also for esx1.xxxx.local

From this logs I asume that something is wrong with VM_store ( on DELL MD3000) and the best way is to recreate it ( I think so). But before I do this I have a doubt.

- why everything work on ESX2 when they share the same store ( with ESX1 ) if there is problem with VM_store ? You can copy , migrate, turn on / off VM without any problems.

- why can I copy VM from VM_store to local store of ESX1 but via verca I can not ?

- the true is when I moved VM to new ( Buffalo ) storage everything worked OK.

- could be problem on ISCSI conected from ESX1 to DELL MD3000 ?

Regards, Alan

[Ma5ek]

Hi,

I have resolved the problem. The problem was in VM_store. So the resolution was :

- attached temporary ISCSI NAS

- create new temporary store

- migrate all VM to temporary store ( cca. 900 GB )

- delete  old store

- recreate new store

- migrate VM to new store

Now everythink works.

Thanks again for your help.

Regards, Alan