Empty DataStore. Need Urgent help!

[TCP]

Hi All!

We have a problem with one of our Datastore.

When I try to browse DataStore from service console "ls" i recieve error message "Invalid argument".

When I try to browse DataStore from DataStore browser i see empty DataStore, but i know DataStore is not empty and several VM's working till now from this DataStore, but VM's inaccessible.

fdisk -l

esxcfg-vmhbadevs -m

lists all volumes correctly and problem DataStore too.

System config:

IBM BladeCenter H

HS21XM blades

VC 2.5 u2

12 - ESX 3.5 u2 hosts

IBM DS4400 SAN.

Please Help.

[NTurnbull]

Hi, have you got any errors in your vmkernel logs? Anything changed on the SAN config/zoning?

[TCP]

Hi.

Yes I have errors like

Nov 4 08:19:15 esx-005-svr vmkernel: 7:21:23:29.380 cpu4:1043)WARNING: FS3: 4979: Reclaiming timed out heartbeat http://HB state abcdef02 offset 3766272 gen 102 stamp 681807788865 uuid 49058253-551cbe34-6067-001a64760058 jrnl <FB 88119> drv 4.31 failed: Invalid metadata

Nov 4 08:19:15 esx-005-svr vmkernel: 7:21:23:29.743 cpu1:1093)<6>Debug scsi underrun

Nov 4 08:19:15 esx-005-svr vmkernel: 7:21:23:29.743 cpu1:1093)<6>Debug scsi underrun

Nov 4 08:19:53 esx-005-svr vmkernel: 7:21:24:07.835 cpu1:1087)<6>Debug scsi underrun

Nov 4 08:20:15 esx-005-svr vmkernel: 7:21:24:29.749 cpu1:1168)<6>Debug scsi underrun

Nov 4 08:21:11 esx-005-svr vmkernel: 7:21:25:25.123 cpu1:1149)<6>Debug scsi underrun

Nov 4 08:21:11 esx-005-svr vmkernel: 7:21:25:25.123 cpu1:1149)<6>Debug scsi

Nov 4 08:46:34 dho-esx-005-svr vmkernel: 7:21:50:49.001 cpu7:1203)WARNING: Swap: vm 1204: 2268: Failed to open swap file '/volumes/48c8c0a1-2c706ad6-74e3-001a64760048/001-VSVR/001-VSVR-300dc447.vswp': Invalid metadata

Nov 4 08:46:34 dho-esx-005-svr vmkernel: 7:21:50:49.001 cpu7:1203)WARNING: Swap: vm 1204: 3586: Failed to initialize swap file '/volumes/48c8c0a1-2c706ad6-74e3-001a64760048/001-VSVR/001-VSVR-300dc447.vswp': Invalid metadata

Nov 4 08:46:34 dho-esx-005-svr vmkernel: 7:21:50:49.001 cpu7:1203)WARNING: World: vm 1204: 910: init fn swap failed with: No swap file!

Nov 4 08:46:34 dho-esx-005-svr vmkernel: 7:21:50:49.001 cpu7:1203)WARNING: World: vm 1204: 1775: WorldInit failed: trying to cleanup.

[polysulfide]

Did you recently bring a new VCB proxy online? If so, hopefully you remembered to disable automount.

[TCP]

No this is not vcb problem.

Powering off of all worked virtual machines solved the probIem.

I can't understand what happened.

[NTurnbull]

TCp thats random as!! Got anything in the messages or kernel logs for anything to do with hba's or SCSI error/info (reservations maybe?) I'm confused

[mikepodoherty]

Do you use IBM Storage Manager? IF so, what do the logs in Storage Manger report for the time period when you couldn't see the datastore?

[Jasemccarty]

Just a heads up... the DS4400 is only on the HCL through ESX 3.0.3. 3.5 isn't supported on DS4400's.

I'm not saying it doesn't work... I have one too. But it isn't supported with the latest builds.

Jase McCarty

http://www.jasemccarty.com

Co-Author of VMware ESX Essentials in the Virtual Data Center

(ISBN:1420070274) from Auerbach

[Jasemccarty]

Alright... I'm going to say it.

Apparently DS4400's/FastT700's don't play well with ESX 3.5. VMware support will tell you to roll back to 3.0.3

Update: With a little help from Mike La Spina... 3.5 Update 2 will work, but Update 3, with the Storage Monitor addition will cause you some issues.

VMware won't support ESX 3.5 on a DS4400, but it will work.

Jase McCarty

http://www.jasemccarty.com

Co-Author of VMware ESX Essentials in the Virtual Data Center

(ISBN:1420070274) from Auerbach

_{Please consider awarding points if this post was helpful or correct}

[GavinConway]

Problem replicated here as well. Unless all the ESXi hosts within that cluster are running then you can't browse the datastores.

Anyone raising a bug report?