ESXi

 View Only
  • 1.  Datastores disappeared after a reboot

    Posted Apr 07, 2019 10:41 AM

    Guys,

    Had to shut down the host to put more memory in. The new memory is detected fine, but both my datastores disappeared, so I can't bring the VMs back online.

    Can't figure out why this happened or how to fix it.


    ESXi v5.5. Running off USB (mpx.vmhba32:C0:T0:L0)

    Terminal outputs:

    # esxcfg-scsidevs -c

    Device UID                            Device Type      Console Device                                            Size      Multipath PluginDisplay Name

    mpx.vmhba32:C0:T0:L0                  Direct-Access    /vmfs/devices/disks/mpx.vmhba32:C0:T0:L0                  1911MB    NMP     Local USB Direct-Access (mpx.vmhba32:C0:T0:L0)

    naa.600605b009617b801cfe4186271b2743  Direct-Access    /vmfs/devices/disks/naa.600605b009617b801cfe4186271b2743  1907200MB NMP     Local LSI Disk (naa.600605b009617b801cfe4186271b2743)

    naa.600605b009617b801cfe4186271b7c33  Direct-Access    /vmfs/devices/disks/naa.600605b009617b801cfe4186271b7c33  2861056MB NMP     Local LSI Disk (naa.600605b009617b801cfe4186271b7c33)

    # partedUtil getptbl /vmfs/devices/disks/naa.600605b009617b801cfe4186271b2743

    gpt

    243133 255 63 3905945600

    1 2048 3905945566 AA31E02A400F11DB9590000C2911D1B8 vmfs 0

    # partedUtil getptbl /vmfs/devices/disks/naa.600605b009617b801cfe4186271b7c33

    gpt

    364733 255 63 5859442688

    1 2048 5859442654 AA31E02A400F11DB9590000C2911D1B8 vmfs 0

    Edit:
    Things attempted so far:
    VMware Knowledge Base - Recreating a missing VMFS datastore partition in VMware vSphere 5.x and 6.x (2046610)

    VMware Knowledge Base - LUN suddenly becomes unavailable to one or more ESXi/ESX hosts and LUN needs to be reset (1000044)



  • 2.  RE: Datastores disappeared after a reboot

    Posted Apr 07, 2019 11:12 AM

    Please see whether the LUN has been detected as a snapshot LUN, and if mounting the datastore manually will work.

    see the "Command line" section at https://kb.vmware.com/kb/1011387

    André



  • 3.  RE: Datastores disappeared after a reboot

    Posted Apr 07, 2019 11:32 AM

    Nope.

    # esxcli storage vmfs snapshot list

    #

    When looking at existing partitions, only the USB one is listed:

    # esxcli system coredump partition list

    Name                    Path                                        Active  Configured

    ----------------------  ------------------------------------------  ------  ----------

    mpx.vmhba32:C0:T0:L0:7  /vmfs/devices/disks/mpx.vmhba32:C0:T0:L0:7   false        true



  • 4.  RE: Datastores disappeared after a reboot

    Posted Apr 07, 2019 11:48 AM

    # offset="128 2048"; for dev in `esxcfg-scsidevs -l | grep "Console Device:" | awk {'print $3'}`; do disk=$dev; echo $disk; partedUtil getptbl $disk; { for i in `echo $offset`; do echo "Checking offset found at $i:"; hexdump -n4 -s $((

    0x100000+(512*$i))) $disk; hexdump -n4 -s $((0x1300000+(512*$i))) $disk; hexdump -C -n 128 -s $((0x130001d + (512*$i))) $disk; done; } | grep -B 1 -A 5 d00d; echo "---------------------"; done

    /vmfs/devices/disks/mpx.vmhba32:C0:T0:L0

    gpt

    243 255 63 3913728

    1 64 8191 C12A7328F81F11D2BA4B00A0C93EC93B systemPartition 128

    5 8224 520191 EBD0A0A2B9E5443387C068B6B72699C7 linuxNative 0

    6 520224 1032191 EBD0A0A2B9E5443387C068B6B72699C7 linuxNative 0

    7 1032224 1257471 9D27538040AD11DBBF97000C2911D1B8 vmkDiagnostic 0

    8 1257504 1843199 EBD0A0A2B9E5443387C068B6B72699C7 linuxNative 0

    ---------------------

    /vmfs/devices/disks/naa.600605b009617b801cfe4186271b2743

    gpt

    243133 255 63 3905945600

    1 2048 3905945566 AA31E02A400F11DB9590000C2911D1B8 vmfs 0

    Checking offset found at 2048:

    0200000 d00d c001

    0200004

    1400000 f15e 2fab

    1400004

    0140001d  64 69 73 6b 32 00 00 00  00 00 00 00 00 00 00 00  |disk2...........|

    0140002d  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|

    ---------------------

    /vmfs/devices/disks/naa.600605b009617b801cfe4186271b7c33

    gpt

    364733 255 63 5859442688

    1 2048 5859442654 AA31E02A400F11DB9590000C2911D1B8 vmfs 0

    Checking offset found at 2048:

    0200000 d00d c001

    0200004

    1400000 f15e 2fab

    1400004

    0140001d  64 69 73 6b 31 00 00 00  00 00 00 00 00 00 00 00  |disk1...........|

    0140002d  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|

    ---------------------

    # fdisk -l /dev/disks/naa.600605b009617b801cfe4186271b2743

    ***

    *** The fdisk command is deprecated: fdisk does not handle GPT partitions.  Please use partedUtil

    ***

    Found valid GPT with protective MBR; using GPT

    Disk /dev/disks/naa.600605b009617b801cfe4186271b2743: 3905945600 sectors, 3725M

    Logical sector size: 512

    Disk identifier (GUID): ffa051fd-46a5-4151-a6d3-b62a3a5b9ec0

    Partition table holds up to 128 entries

    First usable sector is 34, last usable sector is 3905945566

    Number  Start (sector)    End (sector)  Size       Code  Name

       1            2048      3905945566       3724M   0700

    # fdisk -l /dev/disks/naa.600605b009617b801cfe4186271b7c33

    ***

    *** The fdisk command is deprecated: fdisk does not handle GPT partitions.  Please use partedUtil

    ***

    fdisk: device has more than 2^32 sectors, can't use all of them

    Found valid GPT with protective MBR; using GPT

    Disk /dev/disks/naa.600605b009617b801cfe4186271b7c33: 4294967295 sectors, 4095M

    Logical sector size: 512

    Disk identifier (GUID): cd7dd043-9548-4f87-bba7-9b900622d9e4

    Partition table holds up to 128 entries

    First usable sector is 34, last usable sector is 5859442654

    Number  Start (sector)    End (sector)  Size       Code  Name

       1            2048      5859442654       5587M   0700

    # esxcli storage core path list

    usb.vmhba32-usb.0:0-mpx.vmhba32:C0:T0:L0

       UID: usb.vmhba32-usb.0:0-mpx.vmhba32:C0:T0:L0

       Runtime Name: vmhba32:C0:T0:L0

       Device: mpx.vmhba32:C0:T0:L0

       Device Display Name: Local USB Direct-Access (mpx.vmhba32:C0:T0:L0)

       Adapter: vmhba32

       Channel: 0

       Target: 0

       LUN: 0

       Plugin: NMP

       State: active

       Transport: usb

       Adapter Identifier: usb.vmhba32

       Target Identifier: usb.0:0

       Adapter Transport Details: Unavailable or path is unclaimed

       Target Transport Details: Unavailable or path is unclaimed

       Maximum IO Size: 122880

    unknown.vmhba2-unknown.2:1-naa.600605b009617b801cfe4186271b7c33

       UID: unknown.vmhba2-unknown.2:1-naa.600605b009617b801cfe4186271b7c33

       Runtime Name: vmhba2:C2:T1:L0

       Device: naa.600605b009617b801cfe4186271b7c33

       Device Display Name: Local LSI Disk (naa.600605b009617b801cfe4186271b7c33)

       Adapter: vmhba2

       Channel: 2

       Target: 1

       LUN: 0

       Plugin: NMP

       State: active

       Transport: parallel

       Adapter Identifier: unknown.vmhba2

       Target Identifier: unknown.2:1

       Adapter Transport Details: Unavailable or path is unclaimed

       Target Transport Details: Unavailable or path is unclaimed

       Maximum IO Size: 131072

    unknown.vmhba2-unknown.2:0-naa.600605b009617b801cfe4186271b2743

       UID: unknown.vmhba2-unknown.2:0-naa.600605b009617b801cfe4186271b2743

       Runtime Name: vmhba2:C2:T0:L0

       Device: naa.600605b009617b801cfe4186271b2743

       Device Display Name: Local LSI Disk (naa.600605b009617b801cfe4186271b2743)

       Adapter: vmhba2

       Channel: 2

       Target: 0

       LUN: 0

       Plugin: NMP

       State: active

       Transport: parallel

       Adapter Identifier: unknown.vmhba2

       Target Identifier: unknown.2:0

       Adapter Transport Details: Unavailable or path is unclaimed

       Target Transport Details: Unavailable or path is unclaimed

       Maximum IO Size: 131072

    # esxcli storage core device list

    naa.600605b009617b801cfe4186271b7c33

       Display Name: Local LSI Disk (naa.600605b009617b801cfe4186271b7c33)

       Has Settable Display Name: true

       Size: 2861056

       Device Type: Direct-Access

       Multipath Plugin: NMP

       Devfs Path: /vmfs/devices/disks/naa.600605b009617b801cfe4186271b7c33

       Vendor: LSI

       Model: MR9240-4i

       Revision: 2.13

       SCSI Level: 5

       Is Pseudo: false

       Status: on

       Is RDM Capable: false

       Is Local: true

       Is Removable: false

       Is SSD: false

       Is Offline: false

       Is Perennially Reserved: false

       Queue Full Sample Size: 0

       Queue Full Threshold: 0

       Thin Provisioning Status: unknown

       Attached Filters:

       VAAI Status: unsupported

       Other UIDs: vml.0200000000600605b009617b801cfe4186271b7c334d5239323430

       Is Local SAS Device: false

       Is USB: false

       Is Boot USB Device: false

       No of outstanding IOs with competing worlds: 32

    mpx.vmhba32:C0:T0:L0

       Display Name: Local USB Direct-Access (mpx.vmhba32:C0:T0:L0)

       Has Settable Display Name: false

       Size: 1911

       Device Type: Direct-Access

       Multipath Plugin: NMP

       Devfs Path: /vmfs/devices/disks/mpx.vmhba32:C0:T0:L0

       Vendor: General

       Model: USB Flash Disk

       Revision: 1.00

       SCSI Level: 2

       Is Pseudo: false

       Status: on

       Is RDM Capable: false

       Is Local: true

       Is Removable: true

       Is SSD: false

       Is Offline: false

       Is Perennially Reserved: false

       Queue Full Sample Size: 0

       Queue Full Threshold: 0

       Thin Provisioning Status: unknown

       Attached Filters:

       VAAI Status: unsupported

       Other UIDs: vml.0000000000766d68626133323a303a30

       Is Local SAS Device: false

       Is USB: true

       Is Boot USB Device: true

       No of outstanding IOs with competing worlds: 32

    naa.600605b009617b801cfe4186271b2743

       Display Name: Local LSI Disk (naa.600605b009617b801cfe4186271b2743)

       Has Settable Display Name: true

       Size: 1907200

       Device Type: Direct-Access

       Multipath Plugin: NMP

       Devfs Path: /vmfs/devices/disks/naa.600605b009617b801cfe4186271b2743

       Vendor: LSI

       Model: MR9240-4i

       Revision: 2.13

       SCSI Level: 5

       Is Pseudo: false

       Status: on

       Is RDM Capable: false

       Is Local: true

       Is Removable: false

       Is SSD: false

       Is Offline: false

       Is Perennially Reserved: false

       Queue Full Sample Size: 0

       Queue Full Threshold: 0

       Thin Provisioning Status: unknown

       Attached Filters:

       VAAI Status: unsupported

       Other UIDs: vml.0200000000600605b009617b801cfe4186271b27434d5239323430

       Is Local SAS Device: false

       Is USB: false

       Is Boot USB Device: false

       No of outstanding IOs with competing worlds: 32



  • 5.  RE: Datastores disappeared after a reboot

    Posted Apr 07, 2019 07:01 PM

    Hi

    Check if the following VMware KB article helps:VMware Knowledge Base



  • 6.  RE: Datastores disappeared after a reboot

    Posted Apr 07, 2019 07:44 PM

    One of the first things I tried.

    vmkernel.log output when rescanning:

    2019-04-07T19:11:25.957Z cpu3:38976)VC: 2059: Device rescan time 16 msec (total number of devices 6)

    2019-04-07T19:11:25.957Z cpu3:38976)VC: 2062: Filesystem probe time 59 msec (devices probed 6 of 6)

    2019-04-07T19:11:25.957Z cpu3:38976)VC: 2064: Refresh open volume time 0 msec

    ...

    2019-04-07T19:15:06.493Z cpu2:32791)NMP: nmp_ThrottleLogForDevice:2321: Cmd 0x1a (0x412e8081b380, 0) to dev "mpx.vmhba32:C0:T0:L0" on path "vmhba32:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0. Act:NONE

    2019-04-07T19:15:06.493Z cpu2:32791)ScsiDeviceIO: 2338: Cmd(0x412e8081b380) 0x1a, CmdSN 0x418 from world 0 to dev "mpx.vmhba32:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.

    ...

    2019-04-07T19:16:27.080Z cpu6:34890)NMP: nmp_ThrottleLogForDevice:2321: Cmd 0x85 (0x412e8081db80, 34362) to dev "naa.600605b009617b801cfe4186271b7c33" on path "vmhba2:C2:T1:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE

    2019-04-07T19:16:27.080Z cpu6:34890)ScsiDeviceIO: 2338: Cmd(0x412e8081db80) 0x85, CmdSN 0x6 from world 34362 to dev "naa.600605b009617b801cfe4186271b7c33" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.

    2019-04-07T19:16:27.080Z cpu6:34890)ScsiDeviceIO: 2338: Cmd(0x412e8081db80) 0x4d, CmdSN 0x7 from world 34362 to dev "naa.600605b009617b801cfe4186271b7c33" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.

    2019-04-07T19:16:27.081Z cpu6:34890)ScsiDeviceIO: 2338: Cmd(0x412e8081db80) 0x1a, CmdSN 0x8 from world 34362 to dev "naa.600605b009617b801cfe4186271b7c33" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.

    2019-04-07T19:16:27.083Z cpu6:32795)NMP: nmp_ThrottleLogForDevice:2321: Cmd 0x85 (0x412e8081db80, 34362) to dev "naa.600605b009617b801cfe4186271b2743" on path "vmhba2:C2:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE

    2019-04-07T19:16:27.083Z cpu6:32795)ScsiDeviceIO: 2338: Cmd(0x412e8081db80) 0x85, CmdSN 0x9 from world 34362 to dev "naa.600605b009617b801cfe4186271b2743" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.

    2019-04-07T19:16:27.083Z cpu6:32795)ScsiDeviceIO: 2338: Cmd(0x412e8081db80) 0x4d, CmdSN 0xa from world 34362 to dev "naa.600605b009617b801cfe4186271b2743" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.

    2019-04-07T19:16:27.083Z cpu6:32795)ScsiDeviceIO: 2338: Cmd(0x412e8081db80) 0x1a, CmdSN 0xb from world 34362 to dev "naa.600605b009617b801cfe4186271b2743" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.



  • 7.  RE: Datastores disappeared after a reboot

    Posted Apr 08, 2019 05:19 PM

    From the first one of the two KB articles you mentioned in your initial post, did you run vmkfstools -V (step 5), and then check the vmkernel.log for related entries?

    André



  • 8.  RE: Datastores disappeared after a reboot

    Posted Apr 07, 2019 09:44 PM

    Do you can sharing vmkernel.log and hosts.log with the date next to reboot?