ESXi

 View Only

  • 1.  Datastore not accessible

    Posted Jul 13, 2011 01:33 PM

    Hey there,

    we got the same problem two weeks ago. One of our Datastores was suddenly unavailable.

    Back then there where machines running on that Datastore.

    We rebooted the server, the Datastore was back on but we had serious problems with locked files (which still isn't resolved).

    See the Thread here:  http://communities.vmware.com/thread/319421?tstart=150

    We now are in fear that there might be more locked files or some more trouble when rebooting the server.

    We also would like to know the cause of that Datastore beeing unavailable.

    We already tried  "vmkfstools -V" and looking at the problem with esxcfg-scsidevs -l / -a / -m (see output below!)

    I was deleting some old backups of machines when the Datastore vanished (deleting some machines via the Console/SSH on the same Datastore worked a few minutes before).

    I include the Console output at the time the Datastore was gone (see below).

    Maybe someone has some useful tips on how to remount the Datastore / get it back online or can indentify the source of the problem!

    Thanks in advance for replies, asgaroth

    /vmfs/volumes/4bd88e21-e8b0d6ea-6c1a-003048c30e8c # rm -rf serverxyz.subdomain.domain
    

     
    

    rm: cannot remove 'serverxyz.subdomain.domain/serverxyz.subdomain.domain-flat.vmdk': Resource temporarily unavailable
    

    rm: cannot stat 'serverxyz.subdomain.domain/serverxyz.subdomain.domain.vmdk': Device or resource busy
    

    rm: cannot stat 'serverxyz.subdomain.domain/serverxyz.subdomain.domain.nvram': Device or resource busy
    

    rm: cannot stat 'serverxyz.subdomain.domain/vmware-37.log': Device or resource busy
    

    rm: cannot stat 'serverxyz.subdomain.domain/vmware-38.log': Device or resource busy
    

    rm: cannot stat 'serverxyz.subdomain.domain/vmware-39.log': Device or resource busy
    

    rm: cannot stat 'serverxyz.subdomain.domain/serverxyz.subdomain.domain-c1309193.hlog': Device or resource busy
    

    rm: cannot stat 'serverxyz.subdomain.domain/vmware-42.log': Device or resource busy
    

    rm: cannot stat 'serverxyz.subdomain.domain/vmware-41.log': Device or resource busy
    

    rm: cannot stat 'serverxyz.subdomain.domain/vmware-40.log': Device or resource busy
    

    rm: cannot stat 'serverxyz.subdomain.domain/vmware.log': Device or resource busy
    

    rm: cannot remove 'serverxyz.subdomain.domain': Device or resource busy
    

    ash: getcwd: Device or resource busy
    

     
    

     
    

    (unknown) # ls -al
    

    ls: .: Input/output error
    

    ash: getcwd: Input/output error
    

     
    

     
    

    (unknown) # cd /
    

     
    

    ~ # ls -al /vmfs/volumes/
    

     
    

     
    

    ls: /vmfs/volumes/4bd88e21-e8b0d6ea-6c1a-003048c30e8c: Input/output error
    

    drwxr-xr-x    1 root     root                512 Jul 12 13:37 .
    

    drwxr-xr-x    1 root     root                512 Jun 28 15:57 ..
    

    drwxr-xr-x    1 root     root                  8 Jan  1  1970 386a41e4-ff1587ba-1385-2feaeed494ad
    

    drwxr-xr-t    1 root     root               1960 Jul 12 12:56 4bc8414f-3206fe54-f06e-003048c30e8c
    

    drwxr-xr-t    1 root     root               3500 Jul  8 22:30 4bd88e99-c67ad631-6c62-003048c30e8c
    

    drwxr-xr-t    1 root     root                980 May 21 14:43 4dd7cf77-c3e57835-e9ef-003048c30e8c
    

    drwxr-xr-x    1 root     root                  8 Jan  1  1970 8c7e2fe8-f4c58d9f-34d3-8553ffee79a0
    

    lrwxr-xr-x    1 root     root                 35 Jul 12 13:37 Datastore1 -> 4bc8414f-3206fe54-f06e-003048c30e8c
    

    lrwxr-xr-x    1 root     root                 35 Jul 12 13:37 Datastore2 -> 4bd88e21-e8b0d6ea-6c1a-003048c30e8c
    

    lrwxr-xr-x    1 root     root                 35 Jul 12 13:37 Datastore3 -> 4bd88e99-c67ad631-6c62-003048c30e8c
    

    lrwxr-xr-x    1 root     root                 35 Jul 12 13:37 Datastore4 -> 4dd7cf77-c3e57835-e9ef-003048c30e8c
    

    lrwxr-xr-x    1 root     root                 35 Jul 12 13:37 Hypervisor1 -> 386a41e4-ff1587ba-1385-2feaeed494ad
    

    lrwxr-xr-x    1 root     root                 35 Jul 12 13:37 Hypervisor2 -> 8c7e2fe8-f4c58d9f-34d3-8553ffee79a0
    

    lrwxr-xr-x    1 root     root                 35 Jul 12 13:37 Hypervisor3 -> e00f98e1-2bcc0c91-e7a2-3487611c1557
    

    drwxr-xr-x    1 root     root                  8 Jan  1  1970 e00f98e1-2bcc0c91-e7a2-3487611c1557
    

     
    

     
    

    ~ # ls -al /vmfs/volumes/Datastore2
    

     
    

     
    

    lrwxr-xr-x    1 root     root                 35 Jul 12 13:38 /vmfs/volumes/Datastore2 -> 4bd88e21-e8b0d6ea-6c1a-003048c30e8c
    

     
    

     
    

    ~ # ls -al /vmfs/volumes/Datastore2/
    

     
    

     
    

    ls: /vmfs/volumes/Datastore2/: Input/output error
    

     
    

     
    

    ~ # df 
    

    Filesystem           1k-blocks      Used Available Use% Mounted on
    

    visorfs                1338636    267996   1070640  20% /
    

    vmfs3                488112128 328402944  159709184  67% /vmfs/volumes/4bc8414f-3206fe54-f06e-003048c30e8c
    

    nfs                  315829740 135465900  180363840  43% /vmfs/volumes/0d6becf5-2ea0d633
    

    nfs                  1175588244 230332944  945255300  20% /vmfs/volumes/fa828cd2-adbc3d72
    

    vmfs3                976486400    578560  975907840   0% /vmfs/volumes/4dd7cf77-c3e57835-e9ef-003048c30e8c
    

    vfat                    292752    139328    153424  48% /vmfs/volumes/e00f98e1-2bcc0c91-e7a2-3487611c1557
    

    vfat                    255716     90204    165512  35% /vmfs/volumes/8c7e2fe8-f4c58d9f-34d3-8553ffee79a0
    

    vfat                    255716     61804    193912  24% /vmfs/volumes/386a41e4-ff1587ba-1385-2feaeed494ad
    

     
    

     
    

    ~ # vmkfstools -V
    

     
    

     
    

    ~ # ls /vmfs/volumes/
    

     
    

     
    

    0d6becf5-2ea0d633                    4dd7cf77-c3e57835-e9ef-003048c30e8c  Datastore4                           e00f98e1-2bcc0c91-e7a2-3487611c1557
    

    386a41e4-ff1587ba-1385-2feaeed494ad  8c7e2fe8-f4c58d9f-34d3-8553ffee79a0  Hypervisor1                          fa828cd2-adbc3d72
    

    4bc8414f-3206fe54-f06e-003048c30e8c  Datastore1                           Hypervisor2                          nfs-vmds1
    

    4bd88e99-c67ad631-6c62-003048c30e8c  Datastore3                           Hypervisor3                          nfs-vmds2
    

     
    

     
    

    ~ # ls -l /vmfs/volumes/
    

     
    

     
    

    drwxr-x---    1 root     root               4096 Jun 30 12:47 0d6becf5-2ea0d633
    

    drwxr-xr-x    1 root     root                  8 Jan  1  1970 386a41e4-ff1587ba-1385-2feaeed494ad
    

    drwxr-xr-t    1 root     root               1960 Jul 12 12:56 4bc8414f-3206fe54-f06e-003048c30e8c
    

    drwxr-xr-t    1 root     root               3500 Jul  8 22:30 4bd88e99-c67ad631-6c62-003048c30e8c
    

    drwxr-xr-t    1 root     root                980 May 21 14:43 4dd7cf77-c3e57835-e9ef-003048c30e8c
    

    drwxr-xr-x    1 root     root                  8 Jan  1  1970 8c7e2fe8-f4c58d9f-34d3-8553ffee79a0
    

    lrwxr-xr-x    1 root     root                 35 Jul 12 13:59 Datastore1 -> 4bc8414f-3206fe54-f06e-003048c30e8c
    

    lrwxr-xr-x    1 root     root                 35 Jul 12 13:59 Datastore3 -> 4bd88e99-c67ad631-6c62-003048c30e8c
    

    lrwxr-xr-x    1 root     root                 35 Jul 12 13:59 Datastore4 -> 4dd7cf77-c3e57835-e9ef-003048c30e8c
    

    lrwxr-xr-x    1 root     root                 35 Jul 12 13:59 Hypervisor1 -> 386a41e4-ff1587ba-1385-2feaeed494ad
    

    lrwxr-xr-x    1 root     root                 35 Jul 12 13:59 Hypervisor2 -> 8c7e2fe8-f4c58d9f-34d3-8553ffee79a0
    

    lrwxr-xr-x    1 root     root                 35 Jul 12 13:59 Hypervisor3 -> e00f98e1-2bcc0c91-e7a2-3487611c1557
    

    drwxr-xr-x    1 root     root                  8 Jan  1  1970 e00f98e1-2bcc0c91-e7a2-3487611c1557
    

    drwxr-x---    1 root     root               4096 Jun 30 12:36 fa828cd2-adbc3d72
    

    lrwxr-xr-x    1 root     root                 17 Jul 12 13:59 nfs-vmds1 -> 0d6becf5-2ea0d633
    

    lrwxr-xr-x    1 root     root                 17 Jul 12 13:59 nfs-vmds2 -> fa828cd2-adbc3d72
    

     
    

     
    

    ~ # esxcfg-scsidevs -l    
    

    mpx.vmhba0:C0:T0:L0
    

       Device Type: CD-ROM 
    

       Size: 0 MB
    

       Display Name: Local TSSTcorp CD-ROM (mpx.vmhba0:C0:T0:L0)
    

       Multipath Plugin: NMP
    

       Console Device: /vmfs/devices/cdrom/mpx.vmhba0:C0:T0:L0
    

       Devfs Path: /vmfs/devices/cdrom/mpx.vmhba0:C0:T0:L0
    

       Vendor: TSSTcorp  Model: CDDVDW SH-S202N   Revis: SB01 
    

       SCSI Level: 5  Is Pseudo: false Status: on        
    

       Is RDM Capable: false Is Removable: true  
    

       Is Local: true  
    

       Other Names: 
    

          vml.0005000000766d686261303a303a30
    

       VAAI Status: unknown
    

    mpx.vmhba32:C0:T0:L0
    

       Device Type: Direct-Access 
    

       Size: 998 MB
    

       Display Name: Local USB Direct-Access (mpx.vmhba32:C0:T0:L0)
    

       Multipath Plugin: NMP
    

       Console Device: /vmfs/devices/disks/mpx.vmhba32:C0:T0:L0
    

       Devfs Path: /vmfs/devices/disks/mpx.vmhba32:C0:T0:L0
    

       Vendor: TinyDisk  Model: 2005-03-10        Revis: 2.00 
    

       SCSI Level: 2  Is Pseudo: false Status: on        
    

       Is RDM Capable: false Is Removable: true  
    

       Is Local: true  
    

       Other Names: 
    

          vml.0000000000766d68626133323a303a30
    

       VAAI Status: unknown
    

    naa.600050e0d7beda002095000037530000
    

       Device Type: Direct-Access 
    

       Size: 953664 MB
    

       Display Name: Local AMCC Disk (naa.600050e0d7beda002095000037530000)
    

       Multipath Plugin: NMP
    

       Console Device: /vmfs/devices/disks/naa.600050e0d7beda002095000037530000
    

       Devfs Path: /vmfs/devices/disks/naa.600050e0d7beda002095000037530000
    

       Vendor: AMCC      Model: 9650SE-8LP DISK   Revis: 4.10 
    

       SCSI Level: 5  Is Pseudo: false Status: on        
    

       Is RDM Capable: false Is Removable: false 
    

       Is Local: true  
    

       Other Names: 
    

          vml.0200000000600050e0d7beda002095000037530000393635305345
    

       VAAI Status: unknown
    

    t10.AMCC____9QJ0ZCFYD87EFE007FC4
    

       Device Type: Direct-Access 
    

       Size: 953664 MB
    

       Display Name: Local AMCC Disk (t10.AMCC____9QJ0ZCFYD87EFE007FC4)
    

       Multipath Plugin: NMP
    

       Console Device: /vmfs/devices/disks/t10.AMCC____9QJ0ZCFYD87EFE007FC4
    

       Devfs Path: /vmfs/devices/disks/t10.AMCC____9QJ0ZCFYD87EFE007FC4
    

       Vendor: AMCC      Model: 9650SE-8LP DISK   Revis: 4.10 
    

       SCSI Level: 5  Is Pseudo: false Status: on        
    

       Is RDM Capable: false Is Removable: false 
    

       Is Local: true  
    

       Other Names: 
    

          vml.010000000039514a305a434659443837454645303037464334393635305345
    

       VAAI Status: unknown
    

    t10.AMCC____9QJ16PFCD87EFE00CB5E
    

       Device Type: Direct-Access 
    

       Size: 953664 MB
    

       Display Name: Local AMCC Disk (t10.AMCC____9QJ16PFCD87EFE00CB5E)
    

       Multipath Plugin: NMP
    

       Console Device: /vmfs/devices/disks/t10.AMCC____9QJ16PFCD87EFE00CB5E
    

       Devfs Path: /vmfs/devices/disks/t10.AMCC____9QJ16PFCD87EFE00CB5E
    

       Vendor: AMCC      Model: 9650SE-8LP DISK   Revis: 4.10 
    

       SCSI Level: 5  Is Pseudo: false Status: on        
    

       Is RDM Capable: false Is Removable: false 
    

       Is Local: true  
    

       Other Names: 
    

          vml.010000000039514a3136504643443837454645303043423545393635305345
    

       VAAI Status: unknown
    

    t10.AMCC____U34961226AEF5C004A68
    

       Device Type: Direct-Access 
    

       Size: 476827 MB
    

       Display Name: Local AMCC Disk (t10.AMCC____U34961226AEF5C004A68)
    

       Multipath Plugin: NMP
    

       Console Device: /vmfs/devices/disks/t10.AMCC____U34961226AEF5C004A68
    

       Devfs Path: /vmfs/devices/disks/t10.AMCC____U34961226AEF5C004A68
    

       Vendor: AMCC      Model: 9650SE-8LP DISK   Revis: 4.10 
    

       SCSI Level: 5  Is Pseudo: false Status: on        
    

       Is RDM Capable: false Is Removable: false 
    

       Is Local: true  
    

       Other Names: 
    

          vml.01000000005533343936313232364145463543303034413638393635305345
    

       VAAI Status: unknown
    

     
    

     
    

    ~ # esxcfg-scsidevs -a
    

    vmhba0  ata_piix          link-n/a  ide.vmhba0                              (0:0:31.1) Intel Corporation 631xESB/632xESB IDE Controller
    

    vmhba1  3w-9xxx           link-n/a  unknown.vmhba1                          (0:1:0.0) 3ware Inc 3ware 9650SE
    

    vmhba32 usb-storage       link-n/a  usb.vmhba32                             () USB                          
    

    vmhba33 ata_piix          link-n/a  ide.vmhba33                             (0:0:31.1) Intel Corporation 631xESB/632xESB IDE Controller
    

     
    

     
    

    ~ # esxcfg-scsidevs -m
    

    t10.AMCC____U34961226AEF5C004A68:1                               /vmfs/devices/disks/t10.AMCC____U34961226AEF5C004A68:1     4bc8414f-3206fe54-f06e-003048c30e8c  0  Datastore1
    

    naa.600050e0d7beda002095000037530000:1                           /vmfs/devices/disks/naa.600050e0d7beda002095000037530000:1 4dd7cf77-c3e57835-e9ef-003048c30e8c  0  Datastore4
    

    t10.AMCC____9QJ16PFCD87EFE00CB5E:1                               /vmfs/devices/disks/t10.AMCC____9QJ16PFCD87EFE00CB5E:1     4bd88e99-c67ad631-



  • 2.  RE: Datastore not accessible

    Posted Sep 01, 2016 11:58 PM

    I have a similar issue... someone find a answer?



  • 3.  RE: Datastore not accessible

    Posted Sep 04, 2016 11:17 PM

    There is no such thing like a universal answer to this question.
    Sometimes this just means faulty hardware, sometimes it is a fixable error in the VMFS filesystem and sometimes it means that it is still possible to extract your VMs with Linux.
    I can give a better answer once I have seen a dump of the VMFS-metadata.
    Find out the file that corresponds to the inaccessible datastore in /dev/disks.
    Then run
    dd if=/dev/disks/<device> bs=1M count=1536 of=/tmp/eligodoy.1536
    If that does not fit into tmp try to store it in another datastore
    or try
    dd if=/dev/disks/<device> bs=1M count=1536 | hexdump > /tmp/eligodoy.1536.hex
    Download the result to your adminhost, compress it and provide a download.
    With that input I can give you a solid prognosis
    Ulli