ESXi

 View Only
  • 1.  Help - Snapshot Errors following Zero diskspace issue

    Posted Sep 24, 2012 09:45 AM

    Hi,

    I have an issue which occured over the weekend. On my ESXi 4.1 hosts all snapshots are saved into single datastore. Over the weekend this datastore filled up, halting 6 VM which had snapshots running on them. The VM were frozen when I can in this morning.

    Message from 10.0.0.xx: There is no more
    space for virtual disk Server1-000002.vmdk
    . You might be able to continue this session by
    freeing disk space on the relevant volume, and
    clicking Retry. Click Cancel to terminate this
    session.
    info
    24/09/2012 08:31:57
    Server1

    User

    I came in a cleared down a load of ISOs to free up space. I answerved the retry question on the VM to get them back working and all seems fine - data is up to date etc.

    I then tried to remove the snapshots. However I can't remove or create any snapshots on these 6 VM. When i try to I get the following errors:


    Server1

    Cannot complete the
    operation because the file or
    folder <unspecified filename
    > already exists
    Domain\Admin
    Server1.hiqa.local
    24/09/2012 08:53:19
    24/09/2012 08:53:19
    24/09/2012 08:53:24

    Create virtual machine snapshot
    Server1
    Cannot create a quiesced
    snapshot because the create
    snapshot operation
    exceeded the time limit for
    holding off I/O in the frozen
    virtual machine.
    Domain\admin
    Server1.Domain.local
    23/09/2012 00:09:49
    23/09/2012 00:09:49
    23/09/2012 00:10:29

    I don't know where to go from here to get them back into a stable state. Don't want to do anything in case I make it worse.

    Any help would be apprepicated.

    Regards,

    Fastpud



  • 2.  RE: Help - Snapshot Errors following Zero diskspace issue

    Posted Sep 24, 2012 12:16 PM

    In this case I'd recommend you contact VMware support.

    If you like, post (attach) the latest vmware.log file for one of the VM's on which this message appeared to see whether this shows what's wrong. Maybe it's only a synchronization issue between the hosts and vCenter Server which can be fixed with either restarting the Management Agents on the hosts and/or the vCenter Server service. However, be careful with restarting if you use other applications like VMware View which rely on vCenter Server.

    André



  • 3.  RE: Help - Snapshot Errors following Zero diskspace issue

    Posted Sep 24, 2012 01:45 PM

    Andre,

    Here are the last 3 vmware.log files as requested. The disk filled up approx 9:00pm on Sat 22nd, one VM's snapshot file exploded in size due to rogue backups.

    Regards & Thanks.

    Fastpud



  • 4.  RE: Help - Snapshot Errors following Zero diskspace issue

    Posted Sep 24, 2012 02:10 PM

    To me this looks like a delta file has been created without the associated descriptor .vmdk file. To see whether it is save to e.g. delete possibly orphaned files however, I need more details. Please provide a list of files in the VM's folder. Since the Datastore Browser does not show all files, you need to get the list from the command line (e.g. SSH) running ls -lisa. In addition to this list, please post (attach) the current .vmx file.

    André



  • 5.  RE: Help - Snapshot Errors following Zero diskspace issue

    Posted Sep 24, 2012 02:22 PM

    Andre,

    Here is the listing of the VM's files


    /vmfs/volumes/4d4bdbe8-f8ad714b-ddbf-78e7d1e93f4e/HIQASRV2139 # ls -lisa
        836   64 drwxr-xr-x    1 root     root               2520 Sep 24 08:12 .
          4 1024 drwxr-xr-t    1 root     root               3220 Sep 12 08:41 ..
    662708356   64 -rw-r--r--    1 root     root                 13 Sep 24 07:54 HIQASRV2139-aux.xml
    490741892 4096 -rw-------    1 root     root            3277312 Sep 14 13:34 HIQASRV2139-ctk.vmdk
    532682820   64 -rw-r--r--    1 root     root                 37 Jun 20 15:41 HIQASRV2139-d176768e.hlog
    8389444 30393344 -rw-------    1 root     root        53687091200 Sep 18 10:49 HIQASRV2139-flat.vmdk
    704647684   64 -rw-------    1 root     root               8684 Sep 24 07:45 HIQASRV2139.nvram
    671093252   64 -rw-------    1 root     root                611 May  5  2011 HIQASRV2139.vmdk
    25166660   64 -rw-r--r--    1 root     root                 78 Mar  7  2011 HIQASRV2139.vmsd
    16778052   64 -rwxr-xr-x    1 root     root               3768 Sep 24 08:18 HIQASRV2139.vmx
    20972356   64 -rw-r--r--    1 root     root               1575 Sep 24 08:15 HIQASRV2139.vmxf
    146805252 1024 -rw-r--r--    1 root     root             190358 Dec  4  2011 vmware-24.log
    176165380 1024 -rw-r--r--    1 root     root             117164 Dec 11  2011 vmware-25.log
    75498308 2048 -rw-r--r--    1 root     root            1198917 Jun 17 15:40 vmware-26.log
    281022980 1024 -rw-r--r--    1 root     root              72277 Jun 20 15:35 vmware-27.log
    247472260 1024 -rw-r--r--    1 root     root             906351 Sep 18 10:51 vmware-28.log
    398463492 1024 -rw-r--r--    1 root     root             164299 Sep 24 08:03 vmware-29.log
    411046404 1024 -rw-r--r--    1 root     root             156742 Sep 24 13:32 vmware.log

    Here is the listing of the same VM's Snapshot location

    /vmfs/volumes/4ea6b3f1-8a32878b-89bd-78e7d1e93f4c/VM-Snapshots/HIQASRV2139 # ls -lisa
    150996548   64 drwxr-xr-x    1 root     root               1820 Sep 24 14:11 .
       1604   64 drwxr-xr-x    1 root     root               5180 Jun 22 10:16 ..
    482353284 4096 -rw-------    1 root     root            3277312 Sep 18 10:49 HIQASRV2139-000001-ctk.vmdk
    473964676 11653120 -rw-------    1 root     root        11928707072 Sep 24 13:32 HIQASRV2139-000001-delta.vmdk
    478158980   64 -rw-------    1 root     root                458 Sep 18 10:49 HIQASRV2139-000001.vmdk
    729817220    0 -rw-------    1 root     root             106496 Sep 22 23:10 HIQASRV2139-000002-delta.vmdk
    725622916   64 -rw-------    1 root     root              30565 Sep 22 23:10 HIQASRV2139-Snapshot93.vmsn
    788537476   64 -rw-------    1 root     root              30464 Sep 24 07:54 HIQASRV2139-Snapshot94.vmsn
    792731780   64 -rw-------    1 root     root              30564 Sep 24 08:05 HIQASRV2139-Snapshot95.vmsn
    796926084   64 -rw-------    1 root     root              30564 Sep 24 08:10 HIQASRV2139-Snapshot96.vmsn
    805314692   64 -rw-------    1 root     root              30565 Sep 24 08:48 HIQASRV2139-Snapshot97.vmsn
    809508996   64 -rw-------    1 root     root              30565 Sep 24 09:37 HIQASRV2139-Snapshot98.vmsn
    826286212   64 -rw-------    1 root     root              30564 Sep 24 14:11 HIQASRV2139-Snapshot99.vmsn

    I think the snapshot is active as all changes made post snaphot are still on the server.

    Thanks

    Fastpud



  • 6.  RE: Help - Snapshot Errors following Zero diskspace issue
    Best Answer

    Posted Sep 24, 2012 02:41 PM

    To me this looks like "HIQASRV2139-000002-delta.vmdk" was created, but never used due to the disk space issues. For the virtual disk to be usable the has to exist a descriptor file "HIQASRV2139-000002.vmdk" too. What you may do is to shut down the VM (I'm pretty sure this is not really necessary, but for safety reasons I'd recommend it), then move the "HIQASRV2139-000002-delta.vmdk" to a sub folder or rename it and power on the VM again. If the VM powers on without issues you may delete this file.

    André



  • 7.  RE: Help - Snapshot Errors following Zero diskspace issue

    Posted Sep 24, 2012 03:19 PM

    Andre,

    You were spot on.  Once I deleted this file I was able to remove all the snapshots and server is as I expected. This snapshot would of been created by a Backup exec job which runs over the weekend. I will have to review all other VMs for this issue.

    Many thanks, You are a life saver.

    Fastpud.