ESXi

 View Only
Expand all | Collapse all

File VMDK disappear into sesparse and ctk into ESXi

  • 1.  File VMDK disappear into sesparse and ctk into ESXi

    Posted Nov 09, 2021 04:12 PM

    Hello,

    We are a problem with a snapshot. The VMDk from this snapshot has changed into 2 files -ctk and sesparse. How sync this files to restart VM ?

    Have a nice day

     



  • 2.  RE: File VMDK disappear into sesparse and ctk into ESXi

    Posted Nov 09, 2021 04:34 PM

    >>> ... into 2 files -ctk and sesparse.
    Isn't there also a small <vmname>-00000x.vmdk descriptor file for the snapshot in the VM's folder?

    Please run ls -lisa from a command line in the VM's folder to get a complete file listing, and provide the text output in your next reply. The GUI "hides" some files.

    André



  • 3.  RE: File VMDK disappear into sesparse and ctk into ESXi

    Posted Nov 09, 2021 04:53 PM
      |   view attached

    Hello,

    Here the ls -lisa make into the datastore.

    Thanks

     



  • 4.  RE: File VMDK disappear into sesparse and ctk into ESXi

    Posted Nov 09, 2021 05:09 PM

    Anyway, it seems that some important files are missing, which sometimes occur if a datastore runs out of disk space.

    To see what can be done, please run the following commands in the VM's folder:

    • df -h
    • ls -lisa
    • grep vmdk *.vmx

    Please provide (copy&paste) the the commands' output as plain text rather than a screen shot with a truncated output.

    In addition to the commands' output, use e.g. WinSCP, download all of the VM's descriptor .vmdk files (the small ones without flat, sesparse, or ctk in their names), then compress/zip them, and attach the .zip archive to your next reply,.

    André

     

     



  • 5.  RE: File VMDK disappear into sesparse and ctk into ESXi

    Posted Nov 09, 2021 05:26 PM
      |   view attached

    Hello,

     

    Here the logs :

    [root@localhost:/vmfs/volumes/5be4a8eb-27740b64-d75a-20677cd6dd34/SRV-EXCH01] grep vmdk *.vmx
    scsi0:0.fileName = "SRV-EXCH01.vmdk"
    scsi0:1.fileName = "SRV-EXCH01_1.vmdk"

     

    [root@localhost:/vmfs/volumes/5be4a8eb-27740b64-d75a-20677cd6dd34/SRV-EXCH01] df -h
    Filesystem Size Used Available Use% Mounted on
    VMFS-6 830.8G 658.2G 172.6G 79% /vmfs/volumes/datastore1_15K
    VMFS-6 447.0G 364.6G 82.4G 82% /vmfs/volumes/datastore2_SSD
    VMFS-6 558.8G 545.2G 13.6G 98% /vmfs/volumes/datastore3_15K
    vfat 285.8M 209.1M 76.8M 73% /vmfs/volumes/5b96778d-46f5c96a-d744-20677cd6dd34
    vfat 4.0G 22.2M 4.0G 1% /vmfs/volumes/5b967794-c66f7d18-32ee-20677cd6dd34
    vfat 249.7M 183.2M 66.5M 73% /vmfs/volumes/c4424f7d-503e5cff-1dea-9f7ecbbca204
    vfat 249.7M 8.0K 249.7M 0% /vmfs/volumes/7cfb2376-3f54328b-d3ed-40e98635732b

     

    [root@localhost:/vmfs/volumes/5be4a8eb-27740b64-d75a-20677cd6dd34/SRV-EXCH01] ls -lisa
    total 570197504
    3780 128 drwxr-xr-x 1 root root 86016 Nov 9 15:43 .
    4 1024 drwxr-xr-t 1 root root 73728 Nov 9 12:26 ..
    96469956 4096 -rw------- 1 root root 3932672 Nov 3 22:15 SRV-EXCH01-000001-ctk.vmdk
    46138308 10146816 -rw------- 1 root root 10389823488 Nov 9 12:29 SRV-EXCH01-000001-sesparse.vmdk
    83887044 4096 -rw------- 1 root root 3932672 Nov 9 14:06 SRV-EXCH01-000002-ctk.vmdk
    25166788 4683776 -rw------- 1 root root 4795998208 Nov 9 14:06 SRV-EXCH01-000002-sesparse.vmdk
    88081348 0 -rw-r--r-- 1 root root 53 May 8 2020 SRV-EXCH01-18ae0f4e.hlog
    92275652 1024 -rw------- 1 root root 294474 Nov 9 13:35 SRV-EXCH01-Snapshot1283.vmsn
    79692740 4096 -rw------- 1 root root 3932672 Nov 9 13:06 SRV-EXCH01-ctk.vmdk
    16778180 62429184 -rw------- 1 root root 64424509440 Nov 9 13:06 SRV-EXCH01-flat.vmdk
    29361092 1024 -rw------- 1 root root 270840 Nov 9 13:02 SRV-EXCH01.nvram
    964 0 -rw------- 1 root root 567 Nov 9 13:02 SRV-EXCH01.vmdk
    8389572 0 -rw-r--r-- 1 root root 1052 Nov 9 13:09 SRV-EXCH01.vmsd
    20972484 64 -rw-r--r-- 1 root root 3832 Nov 9 15:40 SRV-EXCH01.vmx
    37749700 0 -rw------- 1 root root 3184 Nov 8 2018 SRV-EXCH01.vmxf
    58721220 8192 -rw------- 1 root root 7700992 Nov 9 14:51 SRV-EXCH01_1-000001-ctk.vmdk
    50332612 26355712 -rw------- 1 root root 26987765760 Nov 9 12:32 SRV-EXCH01_1-000001-sesparse.vmdk
    100664260 8192 -rw------- 1 root root 7700992 Nov 9 14:06 SRV-EXCH01_1-000002-ctk.vmdk
    41944004 8192 -rw------- 1 root root 7700992 Nov 9 15:40 SRV-EXCH01_1-ctk.vmdk
    33555396 466540544 -rw------- 1 root root 504658657280 Nov 9 13:06 SRV-EXCH01_1-flat.vmdk
    12583876 0 -rw------- 1 root root 518 Nov 9 15:40 SRV-EXCH01_1.vmdk
    71304132 64 -rw-r--r-- 1 root root 61922 Nov 9 12:56 vmware-11.log
    67109828 64 -rw-r--r-- 1 root root 61922 Nov 9 13:00 vmware-12.log
    75498436 1024 -rw-r--r-- 1 root root 271897 Nov 9 13:06 vmware-13.log
    104858564 64 -rw------- 1 root root 60897 Nov 9 13:09 vmware-14.log
    54526916 64 -rw------- 1 root root 60051 Nov 9 13:36 vmware-15.log
    62915524 64 -rw------- 1 root root 60051 Nov 9 13:36 vmware.log

     

    Have nice day

    Thanks

    Attachment(s)

    zip
    SRV-EXCH01.zip   928 B 1 version


  • 6.  RE: File VMDK disappear into sesparse and ctk into ESXi

    Posted Nov 09, 2021 05:50 PM

    Ok, according the output, it seems that the VM is stored on "datastore3_15K" which ran out of disk space due to active snapshots.

    I may be able to recreate some missing files, but I need to know exactly what happened, and what has been done so far in trying to fix this issue! It looks like some files have been deleted, and the configuration (.vmx) file has been modified.
    Please explain as detailed as possible to avoid (more?) data loss.

    André



  • 7.  RE: File VMDK disappear into sesparse and ctk into ESXi

    Posted Nov 09, 2021 06:34 PM

    Ok I try to explain what happened :

    - VM stop with message : There is no more space for virtual ... You might be able to continue this session by freeing ... Choose Retry or Cancel

    - We try to move some file in another datastore: srv-exch01_Snapshot1282.vmem ... and the some srv-exch1-000X ( not a good idee..)

    after we answer Cancel a the question and the file .vswp has automatically deleted

    The VM not restart always, I change the Hard Disk by the file without -00001.vmdk (in attachment before, now).

    The VM restart but with old data... > so I shut down the VM.

    Unfortunately, I don't have a backup from vmx before. But I share you the version the nearest of the correct config

    I remain available if you are missing any information

    Thanks you for your help

    Guillaume

     

     

    Attachment(s)



  • 8.  RE: File VMDK disappear into sesparse and ctk into ESXi

    Posted Nov 09, 2021 06:39 PM

    You say that you've tried to move some files to another datastore.
    Do you still have these files on the other datastore? Which ones (ls -lisa will help).

    André



  • 9.  RE: File VMDK disappear into sesparse and ctk into ESXi

    Posted Nov 09, 2021 06:55 PM

    We try to move some folder... sorry

    But we normally put all the files back under datastore3 (except the snapshot.vsem)
    Maybe an oversight is causing the problem?

     

    In Datastore1

    [root@localhost:/vmfs/volumes/5b967794-a2638582-cb30-20677cd6dd34] ls -lisa
    total 6170752
    4 1024 drwxr-xr-t 1 root root 73728 Nov 8 10:11 .
    73696723225600 0 drwxr-xr-x 1 root root 512 Nov 9 18:45 ..
    4194308 14336 -r-------- 1 root root 14532608 Sep 10 2018 .fbb.sf
    8388612 132096 -r-------- 1 root root 134807552 Sep 10 2018 .fdc.sf
    33554436 263168 -r-------- 1 root root 268632064 Sep 10 2018 .jbc.sf
    25165828 17408 -r-------- 1 root root 16908288 Sep 10 2018 .pb2.sf
    12582916 1024 -r-------- 1 root root 65536 Sep 10 2018 .pbc.sf
    16777220 1049600 -r-------- 1 root root 1074331648 Sep 10 2018 .sbc.sf
    29360132 1024 drwx------ 1 root root 69632 Sep 10 2018 .sdd.sf
    20971524 7168 -r-------- 1 root root 7340032 Sep 10 2018 .vh.sf
    2052 4683776 -rw------- 1 root root 4795998208 Nov 8 10:12 SRV-EXCH01-000002-sesparse.vmdk
    37748740 0 -rw------- 1 root root 386 Nov 8 10:11 SRV-EXCH01-000002.vmdk
    2244 0 -rw------- 1 root root 399 Nov 8 10:10 SRV-EXCH01_1-000002.vmdk

    Datastore2 - racine

    [root@localhost:/vmfs/volumes/5bb4a993-075def54-f42a-20677cd6dd34] ls -lisa
    total 37836480
    4 1024 drwxr-xr-t 1 root root 77824 Nov 9 13:23 .
    1796 14680064 -rw------- 1 root root 15032385536 Nov 3 18:46 SRV-EXCH01-Snapshot1282.vmem
    2180 4096 -rw------- 1 root root 3658257 Nov 3 19:54 SRV-EXCH01-Snapshot1282.vmsn
    3972 64 -rw------- 1 root root 3883 Nov 3 19:50 SRV-EXCH01.vmx
    1540 21611520 -rw------- 1 root root 22129283072 Nov 3 20:00 SRV-EXCH01_1-000001-sesparse.vmdk

    Datastore2 - racine/old
    [root@localhost:/vmfs/volumes/5bb4a993-075def54-f42a-20677cd6dd34/OLD] ls -lisa
    total 1152
    1028 128 drwxr-xr-x 1 root root 73728 Nov 9 14:51 .
    4 1024 drwxr-xr-t 1 root root 77824 Nov 9 13:23 ..
    4197508 0 -rw------- 1 root root 393 Nov 8 10:07 SRV-EXCH01-000001.vmdk
    16780420 0 -rw------- 1 root root 1052 Nov 9 13:07 SRV-EXCH01.vmsd
    3204 0 -rw------- 1 root root 392 Nov 8 10:05 SRV-EXCH01_1-000001.vmdk

    Datastore2 - racine/old2
    [root@localhost:/vmfs/volumes/5bb4a993-075def54-f42a-20677cd6dd34/old_2] ls -lisa
    total 41187456
    1860 128 drwxr-xr-x 1 root root 73728 Nov 9 11:16 .
    4 1024 drwxr-xr-t 1 root root 77824 Nov 9 13:23 ..
    8388932 10146816 -rw------- 1 root root 10389823488 Nov 9 11:18 SRV-EXCH01-000001-sesparse.vmdk
    324 4683776 -rw------- 1 root root 4795998208 Nov 9 10:56 SRV-EXCH01-000002-sesparse.vmdk
    4194628 26355712 -rw------- 1 root root 26987765760 Nov 9 11:06 SRV-EXCH01_1-000001-sesparse.vmdk

    Thanks to your support



  • 10.  RE: File VMDK disappear into sesparse and ctk into ESXi

    Posted Nov 09, 2021 07:49 PM

    Sorry, I don't know if my last post sent ..

    There are some different folders and 2 date (because same problem yesterday but restart OK)

    Datastore1 ( copid 8 November not today)

    [root@localhost:/vmfs/volumes/5b967794-a2638582-cb30-20677cd6dd34] ls -lisa
    total 6170752
    4 1024 drwxr-xr-t 1 root root 73728 Nov 8 10:11 .
    73696723225600 0 drwxr-xr-x 1 root root 512 Nov 9 18:45 ..
    4194308 14336 -r-------- 1 root root 14532608 Sep 10 2018 .fbb.sf
    8388612 132096 -r-------- 1 root root 134807552 Sep 10 2018 .fdc.sf
    33554436 263168 -r-------- 1 root root 268632064 Sep 10 2018 .jbc.sf
    25165828 17408 -r-------- 1 root root 16908288 Sep 10 2018 .pb2.sf
    12582916 1024 -r-------- 1 root root 65536 Sep 10 2018 .pbc.sf
    16777220 1049600 -r-------- 1 root root 1074331648 Sep 10 2018 .sbc.sf
    29360132 1024 drwx------ 1 root root 69632 Sep 10 2018 .sdd.sf
    20971524 7168 -r-------- 1 root root 7340032 Sep 10 2018 .vh.sf
    2052 4683776 -rw------- 1 root root 4795998208 Nov 8 10:12 SRV-EXCH01-000002-sesparse.vmdk
    37748740 0 -rw------- 1 root root 386 Nov 8 10:11 SRV-EXCH01-000002.vmdk
    2244 0 -rw------- 1 root root 399 Nov 8 10:10 SRV-EXCH01_1-000002.vmdk

    Datastore2 (mixed 8 (old copied) or 9 November (last)

    [root@localhost:/vmfs/volumes] cd datastore2_SSD/
    [root@localhost:/vmfs/volumes/5bb4a993-075def54-f42a-20677cd6dd34] ls -lisa
    total 37836480
    4 1024 drwxr-xr-t 1 root root 77824 Nov 9 13:23 .
    73696723225600 0 drwxr-xr-x 1 root root 512 Nov 9 18:48 ..
    4194308 8192 -r-------- 1 root root 7847936 Oct 3 2018 .fbb.sf
    8388612 132096 -r-------- 1 root root 134807552 Oct 3 2018 .fdc.sf
    33554436 263168 -r-------- 1 root root 268632064 Oct 3 2018 .jbc.sf
    25165828 17408 -r-------- 1 root root 16908288 Oct 3 2018 .pb2.sf
    12582916 1024 -r-------- 1 root root 65536 Oct 3 2018 .pbc.sf
    16777220 1049600 -r-------- 1 root root 1074331648 Oct 3 2018 .sbc.sf
    29360132 1024 drwx------ 1 root root 69632 Oct 3 2018 .sdd.sf
    20971524 7168 -r-------- 1 root root 7340032 Oct 3 2018 .vh.sf
    3332 128 drwxr-xr-x 1 root root 73728 Nov 9 14:06 EXCH0__02
    1028 128 drwxr-xr-x 1 root root 73728 Nov 9 14:51 OLD
    516 128 drwxr-xr-x 1 root root 73728 Nov 9 15:14 SRV-EXCH01
    1796 14680064 -rw------- 1 root root 15032385536 Nov 3 18:46 SRV-EXCH01-Snapshot1282.vmem
    2180 4096 -rw------- 1 root root 3658257 Nov 3 19:54 SRV-EXCH01-Snapshot1282.vmsn
    3972 64 -rw------- 1 root root 3883 Nov 3 19:50 SRV-EXCH01.vmx
    1540 21611520 -rw------- 1 root root 22129283072 Nov 3 20:00 SRV-EXCH01_1-000001-sesparse.vmdk
    580 128 drwxr-xr-x 1 root root 81920 Nov 8 22:51 SRV-TS2016
    1860 128 drwxr-xr-x 1 root root 73728 Nov 9 11:16 old_2
    3908 15360 -rw------- 1 root root 15125822 Nov 3 18:40 vmware-4.log
    452 7168 -rw------- 1 root root 6418994 Nov 3 18:41 vmware-5.log
    3716 1024 -rw------- 1 root root 706626 Nov 3 18:41 vmware-6.log
    2372 1024 -rw------- 1 root root 501686 Nov 3 18:41 vmware-7.log
    2308 1024 -rw------- 1 root root 417883 Nov 3 18:41 vmware-8.log

    Datastore2 > Old2 (other copy)

    [root@localhost:/vmfs/volumes/5bb4a993-075def54-f42a-20677cd6dd34/old_2] ls -lisa
    total 41187456
    1860 128 drwxr-xr-x 1 root root 73728 Nov 9 11:16 .
    4 1024 drwxr-xr-t 1 root root 77824 Nov 9 13:23 ..
    8388932 10146816 -rw------- 1 root root 10389823488 Nov 9 11:18 SRV-EXCH01-000001-sesparse.vmdk
    324 4683776 -rw------- 1 root root 4795998208 Nov 9 10:56 SRV-EXCH01-000002-sesparse.vmdk
    4194628 26355712 -rw------- 1 root root 26987765760 Nov 9 11:06 SRV-EXCH01_1-000001-sesparse.vmdk

    Thank you

     



  • 11.  RE: File VMDK disappear into sesparse and ctk into ESXi

    Posted Nov 09, 2021 08:26 PM

    This is like playing a puzzle, but we're getting closer.

    Anyway, what I'm missing is the "SRV-EXCH01_1-000002-sesparse.vmdk" file.
    Can you please check if it's somewhere in one of the sub-directories?

    What I do need to put the remaining pieces together as good as possible are the following files:

    • Datastore1: SRV-EXCH01-000002.vmdk, SRV-EXCH01_1-000002.vmdk
    • Datastore2: SRV-EXCH01.vmx
    • Datastore3: SRV-EXCH01.vmsd

    Please attach these files compressed as a .zip archive in to reply post.

    André



  • 12.  RE: File VMDK disappear into sesparse and ctk into ESXi

    Posted Nov 09, 2021 08:53 PM
      |   view attached

    Yes a little puzzle..

    The file SRV-EXCH01_1-000002-sesparse.vmdk is missing, I didn't find in any datastore...

    In attachment the asked files.

    Guillaume

     

     

    Attachment(s)

    zip
    SRV-EXCH01-V2.zip   2 KB 1 version


  • 13.  RE: File VMDK disappear into sesparse and ctk into ESXi
    Best Answer

    Posted Nov 09, 2021 10:23 PM
      |   view attached

    Ok, with the files we have, here's what we can do (no guarantees though, due to the missing sesparse file)

    As the first step, please ensure that your Veeam backup server, or a Veeam proxy (in case Hot-Add backup is used) does not still have the VM's virtual disks attached due to an unsuccessful backup attempt. In case it has, remove them from the Veeam VMs (do NOT delete them from disk!). Also make sure that the backup does not kick in while preforming the following tasks!

    1. create a sub-directory in the VM's folder
    2. move the following files to the sub-folder: *ctk.vmdk, *.vmsd, *.vmsn, *.log
    3. extract the 7 files from the attached .zip archive, and upload then to the VM's folder
    4. register Dummy1.vmx, and Dummy2.vmx as VMs from the Datatore Browser
    5. right-click the Dummy1 VM, start Consolidate Disks from the Snapshot ... menu, and wait for it to complete
      --> after the task completes, you should have ~27GB free disk space on the datastore
    6. right-click the Dummy2 VM, start Consolidate Disks from the Snapshot ... menu, and wait for it to complete
      --> now all of the 00000x.vmdk files in the VM's folder should be gone (merged into their parent files)
    7. unregister the two Dummy VMs from the inventory (do NOT delete from disk!)
    8. reload the VM from the command line (see https://kb.vmware.com/s/article/1026043)
      vim-cmd vmsvc/getallvms
      vim-cmd vmsvc/reload <vmid>
    9. delete the files Dummy1.vmx, and Dummy2.vmx from the command line
    10. create a VM snapshot, so that the VM's current files won't get modified
    11. power on the VM

    If things work as expected, you may then delete the previously created sub-directory, and also the current VM snapshot.
    Also remember to delete the files that you've copied to other datastores once you don't need them anymore. Make sure that you don't accidentally delete other files/folder too!

    In case you have questions, please feel free to ask.

    André.

    Attachment(s)

    zip
    SRV-EXCH01-Fixes.zip   3 KB 1 version


  • 14.  RE: File VMDK disappear into sesparse and ctk into ESXi

    Posted Nov 10, 2021 09:41 AM

    bluescreen.png

     Hi, the VM starts now but with bluescreen. Do you have any suggestion now ? 

    Thank you



  • 15.  RE: File VMDK disappear into sesparse and ctk into ESXi

    Posted Nov 10, 2021 04:58 PM

    Not sure what exactly causes this. Due to the broken snapshot, and the fact that the VM had been powered on with the base disk, there might be some data corruption in the file system.

    • Did you try to start Windows in safe mode? If safe mode works, try to see whether the Windows Event logs help finding the cause.
    • When was your latest successful backup? In case the current virtual disk cannot be repaired, it might be an option to restore the 60GB system disk from the backup.

    André



  • 16.  RE: File VMDK disappear into sesparse and ctk into ESXi

    Posted Nov 11, 2021 07:43 AM

    Hello,

    Thank you for your support. All it's OK now.

    Have a nice day.

    Guillaume