There is no more space for virtual disk ...

[johnzapf]

I'm having this same issue. 22 terabytes and it says it's full. I've got it running two times and went in and deleted 7-9 terabytes of data so far and it still says its 100% full, I don't understand.

[johnzapf]

I don't understand it seems to be filling up as fast as I'm deleting data I am lost at what's going on?

[johnzapf]

and when I do it the vm on windows shows there is 11.2 TB's free. so why does esxi think its full?

 

[johnzapf]

There is no more space for virtual disk '/vmfs/volumes/5fc12055-ee0336a3-dfa1-b8ca3a6f7dfc/STORAGE/AIBS2019/AIBS2019_1-000001.vmdk'. You might be able to continue this session by freeing disk space on the relevant volume, and clicking Retry. Click Cancel to terminate this session

And I know there is fourteen terabytes free on that drive I don't know what's going on!

[a_p_]

It looks like you ran out of disk space on the datastore, due to an active snapshot.

Please run the following commands from the ESXi host's command line and post the commands' output.

cd /vmfs/volumes/5fc12055-ee0336a3-dfa1-b8ca3a6f7dfc/STORAGE/AIBS2019
df -h
ls -lisa

André

 

[johnzapf]

 

The ESXi Shell can be disabled by an administrative user. See the
vSphere Security documentation for more information.
[root@ESXi-SERVER:~] cd /vmfs/volumes/5fc12055-ee0336a3-dfa1-b8ca3a6f7dfc/STORAGE/AIBS2019
[root@ESXi-SERVER:/vmfs/volumes/5fc12055-ee0336a3-dfa1-b8ca3a6f7dfc/STORAGE/AIBS2019] df -h
Filesystem Size Used Available Use% Mounted on
VMFS-6 7.3T 4.3T 3.0T 59% /vmfs/volumes/8TB
VMFS-6 21.8T 21.8T 0.0B 100% /vmfs/volumes/23TB
VMFS-6 237.8G 59.7G 178.1G 25% /vmfs/volumes/256-SSD
VMFS-6 109.8G 1.4G 108.3G 1% /vmfs/volumes/ESXi-HOST-SSD
VMFSOS 119.8G 4.0G 115.8G 3% /vmfs/volumes/OSDATA-5fc10f7f-a6bbffdf-c4cf-b8ca3a6f7dfc
vfat 4.0G 279.6M 3.7G 7% /vmfs/volumes/BOOTBANK1
vfat 4.0G 204.6M 3.8G 5% /vmfs/volumes/BOOTBANK2
[root@ESXi-SERVER:/vmfs/volumes/5fc12055-ee0336a3-dfa1-b8ca3a6f7dfc/STORAGE/AIBS2019] ls -lisa
total 23434965248
167772164 128 drwxr-xr-x 1 root root 73728 Feb 19 18:23 .
37748740 128 drwxr-xr-x 1 root root 73728 Nov 30 2020 ..
469765380 6144 -rw------- 1 root root 5636608 Feb 19 18:23 AIBS2019_1-000001-ctk.vmdk
461376772 349503488 -rw------- 1 root root 446743482368 Feb 19 19:20 AIBS2019_1-000001-sesparse.vmdk
465571076 0 -rw------- 1 root root 388 Feb 19 18:23 AIBS2019_1-000001.vmdk
452988164 6144 -rw------- 1 root root 5636608 Feb 19 11:05 AIBS2019_1-ctk.vmdk
171966468 23085449216 -rw------- 1 root root 23639499997184 Feb 19 11:30 AIBS2019_1-flat.vmdk
176160772 0 -rw------- 1 root root 572 Feb 19 06:01 AIBS2019_1.vmdk
[root@ESXi-SERVER:/vmfs/volumes/5fc12055-ee0336a3-dfa1-b8ca3a6f7dfc/STORAGE/AIBS2019]

 

[johnzapf]

This is crazy, the drive is not full.

 

This is what is actually being used on that drive. 

I don't think it's taking a snapshot. There's no snapshots on any of the servers to delete. All snapshots are empty.

That server just shut down again there's no room left and I don't understand why... This is killing me i've spent 4 hours on this so far today I need to get this back up and running. Anybody has any idea please let me know thank you.

 

[a_p_]

Your guest OS may report free disk space, but the datastore ran out of disk space.

You've created a ~21.5 TB virtual disk on a ~21.8 TB datastore. leaving only ~300 GB free disk space.
That disk space is now used by the snapshot.
Please remember that once a snapshot is created, all new writes are stored in that snapshot, i.e. it can grow up to the provisioned size of its base disk.

What you could try - not sure whether it will work with almost no free disk space - is to delete the snapshot from the Snapshot Manager while the VM is powered off.

André

[johnzapf]

 

there is no snapshot that I can see. And this has been running like this for 5 years now not sure why it has been filling up like this?

[a_p_]

There is a snapshot, even though it does not show up in the Snapshot Manager for whatever reason.

469765380 6144 -rw------- 1 root root 5636608 Feb 19 18:23 AIBS2019_1-000001-ctk.vmdk
461376772 349503488 -rw------- 1 root root 446743482368 Feb 19 19:20 AIBS2019_1-000001-sesparse.vmdk
465571076 0 -rw------- 1 root root 388 Feb 19 18:23 AIBS2019_1-000001.vmdk
452988164 6144 -rw------- 1 root root 5636608 Feb 19 11:05 AIBS2019_1-ctk.vmdk
171966468 23085449216 -rw------- 1 root root 23639499997184 Feb 19 11:30 AIBS2019_1-flat.vmdk
176160772 0 -rw------- 1 root root 572 Feb 19 06:01 AIBS2019_1.vmdk

The second line shows the snapshot, and the last line the date/time when the snapshot has (most likely) been created.

The usual way to get rid of such a snapshot would be to create another snapshot, and then click "Delete All" in the Snaphot Manager.
However, with no free disk space this may not work.
Do you see an option "Consolidate" in the VM's context menu?

André

[johnzapf]

yes that option is there, But it won't run insufficient disk space.

 

[johnzapf]

This makes the most sense. So it must have started doing a snapshot this morning at 6AM and of course it's not going to have room to do that, that drive HAD 14TB's on it. But I don't understand I don't see snapshots under any of my servers I didn't know that was happening. I definitely need to turn that off in the future I do not have room for snapshots. Now we need to figure out how to delete that one stop it and delete it?

And I have been deleting terabytes of data off that drive and it's filling up as fast as I'm deleting the data. Have already removed 7TB's and its full again. 

 

 

[a_p_]

As mentioned before, changes are only written into the snapshot (sesparse) file, so even deleting data from within the guest OS will cause modification that will be written into the sesparse file.

Since you say that you are deleting data, I wonder - in case snapshot consolidation does not work - if it would be an option for you to discard todays changes on that large drive!? Just asking.

André

 

[johnzapf]

Like I said above consolidate doesn't work there's no space. Just need to get this server back UP don't care about the data i've deleted I need to figure out how to stop the snapshot process how to clear it and how to get this server running ASAP. 

 And I need to turn off snapshots so they don't try to do this again. It's now one o'clock in the afternoon I've been working on this for 7 hours I have a head breakfast yet I haven't done anything this is very very very important to get this back up.

 

And I just don't understand what's taking up all this space. I have no snapshots on any of my vm's there should be fourteen terabytes free on this drive. Need to figure out how to get this cleared out and back UP and running and I'm not having any luck. I'm losing a lot of money and a lot of data right now I need to get this back up and running I don't care about the data I've deleted.

And I don't care about snapshots. I have never used one in my life. I have backup software that I can restore things from.

Disk consolidation for VM AIBS2022 has failed: Insufficient disk space on datastore

 

[a_p_]

Just to confirm: It is ok for you to lose todays changes?

If the answer is yes, then what you may do is:

1. shut down the VM (unless already done)
2. delete the following snapshot files in /vmfs/volumes/5fc12055-ee0336a3-dfa1-b8ca3a6f7dfc/STORAGE/AIBS2019
   AIBS2019_1-000001-ctk.vmdk
   AIBS2019_1-000001-sesparse.vmdk
   AIBS2019_1-000001.vmdk
3. edit (using e.g. the vi editor) the VM's configuration (.vmx) file in the VM's base directory, and change the virtual disk's file name
   from: /vmfs/volumes/5fc12055-ee0336a3-dfa1-b8ca3a6f7dfc/STORAGE/AIBS2019/AIBS2019_1-000001.vmdk
   to: /vmfs/volumes/5fc12055-ee0336a3-dfa1-b8ca3a6f7dfc/STORAGE/AIBS2019/AIBS2019_1.vmdk
   (since the VM seems to have multiple virtual disks, make sure that you only change this one!
4. reload the VMX by following the steps in https://kb.vmware.com/s/article/1026043
5. create a new snapshot
6. run "Delete All" from the Snapshot Manager (wait for the process to complete)
7. edit the VM's Hard Disk settings and change the "Diskmode" from "Dependent" to "Independent-Persistent"
   (this will exclude the virtual disks from snapshots)

André

 

[a_p_]

... just in case the VM is still running, you may as well copy today's changed data to e.g. a network shared folder, and copy it back after the above procedure.

André

[a_p_]

WAIT: I just realized that the flat file's time stamp is "11:30".

Did you try to run a snapshot consolidation at that time?

In this case parts of the snapshot may have already been merged into the base disk, and deleting the snapshot by following the steps I mentioned may result in data corruption!

André

 

[johnzapf]

I keep trying to consolidate it won't work there's no space there's absolutely zero space it's using it all.

[johnzapf]

the VM will not run, no space, that is the issue. VM is not running. 

[johnzapf]

The VM is not running. No space it won't start. Don't care about any changes today the machine has not been on today that's the problem that's why I'm here trying to get this fixed so I can get it working again. It's a 22 terabyte drive it should have 17 terabytes free and it says it's 100% full!

 

[johnzapf]

OK I'm working on this: 

where is this?  "edit (using e.g. the vi editor) the VM's configuration (.vmx) file in the VM's base directory"

[johnzapf]

I think I found it. 

[johnzapf]

ok its up, still 100% full so how can I take another snapshot?
Failed to create snapshot 'test' on virtual machine AIBS2022

So what is using all this space. this drive 21TB only has 7TB used. its this log files or what? what is using the other 14TB's?

 

[johnzapf]

So I appreciate everything you just told me to do. Did all that and I have 333 gigs free now but I still need to find out what's using the other 14 terabytes is that log files I don't get it?

It just seems impossible that I've deleted 7 terabytes from this drive today and it's still only showing 333 gigs free.

 

 

This is all the data on Storage so why is it full?

[a_p_]

Let me try to explain the physical disk space usage.

The datastore has a size of approximately 21.8TB.
On this datastore you've created a 21.5TB virtual disk in "thick" mode, which means that a ...-flat.vmdk file has been created with that fixed size, leaving only ~333GB free disk space on the datastore.
Once a snapshot gets created, the ...-flat.vmdk file will then be treated as read-only file, and all subsequent changes will be written to the snapshot (sesparse) file. The more changes, the larger this sesparse file will grow.
After deleting the snapshot, changes will then be written to the flat file again, which has it's fixed size, which will not change.

André

 

[johnzapf]

So I just created a new snapshot it completed and then I deleted it and it still says only 333 gigs.

I don't get it all the other drives in this machine are working fine this one is missing a lot space?

I just need to get the space back. I store terabytes of data on this from cameras daily. I need the space back, I don't know where it went. 

I did everything you said and the vm is now set to "Independent-Persistent"

[johnzapf]

 I just have to get this cleared out. it should be reading 14 terabytes free. I have to use this server i'm going to go out of business if I don't get this fixed ASAP, I've been down a day now i'm losing data I have contracts I can't lose any data i'll be screwed. There is 14 terabytes free on this drive where are they and how do I fix it? If anybody knows please help me out here I gotta get this fixed today I still haven't even stopped for breakfast I gotta get this fixed

 

[a_p_]

The 14TB that you see as free disk space within Windows, are data blocks in the ...-flat.vmdk file.

Think of the ...-flat.vmdk for a virtual machine like a physical disk on a physical system. The size of the disk remains the same, no matter how much data you store on it.

André

[johnzapf]

And I see now That the purple bar showing 21.5 terabytes was the allocated amount. So I understand I use the whole drive but with that said I don't understand why the drive said it was full having 21.5 terabytes when I was only using half of that. That's what I'm still having a hard time figuring out.

 

[johnzapf]

thank you. this was the fix.

[johnzapf]

 And not sure what that is it shows up as the 21 terabytes also.