/storage/archive 100% usage

[cberger77]

Good morning,

/storage/archive is 100% full on a vcenter 6.7 appliance

How to free some space on this partition ?

[IRIX201110141]

Welcome to the club..... its quite often that a VCSA partition runs out of free space. :smileysad:

cd /storage/archive
du -sh *

to identify the folder which holds most of the data and than go deeper and deeper. But please be carefull because for normal people its /storage/log which runs out of space.

My VCSA 6.5 doesnt have a /storage/archive to i cant tell you whats in and its save to remove.

Regards,

Joerg

[cberger77]

The files are in /storage/archive/vpostgres/

It looks like DB checkpoints.

The older ones are from July !

The last one is named "partial"

[jwdg]

This happened to me a few weeks ago, and after I expanded the drive a couple of times it kept filling! I logged a call with VMware and they informed me that this is actually by design(!). It's an engineering change to use all available space on that disk (it is transaction log archiving); there is apparently automatic housekeeping that will remove the oldest files but not before the health status of the appliance is flagged as critical and our own in-house monitoring complains...

I was told a KB article is in preparation for this and a fix is possibly coming in 6.7 U2. We just stopped monitoring space on that disk and all remains well!

John

[MikeStoica]

Check this KB VMware Knowledge Base

[bryanvaneeden]

We're in the same boat, our /storage/archive is also full. Has the KB been released yet?

[mtomczak]

Yes - there is VMware Knowledge Base

[JWolberg82]

Unfortunately, I found out about this after increasing the partition a few times and now I have 270GB of *.gz files in this partition.  Are these safe to delete until the fix in U2 is put into place?  Suggestions are welcome.

[GayathriS]

I dont see a supported/recommended way to manually clear or delete files from storage/archive.

Because as it is stated in article: it is aimed at storing as much WAL history as possible, and is automatically cleaned up by the archiver service by removing automatically the oldest WAL segments.

Please consider marking this answer as correct and helpful if this helps you.

regards

GS

[bryanvaneeden]

But what if it isn't automatically cleaning them up :smileyhappy:?

[lukaslang]

The KB mentioned above was updated. U1b has the fix included

[VCenterMyLove]

I have upgraded my Vcenter 6.5 directly to 6.7U1b, and I do experience that issue.

95% of disk usage with files  in /storage/archive/vpostgres and dated from 2019-02-01 to 2019-02-11

[tdubb123]

I am seeing same thing my /storage/archive partition is completely full

I am on 6.7 U1b build 11727113

any idea if I can delete these files?

[IRIX201110141]

Its by design that this /storage/archive  runs up to 100% and than older files are deleted to make place for new ones. The "bug" which exists in 6.7 was that VAMI reports an full filesystem. This is fixed in 6.7u2 i think.

Regards

Joerg

[lingstrom]

I moved them to a dir and after 1 h deleted them.Everything still works fine  : )

[msripada]

BTW, This issue is resolved in 6.7 Update 1b as per the release notes.

• You might see health status warnings for low space in the archive partitionYou might see health status warnings in the Summary tab of the vCenter Server Appliance interface such as File system /storage/archive is low on storage space. Increase the size of disk /storage/archive. You might also see such warnings in the node status in the system configuration settings when using the vSphere Client. System health checks trigger the warning when the partition exceeds the limits specified in the statsMonitor.xml file, but the warning is not indicating a problem, since the archive service automatically cleans old segments, such as WAL history.This issue is resolved in this release.

Thanks,

MS

[lingstrom]

it did not clean the old segments  ;  )  Thats why.    (update2 by the way....)

[GayathriS]

Hi

This doesnt look like a bug however as per design looks like:

The filesystem is intentionally filled up.

We should not run the disk usage check on /storage/archive since it can and will run at 100% full per design.

Please mark this answer as correct and helpfull if this helps you.

regards

Gayathri

[qklx]

find this:

Solution:

1. SSH into the vCSA and login as root and execute the “shell” command to get shell access.
2. Run the “df -h” command and verify the “/storage/archive” mount is at 90+% of use.
3. Access the vCenter Server that the vCSA instance is running on and increase Disk 13 of the vCSA VM Hardware by a significant amount.
4. In the SSH session to the vCSA, run the autogrow script “/usr/lib/applmgmt/support/scripts/autogrow.sh” .
5. Run the “df -h” command and verify the “/storage/archive” mount Use percentage has decreased.
6. After some time has passed, verify the vSphere Client System Configuration Node Health is “Good” .
7. Verify the vCenter Server Appliance VAMI Health Status for Storage is “Good” .

It's ok.

[AScera]

Hello. If, this is the immediate solution, increase the space on the file system; after you must add fix for vCenter Server 6.7 Update 1b,

https://kb.vmware.com/s/article/57829

https://blog.ganser.com/increase-disk-space-on-vcsa-6-7/

Regards, RA

[Raj1988]

The /storage/archive partition can be full by design, as it is aimed at storing as much WAL history as possible, and is automatically cleaned up by the archiver service by removing automatically the oldest WAL segments.

Note:- These files can be removed manually but not recommended as it will be automatically done by the archiver service. In any case it will not cause any issues to VCSA functionality. Also do not increase any partition on VCSA. VMware does not recommend increasing VCSA partition.

WAL data is Write-Ahead logging for the Postgres DB - transaction logging essentially. It should auto-rotate and engineering has designed the partition to fill up to maximize transaction history. The fix was to exclude this partition from the health alerting. It will still fill up but won't throw an alarm.

This issue is resolved in vCenter Server 6.7 Update 1b. As per the fix, we should not get warnings in the Health Status portion of the Summary Tab in the VAMI.

https://kb.vmware.com/s/article/57829

[SolidParkMagnus]

Meanwhile vROPS keeps generating critical alerts like crazy... :smileysad:

Is there a way to create an ignore rule per individual disk on a specific VM in vROPS?

[Raj1988]

Try to suspend the alert on vrops. Check https://bluemedora.com/disabling-alerts-safely-in-vmware-vrealize-operations-manager/

Using vRealize Operations Policies to Disable Alerts on Certain Objects - VMware Cloud Management

Regards,

AJ

[SwanseaUni]

if you find out let me know ! Cant believe they release this (profanity removed by moderator) into production like this !

[Makovar]

Its normal

See VMware Knowledge Base  57829

[vmwaresucksatre]

Look i get the message that its by design and not harmful, but my nimble SAN appliance is alerting me multiple times daily about this partition on this VM as being full.

I can ignore the messages, but damn vmware that is shit design. Why dont you configure it to use 79% space and not give everyone a hard time here. Or maybe someone can tell me how to tweak the garbage cleanup script to run at like 70% as opposed to 100%. I am on vmware version 6.7.0.32000 of the vcenter appliance. Will updating it help? I see there is 6.7.0.40000 available. i would guess no, but correct me if i am wrong.

EDIT: if you are in the same situation, i found a funny hack work around. Simply set the alert warning to 100% of free space, AND THEN increase the datastore size by 1gb. But dont grow the disk :smileyhappy: so then it will never be able to reach 100% because the actual physical disk is larger than the partition!

[NathanosBlightc]

As the VMware said here It's fixed in the VCSA 6.7 U1b

The /storage/archive partition can be full by design, as it is aimed at storing as much WAL history as possible, and is automatically cleaned up by the archiver service by removing automatically the oldest WAL segments.

However you can run ls -lrth in the /storage/archive/vpostgres and remove ( run rm) very old files

[aleex42]

However: to prevent this from happening again, I've written some Nagios / Icinga checks for the vCenter Server Appliance:

https://blog.krogloth.de/vmware-vcenter-server-appliance-monitoring/

Maybe some finds them useful :-)

[NarendarKT]

Hi ,

• The VAMI page shows the warning message:

File system /storage/archive is low on storage space. Increase the size of disk /storage/archive.

This is a Known warning till 6.7 it fixed in 6.7U1b and above.

This Warning is Harmless as it is by Design to full the file system so we can igonore it.

Please refer the below KB for more details :

VMware Knowledge Base : KB 57829

[vmwaresucksatre]

Sorry i am having trouble reading between the lines here. How do i resolve this?

I woke up this AM to my vcenter telling me "Root Disk Exhaustion" which is not a great message.

Then i look at my disk is full, same as you all.

IF this is by design, why is vcenter alerting me?

I am WAY passed u1. i am on vcenter version 6.7.0.44000 which apparently is 6.7 U3g (LATEST)

I had just upgraded the vcenter this week and did not have this problem before that.

how do i resolve, can someone please post the steps that work for you? is it a database in gzip format? i assume if i delete a bunch of database files my database stops working. i have 14M left of the device.

EDIT: reading over the replies, i already had this problem! haha... and fixed the nimble alert portion of it, but now the error has infected my vmware console it seems...

[IRIX201110141]

If you read all the post you should notice that this is some kind of circulation logging and in your up to date vCenter version you shouldnt get an alarm anymore.  Are you sure that the alarm is related to /storage/archive?

Regards,
Joerg

[NathanosBlightc]

If you ensure your problem is related to the VCSA disk space, please read my posts about how to fix your problem:

VCSA Low Disk Space Problem - Part 1

VCSA Low Disk Space Problem - Part 2

[vmwaresucksatre]

I am pretty sure its that partition as none of the others are as full.

[cjabates]

I'm running 6.7 Update 3j on my vCSA, and I can tell you that it's not eating its tail.

Which causes the services all to fail.

Which, in this particular instance, means my userbase can't log into their workstations (VDI).

[vmkfix-SSA]

 

This article describes how to clear space on partition being full on vCenter Appliance

I recently encountered the issue.  /storage/core full got full while checking on vCenter appliance. even we get alert on vCenter alerts.

Following steps to clear the space:-

> Login vCenter appliance putty session.

>  Run the command df -h to get details use of partitions/files.

# df-h

> Change the directory and /storage/core.

# root@vc-01#cd /storage/archive

 

https://vmkfix.blogspot.com/2023/11/vcenter-storagearchive-being-full-on.html 

[oyahyaoglu]

7.0 and later, this partition is controlled by vcenter, so you don't need to take action manually