ESXi Ramdisk Full - [] directory

[Simon Rutherford]

Hi

I hope someone can help as a Google Search is proving fruitless with this.

I have a number of vSphere ESXi hosts that have, as of this morning, started running out of space on the root ramdisk; a vdf -f returns the following:

Ramdisk                   Size      Used Available Use% Mounted on

root                       32M       32M        0B 100% --

etc                        28M      472K       27M   1% --

tmp                       192M       70M      121M  36% --

hostdstats                639M       18M      620M   2% --

Now, I've checked the obvious stuff.  We do not have SNMP enabled on the hosts, so KB2033073 doesn't apply - there is no SNMP directory.  Also, as these are Dell blades, we don't have the issue with the HP Helper log growing out of control as described in KB2040707.  Alongside this I checked all of the obvious locations such as /tmp and /var/log for anything else that looked out of place and couldn't see anything.

I have ran a quick du -h -d 1 on a number of hosts in the same cluster.  The ones that are full are showing the following result:

9.8M    ./etc

448.3M  ./tardisks

4.0K    ./tardisks.noauto

41.9M   ./sbin

17.6M   ./proc

84.3M   ./tmp

140.5M  ./lib

58.2M   ./opt

237.0T  ./dev

14.2M   ./lib64

12.0K   ./vmimages

110.7M  ./usr

62.6M   ./bin

31.0M   ./var

50.7T   ./vmfs

116.0K  ./mbr

4.0K    ./.vmware

29.2M   ./[]

287.7T  .

So, in a round about way, my question is what is the directory listed as /[] (in bold above) and is there anyway I can clear this down without rebooting the host?  All of the servers have the [] directory, however, on a functioning server it hovers around 9MB whereas it sits around 29MB on all affected hosts.  I've not come across this directory before and although an ls -al lists it as a directory:

drwxr-xr-x    1 root     root           512 May  1 15:31 []

Any attempt to cd into it returns the following - That is trying it just as cd [] and cd \[\]:

-sh: cd: can't cd to []

I've checked the ESXi documentation and can't find mention of this particular directory.  We have no third party extensions installed and are running ESXI 5.1.0, 1612806.  If anyone can point me in the right direction it would be much appreciated.

Regards

Simon

[Simon Rutherford]

I'm not sure if this is conclusive or not (I only have one host that boots from an HDD), but this [] directory only seems to appear on hosts that boot from SD Cards as opposed to from those machines with HDDs.  I don't think this should make a difference, but it is one of the few differences between the two hosts.  Any ideas?

[MKguy]

I have HP servers on ESXi 5.1 booting from SD cards and they don't have that /[] directory. Not really sure what's going on there.

Can you also post the output of an esxcli system visorfs ramdisk list ?

Check if you see any file lock activity with lsof  | grep "\[\]" .

Restarting management agents is also worth a try.

You may be able to add a temporary additional ramdisk via esxcli system visorfs ramdisk add but I've never tried this.

[Simon Rutherford]

Hi,

The output of the esxcli command is as follows:

~ # esxcli system visorfs ramdisk list

Ramdisk Name  System   Reserved     Maximum       Used  Peak Used  Free  Reserved Free  Maximum Inodes  Allocated Inodes  Used Inodes  Mount Point

------------  ------  ---------  ----------  ---------  ---------  ----  -------------  --------------  ----------------  -----------  ---------------------------

root            true  32768 KiB   32768 KiB  32764 KiB  32768 KiB   0 %            0 %            8192              4096         2819  /

etc             true  28672 KiB   28672 KiB    420 KiB    628 KiB  98 %           98 %            4096              1024          434  /etc

tmp            false   2048 KiB  196608 KiB      4 KiB   1112 KiB  99 %           99 %            8192               256            7  /tmp

hostdstats     false      0 KiB  654336 KiB  18524 KiB  18528 KiB  97 %            0 %            8192                32            4  /var/lib/vmware/hostd/stats

I ran to check for file locks as well and it returns no results.  I'm going to see if I can try adding a temporary ramdisk to one of our test hosts and, if successful, I'll give it a shot on one of the ones with the issue.  I'm hesistant to do it directly on a "live" host because I'm unable to vMotion the running guests off them when the / partition is full.

[Simon Rutherford]

So, I don't know what has caused it, but I have found a way to workaround the issue and get my hosts working again!

Although I can't interact with the directory in any way through the console or ssh, it turns out that I can through SCP.  I connected into the host with winSCP and could go into the directory.  It appears that it is being used as a Scratch location.  The current scratch location listed in the host advanced settings is configured as /tmp/scratch, however, this is the normal size that I would expect to see this.  I went through, moved the folder off to my local machine and deleted the [] directory through winSCP.  After that, the host was back to having the amount of space that I would expect and all functions now perform correctly.

Thanks for you help with this.