VDP Root filling up, possible tomcat issue?

[phil_harborne]

Hi All,

Quick background, VDP stopped working around a week ago, restarted and it started up again, then stopped again next day, same again. Got looking at logs and saw a load of disk full errors.

Ran df -u and found the below

df -h

Filesystem            Size  Used Avail Use% Mounted on

/dev/sda2             7.9G  7.9G     0 100% /

udev                  3.9G  180K  3.9G   1% /dev

tmpfs                 3.9G     0  3.9G   0% /dev/shm

/dev/sda1             130M   38M   86M  31% /boot

/dev/sda7             1.5G  571M  825M  41% /var

/dev/sda9              77G   57G   16G  79% /space

/dev/sdb1             1.0T  112G  912G  11% /data01

/dev/sde1             1.0T  113G  911G  12% /data02

/dev/sdf1             1.0T  111G  913G  11% /data03

/dev/sdg1             1.0T  111G  913G  11% /data04

/dev/sdh1             1.0T  109G  916G  11% /data05

/dev/sdi1             1.0T  110G  914G  11% /data06

So I read some articles etc... they all pointed towards proxy logs in /usr/local/avarmaclient/  however I saw no such files there, I carried on found some large tomcat logs in /usr/local/avamar-tomcat-7.0.54/webapps/ROOT/logBundle/

I moved (to /space) 1 to clear 2.5g and then restarted emwebapp.sh and was showing as 60% fullish. Back ups worked fine the next day but then died the day after, checked the disk and full again!

So I moved the other file and cleared 3g this time, restarted everything and it worked fine the night.

Came in today and guess what, full again! Checked the /usr/local/avamar-tomcat-7.0.54/webapps/ROOT/logBundle/ location and it went from 1mb of log files to 3gb over night!

Checked some log files and seeing a lot of these errors -

org.apache.catalina.loader.WebappClassLoader clearReferencesThreads

SEVERE: The web application [/vdp-configure] appears to have started a thread named [pool-12-thread-2] but has failed to stop it. This is very likely to create a memory leak.

Can anyone help with pinpointing why the logs are filling up so quickly?

[RyanJMN]

When I run into space issues in the system volumes I follow the instructions on the 6.0 admin guide on how to free up space before upgrade (page 37).  Although enough changed in VDP 6.1 that the instructions don't really work for that version.  6.1 also as twice the space for the sytem vmdk so your less likely to have root fill up.

http://pubs.vmware.com/vsphere-60/topic/com.vmware.ICbase/PDF/vmware-data-protection-administration-guide-60.pdf

Instructions includes stopping enwebapp.sh and etc/init.d/apache2 before deleting log files.  You only mention enwebapp.sh.  Possibly thats related to your issue.  Or you just need to delete the other logs mentioned in the admin guide.

[phil_harborne]

Thanks for the suggestion, I followed the guide but removed 0% space.

I have gotten one of the 3g tomcat files off the server, had to repair it as it was corrupt, but looking at it, it just looks like a copy of load of data, including the /var /usr and /data partitions.

Seems a bit weird to copy stuff off.