vSphere Update Manager Error code 10

[jodykw82]

Not sure why but I can't seem to patch some of my hosts.  I was able to patch it in the past and nothing has changed but I keep getting this error.  Anybody know how to resolve this?

Remediate entity

esxserver1.mydomain.com

The host returns esxupdate error code:10. Cannot create, write or read a

file as expected. Check the Update Manager log files and esxupdate log

files for more details.

DOMAIN\username

VCENTERHOST.mydomain.com

5/15/2015 12:02:33 PM

5/15/2015 12:02:33 PM

5/15/2015 12:02:38 PM

[RyanH84]

Hi,

Have you checked the update manager log and the esxupdate log for the extra information? You'll need to SSH onto the host (or SCP) and grab it from /var/log/esxupdate.log

If you can post them here then we can take a look and see!

I've found a blog that has stated that rebooting the host and trying to remediate again fixed the issue, are you able to try that?

[jodykw82]

I've attached the file you mentioned....

[jodykw82]

Also, wanted to mention this is vSphere 5.5.

[RyanH84]

Ok, well after taking a look at the log I can see that there is the following:

2015-05-15T16:47:36Z esxupdate: esxupdate: ERROR: IOError: [Errno 28] No space left on device

My first check now would be to look at the host and see what the free space is on the service console partitions. On the same host you got the log from, run the command:

df -h

Can you post the output here?

[jodykw82]

Which volume does it need to write to?

~ # df -h

Filesystem   Size   Used Available Use% Mounted on

VMFS-5       9.8G 881.0M      8.9G   9% /vmfs/volumes/SDR-vSphere-SRM-PH  (shared storage)

VMFS-5       1.8T  21.3G      1.8T   1% /vmfs/volumes/SDR-ESX01R-LocalDisk

VMFS-5     499.8G  61.5G    438.2G  12% /vmfs/volumes/SDR-ESX-ISOs (shared storage)

VMFS-5       4.0T  69.1G      3.9T   2% /vmfs/volumes/SDR-ESX-VMFS02-NoReplication (shared storage)

VMFS-5       4.0T   1.8T      2.2T  45% /vmfs/volumes/SDR-ESX-VMFS01-NoReplication (shared storage)

vfat       249.7M 164.3M     85.4M  66% /vmfs/volumes/5c421de1-7c870757-1c93-6142de868aa5

vfat       249.7M 165.4M     84.3M  66% /vmfs/volumes/281910e1-03b7cc92-2a9b-4fdc1c144506

vfat       285.8M 193.4M     92.4M  68% /vmfs/volumes/53d0f05d-3c8bbbaa-90f5-c81f66ea53fe

~ #

[RyanH84]

Hi,

Can you also try:

df -h /tmp

It looks like your vfat partitions are very small. Can you also output:

ls -al /scratch

My gut feeling is that your host doesn't have enough space for the patches to be downloaded to, to then be installed from. You can specify a way of using another datastore for your scratch partition. You could check this KB article which describes the process of changing the location of your scratch. You have a nice big old local VMFS volume that you could use instead. A host reboot would be needed though!

[jodykw82]

df -h /temp came back with the same information as before....

The second command came back with this....

~ # ls -al /scratch

lrwxrwxrwx    1 root     root            57 Feb 18 16:27 /scratch -> /vmfs/volumes/54323d86-26c4820f-b71e-c81f66ea53fe/.locker

~ #

[RyanH84]

I updated my post above with a KB article, you got back to me too quick :smileyhappy:

Specifically the part about running out of space and possibly changing your scratch location to a local VMFS volume on the host.

[jodykw82]

So I checked the scratch config and it is going here....

/vmfs/volumes/54323d86-26c4820f-b71e-c81f66ea53fe/.locker   (this is a local datastore btw)

There is 1.8 TB of free space on that Datastore....  Could it be trying to write somewhere else?

[RyanH84]

Are you saying that 54323d86-26c4820f-b71e-c81f66ea53fe is the VMFS volume SDR-ESX01R-LocalDisk?

If you:

cd /var/tmp

Does that put you in: /vmfs/volumes/54323d86-26c4820f-b71e-c81f66ea53fe/var/tmp

[jodykw82]

~ # cd /var/tmp

/vmfs/volumes/54323d86-26c4820f-b71e-c81f66ea53fe/.locker/var/tmp #

/vmfs/volumes/54323d86-26c4820f-b71e-c81f66ea53fe/

is

SDR-ESX01R-LocalDisk

[RyanH84]

Ok, now what if you try:

vdf -h

I'm interested in the bottom part of the output, Ramdisk, root, tmp, etc.

[jodykw82]

/vmfs/volumes/54323d86-26c4820f-b71e-c81f66ea53fe/.locker/var/tmp # vdf -h

Tardisk                  Space      Used

sb.v00                    148M      148M

s.v00                     295M      295M

misc_cni.v00               24K       21K

net_bnx2.v00              300K      298K

net_bnx2.v01                1M        1M

net_cnic.v00              136K      132K

net_tg3.v00               292K      289K

scsi_bnx.v00              264K      262K

scsi_bnx.v01              196K      192K

dell_eql.v00               11M       11M

dell_eql.v01               16K       13K

dell_eql.v02              136K      133K

elxnet.v00                280K      276K

ima_be2i.v00                1M        1M

lpfc.v00                    1M        1M

scsi_be2.v00              664K      660K

net_ixgb.v00              432K      429K

scsi_mpt.v00              464K      461K

ima_qla4.v00                4M        4M

net_qlcn.v00                1M        1M

qlnative.v00                2M        2M

scsi_qla.v00              508K      504K

dvfilter.v00              688K      687K

ata_pata.v00               40K       39K

ata_pata.v01               28K       27K

ata_pata.v02               32K       30K

ata_pata.v03               32K       30K

ata_pata.v04               36K       35K

ata_pata.v05               32K       31K

ata_pata.v06               28K       27K

ata_pata.v07               36K       32K

block_cc.v00               80K       77K

ehci_ehc.v00               92K       91K

epsec_mu.v00              300K      297K

weaselin.t00               14M       14M

esx_dvfi.v00              404K      401K

xlibs.v00                   1M        1M

ipmi_ipm.v00               40K       38K

ipmi_ipm.v01               88K       87K

ipmi_ipm.v02              100K       97K

lsi_mr3.v00               180K      178K

lsi_msgp.v00              364K      363K

misc_dri.v00                4M        4M

mtip32xx.v00              180K      176K

net_e100.v00              288K      286K

net_e100.v01              232K      230K

net_enic.v00              132K      130K

net_forc.v00              120K      117K

net_igb.v00               296K      293K

net_mlx4.v00              332K      328K

net_mlx4.v01              224K      220K

net_nx_n.v00                1M        1M

net_qlge.v00              340K      336K

net_vmxn.v00              100K       98K

ohci_usb.v00               60K       58K

rste.v00                  740K      737K

sata_ahc.v00               80K       77K

sata_ata.v00               56K       52K

sata_sat.v00               60K       59K

sata_sat.v01               44K       40K

sata_sat.v02               44K       40K

sata_sat.v03               36K       32K

sata_sat.v04               32K       28K

scsi_aac.v00              168K      164K

scsi_adp.v00              412K      409K

scsi_aic.v00              280K      278K

scsi_fni.v00              160K      157K

scsi_hps.v00              164K      160K

scsi_ips.v00               96K       93K

scsi_meg.v00               92K       91K

scsi_meg.v01              164K      160K

scsi_meg.v02               88K       87K

scsi_mpt.v01              500K      497K

scsi_mpt.v02              420K      418K

scsi_qla.v01                1M        1M

uhci_usb.v00               60K       57K

xhci_xhc.v00              168K      164K

dell_con.v00                4K        3K

xorg.v00                    3M        3M

imgdb.tgz                 372K      370K

state.tgz                  40K       37K

-----

Ramdisk                   Size      Used Available Use% Mounted on

root                       32M      660K       31M   2% --

etc                        28M      324K       27M   1% --

tmp                       192M      192M        0B 100% --

hostdstats                803M        5M      797M   0% --

snmptraps                   1M        0B        1M   0% --

[RyanH84]

Hi!

That is a bingo! :smileyhappy:

tmp

192M

192M

0B 100% --

Your tmp directory is full! I'm guessing you are applying a fair few patches?

If you do:

ls -la /tmp

What do you get?

[jodykw82]

This is what it comes back with.....

~ # ls -la /tmp

total 191720

drwxrwxrwt    1 root     root           512 May 15 21:04 .

drwxr-xr-x    1 root     root           512 May 15 16:50 ..

-rw-------    1 root     root            36 May 15 21:05 probe.session

-rw-r--r--    1 root     root     186411672 May 15 21:04 ql_ima.log

-rw-r--r--    1 root     root       4894152 May 15 21:04 ql_ima_sdm.log

-rw-r--r--    1 root     root       5000089 May 15 06:44 ql_ima_sdm.log_old

-rw-r--r--    1 root     root             0 May 15 21:04 snmpd-cvtcimsnmp.xml

-rw-------    1 root     root             0 May 15 19:54 uXypwi

~ #

How can I find out what volume that tmp directory is on?

[RyanH84]

Ok so you have three logs:

-rw-r--r--    1 root     root     186411672 May 15 21:04 ql_ima.log

-rw-r--r--    1 root     root       4894152 May 15 21:04 ql_ima_sdm.log

-rw-r--r--    1 root     root       5000089 May 15 06:44 ql_ima_sdm.log_old

That are filling up all of your space. Specifically the ql_ima.log.

To be honest, I'd be tempted to keep them by doing the following:

mv /tmp /tmp.old   (Keep your files by changing the temp directory to tmp.old

mkdir /tmp      (Creates a new /tmp directory)

services.sh     (restart host services and check it populates with some new files)

vdf -h     (Check that the /tmp is reporting as low)

Try your updates again!

[jodykw82]

That makes sense.  Let me try that and see what happens.

[jodykw82]

I renamed tmp to tmp.old and I created a new tmp directory and restarted services.

I'm running the patches now and so far so good, it's actually patching now.

However, I'm still a little hazy on where that tmp directory actually is?  How can I find that out?

[RyanH84]

Cool, sounds promising. The /tmp directory hangs off root "/" and is 192MB. It is part of a hard coded RamDisk in ESXi.

[jodykw82]

So I guess it's not really possible to increase the size?

[RyanH84]

I do not believe so no!

How are those patches coming along?

[jodykw82]

Patches finished on that host!  I checked the tmp size on my other host that was giving the same error and it was at 99%.  I just did the same to that one and I'm about to try to install patches again.

[RyanH84]

Sounds like great news! In time you could login and delete those /tmp.old folders if you didn't want the logs!

If you can mark the answer as complete when you are happy, that'd be cool :smileyhappy:

[jodykw82]

Thanks!

[brunofernandez1]

It seems as you don't have enought free space on your esxi server. have a look how much free space you have on /tmp

2015-05-15T16:42:59Z esxupdate: esxupdate: ERROR: An esxupdate error exception was caught:
2015-05-15T16:42:59Z esxupdate: esxupdate: ERROR: Traceback (most recent call last):
2015-05-15T16:42:59Z esxupdate: esxupdate: ERROR:   File "/usr/sbin/esxupdate", line 216, in main
2015-05-15T16:42:59Z esxupdate: esxupdate: ERROR:     cmd.Run()
2015-05-15T16:42:59Z esxupdate: esxupdate: ERROR:   File "/build/mts/release/bora-2302651/bora/build/esx/release/vmvisor/sys-boot/lib/python2.6/site-packages/vmware/esx5update/Cmdline.py", line 106, in Run
2015-05-15T16:42:59Z esxupdate: esxupdate: ERROR:   File "/build/mts/release/bora-2302651/bora/build/esx/release/vmvisor/sys-boot/lib/python2.6/site-packages/vmware/esximage/Transaction.py", line 69, in DownloadMetadatas
2015-05-15T16:42:59Z esxupdate: esxupdate: ERROR:   File "/build/mts/release/bora-2302651/bora/build/esx/release/vmvisor/sys-boot/lib/python2.6/site-packages/vmware/esximage/Downloader.py", line 268, in Get
2015-05-15T16:42:59Z esxupdate: esxupdate: ERROR:   File "/build/mts/release/bora-2302651/bora/build/esx/release/vmvisor/sys-boot/lib/python2.6/site-packages/vmware/esximage/Downloader.py", line 170, in _getfromurl
2015-05-15T16:42:59Z esxupdate: esxupdate: ERROR:   File "/build/mts/release/bora-2302651/bora/build/esx/release/vmvisor/sys-boot/lib/python2.6/site-packages/urlgrabber/grabber.py", line 927, in urlgrab
2015-05-15T16:42:59Z esxupdate: esxupdate: ERROR:   File "/build/mts/release/bora-2302651/bora/build/esx/release/vmvisor/sys-boot/lib/python2.6/site-packages/urlgrabber/grabber.py", line 845, in _retry
2015-05-15T16:42:59Z esxupdate: esxupdate: ERROR:   File "/build/mts/release/bora-2302651/bora/build/esx/release/vmvisor/sys-boot/lib/python2.6/site-packages/urlgrabber/grabber.py", line 915, in retryfunc
2015-05-15T16:42:59Z esxupdate: esxupdate: ERROR:   File "/build/mts/release/bora-2302651/bora/build/esx/release/vmvisor/sys-boot/lib/python2.6/site-packages/urlgrabber/grabber.py", line 1219, in _do_grab
2015-05-15T16:42:59Z esxupdate: esxupdate: ERROR: IOError: [Errno 28] No space left on device