VMware vSphere

 View Only

  • 1.  ESXi Update: [OSError] [Errno 28] No space left on device

    Posted Jun 23, 2019 03:16 PM

    Hi,

    I am trying to do an update of my ESXi 6.7.0 U2 host with the command

    esxcli software profile update -p ESXi-6.7.0-20190604001-standard \

    -d https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml

    but I'm getting the error message

    [OSError]

    [Errno 28] No space left on device

    Please refer to the log file for more details.

    I have already rebooted the ESXi host in order to free some memory the ramdisks could be using.

    My vdf -h looks like this:

    ...

    -----

    Ramdisk                   Size      Used Available Use% Mounted on

    root                       32M        2M       29M   7% --

    etc                        28M      196K       27M   0% --

    opt                        32M        0B       32M   0% --

    var                        48M      352K       47M   0% --

    tmp                       256M        8K      255M   0% --

    iofilters                  32M        0B       32M   0% --

    shm                      1024M        0B     1024M   0% --

    hostdstats                204M        2M      201M   1% --

    snmptraps                   1M        0B        1M   0% --

    My df -h looks like this:

    Filesystem   Size   Used Available Use% Mounted on

    VMFS-5     223.5G 196.0G     27.5G  88% /vmfs/volumes/0_OCZ_DARC_240GB

    VMFS-5     931.2G 877.9G     53.3G  94% /vmfs/volumes/0_WD_1TB_RED

    VMFS-5     465.5G 461.0G      4.5G  99% /vmfs/volumes/0_SAMS_500GB

    VMFS-5     465.5G 461.0G      4.5G  99% /vmfs/volumes/1_SAMS_500GB

    VMFS-6     465.5G 391.7G     73.8G  84% /vmfs/volumes/SAMS_850EVO_500G_0

    VMFS-6     465.5G 172.4G    293.1G  37% /vmfs/volumes/SAMS_850EVO_500G_1

    VMFS-6     458.2G 175.1G    283.2G  38% /vmfs/volumes/SAMS_850EVO_500G_2_esxiboot

    vfat       249.7M 147.4M    102.3M  59% /vmfs/volumes/ddc4e178-95bc9fd2-2875-f12db19fbedd

    vfat       249.7M 146.8M    102.9M  59% /vmfs/volumes/c8bcc07e-bfadfa2f-dbdd-6da6bb17b2c7

    vfat         4.0G  21.2M      4.0G   1% /vmfs/volumes/5c433dad-57e1863f-b50b-ac1f6b7a4ea2

    vfat       285.8M 172.9M    112.9M  60% /vmfs/volumes/5c433da5-ece0b664-e9b8-ac1f6b7a4ea2

    I have already enable swapping to disk with mountpoint /vmfs/volumes/SAMS_850EVO_500G_1.

    The /var/log/esxupdate.log says:

    2019-06-23T15:08:27Z esxupdate: 2099595: imageprofile: INFO: Adding VIB VMware_locker_tools-light_10.3.5.10430147-12986307 to ImageProfile (Updated) ESXi-6.7.0-20190404001-standard

    2019-06-23T15:08:27Z esxupdate: 2099595: vmware.runcommand: INFO: runcommand called with: args = '['/sbin/esxcfg-advcfg', '-U', 'host-acceptance-level', '-G']', outfile = 'None', returnoutput = 'True', timeout = '30.0'.

    2019-06-23T15:08:28Z esxupdate: 2099595: Transaction: INFO: Downloading esx-update vib from https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/esx/vmw/vib20/esx-update/VMware_bootbank_esx-update_6.7.0-2.60.13981272.vib

    2019-06-23T15:08:28Z esxupdate: 2099595: vmware.runcommand: INFO: runcommand called with: args = 'localcli system visorfs ramdisk add -m 0 -M 20 -n esx-update-2099595 -p 755 -t /tmp/esx-update-2099595', outfile = 'None', returnoutput = 'True', timeout = '0.0'.

    2019-06-23T15:08:28Z esxupdate: 2099595: downloader: INFO: Downloading https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/esx/vmw/vib20/esx-update/VMware_bootbank_esx-update_6.7.0-2.60.13981272.vib to /tmp/esx-update-2099595/VMware_bootbank_esx-update_6.7.0-2.60.13981272

    2019-06-23T15:08:31Z esxupdate: 2099595: root: DEBUG: Verifying VIB VMware_bootbank_esx-update_6.7.0-2.60.13981272 signature #2

    2019-06-23T15:08:31Z esxupdate: 2099595: Ramdisk: INFO: Mount tardisk /tmp/esx-update-2099595/esxupdt-2099595 in ramdisk /tmp/esx-update-2099595

    2019-06-23T15:08:31Z esxupdate: 2099595: vmware.runcommand: INFO: runcommand called with: args = '/usr/lib/vmware/secureboot/bin/secureMount.py ramdiskMount /tmp/esx-update-2099595/VMware_bootbank_esx-update_6.7.0-2.60.13981272 esxupdt /tmp/esx-update-2099595/esxupdt-2099595 esx-update-2099595', outfile = 'None', returnoutput = 'True', timeout = '0.0'.

    2019-06-23T15:08:31Z esxupdate: 2099595: Ramdisk: INFO: Mount tardisk /tmp/esx-update-2099595/weaselin-2099595 in ramdisk /tmp/esx-update-2099595

    2019-06-23T15:08:31Z esxupdate: 2099595: vmware.runcommand: INFO: runcommand called with: args = '/usr/lib/vmware/secureboot/bin/secureMount.py ramdiskMount /tmp/esx-update-2099595/VMware_bootbank_esx-update_6.7.0-2.60.13981272 weaselin /tmp/esx-update-2099595/weaselin-2099595 esx-update-2099595', outfile = 'None', returnoutput = 'True', timeout = '0.0'.

    2019-06-23T15:08:32Z esxupdate: 2099595: Transaction: DEBUG: Added /tmp/esx-update-2099595/usr/lib/vmware and /tmp/esx-update-2099595/lib64/python3.5/site-packages/vmware to sys.path

    2019-06-23T15:08:32Z esxupdate: 2099595: vmware.runcommand: INFO: runcommand called with: args = 'localcli system visorfs ramdisk list | grep /esx-update-2099595 && localcli system visorfs ramdisk remove -t /tmp/esx-update-2099595', outfile = 'None', returnoutput = 'True', timeout = '0.0'.

    2019-06-23T15:08:32Z esxupdate: 2099595: Ramdisk: INFO: Unmounting manual tardisk /tardisks.noauto/esxupdt-2099595

    2019-06-23T15:08:32Z esxupdate: 2099595: Ramdisk: INFO: Unmounting manual tardisk /tardisks.noauto/weaselin-2099595

    2019-06-23T15:08:32Z esxupdate: 2099595: root: ERROR: Traceback (most recent call last):

    2019-06-23T15:08:32Z esxupdate: 2099595: root: ERROR:   File "/usr/lib/vmware/esxcli-software", line 470, in <module>

    2019-06-23T15:08:32Z esxupdate: 2099595: root: ERROR:     main()

    2019-06-23T15:08:32Z esxupdate: 2099595: root: ERROR:   File "/usr/lib/vmware/esxcli-software", line 461, in main

    2019-06-23T15:08:32Z esxupdate: 2099595: root: ERROR:     ret = CMDTABLE[command](options)

    2019-06-23T15:08:32Z esxupdate: 2099595: root: ERROR:   File "/usr/lib/vmware/esxcli-software", line 213, in ProfileUpdateCmd

    2019-06-23T15:08:32Z esxupdate: 2099595: root: ERROR:     nohwwarning=opts.nohwwarning)

    2019-06-23T15:08:32Z esxupdate: 2099595: root: ERROR:   File "/build/mts/release/bora-13644319/bora/build/esx/release/vmvisor/esxupdate/lib64/python3.5/site-packages/vmware/esximage/Transaction.py", line 376, in UpdateProfileFromDepot

    2019-06-23T15:08:32Z esxupdate: 2099595: root: ERROR:   File "/tmp/esx-update-2099595/usr/lib/vmware/weasel/util/upgrade_precheck.py", line 2161, in cliUpgradeAction

    2019-06-23T15:08:32Z esxupdate: 2099595: root: ERROR:   File "/tmp/esx-update-2099595/usr/lib/vmware/weasel/util/upgrade_precheck.py", line 997, in _parseVmwareVersion

    2019-06-23T15:08:32Z esxupdate: 2099595: root: ERROR:   File "/build/mts/release/bora-13473784/bora/build/esx/release/vmvisor/sys-boot/lib64/python3.5/subprocess.py", line 514, in getoutput

    2019-06-23T15:08:32Z esxupdate: 2099595: root: ERROR:   File "/build/mts/release/bora-13473784/bora/build/esx/release/vmvisor/sys-boot/lib64/python3.5/subprocess.py", line 495, in getstatusoutput

    2019-06-23T15:08:32Z esxupdate: 2099595: root: ERROR:   File "/build/mts/release/bora-13473784/bora/build/esx/release/vmvisor/sys-boot/lib64/python3.5/subprocess.py", line 316, in check_output

    2019-06-23T15:08:32Z esxupdate: 2099595: root: ERROR:   File "/build/mts/release/bora-13473784/bora/build/esx/release/vmvisor/sys-boot/lib64/python3.5/subprocess.py", line 383, in run

    2019-06-23T15:08:32Z esxupdate: 2099595: root: ERROR:   File "/build/mts/release/bora-13473784/bora/build/esx/release/vmvisor/sys-boot/lib64/python3.5/subprocess.py", line 676, in __init__

    2019-06-23T15:08:32Z esxupdate: 2099595: root: ERROR:   File "/build/mts/release/bora-13473784/bora/build/esx/release/vmvisor/sys-boot/lib64/python3.5/subprocess.py", line 1228, in _execute_child

    2019-06-23T15:08:32Z esxupdate: 2099595: root: ERROR: OSError: [Errno 28] No space left on device

    I don't see which mountpoint is running out of space and how to mitigate it.

    Any ideas?

    Thanks in advance.

    Best regards,

    mr_root_



  • 2.  RE: ESXi Update: [OSError] [Errno 28] No space left on device

    Posted Jul 02, 2019 03:32 PM

    Noone an idea?



  • 3.  RE: ESXi Update: [OSError] [Errno 28] No space left on device

    Posted Jul 03, 2019 03:36 PM

    Hello Mr_Root,

    Hope you are doing well.

    I hope below links can help you to resolve the issue.

    https://www.starwindsoftware.com/blog/what-should-you-do-if-no-space-left-on-device-error-occurs-while-updating-vmware-e…

    VMware Knowledge Base



  • 4.  RE: ESXi Update: [OSError] [Errno 28] No space left on device

    Posted Jul 03, 2019 04:43 PM

    Hi @Vikas_VM,

    I have checked the inode count as per your article, but I have enough inodes free for the update. I do not run out of inodes during running the update command.

    I wont do find / -path "/vmfs" -prune -o -type f -size +50000k -exec ls -l '{}' \; since this would delete my .vmdk files.

    I still get the above error message.

    What I found out is that the host cache is enabled on a datastore with enough free space, but not active.

    Command: esxcli sched swap system get

    Output:

       Datastore Active: true

       Datastore Enabled: true

       Datastore Name: SAMS_850EVO_500G_1

       Datastore Order: 1

       Hostcache Active: false

       Hostcache Enabled: true

       Hostcache Order: 0

       Hostlocalswap Active: false

       Hostlocalswap Enabled: true

       Hostlocalswap Order: 2

    Any ideas how I could make ESXi to activate the host cache?

    Thanks in advance.

    Best regards,

    mr_root



  • 5.  RE: ESXi Update: [OSError] [Errno 28] No space left on device

    Posted Jul 03, 2019 05:49 PM

    Hello Mr Root,

    I've found two VMware article for this, these are below.

    Configure Host Cache with VMFS Datastore

    Configuring Host Swap Cache

    I've never faced this issue so taking help from VMware articles.

    Hope this can help you as well.

    Thanks,

    Vikas



  • 6.  RE: ESXi Update: [OSError] [Errno 28] No space left on device

    Posted Jul 03, 2019 06:16 PM

    Hi Vikas_VM,

    thanks for the VMWare articles.

    Though your article Configure Host Cache with VMFS Datastore refers to VMWare Vsphere but I'm running standalone ESXi. I don't have the Configure->Storage-> Host Cache Configuration setting in the ESXi GUI. And your article only refers to flash host cache storage, but I want to bring out host cache to a local SSD storage.

    And your second article Configuring Host Swap Cache refers to flash memory as well. Maybe this is the problem, why my host cache is not active (though enabled), since I am not caching to flash memory?

    Thanks in advance.

    Best regards,

    mr_root



  • 7.  RE: ESXi Update: [OSError] [Errno 28] No space left on device

    Posted Jul 03, 2019 06:37 PM

    Hello Mr Root,

    Hope this can help.

    https://www.experts-exchange.com/articles/10844/HOW-TO-Configure-Host-Cache-Configuration-for-your-VMware-vSphere-5-0-or…

    Thanks,

    Vikas



  • 8.  RE: ESXi Update: [OSError] [Errno 28] No space left on device

    Posted Jul 03, 2019 06:51 PM

    Hi Vikas_VM,

    this article refers to ESXi up to version 6.5, since it describes how to enable Host cache with the VSphere client. My machine is ESXi 6.7 U2, and the VSphere client is not supported there anymore.

    I only have the web interface and the esxcli to configure it. As already said, my host cache is enabled in Host->Manage->Swap and there is also a sysSwap-ds-5c433ca2-1513-dd2b-55a2-ac1f6b7a4ea2.swp file on the selected datastore, but is has a modification time in the past (23/06/2019), shouldn't it then have a modification time of today when swapping would work?

    Best regards,

    mr_root



  • 9.  RE: ESXi Update: [OSError] [Errno 28] No space left on device

    Posted Jul 05, 2019 05:18 PM

    Noone an idea?

    I have already tried the suggestions from other threads where the esx-update caused No space left on device, but it doesn't mitigate my error.

    Best regards,

    mr_root



  • 10.  RE: ESXi Update: [OSError] [Errno 28] No space left on device

    Posted Jul 17, 2019 09:23 PM

    I upgraded from 16GB to 32GB memory and now the update works. Weird that I need 32GB memory for the update :smileyhappy: