Incomplete vCenter Update - Cant finalize it or revert

[Martin Slowikowski]

Hi guys

I wanted to udate a vCenter Appliance 6.7 Update 3n (6.7.0.48000)  to vCenter Appliance 6.7 Update 3o (6.7.0.50000). Unfortunately, something went wrong and the update got stuck and never finished. So once I rebooted the appliance, it came up normally and also the vCenter services came up and I can access the vCenter.

Within the vSphere Client it show the build of 6.7 U3o, but on the VM console it still shows U3n. The problem is, I cant revert to snapshot (shame on me) and I cant restore from backup becuase the last one is from July as the customer disconnected the NFS store where the beackups were made. Perfect, I know ...

 

I tried re-running the update, but it fails:

 

 [2021-09-24T18:35:48.267] : Evaluating packages to stage...
 [2021-09-24T18:35:48.267] : Verifying staging area
 [2021-09-24T18:35:48.267] : ISO unmounted successfully
 [2021-09-24T18:35:49.267] : Validating software update payload
 [2021-09-24T18:35:49.267] : Validation successful
 [2021-09-24 18:35:49,292] : Copying software packages  [2021-09-24T18:35:49.267] : ISO mounted successfully
24/24
 [2021-09-24T18:36:08.267] : ISO unmounted successfully
 [2021-09-24 18:36:08,826] : Running test transaction ....
 [2021-09-24T18:36:09.267] : Installation process failed
 [2021-09-24T18:36:09.267] : Test transaction failed to update packages
Command>

. I tried

software-packages install --iso

and

software-packages stage --iso followed by software-packages install --staged

Both leads to the same output. This is what can be found in /var/log/vmware/applmgmt/software-packages.log:

2021-09-24T18:36:09.267 [15684]DEBUG:vmware.vherd.base.software_update:Test transaction failed to update package
out=Preparing...                          ########################################

error=  package VMware-vpxd-6.7.0-18485166.x86_64 is already installed
        package vmware-eam-6.7.0-18485166.x86_64 is already installed
        package VMware-perfcharts-6.7.0-18485166.x86_64 is already installed
        package vsphere-client-6.7.0-18434431.noarch is already installed
        package VMware-vpxd-svcs-6.7.0-18485166.x86_64 is already installed
        package VMware-vapi-6.7.0-18485166.x86_64 is already installed
        package VMware-UpdateManager-6.7.0-18333053.x86_64 is already installed
        package vmware-studio-appliance-config-3.0.0.7-210719191459.noarch is already installed
        package vmware-psc-health-6.7.0.2137-17947187.x86_64 is already installed
        package VMware-pod-6.7.0-18485166.x86_64 is already installed
        package VMware-mbcs-6.7.0-18485166.x86_64 is already installed
        package vmware-identity-sts-6.7.0.6506-17947188.noarch is already installed
        package vmware-ic-deploy-6.7.0.4073-17947186.x86_64 is already installed
        package vmware-esx-netdumper-6.7.0-0.0.18485166.i386 is already installed
        package vmware-directory-client-6.7.0.5171-17947185.x86_64 is already installed
        package VMware-commonjars-6.7.0-18485166.x86_64 is already installed
        package vmware-cm-6.7.0-18485166.x86_64 is already installed
        package vmware-cam-6.7.0.1543-17947404.x86_64 is already installed
        package VMware-applmon-cloudvm-6.7.0-18485166.x86_64 is already installed
        package vmware-afd-6.7.0.6346-17947183.x86_64 is already installed
        package httpd-2.4.48-1.ph1.x86_64 is already installed

 

I already worked together with the VMware support, but they cant help. They suggest to revert to snap or restore from backup. Otherwise re-install side-by-side and manually migrate stuff. But that is not working

As vCenter seems to run normally, I tried to do a backup through VAMI interface, but there it seems that it still recognizes the old u3n version and the backup fails ...

This is some output from /var/log/vmware/applmgmt/backup.log

2021-09-24T15:33:15.321 [ConfigFilesBackup:PID-2494] ERROR: rc: 0, stderr: /usr/bin/tar: etc/vmware-vcha/vmware-vmon.service.bak: Warning: Cannot stat: No such file or directory
2021-09-24T15:33:36.893 [ComponentScriptsBackup:PID-2493] ERROR: Process returncode is -13, but expected exit codes are [0].
2021-09-24T15:33:36.893 [ComponentScriptsBackup:PID-2493] ERROR: rc: 1, stderr: Traceback (most recent call last):
2021-09-24T15:33:36.893 [ComponentScriptsBackup:PID-2493] ERROR: Process returncode is 1, but expected exit codes are [0].
2021-09-24T15:33:36.893 [ComponentScriptsBackup:PID-2493] ERROR: rc: 1, stderr: b'cat: write error: No space left on device\n'
2021-09-24T15:33:36.904 [ComponentScriptsBackup:PID-2493] ERROR: Component backup command "/etc/vmware/backup/component-scripts/vum/backup_restore.py --startBackup" failed 1.
2021-09-24T15:33:36.904 [ComponentScriptsBackup:PID-2493] ERROR: Unable to perform backup: [Errno 32] Broken pipe
2021-09-24T15:33:36.904 [ComponentScriptsBackup:PID-2493] ERROR: Error during component vum backup
2021-09-24T15:33:37.242 [MainProcess:PID-1390] ERROR: Error at process ComponentScriptsBackup; rc:1.
2021-09-24T15:33:37.242 [MainProcess:PID-1390] ERROR: stderr:Error during component vum backup
2021-09-24T15:33:37.242 [MainProcess:PID-1390] ERROR: BackupManager encountered an exception: Hit exception inside process ComponentScriptsBackup:

 

I will attach the backup.log and software-packages.log to this thread. Maybe someone has an idea, how I can fix that half-updated state or create a backup so I can re-deploy vCenter from backup

Attachment(s)

backup.txt   335 KB 1 version

software-packages.log   786 KB 1 version

[Ajay Raj Achary]

2021-09-24T15:33:36.893 [ComponentScriptsBackup:PID-2493] ERROR: rc: 1, stderr: b'cat: write error: No space left on device\n'

Seems u had space crunch. Fixing a broken update is really tough. Best is to revert back to snapshot before patching and then make sure you have enough space on root and other partitions even and then patch over new  snapshot

[Martin Slowikowski]

Yeah that space thing was taken from the backup.log, will check what exactly is running low on space.

But I cant revert to snap or backup, cause there is none (or at least very old...).

So I would have a chance if I can finalize that broken update or getting a clean configuration backup through VAMI so I can re-deploy with current data.

That is my hope ;>

[Martin Slowikowski]

Good point with the storage

I was able to successfully run a backup from the old environment after adding some space to the target backup space ... I just have overseen this

Now, it created me a backup for 67u3n, so I re-deployed from backup with the corresponding vcsa full iso. Once the backup was restored, it failed at 70% of starting services. Not sure, what the problem was, as all services were online and vCenter accessible. When I try to login to VAMI, I always get redirected to https://vcsa.domain.tld:5480/configure/#/installer?locale=en with the message "vCenter Server has been restored. However, additional steps must be completed before it is available for use. Click the link below to continue.". Then asking for first root pw and then [e-mail] password. So I provided both and it fails again while trying to start services.

I tried running the upgrade to 67u3o, it was successfull. vCenter runs fine. Unfortunately, it still redirects me to this configure page...

So I did another reboot after I followed the steps from this kb https://kb.vmware.com/s/article/67179. As the software_update_state.conf doesnt exist, I skipped that point and created the dir /var/log/vmware/upgrade and also those empty files, did a reboot, got redirected again, entered passwords and it finally went through successfully.

So for now it is looking fine, at least to me ... will get in touch with VMware support again, to double check ...

 

https://kb.vmware.com/s/article/67179