VCenter 8.0.1.00300 Upgrade to 8.0.2. Failed during post upgrade data conversion

[manadrain]

This morning I decided to try and upgrade my VCenter Server from 8.0.1.00300 to 8.0.2.   My appliance is running on Intel Processors which is not like the issues I have been seeing posted about AMD processors and upgrading.  With my experience, I have found the most success with patching VCenter by mounting the the FP ISO and connecting with SSH to run the upgrade.   I always shutdown the VCenter Server and take a snapshot prior to upgrading in case something does not go as expected.

 

The install appeared to go fine with installing the RPMs and Container if I recall from the output in by SSH session  it was stating "post upgrade data conversion" or something very similar.  I do know from past experiences that this part of the process does take a significant amount of time to complete.   However during this process I got a error message that I will include below.   Being that this is a production server I had to shutdown the VM and revert the snapshot.  My VCenter is back at running version 8.0.1.00300 without any issues.

Below is a copy of the message I received in my SSH session.

--------------------------------------------------------------------------------------------------------------------------------------------

Installation failed. Retry to resume from the current state. Or please collect the VC support bundle.
Mismatch:
summary: Internal error occurs during execution of update process Traceback (most recent call last):
File "/storage/seat/software-updateuug4bpa_/stage/scripts/patches/py/vmware_b2b/patching/phases/patcher.py", line 208, in patch
_patchComponents(ctx, userData, statusAggregator.reportingQueue)
File "/storage/seat/software-updateuug4bpa_/stage/scripts/patches/py/vmware_b2b/patching/phases/patcher.py", line 89, in _patchComponents
_startDependentServices(c)
File "/storage/seat/software-updateuug4bpa_/stage/scripts/patches/py/vmware_b2b/patching/phases/patcher.py", line 56, in _startDependentServices
serviceManager.start(depService)
File "/storage/seat/software-updateuug4bpa_/stage/scripts/patches/libs/sdk/service_manager.py", line 909, in wrapper
return getattr(controller, attr)(*args, **kwargs)
File "/storage/seat/software-updateuug4bpa_/stage/scripts/patches/libs/sdk/service_manager.py", line 799, in start
super(VMwareServiceController, self).start(serviceName)
File "/storage/seat/software-updateuug4bpa_/stage/scripts/patches/libs/sdk/service_manager.py", line 665, in start
raise IllegalServiceOperation(errorText)
service_manager.IllegalServiceOperation: Service cannot be started. Error: Error executing start on service vpxd. Details {
"detail": [
{
"id": "install.ciscommon.service.failstart",
"translatable": "An error occurred while starting service '%(0)s'",
"args": [
"vpxd"
],
"localized": "An error occurred while starting service 'vpxd'"
}
],
"componentKey": null,
"problemId": null,
"resolution": null
}
Service-control failed. Error: {
"detail": [
{
"id": "install.ciscommon.service.failstart",
"translatable": "An error occurred while starting service '%(0)s'",
"args": [
"vpxd"
],
"localized": "An error occurred while starting service 'vpxd'"
}
],
"componentKey": null,
"problemId": null,
"resolution": null
}

.
resolution: Send upgrade log files to VMware technical support team for further assistance.

Command>

 

 

 

[ptarnawski]

Hey  ,

 

you have an error with starting vpxd service. This may indicate an issue with the certificate, services won't start if there is a mismatch with the cert or the cert expired

 

[manadrain]

Thank you for the information below.   We are using the built in certificates on VMware.   I can see both from the GUI and if I SSH into the server and ran a command I found in a VMWare KB to verify expiration dates I am showing everything is valid until at least June 19, 2024 or later.

I just want to put this out there as well because I found another post stating about low harddisk space and all of the drives on my VCenter server are not above 29% Utilization looking under Monitor/Disks on the VCenter Server Management.

 

 

 

[DanKane]

We were also investigating an issue upgrading from 8.0.1.00200 to 8.0.2.00000.

 

We found that the local vcenter machine cert was the issue.  

 

FYI

 

[Gilbert Franco]

Similar here going from 8.0.1 to 8.0.2. I've not taken the time to do all the detective work trying to see why my in-place upgrade also is failing. Was hoping to see a few people who've found solutions to there problems to then apply those fixes hoping to catch mine. Its a real pain now upgrading these with how often it fails but then also how it rarely tell's you "why". 

I put in a case but not holding my breath...I'll keep an eye out for any work arounds folks find to try.

 

Thanks all for sharing the issues and posting any known fixes as we figure them out.

[manadrain]

Just a few minutes ago I had time to test running the upgrade again and had the same issue.   This time I was able to gather the support bundle as well as got information from the firstbootStatusFile which is what I found in a KB.  Based on the information I see in the logs I was not able to find something that stood out why it had failed.   However, I may be looking at the incorrect information.  I have since reverted my snapshot, so I have a running 8.0.1.00300 VCenter.

I have opened a case with VMWare Support, and I believe I have provided them enough information to find out why the upgrade to 8.0.2 is failing to install.   I will update this post when I receive an update.

[randysaeks]

We saw the same thing and have since reverted back to 8.0.1 since the 8.0.2 upgrade would not complete successfully at the same spot.

[manadrain]

I did open the case with VMware Support and they had me run several commands looking at the certificates and from what I see everything looks correct.  I didn't receive a response for several days so I asked for an update on the status.  I did receive a response saying there is a known bug on this and when they have an update they would post a response.  I will update once I have a response but I think the response is going to be to upgrade to the next release when they have addressed these issues.

[ksattler]

Hi,

Is there still no update?

[manadrain]

I still have no update on the ticket.   The last response I had from VMWare Support is that this is a known bug and I would be provided an update when they have a fix.

Just for the record, this morning I did update my VCenter Server from 8.0.1.00300 to 8.0.1.00400 which was just released to address the known security issue.  I ran the same process to upgrade and didn't have any issues with the update.

[ksattler]

ok thanks. I also did the upgrade to 8.0.1.00400 successfully. Afterwards I tried again to upgrade to 8.0.2, but I got the same issue.