Upgrade VCSA 6.7 to VCSA 7.0 fails on Stage 2 - Starting VMware Security Token Service...

[Lukas_Ro]

Hello,

I tried to upgrade vCSA 6.7.0.46000  to vCSA 7.0.1.00200 on January 2021 and it fails at Stage 2 -> A problem occured while - Starting VMware Security Token Service. I didn't have time to deal with it. 

I got to it again last week and tried everything with new versions. So i tried from vCSA 6.7.0.47000 to vCSA 7.0.2.00000, but with the same result. All pre-upgrade check passed. Upgrading via build-in account [e-mail].

Error:

Encountered an internal error. Traceback (most recent call last): File "/usr/lib/vmware/site-packages/cis/cisreglib.py", line 249, in securityctx_modifier yield File "/usr/lib/vmware/site-packages/cis/cisreglib.py", line 348, in add_securityctx_to_requests return req_method(self, *args, **kargs) File "/usr/lib/vmware/site-packages/cis/cisreglib.py", line 360, in register_service svc_create_spec) File "/usr/lib/vmware/site-packages/pyVmomi/VmomiSupport.py", line 583, in <lambda> self.f(*(self.args + (obj,) + args), **kwargs) File "/usr/lib/vmware/site-packages/pyVmomi/VmomiSupport.py", line 373, in _InvokeMethod return self._stub.InvokeMethod(self, info, args) File "/usr/lib/vmware/site-packages/pyVmomi/SoapAdapter.py", line 1570, in InvokeMethod raise obj # pylint: disable-msg=E0702 pyVmomi.VmomiSupport.vmodl.fault.SecurityError: (vmodl.fault.SecurityError) { dynamicType = <unset>, dynamicProperty = (vmodl.DynamicProperty) [], msg = '', faultCause = <unset>, faultMessage = (vmodl.LocalizableMessage) [] } During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/vmidentity/firstboot/vmidentity-firstboot.py", line 1861, in main vmidentityFB.boot() File "/usr/lib/vmidentity/firstboot/vmidentity-firstboot.py", line 369, in boot self.registerTokenServiceWithLookupService() File "/usr/lib/vmidentity/firstboot/vmidentity-firstboot.py", line 656, in registerTokenServiceWithLookupService raise e File "/usr/lib/vmidentity/firstboot/vmidentity-firstboot.py", line 652, in registerTokenServiceWithLookupService dynVars=dynVars) File "/usr/lib/vmware-cm/bin/cloudvmcisreg.py", line 710, in cloudvm_sso_cm_register serviceId = do_lsauthz_operation(cisreg_opts_dict) File "/usr/lib/vmware/site-packages/cis/cisreglib.py", line 1118, in do_lsauthz_operation ls_obj.register_service(svc_id, svc_create_spec) File "/usr/lib/vmware/site-packages/cis/cisreglib.py", line 348, in add_securityctx_to_requests return req_method(self, *args, **kargs) File "/usr/lib/python3.7/contextlib.py", line 161, in __exit__ raise RuntimeError("generator didn't stop after throw()") RuntimeError: generator didn't stop after throw()

Resolution

This is an unrecoverable error, please retry install. If you encounter this error again, please search for these symptoms in the VMware Knowledge Base for any known issues and possible resolutions. If none can be found, collect a support bundle and open a support request

Do you have any idea?

 

[virtual-power]

Can u please try to go  for   7.0 update 1d (7.0.1.00300) ?

[Lukas_Ro]

Hi Ajay,

with installer 7.0 update 1d got same result:

Encountered an internal error. Traceback (most recent call last): File "/usr/lib/vmware/site-packages/cis/cisreglib.py", line 249, in securityctx_modifier yield File "/usr/lib/vmware/site-packages/cis/cisreglib.py", line 348, in add_securityctx_to_requests return req_method(self, *args, **kargs) File "/usr/lib/vmware/site-packages/cis/cisreglib.py", line 358, in register_service svc_create_spec) File "/usr/lib/vmware/site-packages/pyVmomi/VmomiSupport.py", line 592, in <lambda> self.f(*(self.args + (obj,) + args), **kwargs) File "/usr/lib/vmware/site-packages/pyVmomi/VmomiSupport.py", line 381, in _InvokeMethod return self._stub.InvokeMethod(self, info, args) File "/usr/lib/vmware/site-packages/pyVmomi/SoapAdapter.py", line 1608, in InvokeMethod raise obj # pylint: disable-msg=E0702 pyVmomi.VmomiSupport.vmodl.fault.SecurityError: (vmodl.fault.SecurityError) { dynamicType = <unset>, dynamicProperty = (vmodl.DynamicProperty) [], msg = '', faultCause = <unset>, faultMessage = (vmodl.LocalizableMessage) [] } During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/vmidentity/firstboot/vmidentity-firstboot.py", line 1843, in main vmidentityFB.boot() File "/usr/lib/vmidentity/firstboot/vmidentity-firstboot.py", line 372, in boot self.registerTokenServiceWithLookupService() File "/usr/lib/vmidentity/firstboot/vmidentity-firstboot.py", line 659, in registerTokenServiceWithLookupService raise e File "/usr/lib/vmidentity/firstboot/vmidentity-firstboot.py", line 655, in registerTokenServiceWithLookupService dynVars=dynVars) File "/usr/lib/vmware-cm/bin/cloudvmcisreg.py", line 710, in cloudvm_sso_cm_register serviceId = do_lsauthz_operation(cisreg_opts_dict) File "/usr/lib/vmware/site-packages/cis/cisreglib.py", line 1116, in do_lsauthz_operation ls_obj.register_service(svc_id, svc_create_spec) File "/usr/lib/vmware/site-packages/cis/cisreglib.py", line 348, in add_securityctx_to_requests return req_method(self, *args, **kargs) File "/usr/lib/python3.7/contextlib.py", line 161, in __exit__ raise RuntimeError("generator didn't stop after throw()") RuntimeError: generator didn't stop after throw()

 

[virtual-power]

Hmm. I would suggest to open a case with VMware Support and upload the  failure log bundle. 

Share the SR number if possible or upload the log bundle here or attach a link to download the bundle

[Lukas_Ro]

We bought licenses from HPE, so i must contact HPE support. When I have more informations, i will put it here.

Log bundle is uploaded

[virtual-power]

Looks like u have sso admin user missing from  SSO Administrators group.

 

[Lukas_Ro]

Hi Ajay,

I don't understand how it happened. But after re-adding [e-mail] to SSO Administrators group everything works. Thank you very much for your help!

 

Lukas

 

[vjrk83]

I have the same issue , but I see the SSO administrator is part of the Administrator Group and also DNS / NTP is correct, but still see the issues. Any thoughts ? 

 

 

[vjrk83]

I see the same issue. Have raised the VMware support case - 21207589703  and attached the logs and bundles. Would you be able to look at that ? 

[virtual-power]

You have a different issue altogether .

Error : ""2021-03-25T03:20:55.832Z Failed to Reregister STS with Lookup Service."

raise Exception('Unable to find the sso endpoint for reregistering') Exception: Unable to find the sso endpoint for reregistering

Seems you have an endpoint issue here. Please collect source vcsa bundle and ldif(https://kb.vmware.com/s/article/2146046) and upload to SR .

[vjrk83]

Hello Ajay, Thanks for your reply. I've uploaded the support bundle from source vcsa to the SR. Also the vcsa is added to only AD as windows integrated authentication and not over LDAP. I suspect the name mismatch since the FQDN is different from the Primary network Identifier. Would it be a suggested way to remove the VCSA from AD upgrade and then add back to AD on the new VCSA on 7.0 ? Please let me know your thoughts . Thanks. 

 

- Vijay 

[virtual-power]

""FQDN is different from the Primary network Identifier"" . This was  exactly what I suspected from upgrade logs but wanted to be sure .  Make sure the PNID and FQDN is same before upgrade. Take cold snapshot > Do changes and then  take another cold snapshot and upgrade.

https://blogs.vmware.com/vsphere/2019/08/changing-your-vcenter-servers-fqdn.html

[vjrk83]

Hi Ajay, 

I wasn't able to update the FQDN , since it doesnt allow to update the FQDN with DNS matching the IP. Instead, I could update the PNID  to match the FQDN. 

/usr/lib/vmware-vmafd/bin/vmafd-cli set-pnid --server-name localhost --pnid <FQDN name> 

I did that before upgrade, but it was failing at the time of starting the vmware secure token services with error "Unable to find the sso endpoint for reregistering" . My question was would it harm to Leave the VCSA from AD , upgrade and Add to AD on new vcsa to overcome this mismatch ? I still need to update PNID to match anyways. 

[virtual-power]

"Leave VCSA from AD" >> Doesn't matter here.   Doing so will cause no harm and u can add that later. 

Ur issue is more on the  FQDN and PNID and that needs to be fixed prior upgrade. Please note that PNID cmd shared is not supported anymore and if you need to change PNID then I already shared the doc with you last .  Let Support team check and help as u already have the SR

[vjrk83]

The issue is fixed. I had to remove the 2nd NIC DNS entry and updated the hostname to match the primary network identifier and also reverted the PNID to original name to match the SSO endpoints. Everything else was fine and after upgrade put the chnages back . However the NIC configurations has to be done manually during and after the upgrade like updating the gateway, making sure the right VLAN Is set on the new VCSA etc. 

 

Thanks for your help. 

[virtual-power]

Cool. Having second NIC is not supported for VCSA  6.x . Not following the docs will introduce new issues

[alan harrington]

Ajay1988 you hit the nail right on the head, we had sts and dns mismatch had to lsdoctor and , after correcting still getting issue above from [e-mail] no longer in administrators group, corrected and good to go. Thank you!!

[TirTul]

 We have a similar issue during our upgrade. SR 22351970308

 

 

[virtual-power]

You have a different issue and I see work is in progress via SR.

[TirTul]

Thanks.  Communication from support/engineering has been poor.

 

I used jxplorer and determined that the service registrations are missing from a couple of groups. I removed the guid the refers to the vCSA in question. Would these cause a upgrade to fail?

"CN=vpxd-extension-*****,CN=ServicePrincipals,DC=vsphere,DC=local" is missing from LicenseService.Administrators and ServiceProviderUsers

"CN=machine-*******,CN=ServicePrincipals,DC=vsphere,DC=local" is missing from Administrators and SyncUsers

"CN=vsphere-webclient-*****,CN=ServicePrincipals,DC=vsphere,DC=local" is missing from vSphereClientSolutionUsers

[virtual-power]

Yes.. Any default solution users removed from SSO groups will cause issues. 

 

Be careful of taking cold snapshots >> https://virtual-power.in/f/29-vcenter-in-enhanced-linked-mode-best-practice-before-changes

[TirTul]

Thanks  . We have been very diligent with doing cold snapshots and verifying replication status before starting the upgrades and after each vCenter in the sso was upgraded.

So we have successfully upgraded 5 out of the 6 vCenters in ELM. So I'm not sure how the registrations for this one vCenter got the way it is. If what I found is the issue, we're still waiting on support to get back to us...while all six still are running on snapshots.

[Mubarak Sardar]

It did work like a charm Thank you  

 

[JairoCA]

I had the same error. my troubleshooting was:

For sure I will make it work without the DNS..... 

2 hrs later,,,,, mmmm maybe the DNS  could help.....

2 hrs later...... Ok, I will install the DNS on a VM with Winserver.....

15 min later...... it worked!!!!!!!

So.... do not loose your time... this error is fixed using the DNS...... it is an obligation.....

 

Hope this helps...

Jairo 

[Mubarak Sardar]

Thanks a ton  It did work like a charm.

I had issue upgrading VCSA7.x to 8.x managed by VCF5.x and this did help me finally. Used the below command on source(old) VCSA to add the missing vSphere local Administrator user.

/usr/lib/vmware-vmafd/bin/dir-cli group modify --name Administrators --add Administrator