upgrade 6.7u3 to 7.0 cert issues

[gjbrown]

attempting to upgrade my lab from 6.7u3.latest to 7.0.latest

new VCSA VM deploys ok, but during pre-check get the following error:

Error

A vCenter Single Sign-On endpoint certificate validation error has occurred.

Resolution

Ensure that the endpoint service registrations in vmdir match their corrsponding machine SSL certificates in VECS. For more information, see Knowledge Base article KB 2121701.

I have already gone through the KB to no avail.  I have also gone through and reset all certs (cert manager option 8).

Anyone have any guidance or suggestions?

Thanks,

-GB

[Alex_Romeo]

Hi,

VMware Knowledge Base

ARomeo

[gjbrown]

thanks for the suggestion, but tried that as well.

GB

[Scott Vessey]

Moderator: Thread moved to the vSphere Upgrade & Install area.

[harry89]

This issue mostly occurs if the SSL trust of the services registered on PSC are having different than the SSL certificate of the node (of which the services is registered).

Please follow steps of the below article

VMware Knowledge Base

you have to basically get the old thumbprint and update the services with ls update cert  script using the new SSL certificate which is currently present

This command will give you all the services registered along with SSL trust they have .

/usr/lib/vmidentity/tools/scripts/lstool.py list --url https://localhost/lookupservice/sdk --no-check-cert --ep-type com.vmware.cis.cs.identity.sso 2>/dev/null

*Please mark the answer as correct if it solves your query

[gjbrown]

Thanks harry89​, I went through the KB no errors, replaced 3 certificates but still the same issue when I attempt to upgrade.

-GB

[harry89]

Can u send the log snippet

[gjbrown]

harry89​ which log snip you want? the log bundle compressed is 16mb and I am sure you don't want to deal with all of it.

Thx

[Sudeshna Sarkar]

Hi gjbrown,

You can run the following command to check if the certificates of the existing environment is fine and valid or not .

#for i in $(/usr/lib/vmware-vmafd/bin/vecs-cli store list); do echo $i; /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store $i --text | grep -i "not after"; done;

If the certs are fine and you continue to face the same issue please go ahead and replace the certificates using the option 8 in the certificate-manager tool.

VMware Knowledge Base

Then continue with the upgrade again.

It still you run into any issue please open a support request with us.

Regards,

Sudeshna Sarkar

Install-Upgrade Specialist

[gjbrown]

Hi sudeshnas​

When I ran the command you provided it only returned back to a prompt with no output.  Not sure if that is good or bad.

I ran through cert replacement, option 8 again, even though I have done already.

Updated 5 service(s)

Status : 60% Completed [Reset vpxd-extension Cert...]

2020-07-22T15:14:46.910Z  Updating certificate for "com.vmware.imagebuilder" extension

Reset status : 100% Completed [Reset completed successfully]

--obviously this is good.

but upgrade still fails

[Sudeshna Sarkar]

Hi gjbrown,

I have attached a script here.

Please download the script and run it on the source machine to fix any ssl trust mismatch in lookup service registrations.

Please take a snapshot before proceeding.

Copy the file to lstool scripts folder.

For vCSA path:

# /usr/lib/vmidentity/tools/scripts

Run the below commands:

# python ls_ssltrust_fixer.py -f scan

#python ls_ssltrust_fixer.py -f fix

Then try running the upgrade.

Note: Make sure you take necessary backup/snapshot. Please try this ls_ssltrust_fixer.py in test environment, do not try this in production environment. Please raise a support request to validate before executing this script in production environment.

Regards,

Sudeshna Sarkar

Install-Upgrade Specialist

_______________________________________________________________________________________________________

"Did you find this helpful? Let us know by completing this survey (takes 1 minute!)"

[gjbrown]

Hi sudeshnas​

The script worked and found 31 mismatches. I ran the fix which let me run the upgrade but failed @ error#2, 89%.  Here is the error

Error

WCP service installation failed : Traceback (most recent call last): File "/usr/lib/vmware-wcp/firstboot/wcp-firstboot.py", line 50, in proxy return func(*args, **kwargs) File "/usr/lib/vmware-wcp/firstboot/wcp-firstboot.py", line 71, in configure wcpconfigure.configure_service() File "/usr/lib/vmware-wcp/py-modules/wcpconfigure.py", line 442, in configure_service create_storage_identity() File "/usr/lib/vmware-wcp/py-modules/wcpconfigure.py", line 438, in create_storage_identity SsoUser(_STORAGE_USER).create_storage_user_and_assign() File "/usr/lib/vmware-wcp/py-modules/wcpconfigure.py", line 330, in create_storage_user_and_assign self._create_storage_user() File "/usr/lib/vmware-wcp/py-modules/wcpconfigure.py", line 298, in _create_storage_user password = svcacctmgmt_client.create_svc_account(self._user_name) File "/usr/lib/vmware-wcp/py-modules/svcacctmgmt.py", line 90, in create_svc_account raise er File "/usr/lib/vmware-wcp/py-modules/svcacctmgmt.py", line 84, in create_svc_account svcacct_pwd_out = svcacct_client.create(create_spec) File "/usr/lib/vmware-wcp/py-modules/vapi-bindings/com/vmware/vcenter/svcaccountmgmt_client.py", line 368, in create 'create_spec': create_spec, File "/usr/lib/vmware-vapi/lib/python/vapi_runtime-2.100.0-py2.py3-none-any.whl/vmware/vapi/bindings/stub.py", line 345, in _invoke return self._api_interface.native_invoke(ctx, _method_name, kwargs) File "/usr/lib/vmware-vapi/lib/python/vapi_runtime-2.100.0-py2.py3-none-any.whl/vmware/vapi/bindings/stub.py", line 298, in native_invoke self._rest_converter_mode) com.vmware.vapi.std.errors_client.InternalServerError: {messages : [LocalizableMessage(id='com.vmware.vapi.authorization.permission.error', default_message='Could not validate permission information for operation com.vmware.vcenter.svcaccountmgmt.service_account.create invocation.', args=['com.vmware.vcenter.svcaccountmgmt.service_account.create'], params=None, localized=None)], data : None, error_type : None}

Resolution

This is an unrecoverable error, please retry install. If you encounter this error again, please search for these symptoms in the VMware Knowledge Base for any known issues and possible resolutions. If none can be found, collect a support bundle and open a support request.

I do have SR 20142056507 open, but just getting started if you would like to review any logs.

Thank you for the help with this.

[Sudeshna Sarkar]

Hi gjbrown,

Thank you for opening a ticket with us.

I have gone through the logs and the errors/backtrace reported.

Well upon researching I see that similar issue has been reported by the other customer too and currently we are working internally to get it fixed.

You will receive all the updates on the ticket.

Regards,

Sudeshna Sarkar

Install-Upgrade Specialist

[gjbrown]

sudeshnas​  Thanks for digging into this.  I'll see what GSS says via ticket.  I'll update this thread with info to guide others towards a KB or solution.

Again thanks for the help and time with this.

-GB

[EdwinMenard]

Hi,

I have the same error : WCP service installation failed.

Where can i find the solution for this problem ?

Thanks for help.

[Mkei]

I am having the same issue, the upgrade stopped at 89% with the error message "WCP service installation failed.....". It seems we have a mismatch sAMAccountName and cn name. I tried https://kb.vmware.com/s/article/82634?lang=en_US using the  6.7 vcenter ip address with the cn name, and it does not work for me. Does anyone how to resolve it?

[ErichWeihrauch]

sudeshnas, that script worked perfectly for me, thank you!

I had some invalid cert, that not even regenerating and resetting existing certs worked to resolve.

Gave your script a shot, and bam!

[AleksejsV]

Hello,

 

your script returns the following error:

root@vcenter [ /usr/lib/vmidentity/tools/scripts ]# python ls_ssltrust_fixer_p3.py -f fix
Running function 'fix'
Fix phase 1: Reading IDs with incorrect certificate from scan results
Using mismatch ID list from: /var/log/ls_ssltrust_fixer/mismatchIDs
SSO administrator user (Default:[e-mail]):[e-mail]
Traceback (most recent call last):
File "ls_ssltrust_fixer_p3.py", line 368, in <module>
main()
File "ls_ssltrust_fixer_p3.py", line 360, in main
_doFix()
File "ls_ssltrust_fixer_p3.py", line 297, in _doFix
user=input("SSO administrator user (Default:[e-mail]):") or "[e-mail]"
File "<string>", line 1
[e-mail]
^
SyntaxError: invalid syntax

[nclifton]

I ran into the same issue.. Adding " " around the user name worked for me.  i.e. "[e-mail]"

 

[IKT_Bee]

Hello,

thx for the script, but it seems on my side some modules are missing, how can i add/import them?
Sorry to ask but i have no phyton knowloge

THX!

root@srvvmvcsa [ /usr/lib/vmidentity/tools/scripts ]# python ls_ssltrust_fixer.py -f scan
Traceback (most recent call last):
File "ls_ssltrust_fixer.py", line 16, in <module>
import lstoolutil
File "/usr/lib/vmidentity/tools/scripts/lstoolutil.py", line 7, in <module>
import urlparse
ModuleNotFoundError: No module named 'urlparse'

[cdsharp]

This fixed my issue. Thanks!

[StefanGM]

Hello,

Unfortunately I have the same issue.

When I run the scan it finds 6 mismatches but when I run the fix it does not update them

Running function 'fix'
Fix phase 1: Reading IDs with incorrect certificate from scan results
Using mismatch ID list from: /var/log/ls_ssltrust_fixer/mismatchIDs
SSO administrator user (Default:[e-mail]):[e-mail]
Password for [e-mail]:
Fix phase 2: Collecting site topology information
Picked up JAVA_TOOL_OPTIONS: -Xms32M -Xmx128M
*** 0 endpoints for 0 service IDs updated with current cetificates and trust ***
Completed running function 'fix'

Any ideas?

 

[MikeBauer]

I keep getting an error when running the scan:

It gives me an UnboundLocalError: local variable 'endpointurl' referenced before assignment

 

 

[MikeBauer]

I was able to make it work by manually creating the local variable with the name of the vCenter with the internal PSC

[Mike Gerolami]

Thanks for the script, it was very helpful for me and others.

[ArneArne]

official vmware lsdoctor wrecked my vcenter 6.7, the upgrade prograde didn't recognize the vcenter appliance anymore as such, reverted the snapshot, tested and used your script. result!!

 

well done

[Odurasler2]

having the same issue. where can i find this file?

thank you!

[ZAKIAH ABDUL MAJID ZAKARIA]

Hi Sudeshna,

can you reshare ls_ssltrust_fixer.py file. i'm having a problem with the same issue when upgrading 6.7 to 7.0 vcenter. i cant seem to find the file anywhere.

Your help is very appreciated.

best regards,

Zakiah

[Dermot Cummings]

Where can i download the script? I dont see it on this thread

[harry89]

There are possibilities that when u ran the reset all the certificates  , some of the endpoints are still having the older machine SSL cert as ssl trust .

This is fairly common occurrence .

But was this done before starting the upgrade or after . (reset all certificates).

If this was done to try to mitigate the issue and solve the upgrade problem , then not sure if this right direction because we need to be sure that prior to upgrade some cert in vecs-cli was surely expired and that was machine ssl .

[vt-vmwaresjo]

Perfect, worked for me - Thanks

[scsigate]

Hi sudeshnas,

thanks for that pyton script. that did the trick for my upgrade.

Cheers,

[systeembhr_sboh]

Hi gjbrown,

I used the ls_ssltrust_fixer_p3.py to fix the certificates but when i run the upgrade it still fails on the on the wcp-firstboot message:

Did you find a fix for this ?

In the vmware kb 82634 , the workaround that would fix this , should be a entry in the /etc/hosts (the new vcsa)

x.x.x.x  vcenter.noa.local > i used this entry but the installer 2 fase removes this entry so i did not work

022-03-12T18:51:40.247Z INFO wcp-firstboot WCP storage user does not exists, create the user.
2022-03-12T18:51:40.247Z INFO wcp-firstboot Creating ServiceAccount client...
2022-03-12T18:51:40.325Z Further filtering retrieved service registration list on hostname : vcenter.noa.local
2022-03-12T18:51:40.335Z INFO wcp-firstboot Creating service account...
2022-03-12T18:51:40.335Z INFO wcp-firstboot Initializing ServiceAccount session...
2022-03-12T18:51:43.356Z ERROR wcp-firstboot Unexpected error creating ServiceAccount {messages : [LocalizableMessage(id='com.vmware.vcenter.svcaccountmgmt.error', default_message='Exception found (Internal Server Error, VMware directory error[9127])', args=['Internal Server Error, VMware directory error[9127]'], params=None, localized=None)], data : None, error_type : ERROR}
2022-03-12T18:51:43.357Z ERROR wcp-firstboot Failed to create service account for workload storage
Traceback (most recent call last):
File "/usr/lib/vmware-wcp/py-modules/wcpconfigure.py", line 362, in _create_storage_user
password = svcacctmgmt_client.create_svc_account(self._user_name)
File "/usr/lib/vmware-wcp/py-modules/svcacctmgmt.py", line 90, in create_svc_account
raise er
File "/usr/lib/vmware-wcp/py-modules/svcacctmgmt.py", line 84, in create_svc_account
svcacct_pwd_out = svcacct_client.create(create_spec)
File "/usr/lib/vmware-wcp/py-modules/vapi-bindings/com/vmware/vcenter/svcaccountmgmt_client.py", line 340, in create

[MRRamsay]

When the script is run in a VCenter for Windows server, it throws the following error:

Traceback (most recent call last):
File "ls_ssltrust_fixer_p3.py", line 16, in <module>
import lstoolutil
ImportError: No module named lstoolutil

lstoolutil is located in the C:\Program Files\VMware\vCenter Server\VMware Identity Services\lstool\scripts folder.

[Ridgy]

Running into same issues

python ls_ssltrust_fixer_p3.py -f scan
Running function 'scan'
Scan Phase1: Getting service IDs
Traceback (most recent call last):
File "ls_ssltrust_fixer_p3.py", line 368, in <module>
main()
File "ls_ssltrust_fixer_p3.py", line 356, in main
_doScan()
File "ls_ssltrust_fixer_p3.py", line 200, in _doScan
rc, ids = lstoolcommunicate(["list","--no-check-cert","--url",lsUrl,"--id-only"])
File "ls_ssltrust_fixer_p3.py", line 43, in lstoolcommunicate
java = lstoolutil._get_java()
File "/usr/lib/vmidentity/tools/scripts/lstoolutil.py", line 215, in _get_java
if os.environ.has_key('VMWARE_JAVA_HOME'):
AttributeError: '_Environ' object has no attribute 'has_key'