Vcenter don't come up after certificates updates

[luizitano]

Hi,

i have a need to update the vsphere machine certificates and after the updates the vsphere don't come up

i have used the the option 4 and after the 8 on /usr/lib/vmware-vmca/bin/certificate-manager

after i try to use the script ./fixsts.sh 

now i don't have any certificate expired:

root@vCSA-001 [ ~/temp ]# python checksts.py

2 VALID CERTS
================

        LEAF CERTS:

        [] Certificate 84:BD:82:F9:31:48:C3:CF:0A:0C:94:48:47:D8:1C:24:25:24:A6:09 will expire in 730 days (2 years).

        ROOT CERTS:

        [] Certificate 1B:3A:EA:78:41:46:2F:A3:82:1D:92:A5:12:C9:2F:E5:9F:69:15:03 will expire in 3646 days (10 years).

0 EXPIRED CERTS
================

        LEAF CERTS:

        None

        ROOT CERTS:

        None

but some services don't come up

the services are in this state:

root@vCSA-001 [ ~ ]# service-control --status --all
Running:
 applmgmt lookupsvc lwsmd observability-vapi vmafdd vmcad vmdird vmonapi vmware-certificateauthority vmware-certificatemanagement vmware-cis-license vmware-eam vmware-envoy vmware-infraprofile vmware-postgres-archiver vmware-rhttpproxy vmware-sca vmware-statsmonitor vmware-stsd vmware-topologysvc vmware-trustmanagement vmware-vapi-endpoint vmware-vmon vmware-vpostgres vtsdb
StartPending:
 vmware-hvc
Stopped:
 observability pschealth vlcm vmcam vmware-analytics vmware-content-library vmware-imagebuilder vmware-netdumper vmware-perfcharts vmware-pod vmware-rbd-watchdog vmware-sps vmware-updatemgr vmware-vcha vmware-vdtc vmware-vpxd vmware-vpxd-svcs vmware-vsan-health vmware-vsm vsphere-ui vstats wcp

on the VPXD service i have:

2024-11-12T16:05:50.747Z info vpxd[37599] [Originator@6876 sub=vpxCrypt] Failed to read X509 cert; err: 151441516
2024-11-12T16:05:50.764Z info vpxd[37599] [Originator@6876 sub=SsoClient] Successfully acquired token: SamlToken [subject={Name: vpxd-8f5a0953-467d-4581-8146-6fd42ca202d6; Domain:vsphere.local}, groups=[{Name: Users; Domain:vsphere.local}, {Name: SolutionUsers; Domain:vsphere.local}, {Name: SystemConfiguration.Administrators; Domain:vsphere.local}, {Name: ComponentManager.Administrators; Domain:vsphere.local}, {Name: LicenseService.Administrators; Domain:vsphere.local}, {Name: ActAsUsers; Domain:vsphere.local}, {Name: Everyone; Domain:vsphere.local}], delegationChain=[], startTime=2024-11-12 16:05:50.747, expirationTime=2024-11-13 00:05:50.747, renewable=false, delegable=false, isSolution=true,confirmationType=1]
2024-11-12T16:05:50.766Z warning vpxd[37719] [Originator@6876 sub=IO.Connection] Failed to connect; <io_obj p:0x00007fe40c003ac8, h:35, <TCP '127.0.0.1 : 47388'>, <TCP '127.0.0.1 : 10080'>>, e: 111(Connection refused), duration: 0msec
2024-11-12T16:05:50.766Z warning vpxd[37719] [Originator@6876 sub=HttpConnectionPool-000027] Failed to get pooled connection; <cs p:00007fe3a475fdf0, TCP:localhost:10080>, (null), duration: 0msec, N7Vmacore15SystemExceptionE(Connection refused: The remote service is not running, OR is overloaded, OR a firewall is rejecting connections.)
--> [context]zKq7AVECAQAAABvdLAEPdnB4ZAAAHug3bGlidm1hY29yZS5zbwAAAYksAIl7LQBX+DIAV9wjAPb3IwCDCCQAuQwkAKYUJAATuSMAl1ojAD+wIwDcczcBh38AbGlicHRocmVhZC5zby4wAALvNQ9saWJjLnNvLjYA[/context]
2024-11-12T16:05:50.768Z info vpxd[37719] [Originator@6876 sub=IO.Http] Set user agent error; state: 1, (null), N7Vmacore15SystemExceptionE(Connection refused: The remote service is not running, OR is overloaded, OR a firewall is rejecting connections.)
--> [context]zKq7AVECAQAAABvdLAEPdnB4ZAAAHug3bGlidm1hY29yZS5zbwAAAYksAIl7LQBX+DIAV9wjAPb3IwCDCCQAuQwkAKYUJAATuSMAl1ojAD+wIwDcczcBh38AbGlicHRocmVhZC5zby4wAALvNQ9saWJjLnNvLjYA[/context]
2024-11-12T16:05:50.769Z error vpxd[37719] [Originator@6876 sub=IO.Http] User agent failed to send request; (null), N7Vmacore15SystemExceptionE(Connection refused: The remote service is not running, OR is overloaded, OR a firewall is rejecting connections.)
--> [context]zKq7AVECAQAAABvdLAEPdnB4ZAAAHug3bGlidm1hY29yZS5zbwAAAYksAIl7LQBX+DIAV9wjAPb3IwCDCCQAuQwkAKYUJAATuSMAl1ojAD+wIwDcczcBh38AbGlicHRocmVhZC5zby4wAALvNQ9saWJjLnNvLjYA[/context]
2024-11-12T16:05:50.769Z warning vpxd[37599] [Originator@6876 sub=Authz] [ConnectAndLogin] Failed to loginBySamlToken: N7Vmacore15SystemExceptionE(Connection refused: The remote service is not running, OR is overloaded, OR a firewall is rejecting connections.)
--> [context]zKq7AVECAQAAABvdLAEPdnB4ZAAAHug3bGlidm1hY29yZS5zbwAAAYksAIl7LQBX+DIAV9wjAPb3IwCDCCQAuQwkAKYUJAATuSMAl1ojAD+wIwDcczcBh38AbGlicHRocmVhZC5zby4wAALvNQ9saWJjLnNvLjYA[/context]
2024-11-12T16:05:50.772Z info vpxd[37599] [Originator@6876 sub=Authz] fallback to loginByCertificate
2024-11-12T16:05:50.772Z warning vpxd[37720] [Originator@6876 sub=IO.Connection] Failed to connect; <io_obj p:0x00007fe3f8003278, h:35, <TCP '127.0.0.1 : 47390'>, <TCP '127.0.0.1 : 10080'>>, e: 111(Connection refused), duration: 0msec
2024-11-12T16:05:50.772Z warning vpxd[37720] [Originator@6876 sub=HttpConnectionPool-000027] Failed to get pooled connection; <cs p:00007fe3a475fdf0, TCP:localhost:10080>, (null), duration: 0msec, N7Vmacore15SystemExceptionE(Connection refused: The remote service is not running, OR is overloaded, OR a firewall is rejecting connections.)
--> [context]zKq7AVECAQAAABvdLAEPdnB4ZAAAHug3bGlidm1hY29yZS5zbwAAAYksAIl7LQBX+DIAV9wjAPb3IwCDCCQAuQwkAKYUJAATuSMAl1ojAD+wIwDcczcBh38AbGlicHRocmVhZC5zby4wAALvNQ9saWJjLnNvLjYA[/context]
2024-11-12T16:05:50.773Z info vpxd[37720] [Originator@6876 sub=IO.Http] Set user agent error; state: 1, (null), N7Vmacore15SystemExceptionE(Connection refused: The remote service is not running, OR is overloaded, OR a firewall is rejecting connections.)
--> [context]zKq7AVECAQAAABvdLAEPdnB4ZAAAHug3bGlidm1hY29yZS5zbwAAAYksAIl7LQBX+DIAV9wjAPb3IwCDCCQAuQwkAKYUJAATuSMAl1ojAD+wIwDcczcBh38AbGlicHRocmVhZC5zby4wAALvNQ9saWJjLnNvLjYA[/context]
2024-11-12T16:05:50.775Z error vpxd[37720] [Originator@6876 sub=IO.Http] User agent failed to send request; (null), N7Vmacore15SystemExceptionE(Connection refused: The remote service is not running, OR is overloaded, OR a firewall is rejecting connections.)
--> [context]zKq7AVECAQAAABvdLAEPdnB4ZAAAHug3bGlidm1hY29yZS5zbwAAAYksAIl7LQBX+DIAV9wjAPb3IwCDCCQAuQwkAKYUJAATuSMAl1ojAD+wIwDcczcBh38AbGlicHRocmVhZC5zby4wAALvNQ9saWJjLnNvLjYA[/context]
2024-11-12T16:05:50.775Z error vpxd[37599] [Originator@6876 sub=httpUtil] Error in sending request: N7Vmacore15SystemExceptionE(Connection refused: The remote service is not running, OR is overloaded, OR a firewall is rejecting connections.)
--> [context]zKq7AVECAQAAABvdLAEPdnB4ZAAAHug3bGlidm1hY29yZS5zbwAAAYksAIl7LQBX+DIAV9wjAPb3IwCDCCQAuQwkAKYUJAATuSMAl1ojAD+wIwDcczcBh38AbGlicHRocmVhZC5zby4wAALvNQ9saWJjLnNvLjYA[/context]
2024-11-12T16:05:50.777Z error vpxd[37599] [Originator@6876 sub=ServerAccess] Remote login failed: N3Vim5Fault9HttpFault9ExceptionE(Fault cause: vim.fault.HttpFault
--> )
--> [context]zKq7AVECAQAAABvdLAEVdnB4ZAAAHug3bGlidm1hY29yZS5zbwAAAYksAIl7LQBX+DIBGbdvdnB4ZAABbfmBgaKgYQEB0AiCAWAi1wHAGtcBssbVAhwsAWxpYmF1dGh6Y2xpZW50LnNvAALjOAECaeABAifVAQHQtNUBz9pvAatLcAEto28DhysCbGliYy5zby42AAERmW8=[/context]
2024-11-12T16:05:50.779Z error vpxd[37599] [Originator@6876 sub=AuthzStorageProvider] [AuthzStorageProvider::CreateAuthzMgr] Failed to connect to Authz service: <N5Vmomi5Fault17HostCommunication9ExceptionE(Fault cause: vmodl.fault.HostCommunication
--> )
--> [context]zKq7AVECAQAAABvdLAEVdnB4ZAAAHug3bGlidm1hY29yZS5zbwAAAYksAIl7LQBX+DIBGbdvdnB4ZAABlXV4AVN2eAFMC4IBYCLXAcAa1wGyxtUCHCwBbGliYXV0aHpjbGllbnQuc28AAuM4AQJp4AECJ9UBAdC01QHP2m8Bq0twAS2jbwOHKwJsaWJjLnNvLjYAARGZbw==[/context]>
2024-11-12T16:05:50.781Z info vpxd[37599] [Originator@6876 sub=AuthzStorageProvider] [AuthzStorageProvider::CreateAuthzMgr] Retry for this error: attempt count 17

the disk space is ok:

Filesystem                                Size  Used Avail Use% Mounted on
devtmpfs                                   14G     0   14G   0% /dev
tmpfs                                      14G  588K   14G   1% /dev/shm
tmpfs                                      14G  1.2M   14G   1% /run
tmpfs                                      14G     0   14G   0% /sys/fs/cgroup
/dev/mapper/vg_root_0-lv_root_0            47G  9.6G   35G  22% /
tmpfs                                      14G  3.0M   14G   1% /tmp
/dev/sda3                                 488M   30M  423M   7% /boot
/dev/mapper/netdump_vg-netdump            9.8G   37M  9.3G   1% /storage/netdump
/dev/mapper/autodeploy_vg-autodeploy       25G   45M   24G   1% /storage/autodeploy
/dev/mapper/dblog_vg-dblog                 25G  1.4G   22G   6% /storage/dblog
/dev/mapper/core_vg-core                   49G  4.4G   43G  10% /storage/core
/dev/mapper/imagebuilder_vg-imagebuilder   25G   45M   24G   1% /storage/imagebuilder
/dev/mapper/vtsdblog_vg-vtsdblog           25G   77M   24G   1% /storage/vtsdblog
/dev/mapper/updatemgr_vg-updatemgr         98G  3.7G   90G   4% /storage/updatemgr
/dev/mapper/lifecycle_vg-lifecycle         98G  4.4G   89G   5% /storage/lifecycle
/dev/sda2                                  10M  2.2M  7.9M  22% /boot/efi
/dev/mapper/archive_vg-archive             98G   16G   77G  18% /storage/archive
/dev/mapper/seat_vg-seat                  541G  2.2G  511G   1% /storage/seat
/dev/mapper/vtsdb_vg-vtsdb                541G  105M  513G   1% /storage/vtsdb
/dev/mapper/log_vg-log                     25G   11G   13G  47% /storage/log
/dev/mapper/db_vg-db                       25G  5.0G   19G  22% /storage/db

i can not find the problem. can someone help me please.

[Navin A]

Hi,

Please use option 8 and reset the certs. 

The services should be up post replacing the certs.

This issue would happen if the chain is not complete.

Regards,

Navin A

[luizitano]

Hi,

Thanks, i have done with option 8 several times and still have the issue.

*] Store : MACHI[*] Store : MACHINE_SSL_CERT
NE_SSL_CERTAlias :      __MACHINE_CERT
            Not After : Nov 13 13:12:36 2026 GMT
[*] Store : TRUSTED_ROOTS
Alias : 507c6a929140b95360b29503d9dce20cf5390f01
            Not After : Nov  3 00:42:35 2030 GMT
Alias : da6d595f3290bda8d6678702d0ecb0fc4b5d70d8
            Not After : Nov  3 06:37:01 2032 GMT
Alias : b9592c4aa626004f5e9c5079b023ea8eb96f92cc
            Not After : Nov  3 06:51:25 2032 GMT
Alias : f1208dd46e4223887e444ab4975f7d8df2dcd3f0
            Not After : Nov  3 06:52:24 2032 GMT
Alias : aff6173cb705c256335b19f3d0303a0920cbdb43
            Not After : Nov  3 10:22:18 2034 GMT
Alias : 5e9efaa86e2391be1c5e672a0bc08b27e501119c
            Not After : Nov  3 10:29:52 2034 GMT
Alias : edc792057763299cfff2b82db65a3a4164cb38f0
            Not After : Nov  3 10:43:22 2034 GMT
Alias : 60efdccba797489d14fe90dad258cb2113f5c308
            Not After : Nov  3 10:46:13 2034 GMT
Alias : 835a8647456caf66f98b0d9bab7bd64d1cf380cd
            Not After : Nov  3 12:09:56 2034 GMT
Alias : 9c2165431c14cfec20bfc37ebb0390d7aa2d672a
            Not After : Nov  3 12:11:08 2034 GMT
Alias : ee19eece717750b01465cedbdeb5daf7c6967d9d
            Not After : Nov  3 13:45:53 2034 GMT
Alias : 9c1ff4755a7ed52943caea2bc86e549824a397e8
            Not After : Nov  3 14:01:30 2034 GMT
Alias : b5a000cba4868e8bf12518888c7c61f7d691e073
            Not After : Nov  4 21:08:02 2034 GMT
Alias : ce732919a95742c49a81aebef27a8c1adda3549b
            Not After : Nov  6 11:32:25 2034 GMT
Alias : 1b3aea7841462fa3821d92a512c92fe59f691503
            Not After : Nov  6 12:14:38 2034 GMT
Alias : 3b8ba019081408cf5478a7ce4372774add373110
            Not After : Nov  6 14:00:59 2034 GMT
Alias : 5421eafe4cc8fb1fd500e0854deae9a8d0074505
            Not After : Nov  6 14:16:26 2034 GMT
Alias : e7bd972af9018f905999f71f3c5a4634e7dea06a
            Not After : Nov  6 20:41:58 2034 GMT
Alias : 9399a66c96aa8d6969ce594ae49646359df4ed72
            Not After : Nov  7 09:05:07 2034 GMT
Alias : 7e5010982a489aabe2ee754d982f5a645f94a976
            Not After : Nov  7 15:49:41 2034 GMT
Alias : efaeb741e4ef3772cf0948909145d13e03c7cb73
            Not After : Nov  7 16:30:58 2034 GMT
Alias : 6eba2bd6a4222e4bc2fc172b4edd47e588a12570
            Not After : Nov  8 12:56:32 2034 GMT
Alias : 98b09befd6c201fea1c0d75c3b160143c9ef27c4
            Not After : Nov  8 13:01:27 2034 GMT
Alias : c13b69acf10eeacccd844981b8fc8dc041ddd2a9
            Not After : Nov  8 13:22:35 2034 GMT
Alias : 64f77e051e12bcf3c6f0d11795ff537766b08383
            Not After : Jan  9 01:50:05 2022 GMT
Alias : b95da3757c9e37e726d5f3539b4210853f548e08
            Not After : Apr  5 08:10:10 2023 GMT
Alias : ea93193f725d1380479144a1e9cbf811e8e5b885
            Not After : Nov  8 14:49:41 2034 GMT
Alias : 7a97332b8671031aaae8f8ba1cd41a7fcf7658c9
            Not After : Nov  8 16:37:23 2034 GMT
[*] Store : machine
Alias : machine
            Not After : Nov 13 13:13:33 2026 GMT
[*] Store : vsphere-webclient
Alias : vsphere-webclient
            Not After : Nov 13 13:13:34 2026 GMT
[*] Store : vpxd
Alias : vpxd
            Not After : Nov 13 13:13:35 2026 GMT
[*] Store : vpxd-extension
Alias : vpxd-extension
            Not After : Nov 13 13:13:35 2026 GMT
[*] Store : hvc
Alias : hvc
            Not After : Nov 13 13:13:38 2026 GMT
[*] Store : data-encipherment
Alias : data-encipherment
            Not After : Nov  3 00:42:35 2030 GMT
[*] Store : APPLMGMT_PASSWORD
Alias : location_password_default
[*] Store : SMS
Alias : sms_self_signed
            Not After : Nov  8 00:46:17 2030 GMT
[*] Store : wcp
Alias : wcp
            Not After : Nov 13 13:13:38 2026 GMT
[*] Store : BACKUP_STORE
Alias : bkp___MACHINE_CERT
            Not After : Nov 13 13:12:36 2026 GMT
Alias : bkp_machine
            Not After : Nov 13 13:13:33 2026 GMT
Alias : bkp_vsphere-webclient
            Not After : Nov 13 13:13:34 2026 GMT
Alias : bkp_vpxd
            Not After : Nov 13 13:13:35 2026 GMT
Alias : bkp_vpxd-extension
            Not After : Nov 13 13:13:35 2026 GMT
Alias : bkp_hvc
            Not After : Nov 13 13:13:38 2026 GMT
Alias : bkp_wcp
            Not After : Nov 13 13:13:38 2026 GMT

[Daniel Moses]

I had an old issue something like this, back on 6.7 with a VASA cert and my storage array, but it was a bad alias name in the Trusted Root store(see below).  I do see two trusted root certs in your list that are showing expired, might want to track down what they go to and see if they are causing the issue.

https://community.broadcom.com/vmware-cloud-foundation/discussion/vcsa-67-vpxd-doesnt-start-after-replacing-machine-ssl-certs

[new_bember]

Hello @Daniel Moses, I think good idea to check all your certs/stores settings with vCert script by [e-mail]

[luizitano]

Hi, @new_bember

where i can find this script?

Thanks and Regards

[new_bember]

Hi, @luizitano, frankly speaking I didnt read forum rules, so not sure if it allowed to post links here, but you can just google "vCert script by [e-mail]" and you will find it in first three rows.

[luizitano]

Hi,

Many thanks for the information, but when i try to run the script i have this warning:

The VMware Directory service is not in NORMAL mode!
Certificate operations should not be actioned until this service
is running correctly in a NORMAL state.

vCenter 7.0 Certificate Management Utility (4.7.0)
-----------------------------------------------------------------
1. Check current certificates status
2. View Certificate Info
3. Manage Certificates
4. Manage SSL Trust Anchors
5. Check configurations
6. Reset all certificates with VMCA-signed certificates
7. ESXi certificate operations
8. Restart services
9. Generate certificate report
E. Exit

 

The VMware Directory service is not in NORMAL mode!
Certificate operations should not be actioned until this service
is running correctly in a NORMAL state.

 

root@vCSA-001 [ ~ ]# service-control --status vmdird
Running:
vmdird

 

[new_bember]

@luizitano do you have any replication with other vCenter?

Check this vmdir logs and this KBs, maybe your case:

https://knowledge.broadcom.com/external/article?legacyId=70756 - this one shows how to set vmdir state to NORMAL

https://knowledge.broadcom.com/external/article/319348/ldap-error-code-49error-49-error-in-vmdi.html

[luizitano]

Hi,

before all, many thanks for your inputs, 

no i don't have any replication. i am doing backup of all and after that i will try to change the state to normal. I have afraid what this changes can do to vsan storage. 

[dabrigo]

@luizitano, we just fixed a similar problem caused by an expired STS certificate, it didn't show as expired from the checksts.py script.

Have a look at this KB: https://knowledge.broadcom.com/external/article/322249/how-to-replace-expired-certificates-on-v.html and maybe try with the fixcerts.py script.

[Santosh Chikkadevaraju]

Try the below to check the vmdird status

# /usr/lib/vmware-vmafd/bin/dir-cli state get

Run the below to change it to Normal if its not

# /usr/lib/vmware-vmafd/bin/dir-cli state set --state NORMAL

Check the status again, if it keeps going back to anything other than Normal then there is an issue with VMDIRD

/var/log/vmware/vmdird/vmdird.log will give some insight

[luizitano]

Hello, thanks for all the answer´s

 

I have changed the state to NORMAL and also try to vcert script by [e-mail] and the checksts.py script.

 

The certificates are renwed sucessfully, the commun problem in all the procedures is that some services dont came up.

 

root@vCSA-001 [ ~ ]# service-control --status --all
Running:
applmgmt lookupsvc lwsmd observability observability-vapi pschealth vlcm vmafdd vmcad vmcam vmdird vmonapi vmware-analytics vmware-certificateauthority vmware-certificatemanagement vmware-cis-license vmware-eam vmware-envoy vmware-hvc vmware-infraprofile vmware-perfcharts vmware-postgres-archiver vmware-rhttpproxy vmware-sca vmware-statsmonitor vmware-stsd vmware-topologysvc vmware-trustmanagement vmware-vapi-endpoint vmware-vmon vmware-vpostgres vmware-vpxd-svcs vmware-vsm vsphere-ui vtsdb
Stopped:
vmware-content-library vmware-imagebuilder vmware-netdumper vmware-pod vmware-rbd-watchdog vmware-sps vmware-updatemgr vmware-vcha vmware-vdtc vmware-vpxd vmware-vsan-health vstats wcp

 

 

Even if i try to start one by one it gives a error:

 

 

root@vCSA-001 [ ~ ]# service-control --start vmware-vpxd
Operation not cancellable. Please wait for it to finish...
Performing start operation on service vpxd...
Error executing start on service vpxd. Details {
    "detail": [
        {
            "id": "install.ciscommon.service.failstart",
            "translatable": "An error occurred while starting service '%(0)s'",
            "args": [
                "vpxd"
            ],
            "localized": "An error occurred while starting service 'vpxd'"
        }
    ],
    "componentKey": null,
    "problemId": null,
    "resolution": null
}
Service-control failed. Error: {
    "detail": [
        {
            "id": "install.ciscommon.service.failstart",
            "translatable": "An error occurred while starting service '%(0)s'",
            "args": [
                "vpxd"
            ],
            "localized": "An error occurred while starting service 'vpxd'"
        }
    ],
    "componentKey": null,
    "problemId": null,
    "resolution": null
}

 

[luizitano]

Hello, thanks for all the answer´s

 

I have changed the state to NORMAL and also try to vcert script by [e-mail] and the checksts.py script.

 

The certificates are renwed sucessfully, the commun problem in all the procedures is that some services dont came up.

 

root@vCSA-001 [ ~ ]# service-control --status --all
Running:
applmgmt lookupsvc lwsmd observability observability-vapi pschealth vlcm vmafdd vmcad vmcam vmdird vmonapi vmware-analytics vmware-certificateauthority vmware-certificatemanagement vmware-cis-license vmware-eam vmware-envoy vmware-hvc vmware-infraprofile vmware-perfcharts vmware-postgres-archiver vmware-rhttpproxy vmware-sca vmware-statsmonitor vmware-stsd vmware-topologysvc vmware-trustmanagement vmware-vapi-endpoint vmware-vmon vmware-vpostgres vmware-vpxd-svcs vmware-vsm vsphere-ui vtsdb
Stopped:
vmware-content-library vmware-imagebuilder vmware-netdumper vmware-pod vmware-rbd-watchdog vmware-sps vmware-updatemgr vmware-vcha vmware-vdtc vmware-vpxd vmware-vsan-health vstats wcp

 

 

Even if i try to start one by one it gives a error:

 

 

root@vCSA-001 [ ~ ]# service-control --start vmware-vpxd
Operation not cancellable. Please wait for it to finish...
Performing start operation on service vpxd...
Error executing start on service vpxd. Details {
    "detail": [
        {
            "id": "install.ciscommon.service.failstart",
            "translatable": "An error occurred while starting service '%(0)s'",
            "args": [
                "vpxd"
            ],
            "localized": "An error occurred while starting service 'vpxd'"
        }
    ],
    "componentKey": null,
    "problemId": null,
    "resolution": null
}
Service-control failed. Error: {
    "detail": [
        {
            "id": "install.ciscommon.service.failstart",
            "translatable": "An error occurred while starting service '%(0)s'",
            "args": [
                "vpxd"
            ],
            "localized": "An error occurred while starting service 'vpxd'"
        }
    ],
    "componentKey": null,
    "problemId": null,
    "resolution": null
}

[Santosh Chikkadevaraju]

Hi luizitano,

Run the vCert again and renew all certs. Choose not to restart services

Let the certificate renew completely and then restart the services manually

# service-control --stop --all && service-control --start --all

If the services are still not coming up, then run lsdoctor -l to find any issues

KB: https://knowledge.broadcom.com/external/article?legacyId=80469

[luizitano]

Hello,

Thanks for the help.

I had run the Vcert again without restarting services

vCenter 7.0 Certificate Management Utility (4.7.0)

-----------------------------------------------------------------

 1. Check current certificates status

 2. View Certificate Info

 3. Manage Certificates

 4. Manage SSL Trust Anchors

 5. Check configurations

 6. Reset all certificates with VMCA-signed certificates

 7. ESXi certificate operations

 8. Restart services

 9. Generate certificate report

 E. Exit

Select an option [1]: 6

Please enter a Single Sign-On administrator account [[e-mail]]:

Please provide the password for [e-mail]:

Certificate Signing Request Information

-----------------------------------------------------------------

Enter the country code [US]:

Enter the Organization name [VMware]:

Enter the Organizational Unit name [VMware Engineering]:

Enter the state [California]:

Enter the locality (city) name [Palo Alto]:

Enter the IP address (optional): *******

Enter an email address (optional):

Replace Machine SSL Certificate

-----------------------------------------------------------------

Generate certool configuration                                 OK

Regenerate Machine SSL certificate                             OK

Backing up certificate and private key                         OK

Updating MACHINE_SSL_CERT certificate                          OK

Replace Solution User Certificates

-----------------------------------------------------------------

Verifying Service Principal entries exist                      OK

Generate new certificates and keys:

   machine                                                     OK

   vsphere-webclient                                           OK

   vpxd                                                        OK

   vpxd-extension                                              OK

   hvc                                                         OK

   wcp                                                         OK

Backup certificate and private key:

   machine                                                     OK

   vsphere-webclient                                           OK

   vpxd                                                        OK

   vpxd-extension                                              OK

   hvc                                                         OK

   wcp                                                         OK

Updating certificates and keys in VECS:

   machine                                                     OK

   vsphere-webclient                                           OK

   vpxd                                                        OK

   vpxd-extension                                              OK

   hvc                                                         OK

   wcp                                                         OK

Updating solution user certificates in VMware Directory:

   machine                                                     OK

   vsphere-webclient                                           OK

   vpxd                                                        OK

   vpxd-extension                                              OK

   hvc                                                         OK

   wcp                                                         OK

Replace Authentication Proxy Certificate

-----------------------------------------------------------------

Generate certool configuration                                 OK

Regenerate Authentication Proxy certificate                    OK

Backing up certificate and private key                         OK

Replace certificate on filesystem                              OK

Replace Auto Deploy CA Certificate

-----------------------------------------------------------------

Regenerate Auto Deploy CA certificate                          OK

Backing up certificate and private key                         OK

Replace certificate on filesystem                              OK

Update vCenter Extension Thumbprints

-----------------------------------------------------------------

ESX Agent Manager                                           FIXED

Auto Deploy                                                 FIXED

VMware Update Manager                                       FIXED

vSAN Health                                                 FIXED

Authentication Proxy                                        FIXED

Replace SSO STS Signing Certificate

-----------------------------------------------------------------

Generate certool configuration                                 OK

Regenerate STS signing certificate                             OK

Backup and delete tenant credentials                           OK

Backup and delete trusted cert chains                          OK

Add new STS signing certifcate to VMDir                        OK

Update SSL Trust Anchors (vCSA-001.vmware.************)

-----------------------------------------------------------------

Updating service: 07fcf7d1-7f44-4ad8-b260-d4d8046b32f4

Updating service: 0a57ad42-63b4-4e8f-b22f-5ee4b35a3f85

Updating service: 0a595090-ade2-49ec-8315-c2e1b100bc44

Updating service: 0b2a5d03-f408-4093-b5fe-2a55bad4104e

Updating service: 0def6524-e335-42aa-bca3-809e0f7fc917

Updating service: 1a0a6190-1b9d-485b-b948-aaf766c20e44

Updating service: 21810db3-0c98-48c9-a72c-f1d9c1142b12

Updating service: 23a772e3-4f9e-432f-9df8-8748d6d4f5dc

Updating service: 30dcf398-8b15-44b6-9a1d-f0020b0ea897

Updating service: 34942b68-e488-4708-b714-eeea8a8f9bb5_authz

Updating service: 34942b68-e488-4708-b714-eeea8a8f9bb5

Updating service: 34942b68-e488-4708-b714-eeea8a8f9bb5_kv

Updating service: 38ffd6ff-9111-428e-9a1b-35f1ecb6ca47

Updating service: 3b4c9bcb-d86e-4155-853f-1b95b32b6930

Updating service: 405d6450-d7b3-4ddd-abba-d52edabae563

Updating service: 42c9b176-e915-449c-9254-101f078443a5

Updating service: 43fe3a4c-2d72-4727-9e56-b8dd7c45d7b0

Updating service: 4dffc9d5-5edc-4515-8467-9e2926e2be66

Updating service: 51eb6333-d11b-48c2-92a6-69e6270bec9f

Updating service: 599d6ca9-a15e-4af8-8da0-fbb03f73c1e6

Updating service: 684d7122-9501-4568-b15a-0d3a03c78511

Updating service: 687a30e9-7c6e-4f9a-b7db-cc69b9b99593

Updating service: 74d4a77e-403d-4b03-bd07-eae503f8e69b

Updating service: 917cc1b2-6196-4aa6-89ec-aa404fc1e09e

Updating service: 97bee298-0266-4f5f-8f9d-80dd107f69f9

Updating service: 98758838-b6b4-4e90-be54-2994e38ca96d

Updating service: 9cd8c5e8-eddb-4fbe-8f12-df16bf59c15e

Updating service: a12abf1f-5a94-4ac3-acf6-e39d02ac6202

Updating service: a5006d33-0ecb-4de5-8250-400e05bb9ee0

Updating service: b5d8cda9-43e6-4e9e-96ef-2e20e8f4687a

Updating service: be874ee7-241d-40ff-80d3-0b89efa44f41

Updating service: bf2807db-c4d3-4a81-ba6d-c36463e60e06

Updating service: cc56d11a-8d0f-41e3-be08-bbf8a0eccacb

Updating service: d2896ae4-8189-4dca-8263-23d24ac2dd7a

Updating service: d3bce6f2-f9e8-49fa-9f90-af4b4789d98e

Updating service: def3baa5-83ed-4f65-8ea2-55bf22b0b3ae

Updating service: default-site:49f6126e-2aca-4ea2-a361-7dec60fb78a0

Updating service: default-site:7ec60460-dc8f-4ddf-809b-4d1c30d31123

Updating service: default-site:cc194ac0-000e-4ffd-b734-9fc1e4c48b21

Updating service: eb5c99f6-b985-4f38-99ad-36a2f4ce2899

Updating service: f0fea641-5c24-4a00-9bce-14d52fb20f05

Updating service: f734f424-fb41-4f90-b169-d98b30418fbf

Updating service: f900ac30-6841-4e94-8187-a7f5b4bfd3a6

Updated 43 service(s)

Restart VMware services [no]:

Then restarted manually the services:

root@vCSA-001 [ ~ ]# service-control --stop --all

Operation not cancellable. Please wait for it to finish...

Performing stop operation on service observability...

Successfully stopped service observability

Performing stop operation on service vmware-pod...

Successfully stopped service vmware-pod

Performing stop operation on service vmware-vdtc...

Successfully stopped service vmware-vdtc

Performing stop operation on profile: ALL...

Successfully stopped service vmware-vmon

Successfully stopped profile: ALL.

Performing stop operation on service vmcad...

Successfully stopped service vmcad

Performing stop operation on service vmdird...

Successfully stopped service vmdird

Performing stop operation on service vmafdd...

Successfully stopped service vmafdd

Performing stop operation on service lwsmd...

Successfully stopped service lwsmd

root@vCSA-001 [ ~ ]# service-control --start --all

Operation not cancellable. Please wait for it to finish...

Performing start operation on service lwsmd...

Successfully started service lwsmd

Performing start operation on service vmafdd...

Successfully started service vmafdd

Performing start operation on service vmdird...

Successfully started service vmdird

Performing start operation on service vmcad...

Successfully started service vmcad

Performing start operation on profile: ALL...

Successfully started service vmware-vmon

Service-control failed. Error: Failed to start services in profile ALL. RC=1, stderr=Failed to start sps, content-library, vstats, vpxd, vsan-health, wcp services. Error: Operation timed out

And run the lsdoctor -l to check for issues, but no problem was detected.

~/lsdoctor-240201 ]# cat /var/log/vmware/lsdoctor/vCSA-001.vmware.5glabaltran.com-2024-11-27-122826.json | grep "problems"

            "Problems Detected": "No problems found.",

                        "Problems Detected": "No problems found.",

                        "Problems Detected": "No problems found.",

                        "Problems Detected": "No problems found.",

                        "Problems Detected": "No problems found.",

                        "Problems Detected": "No problems found.",

                        "Problems Detected": "No problems found.",

                        "Problems Detected": "No problems found.",

                        "Problems Detected": "No problems found.",

                        "Problems Detected": "No problems found.",

                        "Problems Detected": "No problems found.",

                        "Problems Detected": "No problems found.",

                        "Problems Detected": "No problems found.",

                        "Problems Detected": "No problems found.",

                        "Problems Detected": "No problems found.",

                        "Problems Detected": "No problems found.",

                        "Problems Detected": "No problems found.",

                        "Problems Detected": "No problems found.",

                        "Problems Detected": "No problems found.",

                        "Problems Detected": "No problems found.",

                        "Problems Detected": "No problems found.",

                        "Problems Detected": "No problems found.",

                        "Problems Detected": "No problems found.",

                        "Problems Detected": "No problems found.",

                        "Problems Detected": "No problems found.",

                        "Problems Detected": "No problems found.",

                        "Problems Detected": "No problems found.",

                        "Problems Detected": "No problems found.",

                        "Problems Detected": "No problems found.",

                        "Problems Detected": "No problems found.",

                        "Problems Detected": "No problems found.",

                        "Problems Detected": "No problems found.",

                        "Problems Detected": "No problems found.",

                        "Problems Detected": "No problems found.",

                        "Problems Detected": "No problems found.",

                        "Problems Detected": "No problems found.",

                        "Problems Detected": "No problems found.",

                        "Problems Detected": "No problems found.",

                        "Problems Detected": "No problems found.",

                        "Problems Detected": "No problems found.",

                        "Problems Detected": "No problems found.",

                        "Problems Detected": "No problems found.",

                        "Problems Detected": "No problems found.",

                        "Problems Detected": "No problems found.",

                        "Problems Detected": "No problems found.",

                        "Problems Detected": "No problems found.",

                        "Problems Detected": "No problems found.",

                        "Problems Detected": "No problems found.",

                        "Problems Detected": "No problems found.",

                        "Problems Detected": "No problems found.",

                        "Problems Detected": "No problems found.",

                        "Problems Detected": "No problems found.",

    Any more sugestions?

    Apreciate all the help.

[Fausto Sbarra]

Hellow, my name is fausto sbarra from SCHNEIDER ELECTRIC Madrid , i was following all the steps for download my license key to put it into the program in my desktop, but i dont see it in "MY ETITLEMENT"----> HOME FOLDER . Is missing a permissions and i dont know how to get them. 

IS VERY URGENT , THANKS. 

Fausto Sbarra

Execution Technical Project Designer – Digital Energy, Autom Schneider Electric	MS TEAMS  [e-mail]	Avda. Tenerife 4-6 Edificio Prisma 1ª planta San Sebastián de los Reyes 28703