On one of my systems running VCSA 7.0.3.01300, I cannot update the SSL certificates. I'd like to use ones signed by my internal PKI. I've done this successfully in the past, but for some reason, it's not working on this particular system.
What I have done:
Added the issuing CA to the trusted root certificate collection via. This is successful:
# /usr/lib/vmware-vmafd/bin/dir-cli trustedcert publish --chain --cert PKI-CA.crt
Then, I run /usr/lib/vmware-vmca/bin/certificate-manager, go through the options, specify the new machine cert, the private key, the signing certificate again. The process of replacing starts. But when it gets to restarting the services, it fails.
Looking at the log, I have this:
2023-06-08T11:19:07.474Z INFO certificate-manager service endpoints updated with replaced tls certificate successfully
2023-06-08T11:19:07.475Z INFO certificate-manager Running command : ['s', 'e', 'r', 'v', 'i', 'c', 'e', '-', 'c', 'o', 'n', 't', 'r', 'o', 'l', ' ', '-', '-', 's', 't', 'o', 'p', ' ', '-', '-', 'i', 'g', 'n', 'o', 'r', 'e', ' ', ' ', '-', '-', 'a', 'l', 'l', ' ', '-', '-', 'v', 'm', 'o', 'n', '-', 'p', 'r', 'o', 'f', 'i', 'l', 'e', ' ', 'A', 'L', '*****']
2023-06-08T11:19:07.475Z INFO certificate-manager please see service-control.log for service status
2023-06-08T11:19:55.385Z INFO certificate-manager Command executed successfully
2023-06-08T11:19:55.386Z INFO certificate-manager all services stopped successfully.
2023-06-08T11:19:55.386Z INFO certificate-manager None
2023-06-08T11:20:05.397Z INFO certificate-manager Running command :- service-control --start --all
2023-06-08T11:20:05.398Z INFO certificate-manager please see service-control.log for service status
Service-control failed. Error: Failed to start services in profile ALL. RC=2, stderr=Failed to start hvc, vpxd, vpxd-svcs services. Error: Service crashed while starting
2023-06-08T11:26:48.186Z ERROR certificate-manager None
2023-06-08T11:26:48.187Z ERROR certificate-manager Error while starting services, please see service-control log for more details
2023-06-08T11:26:48.187Z ERROR certificate-manager Error while replacing Machine SSL Cert, please see /var/log/vmware/vmcad/certificate-manager.log for more information.
2023-06-08T11:26:48.187Z ERROR certificate-manager {
"detail": [
{
"id": "install.ciscommon.command.errinvoke",
"translatable": "An error occurred while invoking external command : '%(0)s'",
"args": [
"None"
],
"localized": "An error occurred while invoking external command : 'None'"
},
"Error while starting services, please see service-control log for more details"
],
"componentKey": null,
"problemId": null,
"resolution": null
}
It then reverts to the original self-signed certificates.
Is having a certiciate with a 4096 bit public key a problem and VCSA only supports 2048 bit ?
I have a pre-start log showing some errors. The timestamp seems to match that when the error is flagged in the above log:
INFO:__main__:Executing vpxd-svcs pre start commands.
INFO:__main__:Executing vpxd-svcs endpoint registration runner
INFO:__main__:detected vpxdsvcs.version : 8
INFO:__main__:Tagging service grpc enpoint registration : started
INFO:tagging_grpc_registration:Updating tagging service grpc endpoint.
INFO:tagging_grpc_registration:Connecting to Lookup Service
INFO:tagging_grpc_registration:Getting STS endpoint
INFO:tagging_grpc_registration:Logging into SSO AdminClientas machine solution user
INFO:tagging_grpc_registration:Check if gRPC endpoints exist
INFO:tagging_grpc_registration:Tagging service gRPC endpoints exist
INFO:tagging_grpc_registration:Updating spec and re-registering service
INFO:tagging_grpc_registration:gRPC endpoint found. Updating it with the reverse proxy port
INFO:tagging_grpc_registration:attibute local url found , updating the value
ERROR:tagging_grpc_registration:Failed to reregister Tagging service grpc endpoints with Lookup Service
ERROR:tagging_grpc_registration:(vmodl.fault.SystemError) {
dynamicType = <unset>,
dynamicProperty = (vmodl.DynamicProperty) [],
msg = 'LookupFaultServiceFault',
faultCause = <unset>,
faultMessage = (vmodl.LocalizableMessage) [],
reason = 'Invalid fault'
}
Traceback (most recent call last):
File "/usr/lib/vmware-vpxd-svcs/scripts/linux/pre-start/tagging_grpc_registration.py", line 119, in update_endpoints
ls_obj.reregister_service(service_info.serviceId, mutable_spec)
File "/usr/lib/vmware/site-packages/cis/cisreglib.py", line 348, in add_securityctx_to_requests
return req_method(self, *args, **kargs)
File "/usr/lib/vmware/site-packages/cis/cisreglib.py", line 364, in reregister_service
self.service_content.serviceRegistration.Set(svc_id, svc_set_spec)
File "/usr/lib/vmware/site-packages/pyVmomi/VmomiSupport.py", line 595, in <lambda>
self.f(*(self.args + (obj,) + args), **kwargs)
File "/usr/lib/vmware/site-packages/pyVmomi/VmomiSupport.py", line 385, in _InvokeMethod
return self._stub.InvokeMethod(self, info, args)
File "/usr/lib/vmware/site-packages/pyVmomi/SoapAdapter.py", line 1570, in InvokeMethod
raise obj # pylint: disable-msg=E0702
pyVmomi.VmomiSupport.vmodl.fault.SystemError: (vmodl.fault.SystemError) {
dynamicType = <unset>,
dynamicProperty = (vmodl.DynamicProperty) [],
msg = 'LookupFaultServiceFault',
faultCause = <unset>,
faultMessage = (vmodl.LocalizableMessage) [],
reason = 'Invalid fault'
}
looking at the log in /var/log/vmware/vpxd-svcs/vpxd-svcs-runtime.log.stderr, I also have something that looks related (although it appears later):
Starting service process with pid: 35183.
Picked up JAVA_TOOL_OPTIONS: -Xms32M -Xmx128M -Dcom.sun.org.apache.xml.internal.security.ignoreLineBreaks=true -Dorg.apache.xml.security.ignoreLineBreaks=true
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/lib/vmware-vpxd-svcs/lib/log4j-slf4j-impl.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/lib/vmware/common-jars/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Jun 08, 2023 11:29:18 AM org.bouncycastle.jsse.provider.PropertyUtils getStringSecurityProperty
INFO: Found string security property [jdk.tls.disabledAlgorithms]: SSLv3, RC4, DES, MD5withRSA, DH keySize < 1024, EC keySize < 224, 3DES_EDE_CBC, anon, NULL
Jun 08, 2023 11:29:18 AM org.bouncycastle.jsse.provider.PropertyUtils getStringSecurityProperty
INFO: Found string security property [jdk.certpath.disabledAlgorithms]: MD2, MD5, SHA1 jdkCA & usage TLSServer, RSA keySize < 1024, DSA keySize < 1024, EC keySize < 224
Jun 08, 2023 11:29:18 AM org.bouncycastle.jsse.provider.DisabledAlgorithmConstraints create
WARNING: Ignoring unsupported entry in 'jdk.certpath.disabledAlgorithms': SHA1 jdkCA & usage TLSServer
Jun 08, 2023 11:29:28 AM org.bouncycastle.jsse.provider.ProvTrustManagerFactorySpi getDefaultTrustStore
INFO: Initializing with trust store at path: /usr/java/jre-vmware/lib/security/cacerts
Jun 08, 2023 11:29:29 AM org.apache.catalina.core.StandardContext setPath
WARNING: A context path must either be an empty string or start with a '/' and do not end with a '/'. The path [/] does not meet these criteria and has been changed to []
Jun 08, 2023 11:29:29 AM org.apache.catalina.core.AprLifecycleListener lifecycleEvent
INFO: The Apache Tomcat Native library which allows using OpenSSL was not found on the java.library.path: [../lib:../../bin]
Jun 08, 2023 11:29:29 AM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["http-nio-127.0.0.1-10080"]
Jun 08, 2023 11:29:29 AM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["http-nio-0:0:0:0:0:0:0:1-10080"]
Jun 08, 2023 11:29:29 AM org.apache.catalina.core.StandardService initInternal
SEVERE: Failed to initialize connector [Connector[HTTP/1.1-10080]]
org.apache.catalina.LifecycleException: Protocol handler initialization failed
at org.apache.catalina.connector.Connector.initInternal(Connector.java:1115)
at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136)
at org.apache.catalina.core.StandardService.initInternal(StandardService.java:571)
at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136)
at org.apache.catalina.core.StandardServer.initInternal(StandardServer.java:874)
at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:173)
at org.apache.catalina.startup.Tomcat.start(Tomcat.java:440)
at com.vmware.vim.vmomi.server.http.impl.TcServer.start(TcServer.java:275)
at com.vmware.vim.dataservices.DataService.init(DataService.java:62)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeCustomInitMethod(AbstractAutowireCapableBeanFactory.java:1925)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1867)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1795)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:594)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:516)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:342)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1109)
at com.vmware.vim.dataservices.DataService.getService(DataService.java:160)
at com.vmware.vim.dataservices.VpxdSvcsMain$Main_Vmon.start(VpxdSvcsMain.java:54)
at com.vmware.vim.dataservices.VpxdSvcsMain.main(VpxdSvcsMain.java:41)
Caused by: java.net.SocketException: Protocol family unavailable
at sun.nio.ch.Net.bind0(Native Method)
at sun.nio.ch.Net.bind(Net.java:461)
at sun.nio.ch.Net.bind(Net.java:453)
at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:222)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:85)
at org.apache.tomcat.util.net.NioEndpoint.initServerSocket(NioEndpoint.java:225)
at org.apache.tomcat.util.net.NioEndpoint.bind(NioEndpoint.java:201)
at org.apache.tomcat.util.net.AbstractEndpoint.bindWithCleanup(AbstractEndpoint.java:1221)
at org.apache.tomcat.util.net.AbstractEndpoint.init(AbstractEndpoint.java:1234)
at org.apache.tomcat.util.net.AbstractJsseEndpoint.init(AbstractJsseEndpoint.java:230)
at org.apache.coyote.AbstractProtocol.init(AbstractProtocol.java:633)
at org.apache.coyote.http11.AbstractHttp11Protocol.init(AbstractHttp11Protocol.java:80)
at org.apache.catalina.connector.Connector.initInternal(Connector.java:1112)
... 24 more
Jun 08, 2023 11:29:29 AM org.apache.catalina.core.StandardService startInternal
INFO: Starting service [Tomcat]
Jun 08, 2023 11:29:29 AM org.apache.catalina.core.StandardEngine startInternal
INFO: Starting Servlet engine: [Apache Tomcat/8.5.82]
Jun 08, 2023 11:29:31 AM org.apache.coyote.AbstractProtocol start
INFO: Starting ProtocolHandler ["http-nio-127.0.0.1-10080"]
Jun 08, 2023 11:29:32 AM org.bouncycastle.jsse.provider.ProvTrustManagerFactorySpi getDefaultTrustStore
INFO: Initializing with trust store at path: /usr/java/jre-vmware/lib/security/cacerts
Jun 08, 2023 11:34:28 AM org.bouncycastle.jsse.provider.ProvTrustManagerFactorySpi getDefaultTrustStore
INFO: Initializing with trust store at path: /usr/java/jre-vmware/lib/security/cacerts
Any ideas where to look?