Tried out CU7 but startup of controller fails with "error:0x0A00018E: SSL routines: func(167772558): ca md too weak"

[Garin Walsh]

Mar 11 09:17:10:673 [5060] 0 Controller:  Loglevel = 0, Logfile = controller.log
Mar 11 09:17:11:462 [5060] 0 Controller:  Running as user SYSTEM
Mar 11 09:17:11:462 [5060] 0 Controller: ----- 
Mar 11 09:17:11:462 [5060] 0 Controller: ssl_log_error - Could not read public key file., [1] error:0x0A00018E: SSL routines: func(167772558): ca md too weak
Mar 11 09:17:11:462 [5060] 0 Controller: (controller proxy ssl setup): Failed to initialize ssl context.
Mar 11 09:17:11:462 [5060] 0 Controller: proxy setup: Failed to setup ssl context
Mar 11 09:17:11:462 [5060] 0 Controller: Proxy initialization failed - rc=1 error

Central hub is running secure robot and hub and it appears that in the controller distributed with CU7, the minimum allowed security for the CA has been increased.

Any suggestions on a reasonable process to update the robot CA certificate to an appropriate security level and what that level might be?

The release notes seem to be void of any mention of this change Robot Release Notes

Broadcom		remove preview
Robot Release Notes The controller probe is a core component of the DX UIM robot. View this on Broadcom >

-------------------------------------------

[Jason Allen]

Hi Garin,

Thanks for bringing this to our attention.  I am looking into it with the internal team.  Are you using Third-Party/CA-Signed certificates in this environment?

-------------------------------------------

[Garin Walsh]

No to the Third Party question.

The specific server/environment was originally deployed long ago (controller 5.4 days) and upgraded. In Feb or 2023 it was converted to secure hub/controller using the then current UIM installer.

The relevant section of the robot.cfg file looks like:

   robot_mode = normal
   proxy_ca_location = C:\Program Files (x86)\Nimsoft\.\hub\certs\cert02.pem
   proxy_cert = C:\Program Files (x86)\Nimsoft\.\hub\certs\cert02.pem
   proxy_private_key = C:\Program Files (x86)\Nimsoft\.\hub\certs\cert02.pem
   proxy_private_key_password = NotARealPassword==

I downgraded controller to the 23.4.6.S version while leaving the 23.4.7.S hub in place and the controller and whatnot start.

I then created another certificate essentially identical to the cert02.pem - because in creating the new cert you can only affect the common name and expiration - using the hub and the steps from the manual convert to secure KB.

If I then replace the cert02.pem with cert03.pem (the newly created cert) then the restart, the 23.4.6.S controller starts and runs with the new cert.

Using this new cert, the upgrade then appears to work.

Between the two certs, the first section of the new cert is 4 lines shorter than the old. 

The middle section of the old cert refers to "-----BEGIN RSA PRIVATE KEY-----" and the new one refers to "-----BEGIN EC PRIVATE KEY-----" and the section from the new cert is 21 lines shorter tan the old.

The third section of the cert is identical between the two.

So I am guessing that there's something in the internal parameters that the hub uses to create a cert that's changed that works OK with the latest controller.

-------------------------------------------

[Jason Allen]

Thanks for the additional details - the workaround information is very helpful.  We are looking into this, and I will update this thread once we have further details.

-------------------------------------------

[Jan Olav Melbye]

I see the same issue. Both in a newly upgraded to CU6 (from CU3) production environment and in a CU7 (from CU5) in lab environment. It looks like all tunnels are asking for a private/public key depending on if it is a server or a client. I've not had the time to look into the details like Garin have (thanks for that mate!). I also had the same guess that there is an internal parameter for the key pointing in the wrong direction when establishing the tunnels. In other words, it is not looking in the issued certificate file but are trying to find it elsewhere?

-------------------------------------------

[Garin Walsh]

Note that, as would be expected, this also impacts the tunnel server portion of the hub which fails to start:

Mar 13 08:47:52:381 [10592] 2 hub: nimVerifyLogin - probe name from SID:administrator
Mar 13 08:47:52:381 [10592] 2 hub: nimVerifyLogin hub
Mar 13 08:47:52:381 [10592] 1 hub: verify - [target] prid=hub cmd=tunnel_start ip=127.0.0.1 perm=3
Mar 13 08:47:52:381 [10592] 1 hub: verify - [source] id=administrator ip=1.1.1.1 hub=chub01hub (OK)
Mar 13 08:47:52:381 [10592] 2 hub: tunnel_start - type=1 from 127.0.0.1/52632
Mar 13 08:47:52:381 [10592] 1 hub: CORE adding command type=1, action=1 id=N/A
Mar 13 08:47:52:390 [9952] 1 hub: CORE popping command type=1, action=1 id=N/A
Mar 13 08:47:52:390 [9952] 1 hub: CORE start server received
Mar 13 08:47:52:390 [9952] 4 hub: cfgReader file open: hub.cfg
Mar 13 08:47:52:391 [9952] 4 hub: cfgReader file close: hub.cfg
Mar 13 08:47:52:391 [9952] 4 hub: cfgReader file open: hub.cfg
Mar 13 08:47:52:392 [9952] 4 hub: cfgReader file close: hub.cfg
Mar 13 08:47:52:392 [9952] 0 hub: CORE load -- (SERVER) -- public certificate () failed. Reason=(not found)
Mar 13 08:47:52:392 [9952] 3 hub: CORE - server common name: 1.1.1.1
Mar 13 08:47:52:392 [9952] 0 hub: CORE server weak cipher configuration upgraded
Mar 13 08:47:52:392 [9952] 1 hub: CORE server cipher=EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH:!RC4:!DSS
Mar 13 08:47:52:392 [9952] 1 hub: (ssl_ctx_setup) ca location -> certs/server.pem
Mar 13 08:47:52:392 [9952] 1 hub: (ssl_ctx_setup) certificate -> certs/server.pem
Mar 13 08:47:52:392 [9952] 1 hub: (ssl_ctx_setup) private key file -> certs/server.pem
Mar 13 08:47:52:393 [9952] 0 hub: ssl_log_error - Could not read public key file., [1] error:0x0A00018E: SSL routines: func(167772558): ca md too weak
Mar 13 08:47:52:393 [9952] 0 hub: CORE server failed to create SSL context for server
Mar 13 08:47:52:393 [9952] 5 hub: ciOpen - cache path: C:\Program Files (x86)\Nimsoft/niscache

-------------------------------------------

[Garin Walsh]

It looks like in order to get the upgrade to work correctly with the existing certificates, you need to go all the way back and issue the tunnel_initialize_ca hub callback to regenerate the CA and then reissue all new certificates and propagate those to where they're used.

Pending any kind of fix/alternate process my upgrade sequence was:

1. Make copies of the robot and hub directories
2. Run the UIM CU7 installer and let it fail
3. Restore the executables and dlls from the copy of the robot dir overwriting the ones put in place by the upgrade
4. Restart UIM
5. Issue the tunnel_initialize_ca hub callback to regenerate the CA - note this wipes everything from the <tunnels> section except for the CA info and a couple settings
6. Start the hub GUI and reapply the correct hub tunnel server settings and start the tunnel server
7. Use the "new" button to create a new certificate
8. Ensure that anywhere that was using a self signed tunnel certificate is updated either with the contents or new name (it'll be cert02.pem now since everything was reinitialized)
9. Restore third party signed certs and related hub config sections from backup
10. Restart UIM to get the new certs in place and verify working
11. Rerun the CU7 upgrade which should now complete
12. Update any tunnel clients using the self signed certificates with the new tunnel server cert.

 

-------------------------------------------

[Jason Allen]

Thanks Garin for your inputs here - we really appreciate it!

We have been working through this internally as well and have published a couple of related KB's.

The long and short of it is that prior to hub 23.4.0, tunnel certificates were signed with the SHA1 algorithm and in hub 23.4.0 and forward, they are signed with SHA384.

Starting with hub 23.4.7, there's an OpenSSL upgrade and the new version doesn't accept SHA1 certificates.

This was unfortunately not caught during testing as all our test systems are using SHA384 already.

Tunnel servers and clients which started their lives as hub 23.4.0 or later will therefore not be impacted, but if your tunnels were originally set up under a prior version of the hub, the certificates will be SHA1 and upgrading the hub will cause the tunnels to fail.

Below are some links which should hopefully help - unfortunately this is not an easy problem to deal with, and even the prevention requires the re-issue of all tunnel CA's and certificates, which invalidates the existing client certificates and requires touching every tunnel client.   You can (as described in one of the below KB's, and thanks again to Mr. Walsh for his inputs here) set up a secondary tunnel server and use that to push new certificates for the first one, but it's still quite a bit of work unfortunately.

CU7 Upgrade Stuck/Failed on Starting Server phase (Secure Bus)

If you have not yet upgraded to CU7/hub 23.4.7 the following KB will help you validate whether you need to re-issue the tunnel certificates:

Validating tunnel certificate readiness for CU7 Upgrade

For tunnel hubs refer to:

Tunnel Failure after upgrading hub to 23.4.7 / CU7

Hopefully this (along with the fantastic information provided by Garin above) will be helpful.

-------------------------------------------

[Joakim E]

Is it not possible to create a new SHA384 certificate on the hub tunnel server and simply replace the SHA1 certificate in the client config with the SHA384 certificate?

-------------------------------------------

[Jason Allen]

Yes, that's the actual fix - but it can be difficult at scale, because as soon as you create the new SHA384 cert on the server, the client certificates previously issued all become invalid, so unfortunately you still have to touch every client to replace the certs on the client side.

-------------------------------------------

[Joakim E]

Thanks Jason.

Hmm, the IM hub gui crashes when trying to view the certificate. Is there a fix? We're on 23.4.6.

-------------------------------------------

[Jason Allen]

I haven't seen that before - you can try holding down SHIFT on your keyboard and double-click the probe which will force the GUI files to reload, sometimes that helps with odd issues.  Maybe try reloading IM entirely or try from another machine. Otherwise you may need to work with the certificates directly on the filesystem itself, and it may be worth opening a support case for the GUI crash if you can't get around it - I don't think anyone else has reported that particular behavior.

-------------------------------------------

[Garin Walsh]

I do not see the crash in either 23.4.6.S or 23.4.7.S

The certificate is just a human readable text file and the display of it in the GUI doesn't appear to care what the contents of the certificate file actually are.

I'd suggest that you start with looking at the certificate file on the server and verify that there are no special characters in it that might be tripping up the display. And not that it should matter, but also make sure that there's a carriage return following the final text line in the certificate.  

-------------------------------------------

[Olaf Pape]

Are there any plans to update the installation files of CU7?

When I read Jason Allen's comment in particular ("Yes, that's the actual fix - but it can be difficult at scale, because as soon as you create the new SHA384 cert on the server, the client certificates previously issued all become invalid, so unfortunately you still have to touch every client to replace the certs on the client side."), I cannot recommend an update to CU7 to my customers.

 

-------------------------------------------

[Jason Allen]

We are currently exploring the feasibility of a patch for the hub which would allow the old certificates to be used at least temporarily until new ones can be generated. It is not yet 100% clear whether this will be possible to do without introducing other security concerns, but we are investigating it.

-------------------------------------------

[Ravishu Arora]

The CU7 release solution document has been updated with a hotfix for the hub to address this issue about support for old SHA1 certificates.

------------------------------
Principal Product Manager
Broadcom Software
------------------------------

[Jason Allen]

Note for those customers who have not yet upgraded; you will have the best experience by following the below steps:

1. import the hub 23.4.7.1 to your archive, but don't deploy it yet
2. On each tunnel hub, edit hub.cfg and in the <tunnel> section, add enable_cert_compatibility = yes as described in the solution document.
3. Now deploy hub 23.4.7.1 to these hubs and they will successfully upgrade and reconnect.

If you deploy the hub before adding the config key, the tunnels will fail and you will need to SSH/RDP to the target system and edit the config file locally, so it's a good idea to add the key prior to deployment so that it will be there on startup.

-------------------------------------------

[Marius Nitu]

Hello,

So the upgrade to 23.4.7 on the primary hub should be done after the 3 steps you mentioned or before?

Thanks,

Marius

-------------------------------------------

[Jason Allen]

So, just for the sake of clarity - let me outline a few different scenarios, as it will depend on how your specific environment is configured.

The following assumes that you are not running the "Secure Bus" or "S" version of the hub/robot in your environment (e.g. "23.4.6.S")  I will address that further at the end of this post, but for most customers who are running the standard hub and robot, the following considerations apply:

Scenario 1:  Your Primary Hub is also a Tunnel Server, which has one or more clients connected to it.

In this scenario, you should deploy the CU7 upgrade along with the 23.4.7.1 hotfix in the following order:

- make the configuration change for "enable_cert_compatibility = yes" in the <tunnel> section of hub.cfg on all hubs with tunnels, but do not upgrade anything yet.  This means - any tunnel client and any tunnel server, not just the primary;  make the configuration change to all hubs with tunnels ahead of time.

- deploy the hub 23.4.7.1 hotfix to the tunnel clients only.  If desired, you can skip secondary hubs and clients for now, and do them later - but any clients which connect to the primary hub itself must be upgraded to 23.4.7.1 at this point.   They will reconnect successfully to the primary hub/tunnel server.  If you do want to upgrade all the hubs at once, always do the tunnel clients first before the servers.  If you have multiple "tiers" of hubs, where some are acting as both client and server, upgrade the farthest-away or "edge" clients first.  Upgrade tunnel servers only after all of the clients have been upgraded, otherwise you risk losing connectivity to the clients.

- deploy the CU7 upgrade to the primary hub by running the installer.    This will upgrade the primary hub to 23.4.7 and the tunnel clients will drop off, as the tunnel server portion of the hub will not start until upgraded to 23.4.7.1.

- After the CU7 upgrade completes, redeploy hub 23.4.7.1 to the primary hub and the "enable_cert_compatibility" key which you already added will take effect, and the tunnel clients will be able to reconnect successfully.

Scenario 2: Your Primary Hub is a Tunnel Client connecting to one or more tunnel servers

In this scenario, use the following order:

- make the configuration change for "enable_cert_compatibility = yes" in the <tunnel> section of hub.cfg on all hubs with tunnels, but do not upgrade anything yet.  This means - any tunnel client and any tunnel server, not just the primary;  make the configuration change to all hubs with tunnels ahead of time.

- deploy the hub 23.4.7.1 hotfix to the "other" tunnel clients first;  not the primary hub, but any other clients which connect to the same tunnel servers that the primary connects to.   They will reconnect successfully to their respective tunnel servers.  If desired, you can skip secondary hubs and clients for now, and do them later - but any servers which the primary hub itself is a client to must be upgraded to 23.4.7.1 and before you do that, you must also upgrade all the other clients which connect to those same tunnel servers.

- after deploying the 23.4.7.1 hotfix to the "other" tunnel clients, deploy it to the tunnel servers.  At this point your primary hub/client should still be running a pre-23.4.7 hub, so it should reconnect successfully to those tunnel servers after they are upgraded.

- deploy the CU7 upgrade to the primary hub by running the installer.    This will upgrade the primary hub to 23.4.7 and it will then fail to reconnect to the tunnel server(s) that it is a client to.

- now deploy hub 23.4.7.1 to the primary hub and the "enable_cert_compatibility" key which you already added will take effect, and it will connect to the tunnel server(s) successfully.

Scenario 3: Your Primary Hub is Both a Tunnel Server and a Client to Other Tunnel Servers

In this scenario, use the following order:

- make the configuration change for "enable_cert_compatibility = yes" in the <tunnel> section of hub.cfg on all hubs with tunnels, but do not upgrade anything yet.  This means - any tunnel client and any tunnel server, not just the primary;  make the configuration change to all hubs with tunnels ahead of time.

- deploy the hub 23.4.7.1 hotfix to "other" tunnel clients first;  not the primary hub, but any other clients which connect to the same tunnel servers that the primary connects to.    They will reconnect successfully to their respective tunnel servers.

- Also deploy the 23.4.7.1 hotfix to any hub that is a client of the primary hub.

- If desired, you can skip secondary hubs and clients for now, and do them later - but any servers which the primary hub itself is a client to must be upgraded to 23.4.7.1 and before you do that, you must also upgrade all the other clients which connect to those same tunnel servers.  Any server that is a client directly to the primary hub should also be upgraded at this time.

- after deploying the 23.4.7.1 hotfix to the "other" tunnel clients, deploy it to the tunnel servers.  At this point your primary hub/client should still be running a pre-23.4.7 hub, so it should reconnect successfully to those tunnel servers after they are upgraded.

- now deploy the 23.4.7.1 hotfix to the tunnel clients that are connecting directly to the primary hub; again, they should successfully reconnect to the primary hub after the deployment.

- finally, deploy the CU7 upgrade to the primary hub by running the installer.    This will upgrade the primary hub to 23.4.7 and it will then fail to reconnect to the tunnel server(s) that it is a client to, and all direct clients will fail to reconnect to it.

- now deploy hub 23.4.7.1 to the primary hub and the "enable_cert_compatibility" key which you already added will take effect, and it will connect to the tunnel server(s) successfully.

Scenario 4: Your Primary Hub Is Not A Tunnel Server or Client, but you have secondary hubs which are, and Name Services entries connecting some of them to the Primary Hub

This is the easiest scenario to deal with, here are the steps:

- make the configuration change for "enable_cert_compatibility = yes" in the <tunnel> section of hub.cfg on all hubs with tunnels, but do not upgrade anything yet.  This means - any tunnel client and any tunnel server;  make the configuration change to all hubs with tunnels ahead of time.

- deploy the 23.4.7.1 hotfix to all tunnel clients first;  if you are in a "multi-tier" environment with some hubs acting as both server and client, upgrade the "farthest-away" or edge clients first, then work your way in toward the primary hub.

- once all the tunnel clients are upgraded, upgrade the tunnel servers.

- This can be done either before or after the upgrade to CU7 is run on the primary hub, and it will not be necessary to apply the 23.4.7.1 hotfix to the primary hub if there are no tunnels present.

For Secure Hub/Robot Installations:

Unfortunately there's no easy way to deal with this;  Garin Walsh has outlined the steps he took in his environment, and those are essentially the necessary steps to take.   If you have a Secure Bus environment, my best advice at this point would be to wait for CU8 (tentatively due around July 2026 approximately), because the installer itself will include a hub_secure build that supports the enable_cert_compatibility key, and there will be a companion build for the robot_secure build as well, so it will only be necessary to make the configuration change and then run the installer.

That being said, if you are running the Secure Bus and cannot wait that long to upgrade to CU8 and require CU7, please open a support ticket and we can help guide you through the process.

-------------------------------------------