Dear Petteri,
I’m in the exact same situation you are. Running ESXi in my home lab, but this architecture is linked to home automation developed over the years, meaning quite sensitive. Since I can’t easily stop the system, install Proxmox (I don’t even know about so far), I ordered a new server not later than yesterday to build the new Proxmox based architecture. Anyway, this will allow me to have a real hardware failover solution one the new server in production.
But sorry this is not your topic :-) Regarding your certificate issue, I guess you’re still using the WMWare self-signed certificate embedded in your EXSi? My architecture is setup with a private PKI, meaning I generate all my SSL certificates with step-ca, which deployment is done with Ansible. You can easily generate your own self signed certificates using openssl, you have plenty of tutos over the web, that’s quite easy. Basically you need a private key and a server certificate.
Once you have your certificates done, let’s say esxi.crt and esxi.key for the private key, rename those certificates into rui.crt and rui.key, then log on your ESXi and cd into /etc/vmware/ssl . You will find the current outdated certificates there: rui.crt and rui.key. Rename them to something like .old, then copy both certificates you’ve just created (you can use scp command or copy paste the content, certificates are text files).
Restart the web UI with the following command : /etc/init.d/hostd restart
Try to connect to web interface, you should be prompted about a non trusted certificate, just accept this exception.
Let me know if that works.