Maybe a bit too late, but I still wanted to reply to this thread as some people might experience the same issues when following the KB. There are some errors in it. You probably followed the guide to create the OpenSSL configuration file (http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&docType=kc&docTypeID=DT_KB_1_1&externalId=2015387). This line is quite confusing:
subjectAltName = DNS: vc.domain.com, DNS: vc41.domain.com, DNS: vc41 #examples
You probably ended up entering the vCenter server FQDN and parameters. In fact, it has to be the ESXi server's details.
I create the certificates like this (a bit different approach):
We will start off by creating a configuration file to generate our certificate request file. In the D:\Certificates\ESXi folder - this is where I keep my certificates - , create a new text file called CC-ESX01.cfg (you will need a configuration file for each of your hosts - mine is called CC-ESX01, or you need to change the contents before generating a certifiate for another host). Put the following in the file:
[ req ]
default_bits = 2048
default_keyfile = rui.key
distinguished_name = req_distinguished_name
encrypt_key = no
prompt = no
string_mask = nombstr
req_extensions = v3_req
[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = digitalSignature, keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth
subjectAltName = DNS:CC-ESX01, DNS:cc-esx01.xylos.training, DNS:172.16.2.1, IP:172.16.2.1
[ req_distinguished_name ]
countryName = BE
stateOrProvinceName = Antwerp
localityName = Training
0.organizationName = Xylos
organizationalUnitName = ESXiHost
commonName = cc-esx01.xylos.training
Change the values marked to your specific settings. Save the file.
D:\OpenSSL-Win64\bin\openssl req -new -nodes -out rui.csr -keyout rui.key -config D:\Certificates\ESXi\CC-ESX01.CFG
(I use OpenSSL 64 bit 0.9.8, located in D:\OpenSSL-Win64)
This will generate the rui.key and rui.csr files
Open the 'rui.csr' file and copy the contents to the clipboard.
Log in to the Microsoft CA certificate authority website and Request a certificate > Submit a certificate request by using a base-64 encoded... Paste the contents of the clipboard in the 'Saved Request' box.
In my Certificate Authority Server, I created a template 'VMware SSL Template', which is a duplicate of the 'Web Server' template, in Windows 2003 Enterprise format. On the 'Extensions' tab, click on 'Key Usage' and select 'Allow Encryption of User data'. This is the only difference with the web server template.
Submit the request and download the file as a 'Base 64 encoded' file and save it as 'rui.crt' in the same folder.
Start SSH on your host, and start a WinSCP connection to it.
In the right-hand pane, navigate to '/etc/vmware/ssl', on the left side, navigate to the folder where your certificates are located. (In my case it is D:\Certificates\ESXi)
Delete the rui.crt and rui.key on the ESXi server (right hand side pane) and upload the new ones. Change the Transfer settings to 'Text'
Close WinSCP and start a SSH session to the host (i.e. with PuTTY). Log on with the 'root' credentials.
Make sure there are no ^M characters by typing in the following command: less /etc/vmware/ssl/rui.crt
If not, then you are ready.
Restart the Management agents by typing: ./sbin/services.sh restart
After the management agents have been restarted (which should work now), you can connect to the https://<FQDN_of_your_host> to verify if it actually uses the certificate:
I have noticed that sometimes, after replacing the certificates and restarting the Management Agents, you cannot connect to the Console anymore of of VM using the vSphere Client. You only get a black screen and the VM does not accept any console input (although it works fine and you can RDP to it). After restarting the host, this problem is solved.
I hope this clears up things a bit. These are exerpts of my Blog, that covers a complete installation of ESXi 5.1 from scratch, including Active Directory and CA setup.