vCenter Server Appliance: Where do I upload SSL certificate?

[sofakng]

I'm using the new vCenter Server Appliance but I'm having trouble with the SSL certificates.

Where do I upload the SSL certificate for the vCenter, and where do I upload the SSL certificates for my ESXi hosts?

I can't seem to find any documentation on this...

Because I can't add the SSL certificates, when I try to add an ESXi host to vCenter it gives an error about "Authenticity of the host's SSL" and "certificate is not verified".

(My SSL CA uses an intermediate root certificate which I'm also unsure if I need to upload somewhere...)

[sofakng]

Anybody?

[Doug Baer]

Put the new cert here... (using sftp works):

/opt/vmware/etc/lighttpd/server.pem

then reboot (probably a cleaner way to do this, but a reboot works)

===

For more detail, read on.  My test machine is called "s01-vc01":

s01-vc01:~/KEYS # openssl req -out s01-vc01.csr -new -newkey rsa:2048 -nodes -keyout s01-vc01.key
Generating a 2048 bit RSA private key
.....+++
.......................................+++
writing new private key to 's01-vc01.key'
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:US
State or Province Name (full name) [Some-State]:Arizona
Locality Name (eg, city) []:Tempe
Organization Name (eg, company) [Internet Widgits Pty Ltd]:ITPLAB
Organizational Unit Name (eg, section) []:ITPLAB
Common Name (eg, YOUR name) []:s01-vc01.itplab.local
Email Address []:[e-mail]

Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:
An optional company name []:
s01-vc01:~/KEYS # ls
s01-vc01.csr  s01-vc01.key

(Copy the CSR to the CA and issue a key ... Base-64 encoded is what you want.. then get the CER file back here)

s01-vc01:~/KEYS # ls
s01-vc01.cer  s01-vc01.csr  s01-vc01.key

(Concatenate the KEY and the CER file into a PEM file in the appropriate location ... you may want to backup the original first)

s01-vc01:~/KEYS # cat s01-vc01.key s01-vc01.cer > /opt/vmware/etc/lighttpd/server.pem

(Check the fingerprint of the new key ... )

s01-vc01:~/KEYS # /usr/bin/openssl x509 -fingerprint -noout -in /opt/vmware/etc/lighttpd/server.pem
SHA1 Fingerprint=40:3F:CE:3E:0B:CD:F3:8D:B5:D7:C1:73:51:5C:6E:77:7C:0D:0A:75

Reboot the appliance and you should be good.  <- there's probably another way to kick the lighttpd daemon, but a reboot works.

[sofakng]

Thanks for the information!

That will assign an SSL certificate to vCenter itself, right?  (both the web server and vCenter server?)

What about placing my ESXi host's SSL certificate somewhere on the server?

According to vSphere 4, you had to tell vCenter Server about the ESXi hosts by uploading all of their SSL certificates.  Somebody please tell me if I'm wrong about this though...

[Doug Baer]

I'll answer this in a few posts as I get a chance to test them.  As you have noticed, all of the information that seems to be available has to do with the Windows version of vCenter.

I realized this morning that the above procedure seems to only update the key for the web service portion of the appliance.  In order to update the vCenter client portion, you'd need to also update the

  /etc/vmware-vpx/ssl/rui.crt

  /etc/vmware-vpx/ssl/rui.key

files.  If you want to update the pfx, you can, but that's supposed to be a Windows thing (not sure why the file is included on the appliance).

This is best done before you create the database or add anything to it.  Unfortunately, I had already populated my database and got an "invalid key, fail" message in the log when I tried to restart the vpxd.

NOTE: you can run

#  service vmware-vpxd restart

to get the service to reload w/o rebooting the appliance

In poking around, I found a command called vpxd_servicecfg that has an option to replace certificates:

#  vpxd_servicecfg certificate change new-rui.crt new-rui.key

I think that may be useful, but it doesn't have much (any) UI and I can't find any documentation on it.  When I run that command, I get the following:

VC_CFG_RESULT=653

Not sure what that means, and restarting vmware-vpxd after that seems to use the new certs for a portion of the vSphere client login, but there is another one being presented as well.  I'm looking into that.

As for the host certs, I'll get there eventually...

[Doug Baer]

Interesting. Tracking down the other cert with its SHA1 thumbprint, I found it in

/usr/lib/vmware-vpx/inventoryservice/ssl/rui.crt

I replaced that one with the same one I used above in

/etc/vmware-vpx/ssl/rui.crt

bounced the vmware-vpxd service and tried to connect with the vSphere Client.

Partial success.  That service wants to use the IP address rather than the FQDN for identification, so I get a different certificate warning message now.

I'm not sure how to change that.

[ChBehma]

Hi,

thank you for this interisting command.

I used it to change the vpxd (vCenter Service) SSL certificate.

If you have your two new files

• newrui.key
• newrui.crt

Do the following Steps:

1. /etc/init.d/vmware-vpxd stop
2. vpxd_servicecfg certificate change newrui.key newrui.crt
3. /etc/init.d/vmware-vpxd start

The vpxd_servicecfg don't change the certificate if the service is up and running.

If the script shows something like this:

VC_CFG_RESULT=653

the job wasn't successfull.

It MUST result with 0!

Kind regards,

Christian

[nbucsi]

Hi!

I did all of written before in this post but it did not work for me.

I generally get VC_CFG_RESULT=650 or VC_CFG_RESULT=651 ...

Do you know a detailed workaround (post) about changing vcenter server (appliance) self-signed certificate to a ca issued one?

thanks,

[Doug Baer]

I would love to know where that information lives. I kind of muddled through the process myself and the above worked for me. To date, I have seen nothing official from VMware on this, but I will redeploy an appliance and walk through the process again to see if I can streamline it a little and re-post here.

Doug

[nbucsi]

Thanks Doug.  I am curious for the steps of the process. But I try to deploy again

[ChBehma]

Hi,

I just looked in the vpxd_servicecfg file.

It's a regular bash script.

From line 56 to 99 you will find an answer what the error number could mean.

650 = CERTIFICATE PASS REENCRYPT FAILED

651 = MISSING CERTIFICATE

If you got error 650 the script starts the vpxd deamon with your certificate files.

If this failes, the script returns error 650.

Maybe your files are corrupt or the chain is not complete.

On error 651 the script can't find your certificate file.

:-)  

[Doug Baer]

DOH! I didn't even think of doing that :smileyhappy:

[nbucsi]

Thanks, I will check out in our env.

[BobbyW]

Thanks for telling us what the errors mean. It's a little frustrating that the script knows exactly why it's failing, but instead of telling us in english, it just spits out a number.

[lobo519]

I would love for someone to figure this out (or be properly doucmented by VMware). I have been trying the the last 3 days. Submitted a ticket to VMware but they have been less than helpful. Basically gave me all the same info that is already here.

When I restart vmware-vxpd - it either hangs on "waiting for vpxd to initialize..." or failed immediatly!

[Doug Baer]

Let's see if I can help out some more here. I have a feeling this is going to become a blog posting when I get the time, but I started from scratch with a new VCMA and went through the process:

1. Import the appliance
2. Configure the IP address and hostname (stuff the hostname into /etc/HOSTNAME and configure a lookup in /etc/hosts or DNS)
3. (I rebooted here since I was trying to keep things clean -- don't know if it is required)
4. Assume you follow the directions to get a CSR generated and use that to get a CER issued (Base64 encoded).
5. Copy that CER to the VCMA
6. Login as root to the VCMA and change to a directory wher the .cer and .key files live

NOTE: make sure you convert the line endings to UNIX from DOS if you issued the certs from Windows.

awk is fantastic for this:

# awk '{sub(/\r$/,"");print}' vcma01.cer_FROM_WINDOWS >vcma01.cer_FOR_UNIX

From here, you need to do a couple of things. You can use the web UI or the CLI, whichever works for you.

1. Accept the EULA
   

   vpxd_servicecfg eula accept

2. Initialize the database -- I'm using the embedded one

   vpxd_servicecfg db write embedded

3. Swap out the certificates

   vpxd_servicecfg certificate change vcma01.cer vcma01.key

Each of the above 3 returned VC_CFG_RESULT=0 and things looked pretty good.  Checking the fingerprints of my CER and the two at the following locations showed a successful swap (fingerprints matched):

# /usr/bin/openssl x509 -fingerprint -noout -in /opt/vmware/etc/lighttpd/server.pem

# /usr/bin/openssl x509 -fingerprint -noout -in /etc/vmware-vpx/ssl/rui.crt

Next was to start up the vCenter services (again, you can also use the web UI):

# /usr/sbin/vpxd_servicecfg service start

Another result of VC_CFG_RESULT=0

So far, so good.

Let me know if this works for you...

UPDATE:  I have tested the VCMA's HTTPS web page, vSphere client authentication, and PowerCLI.  I have not been presented with the 'untrusted certificate' warnings in any of those cases.

NOTE: it may or may not be obvious to people that your root CA's (and any intermediate issuing CA's) certificates need to be in the proper place on your client(s) in order for the chain of trust to be recognized properly.

Message was edited by: DougBaer

[Doug Baer]

Unfortunately, the vSphere Web Client

     https://VCMA_DNS_NAME:9443/vsphere-client/#

must use a different certificate. I'll get to that soon, I suppose :smileyhappy:

[nbucsi]

Hi Doug,

Great thanks for the detailed steps. I am trying it.

[Doug Baer]

I've run through the process again and posted a blog article on this topic, along with a bunch of screenshots.

http://www.goitpartners.com/blog/?p=217

Doug