"The new host TPM endorsement key doesn't match the one stored in the DB" error

[lspin]

We recently had one of our hosts system board replaced by HP. However, when they replaced the system board they did not install a new TPM chip. The old board had a TPM chip that was already managed by vSphere. They recently came out and replaced the system board and installed a new TPM chip. However, now we're getting the following error in vSphere "The new host TPM endorsement key doesn't match the one stored in the DB." We have a vDS configured on the cluster and the host is also part of a vSAN cluster. However, the host has been placed in M-mode with full-data migration. Any suggestions on how to resolve?

[lspin]

I've tried disconnecting and reconnecting the host, but the alert is still present.

[lspin]

Resolved with the help of VMware support. Stated there is a Dell VxRail KB out there somewhere with the resolution if the "Disconnect" and "Reconnect" does not work. Steps we took are below. Hope this helps someone. STEP 1: Take a snapshot of the vCenter STEP 2: SSH to vcenter and ran commands: cat /etc/vmware-vpx/vcdb.properties (you'll need the outputted password for the "psql -d VCDB vc" command below) cd /opt/vmware/vpostgres/current psql -d VCDB vc select ID,DNS_NAME,endorsement,_key,attestation_identity_key from VPX_HOST; (you'll need the ID# listed at the start of each host line item) STEP 3: Ctrl + Z to back out. STEP 4: Backup file: /opt/vmware/vpostgres/current/bin/pg_dump -U postgres -t VPX_HOST VCDB > /tmp/VPX_HOST.sql STEP 4: Then back into KB to make one change: update VPX_HOST set endorsement_key=' ', attestation_identity_key=' ' where id =ID#; [ID# is the ID of the affected host from the "select ID,DNS,_NAME..." list. Example: 726 | 10.10.10.10 | AToAAQALAAMAsgAgg3GXZ0SEs/g....] STEP 5: Lastly, from vsphere client, we disconnected and reconnected the host.

[hyperwyrm]

Hello!

 

We have the same issue and I tried to fix it with your approach. I slightly modified your commands, because there are errors in SQL syntax. So we have warning message "The new host TPM endorsement key doesn't match the one stored in the DB" for host "esx01.domain.tld" and we have to do (under [e-mail])

 

 

# Get VCDB password
cat /etc/vmware-vpx/vcdb.properties | grep 'password =' | awk '{print $3}'

# Check presence of "endorsement_key" and "attestation_identity_key" for our host
/opt/vmware/vpostgres/current/bin/psql -U vc -c "SELECT dns_name,endorsement_key,attestation_identity_key FROM vpx_host WHERE dns_name = 'esx01.domain.tld';" -d VCDB

# Backup table vpx_host from VCDB
/opt/vmware/vpostgres/current/bin/pg_dump -U postgres -t vpx_host -d VCDB >/tmp/vpx_host.sql

# Update "endorsement_key" and "attestation_identity_key" values
/opt/vmware/vpostgres/current/bin/psql -U vc -c "UPDATE vpx_host SET endorsement_key=' ', attestation_identity_key=' ' WHERE dns_name = 'esx01.domain.tld';" -d VCDB

 

 

Unfortunately after issuing those commands and disconnect/reconnect host warning message was still there. I have to remove host from inventory and add it again and warning has disappeared

[William Lam]

Not sure why support had you messing w/VCDB, this should really be a last resort ...

I recently had to resolve the TPM alarm in vCenter Server for clearing the TPM keys but the workflow to resolve is the same for when TPM chip is also replaced. 

While it was not easy to find, the workflow is documented in this KB https://kb.vmware.com/kb/81446 which includes your use case but it did lack the details on getting the recovery key which is done on ESXi host by running: esxcli system settings encryption recovery list

In addition, rather than typing out the long 96-digit key, I think easier approach (one I took) was to append that to boot.cfg while I was SSH'ed to ESXi host, reboot and then re-attach ESXi host to cluster and you'll need to clear the alarm once, but after that you can remove the boot.cfg entry and upon next reboot, you won't see error again.

I was going to blog about this later when I had a chance but came across this post, so figure I share for others

[William Lam]

Here's a blog post on using slightly modified method per the VMware KB mentioned https://williamlam.com/2023/06/clearing-tpm-alarms-after-replacing-tpm-chip-or-resetting-tpm-keys-for-esxi.html 

[hyperwyrm]

Thanks, William

[Sayenath N]

Hello William, 

I believe the, the DB clean up method will be necessary on VCF infrastructure as we will not be able to remove the host from the VC inventory.  Is there any other alternative method you would suggest for VCF infrastructure? 

[William Lam]

I see that you're an employee, best to reach out to Engr team for further guidance on VCF environment. 

[vmjoe]

The commands do not work with vCSA 8 (8.0U2 at least), there's no vc user password anymore in /etc/vmware-vpx/vcdb.properties.

But luckily it works with the postgres user - without needing a password

/opt/vmware/vpostgres/current/bin/psql -U postgres -c "SELECT dns_name, endorsement_key, attestation_identity_key FROM vpx_host WHERE dns_name = 'esx01.domain.tld';" -d VCDB

Also, the two keys need to be replaced with '' (nothing), it doesn't work with a space between the single quotation marks:

/opt/vmware/vpostgres/current/bin/psql -U postgres -c "UPDATE vpx_host SET endorsement_key='', attestation_identity_key='' WHERE dns_name = 'esx01.domain.tld';" -d VCDB

 A host reboot is needed afterwards, and TPM attestation should be back to "Passed":