Db health is UNHEALTHY, Backup Failed. Disable health check to take backup in current state.

[SMcT]

Hi All

My VCSA backups have started failing recently with the following error in the VAMI - Db health is UNHEALTHY, Backup Failed. Disable health check to take backup in current state. 

I'm not seeing any function problems with the operation of my vcsa and can backup successfully when I unselect the DB checks.  I haven't found anything compelling in the log files either. 

Can anyone direct me to a specific log file or provide any thoughts?  I'm either not searching for the right thing on Google or there isn't much out there on this...

VCSA 7.0.2 - Build 17694817

(Its a lab so no support)

Thanks 

[virtual-power]

Use the script from  https://kb.vmware.com/s/article/53062   and check for any issues. 

[SMcT]

Hi Ajay

thanks for the imformation.  I ran the script and got the following output.

root@vm-vcsa-01 [ / ]# /scripts/vpgsql-consistency-check.sh -a -b -f /tmp/ -v
amcheck extension is not installed
amcheck extension is not installed

05-10-2021T10:48:15 Starting the catalog diagnostic
05-10-2021T10:48:16 Successfully finished the catalog diagnostic

Taking a backup of the VCDB to: /tmp/

05-10-2021T10:48:16 Starting backup of the VCDB database.
05-10-2021T10:48:23 Successfully finished backup of the VCDB database.

05-10-2021T10:48:23 Starting backup of the AFD database.
05-10-2021T10:48:24 Successfully finished backup of the AFD database.

05-10-2021T10:48:24 Starting PSQL Instance Vacuum
vCenter is currently running.
05-10-2021T10:48:46 Finished PSQL Instance Vacuum

05-10-2021T10:48:46 Starting Consistency Check
05-10-2021T10:48:54 Finished Consistency Check.

Consistency Check PASSED. The database has been found to be consistent. No further action required.

However I am still getting the error when backing up the DB via the VAMI.  I have attached the log file which mentions 3 circular foreign-key constraints on the following 3 tables - vpx_task, vpx_entity, pm_config_commits.

Thanks 

[virtual-power]

Attach the  backup log from  /var/log/vmware/applmgmt

[SMcT]

[virtual-power]

2021-05-10T09:50:35.541 [20210510-094943-17694817] [MainProcess:PID-46867] [BackupManager::getDbHealthStatus:BackupManager.py:480] INFO: getDbHealthStatus returns status as UNHEALTHY
2021-05-10T09:50:35.542 [20210510-094943-17694817] [MainProcess:PID-46867] [BackupManager::HealthCheckStatus:BackupManager.py:493] ERROR: HealthCheckStatus Overall status: Failed. Check /var/log/vmware/applmgmt/applmgmt.log for more details.
2021-05-10T09:50:35.542 [20210510-094943-17694817] [MainProcess:PID-46867] [BackupManager::main:BackupManager.py:625] ERROR: BackupManager encountered an exception: Db health is UNHEALTHY, Backup Failed.Disable health check to take backup in current state.

 

Please grab the   /var/log/vmware/applmgmt/applmgmt.log and   /var/log/vmware/cloudvm/fileintegrity.log  . Also   /etc/vmware/appliance/fileintegrity_config.json

[SMcT]

Thanks for taking the time to assist 

[virtual-power]

fileintegrity.log

2021-05-10T13:00:01.675Z ERROR: /etc/vmware/appliance/fileintegrity_config.json config file verification failed
2021-05-10T13:00:01.675Z ERROR: Exception occurred during check functionality. Exception: Failed verify config file
2021-05-10T13:00:01.675Z ERROR: Exception in IntegrityChecker, [Errno 2] No such file or directory: '/var/vmware/applmgmt/fileintegrity/integrity_check.sqlite_bkp'

 

Also the  applmgmt.log has rolled over .. Need the log which covers the below timestamp.

2021-05-06T12:49:38.443 [20210506-124823-17694817] [MainProcess:PID-43465] [integrity_checker::log_and_print:integrity_checker.py:75] ERROR: /etc/vmware/appliance/fileintegrity_config.json config file verification failed
2021-05-06T12:49:38.444 [20210506-124823-17694817] [MainProcess:PID-43465] [BackupManager::CheckFileIntegrity:BackupManager.py:420] ERROR: Fileintegrity [FILE_INTEGRITY_CHECK] Overall status: Failed. Check /var/log/vmware/cloudvm/fileintegrity.log for more details.

[SMcT]

[SMcT]

Here is the snippet from the time stamp you mentioned.

 

2021-05-06T12:49:37.227 [3798]INFO:vmware.appliance.vapi.auth:Authorization request for service_id: com.vmware.appliance.recovery.backup.job.details, operation_id: list
2021-05-06T12:49:37.228 [3798]DEBUG:vmware.vherd.base.authorization_local:Verify privileges user (root) privilege ['ModifyConfiguration']
2021-05-06T12:49:37.229 [3798]DEBUG:root:Validated user privileges in localstore or SSO
2021-05-06T12:49:37.230 [3798]DEBUG:vmware.appliance.update.update_state:In State._get using state file /etc/applmgmt/appliance/software_update_state.conf
2021-05-06T12:49:38.264 [3798]INFO:vmware.appliance.vapi.auth:Authorization request for service_id: com.vmware.appliance.recovery.backup.job.details, operation_id: list
2021-05-06T12:49:38.264 [3798]DEBUG:vmware.vherd.base.authorization_local:Verify privileges user (root) privilege ['ModifyConfiguration']
2021-05-06T12:49:38.264 [3798]DEBUG:root:Validated user privileges in localstore or SSO
2021-05-06T12:49:38.265 [3798]DEBUG:vmware.appliance.update.update_state:In State._get using state file /etc/applmgmt/appliance/software_update_state.conf
2021-05-06T12:49:39.290 [3798]INFO:vmware.appliance.vapi.auth:Authorization request for service_id: com.vmware.appliance.recovery.backup.job.details, operation_id: list
2021-05-06T12:49:39.291 [3798]DEBUG:vmware.vherd.base.authorization_local:Verify privileges user (root) privilege ['ModifyConfiguration']
2021-05-06T12:49:39.292 [3798]DEBUG:root:Validated user privileges in localstore or SSO
2021-05-06T12:49:39.293 [3798]DEBUG:vmware.appliance.update.update_state:In State._get using state file /etc/applmgmt/appliance/software_update_state.conf

[SMcT]

Inadvertently fixed this by restoring the vcsa from file backup for another issue.  The backup I restored was taken when the issue existed.  I'm assuming the recreation of the appliance removed an issue, that wasn't something present in the backup.

[virtual-power]

Hello. I encountered the same issue in my lab and noticed it was caused due to proxy. I have blogged about it here >> https://virtual-power.in/f/30-vami-backup-fails-on-vcsa-configured-with-proxy

[BenLor00]

Hi SMcT,

Today I encountered the same error on two distinct vCenters 7.0U3a and I solved it following the workaround described in the article 86174

Ciao!

--
Lorenzo

[CLS78]

The same error occurred in 3 out 5 vSphere environments upgrading to vCSA 7.0 Update 3a.   The fix action is to run the following command via shell on the vCSA:  /usr/bin/dbcc -fbss embedded

This command refreshes the latest db schema on the vCSA.  The reason why this occurred is because there has been change in DB schema which is not refreshed post patching.

Reference URL:  https://vinfrastructure.it/2021/10/file-level-backup-issue-with-vcenter-7-0-3/

 

 

[CLS78]

However, running the command only I see on 3 on my environment only resolves it for a day.  I wonder if anybody else is encountering the same issue?

[BenLor00]

Hi CLS78,

Yes, after a couple of days the issue returns. We just have to wait for the new patch, in the hope that the problem will be definitively resolved.

Lorenzo

 

[CLS78]

Lorenzo,

Thanks for the quick response.  I had the same idea,

 

Christian

[marsherian]

The script referenced in the original KB did not work for me, but this KB did:

https://kb.vmware.com/s/article/86084