A failure occurred when starting a host compliance check operation when during run pre-check

[dbutch1976]

Hello,

I am getting this error on only two nodes in a 4 node cluster:

A failure occurred when starting a host compliance check operation on host 'host1.com' : Error: com.vmware.vapi.std.errors.internal_server_error Messages: com.vmware.esx.task.create.error<Failed to create task. Please retry later.>

So far I have removed the host from vCenter, entered/exited maintenance mode. Everything looks fine.

Any ideas what I can check next?

[dbutch1976]

Checking the /var/log/hostd.log I found the following:
tail -n 15 hostd.log

Hostd[2099325]: [Originator@6876 sub=Vimsvc opID=b514cbc0 sid=520af440] [Auth]: User root 2024-07-25T15:23:35.637Z Wa(164) Hostd[2099325]: [Originator@6876 sub=Vimsvc opID=b514cbc0 sid=520af440] Refresh function is not configured.User data can't be added to scheduler.User name: root 2024-07-25T15:23:35.637Z In(166) Hostd[2099325]: [Originator@6876 sub=Vimsvc.ha-eventmgr opID=b514cbc0sid=520af440] Event 5391 : User root@127.0.0.1 logged in as VMware-client/8.0.32024-07-25T15:23:35.672Z In(166) Hostd[2099569]: [Originator@6876 sub=Vimsvc.ha-eventmgr opID=b514cbc4 sid=520af440 user=root] Event 5392 : User root@127.0.0.1 logged out (login time: Thursday, 25 July, 2024 03:23:35 PM, number of API invocations: 1, user agent: VMware-client/8.0.3)
2024-07-25T15:23:45.001Z Er(163) Hostd[2136491]: [Originator@6876 sub=VMkernelStatsProvider(00000029cc3bb270)] GetKernelStatValues: Detected error while retrieving stats: VSINode(2647): Not found (status=195887107)
2024-07-25T15:23:50.175Z In(166) Hostd[2099353]: [Originator@6876 sub=Internalsvc.HostsvcManager opID=b514cbd0 sid=52104980 user=dcui:vsanmgmtd] Start refresh VSAN datastore in Hostd
2024-07-25T15:23:50.175Z In(166) Hostd[2099118]: [Originator@6876 sub=Libs opID=b514cbd0 sid=52104980 user=dcui:vsanmgmtd] GetTypedFileSystems: fstype vsan 2024-07-25T15:23:50.176Z In(166) Hostd[2099118]: [Originator@6876 sub=Libs opID=b514cbd0 sid=52104980 user=dcui:vsanmgmtd] GetTypedFileSystems: uuid vsan:523fee61ca759436-d5b80e248d50647a
2024-07-25T15:23:50.180Z In(166) Hostd[2099118]: [Originator@6876 sub=Libs opID=b514cbd0 sid=52104980 user=dcui:vsanmgmtd] VsanGetPersonality Get personality status Success, personality value 1
2024-07-25T15:23:50.184Z In(166) Hostd[2099118]: [Originator@6876 sub=Libs opID=b514cbd0 sid=52104980 user=dcui:vsanmgmtd] VsanFileSystemImpl: vSAN datastore cid 523fee61ca759436-d5b80e248d50647a, aid 523fee61ca759436-d5b80e248d50647a total RawCapacity: 7001362923520, usedRawCapacity: 3009876800306
2024-07-25T15:23:50.184Z In(166) Hostd[2099118]: [Originator@6876 sub=Libs opID=b514cbd0 sid=52104980 user=dcui:vsanmgmtd] VsanInfoImpl: Refresh config generation
2024-07-25T15:23:50.187Z In(166) Hostd[2099118]: [Originator@6876 sub=Libs opID=b514cbd0 sid=52104980 user=dcui:vsanmgmtd] VsanInfoCluster: vSan mode is set to  : Mode_None
2024-07-25T15:23:50.188Z In(166) Hostd[2099118]: [Originator@6876 sub=Libs opID=b514cbd0 sid=52104980 user=dcui:vsanmgmtd] VsanInfoEncrypt: LoadEncryptionConfig FromConfigStore enabled 0 kekId  hostKeyId  dekId  kmsId
2024-07-25T15:23:50.188Z In(166) Hostd[2099118]: [Originator@6876 sub=Libs opID=b514cbd0 sid=52104980 user=dcui:vsanmgmtd] VsanInfoDatastore: Load default datastore name vsanDatastore
2024-07-25T15:24:05.003Z Er(163) Hostd[2136491]: [Originator@6876 sub=VMkernelStatsProvider(00000029cc3bb270)] GetKernelStatValues: Detected error while retrieving stats: VSINode(2647): Not found (status=195887107)


I think this is relevant. 
Refresh function is not configured.User data can't be added to scheduler.User name: root 2024-07-25T15:23:35.637Z

Could this be causing my issue? Anything I can to to troubleshoot?

[dbutch1976]

Still hoping to get some traction on this. I have tried placing the hosts in maintenance mode, removing from the vCenter. Once the two affected hosts were removed I successfully turned HA on without any issues. When I rebooted and then added the two affected nodes back to the cluster the HA agent could not install successfully. Whatever issue is affecting the update pre-check also appears to be affecting the install of the HA agent. There must be some basic troubleshooting steps I could try for this type of issue, but I'm at a bit of a loss, would appreciate any ideas.

[Thor Quiocho]

Had something similar. What's the error on the eam.log?

This fixed it for me.

https://knowledge.broadcom.com/external/article?legacyId=2112577

[dbutch1976]

I disconnected and removed one of the affected hosts called dessloch.lebrine.local, then checked the eam log as I was doing it. Could this be relevant?

2024-07-28T11:59:30.497Z |  INFO | vim-inv-update | VcHostSystem.java | 110 | Ignoring state change due to disabled listeners on VcHostSystem(ID: host-34015)

The host isn't able to take workloads because of the HA agent failure. Here are the full contents of the logs:

root@vcnuc [ /var/log/vmware/eam ]# tail -n 30 eam.log
2024-07-28T11:59:30.456Z |  INFO | vim-async-1 | OpIdLogger.java | 35 | [Filter(session[52464a6a-d2da-2936-de92-7d3028156819]52147b48-a824-8ff7-b1ee-1bad3b77d562)->DestroyFilter:f159305b752c3cd7] Completed.
2024-07-28T11:59:30.497Z |  INFO | vim-inv-update | VcHostSystem.java | 110 | Ignoring state change due to disabled listeners on VcHostSystem(ID: host-34015)
2024-07-28T11:59:30.516Z |  INFO | vim-inv-update | VcHostSystem.java | 1716 | Host dessloch.lebrine.local has been removed from the VC inventory
2024-07-28T11:59:30.516Z |  INFO | vim-monitor | OpId.java | 37 | [PropertyCollector(session[52464a6a-d2da-2936-de92-7d3028156819]5247f8b4-a8f9-edea-ce27-d373a1010246)->WaitForUpdatesEx:fdca0ed4f8985c21] created from [WaitForUpdatesEx:5661cb6f20883be4]
2024-07-28T11:59:30.533Z |  INFO | vim-async-0 | OpIdLogger.java | 35 | [PropertyCollector(session[52464a6a-d2da-2936-de92-7d3028156819]5247f8b4-a8f9-edea-ce27-d373a1010246)->WaitForUpdatesEx:fdca0ed4f8985c21] Completed.
2024-07-28T11:59:30.618Z |  INFO | vim-inv-update | ComputeResourceBase.java | 737 | [1de6f209-1c17-4931-9a86-5ec9f7524db4::ClusterComputeResource:domain-c15036] datastores changed to [ManagedObjectReference: type = Datastore, value = datastore-15058, serverGuid = 1de6f209-1c17-4931-9a86-5ec9f7524db4, ManagedObjectReference: type = Datastore, value = datastore-15048, serverGuid = 1de6f209-1c17-4931-9a86-5ec9f7524db4]
2024-07-28T11:59:30.830Z |  WARN | vim-inv-update | ClusterVMAgency.java | 1317 | Host(HostSystem:host-34015) listener not present for unregistration in ClusterVMAgency(ID:db9cad38-9724-47ea-a8d6-de9286396203)
2024-07-28T11:59:30.847Z |  INFO | vim-inv-update | ComputeResourceBase.java | 744 | [1de6f209-1c17-4931-9a86-5ec9f7524db4::ClusterComputeResource:domain-c15036] networks changed to [ManagedObjectReference: type = Network, value = network-15041, serverGuid = 1de6f209-1c17-4931-9a86-5ec9f7524db4]
2024-07-28T11:59:30.847Z |  INFO | vim-monitor | OpId.java | 37 | [PropertyCollector(session[52464a6a-d2da-2936-de92-7d3028156819]5247f8b4-a8f9-edea-ce27-d373a1010246)->WaitForUpdatesEx:353eef551c5c6eef] created from [WaitForUpdatesEx:f795d35ecaf56da5]
2024-07-28T11:59:41.188Z |  WARN | vlsi | AgencyBase.java | 467 | Invocation of disabled API Agency.Update
2024-07-28T11:59:57.565Z |  INFO | vim-async-1 | OpIdLogger.java | 35 | [PropertyCollector(session[52464a6a-d2da-2936-de92-7d3028156819]5247f8b4-a8f9-edea-ce27-d373a1010246)->WaitForUpdatesEx:353eef551c5c6eef] Completed.
2024-07-28T11:59:57.566Z |  INFO | vim-monitor | OpId.java | 37 | [PropertyCollector(session[52464a6a-d2da-2936-de92-7d3028156819]5247f8b4-a8f9-edea-ce27-d373a1010246)->WaitForUpdatesEx:98a052521183ec7b] created from [WaitForUpdatesEx:b0a2abca7ec10ed4]
2024-07-28T12:00:11.191Z |  WARN | vlsi | AgencyBase.java | 467 | Invocation of disabled API Agency.Update
2024-07-28T12:00:30.940Z |  INFO | vim-async-2 | OpIdLogger.java | 35 | [PropertyCollector(session[52464a6a-d2da-2936-de92-7d3028156819]526acce6-1328-94fa-7f04-1bcc9643b288)->WaitForUpdatesEx:637737a8189c3c0c] Completed.
2024-07-28T12:00:30.940Z |  INFO | vim-monitor | OpId.java | 37 | [PropertyCollector(session[52464a6a-d2da-2936-de92-7d3028156819]526acce6-1328-94fa-7f04-1bcc9643b288)->WaitForUpdatesEx:c2581c46d3b30210] created from [WaitForUpdatesEx:6544af5353d1b210]
2024-07-28T12:00:41.189Z |  WARN | vlsi | AgencyBase.java | 467 | Invocation of disabled API Agency.Update
2024-07-28T12:01:11.191Z |  WARN | vlsi | AgencyBase.java | 467 | Invocation of disabled API Agency.Update
2024-07-28T12:01:33.589Z |  INFO | vim-async-0 | OpIdLogger.java | 35 | [PropertyCollector(session[52464a6a-d2da-2936-de92-7d3028156819]5247f8b4-a8f9-edea-ce27-d373a1010246)->WaitForUpdatesEx:98a052521183ec7b] Completed.
2024-07-28T12:01:33.590Z |  INFO | vim-monitor | OpId.java | 37 | [PropertyCollector(session[52464a6a-d2da-2936-de92-7d3028156819]5247f8b4-a8f9-edea-ce27-d373a1010246)->WaitForUpdatesEx:df8ab101f626341] created from [WaitForUpdatesEx:10ff16944cf1c468]
2024-07-28T12:01:33.603Z |  INFO | vim-async-2 | OpIdLogger.java | 35 | [PropertyCollector(session[52464a6a-d2da-2936-de92-7d3028156819]5247f8b4-a8f9-edea-ce27-d373a1010246)->WaitForUpdatesEx:df8ab101f626341] Completed.
2024-07-28T12:01:33.604Z |  INFO | vim-monitor | OpId.java | 37 | [PropertyCollector(session[52464a6a-d2da-2936-de92-7d3028156819]5247f8b4-a8f9-edea-ce27-d373a1010246)->WaitForUpdatesEx:fc4e7928a12aea56] created from [WaitForUpdatesEx:b8b08848a59566cb]
2024-07-28T12:01:37.770Z |  INFO | vim-async-0 | OpIdLogger.java | 35 | [PropertyCollector(session[52464a6a-d2da-2936-de92-7d3028156819]5247f8b4-a8f9-edea-ce27-d373a1010246)->WaitForUpdatesEx:fc4e7928a12aea56] Completed.
2024-07-28T12:01:37.771Z |  INFO | vim-monitor | OpId.java | 37 | [PropertyCollector(session[52464a6a-d2da-2936-de92-7d3028156819]5247f8b4-a8f9-edea-ce27-d373a1010246)->WaitForUpdatesEx:4f95e2c111b9e76] created from [WaitForUpdatesEx:a08cec77e69112c6]
2024-07-28T12:01:37.783Z |  INFO | vim-async-2 | OpIdLogger.java | 35 | [PropertyCollector(session[52464a6a-d2da-2936-de92-7d3028156819]5247f8b4-a8f9-edea-ce27-d373a1010246)->WaitForUpdatesEx:4f95e2c111b9e76] Completed.
2024-07-28T12:01:37.783Z |  INFO | vim-monitor | OpId.java | 37 | [PropertyCollector(session[52464a6a-d2da-2936-de92-7d3028156819]5247f8b4-a8f9-edea-ce27-d373a1010246)->WaitForUpdatesEx:4ff2254a36e62061] created from [WaitForUpdatesEx:e08bb8144354d882]
2024-07-28T12:01:38.311Z |  INFO | vim-async-0 | OpIdLogger.java | 35 | [PropertyCollector(session[52464a6a-d2da-2936-de92-7d3028156819]5247f8b4-a8f9-edea-ce27-d373a1010246)->WaitForUpdatesEx:4ff2254a36e62061] Completed.
2024-07-28T12:01:38.311Z |  INFO | vim-monitor | OpId.java | 37 | [PropertyCollector(session[52464a6a-d2da-2936-de92-7d3028156819]5247f8b4-a8f9-edea-ce27-d373a1010246)->WaitForUpdatesEx:51c59c2d01324e35] created from [WaitForUpdatesEx:4cce55ba997a7b07]
2024-07-28T12:01:38.318Z |  INFO | vim-async-2 | OpIdLogger.java | 35 | [PropertyCollector(session[52464a6a-d2da-2936-de92-7d3028156819]5247f8b4-a8f9-edea-ce27-d373a1010246)->WaitForUpdatesEx:51c59c2d01324e35] Completed.
2024-07-28T12:01:38.319Z |  INFO | vim-monitor | OpId.java | 37 | [PropertyCollector(session[52464a6a-d2da-2936-de92-7d3028156819]5247f8b4-a8f9-edea-ce27-d373a1010246)->WaitForUpdatesEx:c7b620d31c3a75b4] created from [WaitForUpdatesEx:9778e6dc74106656]
2024-07-28T12:01:41.191Z |  WARN | vlsi | AgencyBase.java | 467 | Invocation of disabled API Agency.Update
You have new mail in /var/mail/root

[dbutch1976]

Checking the eam.log pushed me towards this article:

vCLS VMs do not deploy due to the EAM Agency showing as "Disabled" following a VSAN Cluster Shutdown (broadcom.com)

Although I'm running esxi8 I am running VSAN, and I strongly suspect due to issues I've had with the cluster two of the nodes may have not been added back into the cluster properly. I'm trying to shut the cluster down and restart it, but I'm still having issues because the HA agent will not start the two affect nodes. I'm going to try removing the nodes from vCenter, restarting, disabling HA, then trying the whole thing over again. I think I'm on the right track, thanks for the tip about the eam.log.