VM shows alert icon with no alarms triggered

[stuartcallison]

I have been having phantom alarms lately. Virtual machines will show the red alert icon with no alarms triggered. I checked the VM, the host, the cluster, the folder, the datacenter, and vcenter but none of those objects have any triggered alarms either. The cluster does not have HA or DRS enabled.

I had several VMs in this condition so I restarted vCenter server, which usually solves the problem for a while. However, this time two of the VMs have retained their alert state through the vCenter server restart.

I ran this powercli script on the trouble VMs and every alarm state was either gray or green. How can I find out what is causing the VMs to red exclamation point alert symbol?

$vm = Get-VM -Name VM | Get-View

foreach($state in $vm.DeclaredAlarmState){

    $alarm = Get-View $state.Alarm

    Write-Host "name:"

    Write-Host $alarm.info.name

    Write-Host "overall status:"

    Write-Host $state.OverallStatus

}

[stuartcallison]

I forgot to mention, the hardware status tab of each host shows all green checkmarks, there is nothing indicated wrong there.

[mschau]

My Lab environment did this to me today.   I had recabled my environment so I had shutdown the SAN and my two lab hosts.  When they came up I manually started my DCs, SQL and vCenter etc directly on the hosts. When I signed into vCenter I saw alerts on ALL of my VMs that were located on my SAN. Some how the VMs did not register correctly. (Maybe my hosts came up before the SAN).  A quick vmotion between hosts cleared the alert without having to remove the VM and re-adding it to Inventory.

[PMI_IT]

I had the same issue in my v5 lab and a vMotion was just what the Dr ordered.

Thanks!!

[Jt3rry]

I've also found that vMotion clears the phantom alarm... Is this an ESX 5 bug? We're running 5.0.0 build 504890 and have seen this behavior on 4 hosts running 2008 R2 VMs.

The left-hand red alert doesn't clear on the VM, yet when you click into the VM's alarms tab the event is missing making it hard to troubleshoot what causes the alarm in the first place.

Any ideas?

[kruddy]

I saw this in our DR site this morning after upgrading the site to ESXi 5 Update 1 and also upgrading the SAN's firmware as well.

I think mschau has it right, I think the host came up before the SAN did and all of the systems were probably disconnected for a couple seconds.

Eitherway, a reboot fixed everything and cleared all the alerts.

[mschau]

If you don't want to reboot I found that just vmotioning the VMs from one host to another cleared up the alerts.

[onerobertone]

I had this problem today.  The fix was to reconfigure HA at the cluster level and then the alarm could be seen and then removed.

[lm31]

I saw this problem as well - services.sh restart fixed it for me.

[Northsaloplad]

I had the same issue and it turned out to be licencing

[richydom]

"Restart Management Agents" on the esx host, where the alarm vm is running solves the problem.

[allison_gibb]

vMotion didn't clear my alarm but unticking HA cleared the alarm and then I was able to tick HA again OK. Thanks

[govandsinjari]

One of my ESXis went offline after hardware failure, after fixing it, the red alert triangle appeared on all VMs, and there is no way to clear it, this is if I am logged in through the vCenter, BUT there were no alerts if you login directly to ESXi.

Solution:

From the vCenter, click on the ESXi host "Configuration" tab, select "Security Profile", then from Services properties I have restarted VPXA services which is vCenter account on the ESXi, I lost connection to ESXi host temporary but it recovered after "Reconnect" and the alerts cleared!

[cgprime]

I have confirmed that the answer provided by govandsinjari will resolve the issue.

In my case, I brought up the ESXi host before the HNAS storage had come back online.

- Restart VPXA for the ESXi host the VM's that display the alert reside

- A new alert will show up for the disconnected host, Connect the host to vCenter

- Clear the disconnected host alert and you will be clear

[Nigel Hickey]

This is a valuable thread as I have found myself digging this out twice before.

As govandsinjari states below you can do this method from within the vSphere C# client or you can also perform the same function via SSH to your ESXi host.

govandsinjari wrote:

Solution:

From the vCenter, click on the ESXi host "Configuration" tab, select "Security Profile", then from Services properties I have restarted VPXA services which is vCenter account on the ESXi, I lost connection to ESXi host temporary but it recovered after "Reconnect" and the alerts cleared!

-SSH to ESXi

     Command to type:
     /etc/init.d/vpxa restart

You will get the same results as if using the vSphere client less all the errors from the disconnects that the client spits out since you are not connected to the vCenter and performing the task.

Thanks you and Good Luck!

Nigel