I logged in to the VIC this morning and I noticed that 2 of my templates are greyed out and listed as "orphaned". I looked at the Events within VirtualCenter and noticed the following:
info 11/10/2007 1:21:19 AM Win2003 R2 SP2 STD x64 10GB on host esx102.something.com in Datacenter is disconnected
warning 11/10/2007 1:21:32 AM Win2003 R2 SP2 STD x64 10GB does not exist on esx102.something.com in Datacenter
So from there I did some digging with the logs on that particular ESX host and this is what I came up with:
From /var/log/messages:
Nov 10 01:21:10 esx102 watchdog-vpxa: '/usr/sbin/vpxa' exited after 1536438 seconds
Nov 10 01:21:10 esx102 watchdog-vpxa: Executing '/usr/sbin/vpxa'
Nov 10 01:21:21 esx102 /usr/lib/vmware/hostd/vmware-hostd[1438]: Accepted password for user vpxuser from 127.0.0.1
From /var/log/secure:
Nov 10 01:21:20 esx102 xinetd[1326]: START: vmware-authd pid=27739 from=172.21.2.132
Nov 10 01:21:20 esx102 vmware-authd[27739]: login from 172.21.2.132 as vpxuser
From /var/log/vmware/vpx/vpxa.log:
Unexpected return result. Expect 1 sample, receive 2
Unexpected return result. Expect 1 sample, receive 2
Unexpected return result. Expect 1 sample, receive 2
Unexpected return result. Expect 1 sample, receive 2
Unexpected return result. Expect 1 sample, receive 2
Unexpected return result. Expect 1 sample, receive 2
Unexpected return result. Expect 1 sample, receive 2
Unexpected return result. Expect 1 sample, receive 2
Unexpected return result. Expect 1 sample, receive 2
Unexpected return result. Expect 1 sample, receive 2
So it looks like vpxa failed if I'm reading these logs correctly, but I'm not entirely sure. The same exact thing (4 templates showed up as "orphaned") happened earlier in the week with a different ESX host in the DataCenter (it was part of a different Cluster as well), so I'm trying to nail this issue down. Lastly, while I'm certain that I could simply remove the Orphaned templates from the VIC and re-add them by browsing the Datastore, I am unable to make changes at this company due to a freeze in IT changes. So the more investigative "look but don't touch" work I can do, the better.