"Find entity by IP" when PCNS gives shutdown command.

[Glitchy]

I recently installed PCNS 3.1 on vMA 5.5. This in an ESXi version 5.5.

I installed vMA and added my ESXi  with 'vifp addserver server.domain.name' .

with 'vifp listserver'I can see that it is succesfully added.

Then, I installed Powerchute and configured it like this:

-IPV4

-Standalone VMware Host

-information to the ESXi server (IP address, username and password)

-Single UPS configuration.

-login information to the ups

the PCNS succesfully regsitered the UPS' management card.

The problem

------------------

I now go to the UPS' web ui and comamnd it to reboot the UPS (to test my PCNS obviously).

So I go to the event log in the PCNS and it show 'shutting down host IP ADDRESS'.

But when I view my ESXi it says 'Find entity by IP' and it doesn't do anything.

Both the PCNS and ESXi view these events every 30 seconds.

What can I do to make it actually work?

Thanks in advance.

[dgrehan]

Hi,

PCNS 3.1 no longer requires you to add the ESXi host using  vifp addserver command.

Can you enable debug logging in PCNS 3.1 as follows:

1. sudo service PowerChute stop.

2. Edit /opt/APC/PowerChute/group1/log4j.xml using sudo vi and change:

<root>

        <priority value="error" />

        <appender-ref ref="rollingFile" />

    </root>

to

<root>

        <priority value="debug" />

        <appender-ref ref="rollingFile" />

    </root>

3. Save log4.xml and re-start the daemon - sudo service PowerChute start.

4. Try to shutdown the UPS again.

Attach a copy of /opt/APC/PowerChute/group1/error.log

[dgrehan]

Can you check if the following errors appear in /opt/APC/PowerChute/group1/error.log:

ERROR Thread-35 com.apcc.m11.components.Shutdowner.ShutdownSequencer - performShutdownForESXiStandaloneConfiguration

com.vmware.vim25.RestrictedVersion

ERROR pool-2-thread-1 com.apcc.m11.components.Shutdowner.CheckIfCriticalEventActive - processCriticalEventExceptClusterMode

com.vmware.vim25.RestrictedVersion

This error - "com.vmware.vim25.RestrictedVersion" usually points to a licensing issue e.g. attempting to shutdown the Free/Unlicensed version of ESXi.

https://www.vmware.com/support/developer/vc-sdk/visdk41pubs/ApiReference/vim.fault.RestrictedVersion.html

http://sourceforge.net/p/vijava/discussion/823526/thread/0224ed4c

http://sourceforge.net/p/vijava/discussion/826592/thread/3ee70cd7

http://www.virtuallyghetto.com/2011/06/dreaded-faultrestrictedversionsummary.html

Can you confirm if it is the free version of ESXi 5.5 or a licensed version? If licensed, can you specify the license type?

PowerChute Network Shutdown does not support the free version of ESXi. The following Kbase provides details on how to perform ESXi host shutdown for the free version:

shutdownHostViaSOAPAPICall.pl for ESX(i) licensed and free version

[dgrehan]

We have discovered an issue in PCNS 3.1 that affects Standalone ESX/ESXi hosts only. NB: This does not occur if you are using the Managed by vCenter Server option.

PowerChute opens an API session to communicate with the ESX/ESXi host that later times out and PowerChute fails to re-establish the session. This prevents PowerChute from issuing a shutdown command to the Host when a critical UPS event occurs. A fix has been identified and an update to address this issue will be published on the APC Web site shortly. v3.0.1 should be used for Standalone ESXi hosts until the update has been released.

[stanj]

I am running ESXi 5.1 with a vMA along with vCenter Server 5.1.

I have PCNS 3.0.1 installed on the vMA.

I do not recall seeing or selecting an option for Managed by vCenter Server.

The current installation is working as when there is a critical battery event, the VMs do an orderly shutdown.

My goal is to upgrade to vSphere 5.5.

APC support indicted that vSphere / ESXi 5.5 is not supported with PCNS but didn’t say it would not work.

Is it possible to upgrade to 5.5 and use PCNS to shutdown the VMs if the batteries reach the critical threshold?

thanks

[dgrehan]

Yes it should be possible to upgrade to vSphere 5.5 and use PCNS to shutdown the VMs. You should be able to use PCNS 3.0.1 to achieve this.

PCNS 3.1 is the latest version and provides support for HA Clusters.

[defkev]

As much as i hate resurrecting old topics but as of today, running:

* ESXi 5.5 (licensed)

* vMA 5.5

* PCNS 3.10

the exact same Problem still exists and the announced patch has yet to be released, or at least i am not seeing anything over at apc.com...

The problem just hit me on a setup i did a while ago, and never truly tested (yeah, yeah, shame on me...never ever trust a Whitepaper) with the site having a one hour lasting power-outtake this morning and none of the configured VMs were gracefully shutdown and the host just turning off after sucking the batteries dry.

The vSphere Client is only reporting "Find entity by IP" and "Find entity by DNS" again and again once the shutdown gets called by PCNS but nothing actually happens, no Host shutdown gets initiated thus no VM gets shutdown either.

As a "workaround" i have now setup PCNS to call the actual shutdown bash file /opt/APC/PowerChute/group1/bin/shutdown explicit as a Run command under Shutdown Settings in the PCNS frontend...and i am pretty certain that this shouldn't be required to get it to work.

Cheers!

[stanj]

I just deployed PCNS 3.1 Appliance (not powered up yet)  and also have a vMA running on ESXi 5.5 U1 build 1881737 with plans to move to ESXi U2.

I was going to shutdown the vMA and bring the PCNS Appliance on-line to handle the automated shutdown.

What problem are you referring to that requires a patch?

thanks

[dgrehan]

The issue is outlined in the following Kbase FAQ - FA222090

It has since been fixed - This update was included as part of the main product on the Software/Firmware downloads page on February 17th 2014 so any downloads since this date are not exposed to this issue.

[stanj]

ok,

Looks like I need to then download and apply the patch PCNS3.1.0EN.tar.gz listed in the FAQ?

[dgrehan]

If you downloaded the Appliance before Feb 17th then yes you'll need to install the patch or download the Appliance again to replace your existing copy.

[dgrehan]

@defkev: FindByIP/FindByDNSName: Could you attach a copy of error.log, pcnsconfig.ini and EventLog.txt please? These are located in /opt/APC/PowerChute/group1 folder on vMA.

The /opt/APC/PowerChute/group1/bin/shutdown is no longer used in 3.1 for shutting down the ESXi host.

[stanj]

The one i have unnzippd is dated 2/4/14.

What is in the logs??

[dgrehan]

The affected version is build 4512 (See About Box in PowerChute UI).

The fixed version is build 4844. The fixed version of the Appliance is available to download from the APC Software/Firmware downloads page.

The request for the logs was in response to the comments posted by user defkev  on Sep 11, 2014 1:12 AM.

[stanj]

ok,

i see the new appliance is dated 9/10/14 at the APC site

It would be nice if APC would let users know when thee are new patches or potential issues once they are discovered

[dgrehan]

My apologies,  I should have updated this thread when the fixed version was released. We added entries on the APC Forums and VMware Forums when the issue was discovered.

[stanj]

no problem,

I am glad you updated the forum because I did receive the email alert to check..

Where in the appliance can I find the build number?

[dgrehan]

The build number is shown in the PowerChute UI->Help->About.

For the next and future releases we will look at including the Build Number in the Appliance Name on the Downloads page so as to make it easier to distinguish if there are updates required in future.

[stanj]

Ok,

So I was going to do some UPS testing and was looking over the PowerChute VMware User Guide. Pg 23 showed the Plugin in vCenter.

I logged into PCNS (https://xxx.xxx.xxx.x:6547)  and checked the Communication Settings tab and

I did not see the plugin, so I went back thorough the setup and added my vCenter.

The plugin is now there. 

Prior to installing the new PCNS VM App via OVA, I was using the vMA and had to go in to the vcenter and set up the VM shutdown order (attached screen)

The question I have is, I can't recall if I made these setting or if configuring PCNS to vCenter made them?

Are these still required ?

thanks

[defkev]

As i originally wrote i am running PCNS on vMA (as a service, installed using pcns310ESXi.tar downloaded on July 7 2014) and not the PCNS appliance, as i make use of the vMA anyway.

The build number is 4844

If 3.1 doesn't use /opt/APC/PowerChute/group1/bin/shutdown to initiate the host shutdown, what should i be looking for within PCNS instead? What script/binary calls the shutdown? Does it even use the vi-fastpass target any longer?

Find attached snippets of the log files during a NMC initiated UPS restart, which didn't shutdown the host/prior to adding the (now obsolete) shutdown script as a Run command.

Eventlog.txt:

...
09/11/2014      01:23:16        Communication has been established.     .3.5.1.5.6.1
09/11/2014      01:23:16        Communication established with Environmental Monitor.   .3.5.1.7.20
09/11/2014      01:23:18        PowerChute successfully opened UDP port 3052.   .3.4.9.9
09/11/2014      01:23:18        PowerChute Network Shutdown version 3.1.0 monitoring started.   .3.5.1.5.6.10
09/11/2014      01:23:19        PowerChute successfully opened TCP port 3052.   .3.4.9.9
09/11/2014      01:23:19        PowerChute successfully opened TCP port 6547.   .3.4.9.9
09/11/2014      01:43:53        UPS turn off has been initiated.        .3.5.1.4.6
09/11/2014      01:43:53        UPS critical event: <b>UPS turn off initiated</b> occurred on Host: <b>esxi.fully.qualified.domainname</b>.     .3.4.9.9
09/11/2014      01:46:58        UPS critical event: <b>UPS turn off initiated</b> resolved on Host: <b>esxi.fully.qualified.domainname</b>.     .3.4.9.9

errorlog.txt

...

ERROR main com.apcc.m11.components.Shutdowner.Shutdowner - bad interface ID: .1.3.7
ERROR main com.apcc.m11.components.Shutdowner.Shutdowner - bad interface ID: .1.3.8
ERROR pool-1-thread-6 com.apcc.m11.components.Omaha.OmahaComponent - com.apcc.m11.arch.CommunicationException
ERROR pool-1-thread-18 com.apcc.m11.components.AutoUpdate.SoftwareUpdateChecker_HTTPS - http 404
ERROR 1366201742@qtp-1777521044-3 com.apcc.m11.components.AutoUpdate.SoftwareUpdateChecker_HTTPS - http 404
ERROR Thread-31 com.apcc.m11.components.CommandFileRunner.DelayedExecutor - No event formatter
ERROR main com.apcc.m11.components.EventLogger.Logger.TextFileLogger - java.io.FileNotFoundException: EventLog (No such file or directory)

Furthermore the pcnsconfig.ini is still pointing the "shutdownCommand" to "/opt/APC/PowerChute/group1/bin/shutdown"

pcnsconfig.ini

[NetworkManagementCard]
port = 80
protocol = http
enrolWithNMC = true
acceptCerts = false
IP_1 = the.nmc.ipv4.address
mode = single
IP_1_Outlet = 0

[EventLog]
logsize = 1000
logfile = EventLog.txt

[Shutdown]
shutdownCommandDuration = 60
turnOffUps = false
turnOffSOG = false
shutdownDelay = 10
event_ShutdownStarting_enableLogging = true
shutdownCommand = /opt/APC/PowerChute/group1/bin/shutdown
runtimeRemainingThreshold = 0
shutdownCommandFile = /opt/APC/PowerChute/group1/bin/shutdown
shutdownCommandFileDelay = 0
shutdownCommandFileDuration = 0
shutdownCommandFileEnabled = true

...

[AdvancedConfig]
SinglePowerFailed_enableLogging = true
SinglePowerFailed_turnOffUps = false
SinglePowerFailed_turnOffUpsDelay = 0
SinglePowerFailed_enableCommandFile = false

[HostDetails]
ESXiConfigurationMode = Standalone Host.
ESXi_IP = esxi.fully.qualified.domainname

So who or what am i supposed to believe now?

Thanks for looking into this.

Regards

PS: Looks like the Update Check also isn't working right now, the server only returns a 404 querying https://updates.apc.com/pcns/autoupdate.cfm

[dgrehan]

Hi Defkev,

Starting in version 3.1, PowerChute no longer uses /opt/APC/PowerChute/group1/shutdown script which called shutdownhost.pl. The is a PERL script that used VMWare PERL SDK and vi-fastpass functionality to connect to the ESXi host to perform the shutdown.

In 3.1 PowerChute uses VI-Java which is a wrapper library that interacts with the VMware Web SDK. In order to troubleshoot the issue you are having I would need to have a look at the debug logs from PowerChute. If you'd like to send me a private message I can provide instructions on how to enable debug logging and you can send me the logs directly rather than posting them publicly.

I'm guessing that there may be an issue trying to resolve the ESXi hostname or a permissions issue but I cannot tell for sure without access to the debug log.

[defkev]

Hi dgrehan,

I got it working without the shutdown Perl script!

After removing the Run Command from Shutdown Settings, enabling debug logging in log4j.xml, restarting PCNS and restarting the APC trough the NMC the following catches my eye in error.log:

...
DEBUG pool-2-thread-3 com.apcc.pcns.configservice.configurationmanagers.ConfigurationManagerImpl - containsKey(String,String) - Section: NetworkManagementCard, Key: mode, Value: true
DEBUG pool-2-thread-3 com.apcc.pcns.configservice.configurationmanagers.ConfigurationManagerImpl - getString(String,String) - Section: NetworkManagementCard, Key: mode, Value: single
DEBUG pool-2-thread-3 com.apcc.m11.components.Shutdowner.CheckIfCriticalEventActive - processCriticalEventExceptClusterMode
DEBUG pool-2-thread-3 com.apcc.pcns.configservice.configurationmanagers.ConfigurationManagerImpl - containsKey(String,String) - Section: HostDetails, Key: ESXiConfigurationMode, Value: true
DEBUG pool-2-thread-3 com.apcc.pcns.configservice.configurationmanagers.ConfigurationManagerImpl - getString(String,String) - Section: HostDetails, Key: ESXiConfigurationMode, Value: Standalone Host.
DEBUG pool-2-thread-3 com.apcc.pcns.configservice.configurationmanagers.ConfigurationManagerImpl - containsKey(String,String) - Section: HostDetails, Key: ESXiConfigurationMode, Value: true
DEBUG pool-2-thread-3 com.apcc.pcns.configservice.configurationmanagers.ConfigurationManagerImpl - getString(String,String) - Section: HostDetails, Key: ESXiConfigurationMode, Value: Standalone Host.
DEBUG pool-2-thread-3 com.apcc.m11.components.WebServer.CfgWizard.pages.VCenterServerConnection - Checking ESXi service instance session is valid or not
DEBUG pool-2-thread-3 com.apcc.m11.components.WebServer.CfgWizard.pages.VCenterServerConnection - ESXi Service instance session is valid
DEBUG pool-2-thread-3 com.apcc.pcns.configservice.configurationmanagers.ConfigurationManagerImpl - containsKey(String,String) - Section: HostDetails, Key: ESXi_IP, Value: true
DEBUG pool-2-thread-3 com.apcc.pcns.configservice.configurationmanagers.ConfigurationManagerImpl - getString(String,String) - Section: HostDetails, Key: ESXi_IP, Value: esxi.fully.qualified.domainname
DEBUG pool-2-thread-3 com.apcc.m11.components.WebServer.util.VirtualizationUtils - getHostUsingIPOrDNSName - Host IP - esxi.fully.qualified.domainname
DEBUG pool-1-thread-13 com.apcc.pcns.configservice.configurationmanagers.ConfigurationManagerImpl - getString(String,String) - Section: Networking, Key: NetworkConfig, Value: IPV4
DEBUG pool-1-thread-13 com.apcc.pcns.configservice.configurationmanagers.ConfigurationManagerImpl - getString(String,String) - Section: Networking, Key: IPv6NetworkConfig, Value: unicast
DEBUG pool-1-thread-5 com.apcc.m11.components.StdPowerSource.BroadcastReceiver - Received Data:
INFO pool-1-thread-5 com.apcc.m11.components.StdPowerSource.SecureHttpTransportAdapter - StdPowerSource: GET VERIFICATION PASSED
INFO pool-1-thread-5 com.apcc.m11.components.StdPowerSource.BroadcastReceiver - packet_target :vma.networks.broadcast.ip
DEBUG pool-1-thread-5 com.apcc.m11.components.StdPowerSource.BroadcastReceiver - Checking if vma.networks.broadcast.ip is a valid localhost address.
DEBUG pool-1-thread-5 com.apcc.m11.components.StdPowerSource.BroadcastReceiver - returning existing theLocalIpSet
DEBUG pool-1-thread-5 com.apcc.m11.components.StdPowerSource.BroadcastReceiver - vma.networks.broadcast.ip does not match vma.ip.v4.address
DEBUG pool-1-thread-5 com.apcc.m11.components.StdPowerSource.BroadcastReceiver - vma.networks.broadcast.ip is not a valid localhost address.
...

The ESXi_IP was set to the ESXi hosts fully qualified domain name, this FQDN is resolvable by the vMA and pointing to the ESXi hosts IP address of course.

Now, after changing the FQDN to the actual IP address of the ESXi host under vma -> Communication Details -> ESXi Host Details -> VMware Host IP Address/Hostname (changing ESXi_IP to esxi.ip.v4.address in pcnsconfig.ini) the shutdown is working as intended.

No more "Find entity by IP/DNS" gets logged in the vSphere Client and the host gets shutdown after gracefully shutting down the configured VMs.

I dunno what PCNS is really trying to do here, even true the network in question is considering of multiple Subnets all relevant equipment - ESXi/vMA/NMC - are on the exact same Subnet and thus a broadcast shouldn't even be necessary.

Regards

defkev

[stanj]

I have the PCNS Appliance running.

In vCenter, the VM Startup and Shutdown is set as this was used with the vMA.

Is there a way to test the new PCNS Appliance in order to shutdown the VMs without having to pull the power to the UPS ?

[defkev]

@stanj

Log on to the NMC controlling your vMA (the one shown in PCNS -> UPS Communication) and verify that the vMA is registered as a "PowerChute Network Shutdown Client" under UPS -> Configuration -> PowerChute clients

Than simply reboot the UPS Outlet Group (UPS -> Control -> UPS) and make sure "Signal PowerChute Network Shutdown Clients" is checked.

After clicking [Next] PCNS should log an event along the line "UPS critical event: UPS turn off initiated occurred on ..." followed by a "Shutting down ..."

Now the vSphere Client/Web Client should queue the host(s) shutdown followed by (gracefully) shutting down all configured VMs.

Cheers!

[stanj]

thanks 

I will give it a shot.

since I am not in the same location as the server, I assume I will have to be in the building to power up the ESXi Server and then restart the VMs?

Also, I have a PC set to power down.

I believe it is running a PCNS version for Windows 7.

[stanj]

Note, I am not using vMA,

[defkev]

Depends on your equipment/setup, if your remote connection isn't affected by the host shutdown (Gateway as a hardware appliance/still running native or on another host) and you have a IP-KVM (e.g. HP Server w/ configured ILO) setup you should be fine.

Alternative call an employe onside with physical access to the server and tell him/her to power the "currently not running thing in rack whatever" up.

If everything fails, you have to get your "hands dirty"

VMs are the more easy part here, as every Virtual Machine you have setup to gracefully shutdown in vSphere will be started automatically (in reverse order) once the host gets powered up, as long as the host isn't in maintenance mode of course.

[stanj]

When you say in the previous message.. ...

simply reboot the UPS Outlet Group (UPS -> Control -> UPS) and make sure "Signal PowerChute Network Shutdown Clients" is checked.

Is this accomplished in the Action column in the attached screen shot ?

[defkev]

Hi

you don't need to explicit tinker with the outlet groups, just reboot them all and tell the clients to shutdown.

UPS > Control > UPS (same sub menu, one point above, see attached screenshot)

Cheers