Hi EveryBody;
I use VMware ESXi 1.4 u2. Saturday night was a server reboot for some reason can not be determined. could not make something clear look at its log files.
Dec 16 02:22:13 rentHost><AlternateHost>typ-esx05</AlternateHost><StateHint>RUNNING</StateHint><StateHintReason>UNSET</StateHintReason><FTStateHint>FT_NOTCONFIGURED</FTStateHint><FTStateHintReason>UNSET</FTStateHintReason></VM><VM><Name>/%2fvmfs%2fvolumes%2f4fbe3fa1%2d7f8b67cb%2da47e%2dd485644aa694%2fTY%2dMNG02%2fTY%2dMNG02%2evmx</Name><Type>1</Type><MinCpu>0</MinCpu><MinMemory>388</MinMemory><Priority>3</Priority><FtPair>/</FtPair><CurrentHost>typ-esx05</CurrentHost><AlternateHost>typ-esx05</AlternateHost><StateHint>RUNNING</StateHint><StateHintReason>UNSET</StateHintReason><FTStateHint>FT_NOTCONFIGURED</FTStateHint><FTStateHintReason>UNSET</FTStateHintReason></VM><VM><Name>/%2fvmfs%2fvolumes%2f4fbe0073%2d294c9abd%2d5d60%2dd485644aa694%2fTY%2dMRKWEB%2fTY%2dMRKWEB%2evmx</Name><Type>1</Type><MinCpu>0</MinCpu><MinMemory>287</MinMemory><Priority>3</Priority><FtPair>/</FtPair><CurrentHost>typ-esx05</CurrentHost><AlternateHost>typ-esx05</AlternateHost><StateHint>RUNNING</StateHint><StateHintReason>UNSET</StateHintReason><FTState
Dec 16 02:22:13 Hint>FT_NOTCONFIGURED</FTStateHint><FTStateHintReason>UNSET</FTStateHintReason></VM><VM><Name>/%2fvmfs%2fvolumes%2f4fbe3b37%2db11f42d1%2d97b0%2dd485644aa694%2fTY%2dMRKWEB01%20%28sen%2etrendyol%29%2fTY%2dMRKWEB01%20%28sen%2etrendyol%29%2evmx</Name><Type>1</Type><MinCpu>0</MinCpu><MinMemory>618</MinMemory><Priority>3</Priority><FtPair>/</FtPair><CurrentHost>typ-esx05</CurrentHost><AlternateHost>typ-esx05</AlternateHost><StateHint>RUNNING</StateHint><StateHintReason>UNSET</StateHintReason><FTStateHint>FT_NOTCONFIGURED</FTStateHint><FTStateHintReason>UNSET</FTStateHintReason></VM><VM><Name>/%2fvmfs%2fvolumes%2f4fbcf0de%2ddd1b9286%2d22be%2dd485644aa694%2fTY%2dNSSRV01%20CLONE%2fTY%2dNSSRV01%20CLONE%2evmx</Name><Type>1</Type><MinCpu>0</MinCpu><MinMemory>202</MinMemory><Priority>3</Priority><FtPair>/</FtPair><CurrentHost>typ-esx05</CurrentHost><AlternateHost>typ-esx05</AlternateHost><StateHint>ABSENT</StateHint><StateHintReason>UNCLEAN</StateHintReason><FTStateHint>FT_NOTCONFIGURED</FTStateHint><FTStateHintReason>UN
Dec 16 02:22:13 SET</FTStateHintReason></VM><VM><Name>/%2fvmfs%2fvolumes%2f4fbe3eaf%2d053a9cc3%2d8c10%2dd485644aa694%2fTYP%2dAPPFABRIC01%2fTYP%2dAPPFABRIC01%2evmx</Name><Type>1</Type><MinCpu>0</MinCpu><MinMemory>339</MinMemory><Priority>3</Priority><FtPair>/</FtPair><CurrentHost>typ-esx05</CurrentHost><AlternateHost>typ-esx05</AlternateHost><StateHint>RUNNING</StateHint><StateHintReason>UNSET</StateHintReason><FTStateHint>FT_NOTCONFIGURED</FTStateHint><FTStateHintReason>UNSET</FTStateHintReason></VM><VM><Name>/%2fvmfs%2fvolumes%2f4fbe3fa1%2d7f8b67cb%2da47e%2dd485644aa694%2fTYP%2dDC03%2fTYP%2dDC03%2evmx</Name><Type>1</Type><MinCpu>0</MinCpu><MinMemory>202</MinMemory><Priority>3</Priority><FtPair>/</FtPair><CurrentHost>typ-esx05</CurrentHost><AlternateHost>typ-esx05</AlternateHost><StateHint>RUNNING</StateHint><StateHintReason>UNSET</StateHintReason><FTStateHint>FT_NOTCONFIGURED</FTStateHint><FTStateHintReason>UNSET</FTStateHintReason></VM><VM><Name>/%2fvmfs%2fvolumes%2f4fbe0073%2d294c9abd%2d5d60%2dd485644aa694%2fTYP%2dEMCESR
Dec 16 02:22:13 S02%2fTYP%2dEMCESRS02%2evmx</Name><Type>1</Type><MinCpu>0</MinCpu><MinMemory>198</MinMemory><Priority>3</Priority><FtPair>/</FtPair><CurrentHost>typ-esx05</CurrentHost><AlternateHost>typ-esx05</AlternateHost><StateHint>RUNNING</StateHint><StateHintReason>UNSET</StateHintReason><FTStateHint>FT_NOTCONFIGURED</FTStateHint><FTStateHintReason>UNSET</FTStateHintReason></VM><VM><Name>/%2fvmfs%2fvolumes%2f502ce327%2dfb507ca3%2de41f%2d78e7d1580b60%2fTYP%2dWEEAMSRV01%2fTYP%2dWEEAMSRV01%2evmx</Name><Type>1</Type><MinCpu>0</MinCpu><MinMemory>291</MinMemory><Priority>3</Priority><FtPair>/</FtPair><CurrentHost>typ-esx05</CurrentHost><AlternateHost>typ-esx05</AlternateHost><StateHint>RUNNING</StateHint><StateHintReason>UNSET</StateHintReason><FTStateHint>FT_NOTCONFIGURED</FTStateHint><FTStateHintReason>UNSET</FTStateHintReason></VM><VM><Name>/%2fvmfs%2fvolumes%2f501fd073%2d74870661%2d8c25%2df4ce46a85c4a%2fTY%2dSTN01%2fTY%2dSTN01%2evmx</Name><Type>1</Type><MinCpu>0</MinCpu><MinMemory>291</MinMemory><Priority>3</Priority><Ft
Dec 16 02:22:13 Pair>/</FtPair><CurrentHost>typ-esx05</CurrentHost><AlternateHost>typ-esx05</AlternateHost><StateHint>RUNNING</StateHint><StateHintReason>UNSET</StateHintReason><FTStateHint>FT_NOTCONFIGURED</FTStateHint><FTStateHintReason>UNSET</FTStateHintReason></VM><VM><Name>/%2fvmfs%2fvolumes%2f4fbcf0de%2ddd1b9286%2d22be%2dd485644aa694%2fTY%2dSTN02%2fTY%2dSTN02%2evmx</Name><Type>1</Type><MinCpu>0</MinCpu><MinMemory>291</MinMemory><Priority>3</Priority><FtPair>/</FtPair><CurrentHost>typ-esx05</CurrentHost><AlternateHost>typ-esx05</AlternateHost><StateHint>RUNNING</StateHint><StateHintReason>UNSET</StateHintReason><FTStateHint>FT_NOTCONFIGURED</FTStateHint><FTStateHintReason>UNSET</FTStateHintReason></VM><VM><Name>/%2fvmfs%2fvolumes%2f4fbce4ea%2d2e826453%2dbd9c%2dd485644aa694%2fTY%2dSTN03%2fTY%2dSTN03%2evmx</Name><Type>1</Type><MinCpu>0</MinCpu><MinMemory>291</MinMemory><Priority>3</Priority><FtPair>/</FtPair><CurrentHost>typ-esx05</CurrentHost><AlternateHost>typ-esx05</AlternateHost><StateHint>RUNNING</StateHint><StateHi
Dec 16 02:22:13 ntReason>UNSET</StateHintReason><FTStateHint>FT_NOTCONFIGURED</FTStateHint><FTStateHintReason>UNSET</FTStateHintReason></VM><VM><Name>/%2fvmfs%2fvolumes%2f502ce327%2dfb507ca3%2de41f%2d78e7d1580b60%2fTY%2dWEB05%2fTY%2dWEB05%2evmx</Name><Type>1</Type><MinCpu>0</MinCpu><MinMemory>618</MinMemory><Priority>3</Priority><FtPair>/</FtPair><CurrentHost>typ-esx05</CurrentHost><AlternateHost>typ-esx05</AlternateHost><StateHint>RUNNING</StateHint><StateHintReason>UNSET</StateHintReason><FTStateHint>FT_NOTCONFIGURED</FTStateHint><FTStateHintReason>UNSET</FTStateHintReason></VM><VM><Name>/%2fvmfs%2fvolumes%2f4fbe3f31%2d646a2a71%2d3c72%2dd485644aa694%2fTY%2dWEBSRV03%2fTY%2dWEBSRV03%2evmx</Name><Type>1</Type><MinCpu>0</MinCpu><MinMemory>291</MinMemory><Priority>3</Priority><FtPair>/</FtPair><CurrentHost>typ-esx05</CurrentHost><AlternateHost>typ-esx05</AlternateHost><StateHint>RUNNING</StateHint><StateHintReason>UNSET</StateHintReason><FTStateHint>FT_NOTCONFIGURED</FTStateHint><FTStateHintReason>UNSET</FTStateHintReason></VM
Dec 16 02:22:13 ><VM><Name>/%2fvmfs%2fvolumes%2f4fbe3b37%2db11f42d1%2d97b0%2dd485644aa694%2fTY%2dWEBSRV05%2fTY%2dWEBSRV05%2evmx</Name><Type>1</Type><MinCpu>0</MinCpu><MinMemory>291</MinMemory><Priority>3</Priority><FtPair>/</FtPair><CurrentHost>typ-esx05</CurrentHost><AlternateHost>typ-esx05</AlternateHost><StateHint>RUNNING</StateHint><StateHintReason>UNSET</StateHintReason><FTStateHint>FT_NOTCONFIGURED</FTStateHint><FTStateHintReason>UNSET</FTStateHintReason></VM><VM><Name>/%2fvmfs%2fvolumes%2f4fbe0073%2d294c9abd%2d5d60%2dd485644aa694%2fTY%2dWEBSRV08%2fTY%2dWEBSRV08%2evmx</Name><Type>1</Type><MinCpu>0</MinCpu><MinMemory>484</MinMemory><Priority>3</Priority><FtPair>/</FtPair><CurrentHost>typ-esx05</CurrentHost><AlternateHost>typ-esx05</AlternateHost><StateHint>RUNNING</StateHint><StateHintReason>UNSET</StateHintReason><FTStateHint>FT_NOTCONFIGURED</FTStateHint><FTStateHintReason>UNSET</FTStateHintReason></VM><VM><Name>/%2fvmfs%2fvolumes%2f4fbe3eaf%2d053a9cc3%2d8c10%2dd485644aa694%2fTY%2dWEBSRV12%2fTY%2dWEBSRV12%2evmx</Name
Dec 16 02:22:13 ><Type>1</Type><MinCpu>0</MinCpu><MinMemory>388</MinMemory><Priority>3</Priority><FtPair>/</FtPair><CurrentHost>typ-esx05</CurrentHost><AlternateHost>typ-esx05</AlternateHost><StateHint>RUNNING</StateHint><StateHintReason>UNSET</StateHintReason><FTStateHint>FT_NOTCONFIGURED</FTStateHint><FTStateHintReason>UNSET</FTStateHintReason></VM><VM><Name>/%2fvmfs%2fvolumes%2f4fbe3eaf%2d053a9cc3%2d8c10%2dd485644aa694%2fTY%2dWEBSRV14%2fTY%2dWEBSRV14%2evmx</Name><Type>1</Type><MinCpu>0</MinCpu><MinMemory>618</MinMemory><Priority>3</Priority><FtPair>/</FtPair><CurrentHost>typ-esx05</CurrentHost><AlternateHost>typ-esx05</AlternateHost><StateHint>RUNNING</StateHint><StateHintReason>UNSET</StateHintReason><FTStateHint>FT_NOTCONFIGURED</FTStateHint><FTStateHintReason>UNSET</FTStateHintReason></VM><VM><Name>/%2fvmfs%2fvolumes%2f4fbe3f31%2d646a2a71%2d3c72%2dd485644aa694%2fTY%2dWEBSRV18%2fTY%2dWEBSRV18%2evmx</Name><Type>1</Type><MinCpu>0</MinCpu><MinMemory>618</MinMemory><Priority>3</Priority><FtPair>/</FtPair><CurrentHost>typ-e
Dec 16 02:22:13 sx05</CurrentHost><AlternateHost>typ-esx05</AlternateHost><StateHint>RUNNING</StateHint><StateHintReason>UNSET</StateHintReason><FTStateHint>FT_NOTCONFIGURED</FTStateHint><FTStateHintReason>UNSET</FTStateHintReason></VM><VM><Name>/%2fvmfs%2fvolumes%2f502ce327%2dfb507ca3%2de41f%2d78e7d1580b60%2fAV%2dUSM%2fAV%2dUSM%2evmx</Name><Type>1</Type><MinCpu>0</MinCpu><MinMemory>383</MinMemory><Priority>3</Priority><FtPair>/</FtPair><CurrentHost>typ-esx05</CurrentHost><AlternateHost>typ-esx05</AlternateHost><StateHint>RUNNING</StateHint><StateHintReason>UNSET</StateHintReason><FTStateHint>FT_NOTCONFIGURED</FTStateHint><FTStateHintReason>UNSET</FTStateHintReason></VM><VM><Name>/%2fvmfs%2fvolumes%2f5061f524%2d05f8449a%2db7b7%2d1cc1de7597e8%2fTY%2dWEB01%2eprod%2etrendyol%2ecom%20convert%5freplica%2fTY%2dWEB01%2eprod%2etrendyol%2ecom%20conve%2evmx</Name><Type>1</Type><MinCpu>0</MinCpu><MinMemory>778</MinMemory><Priority>3</Priority><FtPair>/</FtPair><CurrentHost>typ-esx05</CurrentHost><AlternateHost>typ-esx05</AlternateHost
Dec 16 02:22:13 ><StateHint>RUNNING</StateHint><StateHintReason>UNSET</StateHintReason><FTStateHint>FT_NOTCONFIGURED</FTStateHint><FTStateHintReason>UNSET</FTStateHintReason></VM><VM><Name>/%2fvmfs%2fvolumes%2f4fc66cd1%2d72bb3ebb%2db33c%2dd485644aa694%2fMONGO01%2fMONGO01%2evmx</Name><Type>1</Type><MinCpu>0</MinCpu><MinMemory>287</MinMemory><Priority>3</Priority><FtPair>/</FtPair><CurrentHost>typ-esx05</CurrentHost><AlternateHost>typ-esx05</AlternateHost><StateHint>RUNNING</StateHint><StateHintReason>UNSET</StateHintReason><FTStateHint>FT_NOTCONFIGURED</FTStateHint><FTStateHintReason>UNSET</FTStateHintReason></VM><VM><Name>/%2fvmfs%2fvolumes%2f50bb5b3e%2d3114ea4b%2df79d%2d78e7d1580b60%2fTYP%2dSQLCLU01%2fTYP%2dSQLCLU01%2evmx</Name><Type>1</Type><MinCpu>0</MinCpu><MinMemory>1581</MinMemory><Priority>3</Priority><FtPair>/</FtPair><CurrentHost>typ-esx05</CurrentHost><AlternateHost>typ-esx05</AlternateHost><StateHint>RUNNING</StateHint><StateHintReason>UNSET</StateHintReason><FTStateHint>FT_NOTCONFIGURED</FTStateHint><FTStateHint
Dec 16 02:22:13 Reason>UNSET</FTStateHintReason></VM></VMList></NodePolicy>
Dec 16 02:22:16 12/16/12 02:22:15 [print_args ] PWD=/var/log/vmware/vpx 12/16/12 02:22:15 [print_args ] PS_OPTIONS= 12/16/12 02:22:15 [print_args ] FT_NO_CONSOLE_TRACE=1 12/16/12 02:22:15 [print_args ] PATH=/sbin:/usr/sbin:/bin:/usr/bin:/opt/vmware/aam/bin:/bin 12/16/12 02:22:15 [print_args ] FT_LOG_DIR=/var/log/vmware/aam 12/16/12 02:22:15 [print_args ] FT_DOMAIN=vmware 12/16/12 02:22:15 [print_args ] Parsed arguments: 12/16/12 02:22:15 [print_args ] cmd=listnodes 12/16/12 02:22:15 [print_args ] -z=1 12/16/12 02:22:15 [print_args ] uname=VMkernel 12/16/12 02:22:15 [print_args ] shortname=typ-esx05 12/16/12 02:22:15 [print_args ] domain=vmware 12/16/12 02:22:15 [issue_cmd ] CMD: /opt/vmware/aam/bin/ft_gethostbyname typ-esx05 |grep FAILED 12/16/12 02:22:15 [issue_cmd ] STATUS: 1 12/16/12 02:22:15 [issue_cmd ] RESULT:
Dec 16 02:22:16 12/16/12 02:22:15 [issue_cmd ] 12/16/12 02:22:15 [list_nodes ] 12/16/12 02:22:15 [issue_cli_cmd ] command is '/opt/vmware/aam/bin/ftcli -domain vmware -port 8042 -timeout 15 -cmd listnodes' 12/16/12 02:22:15 [issue_cmd ] CMD: /opt/vmware/aam/bin/ftcli -domain vmware -port 8042 -timeout 15 -cmd listnodes 12/16/12 02:22:15 [issue_cmd ] STATUS: 0 12/16/12 02:22:15 [issue_cmd ] RESULT: 12/16/12 02:22:15 [issue_cmd ] *** Node typ-esx05 is the master primary *** 12/16/12 02:22:15 [issue_cmd ] Node Type State 12/16/12 02:22:15 [issue_cmd ] ----------------------- ------------ -------------- 12/16/12 02:22:15 [issue_cmd ] typ-esx05 Primary Agent Running 12/16/12 02:22:15 [issue_cmd ] VMwarenodestatus=typ-esx05,Primary,AgentRunning 12/16/12 02:22:15 [issue_cmd ] CMD: /bin/ping -c 1 10.0.0.10 12/16/12 02:22:15 [issue_cmd
Dec 16 02:22:16 ] STATUS: 0 12/16/12 02:22:15 [issue_cmd ] RESULT: 12/16/12 02:22:15 [issue_cmd ] PING 10.0.0.10 (10.0.0.10): 56 data bytes 12/16/12 02:22:15 [issue_cmd ] 64 bytes from 10.0.0.10: icmp_seq=0 ttl=255 time=0.155 ms 12/16/12 02:22:15 [issue_cmd ] 12/16/12 02:22:15 [issue_cmd ] --- 10.0.0.10 ping statistics --- 12/16/12 02:22:15 [issue_cmd ] 1 packets transmitted, 1 packets received, 0% packet loss 12/16/12 02:22:15 [issue_cmd ] round-trip min/avg/max = 0.155/0.155/0.155 ms 12/16/12 02:22:15 [issue_cmd ] 12/16/12 02:22:15 [myexit ] VMwareresult=success 12/16/12 02:22:15 [elapsed_time ] Total time for script to complete: 0 minute(s) and 0 second(s)
Dec 16 02:22:17 :21:53 [print_args ] FT_PERSISTED_CONFIG_DIR=/etc/opt/vmware/aam 12/16/12 02:21:53 [print_args ] PWD=/var/log/vmware/vpx 12/16/12 02:21:53 [print_args ] PS_OPTIONS= 12/16/12 02:21:53 [print_args ] FT_NO_CONSOLE_TRACE=1 12/16/12 02:21:53 [print_args ] PATH=/sbin:/usr/sbin:/bin:/usr/bin:/opt/vmware/aam/bin:/bin 12/16/12 02:21:53 [print_args ] FT_LOG_DIR=/var/log/vmware/aam 12/16/12 02:21:53 [print_args ] FT_DOMAIN=vmware 12/16/12 02:21:53 [print_args ] Parsed arguments: 12/16/12 02:21:53 [print_args ] cmd=addnode 12/16/12 02:21:53 [print_args ] hostnet=10.0.0.215/255.255.0.0 12/16/12 02:21:53 [print_args ] uname=VMkernel 12/16/12 02:21:53 [print_args ] sensorpollingfreq=10 12/16/12 02:21:53 [print_args ] -z=1 12/16/12 02:21:53 [print_args ] iso=10.0.0.10 12/16/12 02:21:53 [print_args ] shortname
Dec 16 02:22:17 =typ-esx05 12/16/12 02:21:53 [print_args ] domain=vmware 12/16/12 02:21:53 [active_primary_ftcli] active primary is '' 12/16/12 02:21:53 [active_primary_ftcli] command is 'listnodes' 12/16/12 02:21:53 [find_active_primary ] attempting to find an active primary. 12/16/12 02:21:53 [issue_cli_cmd ] command is '/opt/vmware/aam/bin/ftcli -domain vmware -cmd "la -l"' 12/16/12 02:21:53 [issue_cli_cmd ] Trying to talk to self ('typ-esx05'), yet there is no sites file. 12/16/12 02:21:53 [find_active_primary ] Could not connect to any node to find an active primary . 12/16/12 02:21:53 [active_primary_ftcli] could not find an active primary on which to execute the command. 12/16/12 02:21:53 [add_aam_node ] 12/16/12 02:21:53 [issue_cmd ] CMD: cp /opt/vmware/aam/bin/generateConfigBackup.pl /var/log/vmware/aam/zgenerateConfigBackup.pl 12/16/12 02:21:53 [issue_cmd ] STATUS: 0 12/16/12 02:21:53 [issue_cmd ] RESULT: 12/16/12 02:21:53 [issue_cmd
Dec 16 02:22:17 ] 12/16/12 02:21:53 [issue_cmd ] CMD: cp /var/log/vmware/aam/zgenerateConfigBackup.pl /var/lib/vmware/aam/runOnce 12/16/12 02:21:53 [issue_cmd ] STATUS: 0 12/16/12 02:21:53 [issue_cmd ] RESULT: 12/16/12 02:21:53 [issue_cmd ] 12/16/12 02:21:53 [add_aam_node ] this is the primary agent -- 1st node in cluster. 12/16/12 02:21:53 [add_aam_node ] primary agent: typ-esx05 12/16/12 02:21:53 [setup_ft_hosts ] setup_ft_hosts adding host typ-esx05 IP info: 10.0.0.215/255.255.0.0 12/16/12 02:21:53 [setup_ft_hosts ] Added shortname 'typ-esx05' to /etc/hosts 12/16/12 02:21:53 [setup_ft_hosts ] Correct 'order' line found in /etc/host.conf 12/16/12 02:21:53 [setup_ft_hosts ] Correct 'hosts' line found in /etc/nsswitch.conf 12/16/12 02:21:53 [issue_cmd ] CMD: /opt/vmware/aam/bin/ft_gethostbyname typ-esx05 |grep FAILED 12/16/12 02:21:53 [issue_cmd ] STATUS: 1 12/16/12 02:21:53 [issue_cmd ] RESU
Dec 16 02:22:17 LT: 12/16/12 02:21:53 [issue_cmd ] 12/16/12 02:21:53 [add_aam_node ] ports 8042-8045 are free for use. 12/16/12 02:21:55 [issue_cmd ] CMD: /opt/vmware/aam/bin/ft_setup -domain=vmware -upgrade=n -noprompt=y -hostname=typ-esx05 -port1=8042 -licensekey=AMCFNEET-4YRDDN53CTHMBDSJ -mailserver=none -primaryagent=typ-esx05 -isos=10.0.0.10 -nics=10.0.0.215/255.255.0.0 12/16/12 02:21:55 [issue_cmd ] STATUS: 0 12/16/12 02:21:55 [issue_cmd ] RESULT: 12/16/12 02:21:55 [issue_cmd ] Setting up the AAM agent for domain vmware 12/16/12 02:21:55 [issue_cmd ] 12/16/12 02:21:55 [issue_cmd ] Welcome to VMware HA Agent. (Release 5.1 ) 12/16/12 02:21:55 [issue_cmd ] 12/16/12 02:21:55 [issue_cmd ] Configuring Agent for current node: typ-esx05 12/16/12 02:21:55 [issue_cmd ] 12/16/12 02:21:55 [issue_cmd ] Configuration requires the node name of a primary agent. If you 12/16/12 02:21:55
Dec 16 02:22:17 [issue_cmd ] are configuring the first node in the domain, enter the name 12/16/12 02:21:55 [issue_cmd ] of this node. (i.e. typ-esx05) If this is a subsequent installation 12/16/12 02:21:55 [issue_cmd ] enter the name of an existing primary agent node. 12/16/12 02:21:55 [issue_cmd ] 12/16/12 02:21:55 [issue_cmd ] Enter the name of a Primary Agent Node [typ-esx05] : 12/16/12 02:21:55 [issue_cmd ] Using input argument of typ-esx05 for Primary Agent 12/16/12 02:21:55 [issue_cmd ] Performing a primary node configuration. 12/16/12 02:21:55 [issue_cmd ] 12/16/12 02:21:55 [issue_cmd ] Agents require the use of 4 network ports through which to 12/16/12 02:21:55 [issue_cmd ] communicate. These port numbers must be available and consistent 12/16/12 02:21:55 [issue_cmd ] across each of the nodes in the domain. If you are unsure about 12/16/12 02:21:55 [issue_cmd ] specifying port numb
Dec 16 02:22:17 ers or defining primary nodes please read the 12/16/12 02:21:55 [issue_cmd ] appropriate sections of the user documentation provided with this 12/16/12 02:21:55 [issue_cmd ] product. 12/16/12 02:21:55 [issue_cmd ] 12/16/12 02:21:55 [issue_cmd ] Specify the first of the 4 port numbers: [8042] 12/16/12 02:21:55 [issue_cmd ] Using argument for port1: 8042 12/16/12 02:21:55 [issue_cmd ] 12/16/12 02:21:55 [issue_cmd ] Ports 8042, 8043, 8044 and 8045 will be used. 12/16/12 02:21:55 [issue_cmd ] 12/16/12 02:21:55 [issue_cmd ] Installation for this node is complete. 12/16/12 02:21:55 [issue_cmd ] 12/16/12 02:21:55 [issue_cmd ] To start the Agent run the "ft_startup" command. 12/16/12 02:21:55 [issue_cmd ] 12/16/12 02:21:55 [issue_cmd ] CMD: /bin/ping -c 1 10.0.0.10 12/16/12 02:21:55 [issue_cmd ] STATUS: 0 12/16/12 02:21:55 [issue_cmd ] RESULT:
Dec 16 02:22:17 12/16/12 02:21:55 [issue_cmd ] PING 10.0.0.10 (10.0.0.10): 56 data bytes 12/16/12 02:21:55 [issue_cmd ] 64 bytes from 10.0.0.10: icmp_seq=0 ttl=255 time=0.177 ms 12/16/12 02:21:55 [issue_cmd ] 12/16/12 02:21:55 [issue_cmd ] --- 10.0.0.10 ping statistics --- 12/16/12 02:21:55 [issue_cmd ] 1 packets transmitted, 1 packets received, 0% packet loss 12/16/12 02:21:55 [issue_cmd ] round-trip min/avg/max = 0.177/0.177/0.177 ms 12/16/12 02:21:55 [issue_cmd ] VMwareprogress=0 VMwareprogress=2 VMwareprogress=2 VMwareprogress=5 12/16/12 02:21:55 [issue_cmd ] CMD: cp -f /var/lib/vmware/aam/ftbb.prm /var/lib/vmware/aam/ftbb.prm.bck 12/16/12 02:21:55 [issue_cmd ] STATUS: 0 12/16/12 02:21:55 [issue_cmd ] RESULT: 12/16/12 02:21:55 [issue_cmd ] 12/16/12 02:21:55 [ft_startup_monitor ] Waiting for /opt/vmware/aam/bin/ft_startup to complete VMwareprogress=5 VMwareprogress=5 12/16/12 02:22:16 [ft
Dec 16 02:22:17 _startup_monitor ] /var/log/vmware/aam/startAamExitCode.txt contained 0 12/16/12 02:22:16 [ft_startup_monitor ] 12/16/12 02:22:16 [ft_startup_monitor ] /var/log/vmware/aam/startAamOutput.txt contained Starting agent for domain vmware 12/16/12 02:22:16 [ft_startup_monitor ] 12/16/12 02:22:16 [ft_startup_monitor ] 12/16/12 02:22:16 [ft_startup_monitor ] Sun Dec 16 02:21:55 2012: Starting Backbone... 12/16/12 02:22:16 [ft_startup_monitor ] ... 12/16/12 02:22:16 [ft_startup_monitor ] Sun Dec 16 02:22:10 2012: Backbone started successfully. 12/16/12 02:22:16 [ft_startup_monitor ] 12/16/12 02:22:16 [ft_startup_monitor ] Sun Dec 16 02:22:10 2012: Starting Agent... 12/16/12 02:22:16 [ft_startup_monitor ] 12/16/12 02:22:16 [ft_startup_monitor ] Sun Dec 16 02:22:13 2012: Agent started successfully. 12/16/12 02:22:16 [ft_startup_monitor ] 12/16/12 02:22:16 [ft_startup_monitor ] ft_startup_ret has evaluated to 2 VMwareprogress=19 12/16/12 02:22:16 [elapsed_time ] ft_sta
Dec 16 02:22:17 rtup_monitor: elapsed time 0 minute(s) and 18 second(s) 12/16/12 02:22:16 [remove_from_dead_hos] hosts without running agents: VMwareprogress=19 12/16/12 02:22:16 [active_primary_ftcli] active primary is 'typ-esx05' 12/16/12 02:22:16 [active_primary_ftcli] command is 'listnodes' 12/16/12 02:22:16 [issue_cli_cmd ] command is '/opt/vmware/aam/bin/ftcli -domain vmware -connect typ-esx05 -port 8042 -timeout 15 -cmd "listnodes"' 12/16/12 02:22:16 [issue_cmd ] CMD: /opt/vmware/aam/bin/ftcli -domain vmware -connect typ-esx05 -port 8042 -timeout 15 -cmd "listnodes" 12/16/12 02:22:16 [issue_cmd ] STATUS: 0 12/16/12 02:22:16 [issue_cmd ] RESULT: 12/16/12 02:22:16 [issue_cmd ] *** Node typ-esx05 is the master primary *** 12/16/12 02:22:16 [issue_cmd ] Node Type State 12/16/12 02:22:16 [issue_cmd ] ----------------------- ------------ -------------- 12/16/12 02:22:16 [issue_cmd ] typ-esx05
Dec 16 02:22:17 Primary Agent Running 12/16/12 02:22:16 [issue_cmd ] 12/16/12 02:22:16 [active_primary_ftcli] command ran successfully on 'typ-esx05'. 12/16/12 02:22:16 [wait_agent_startup ] waiting for agent 'typ-esx05' to come alive, status is : 'running' 12/16/12 02:22:16 [issue_cmd ] CMD: /opt/vmware/aam/bin/Cli -cmd "getnode typ-esx05" 12/16/12 02:22:16 [issue_cmd ] STATUS: 0 12/16/12 02:22:16 [issue_cmd ] RESULT: 12/16/12 02:22:16 [issue_cmd ] 12/16/12 02:22:16 [issue_cmd ] 12/16/12 02:22:16 [issue_cmd ] Description : Entry for node typ-esx05 12/16/12 02:22:16 [issue_cmd ] System Name : typ-esx05 12/16/12 02:22:16 [issue_cmd ] Operating System : VMkernel 4.1 12/16/12 02:22:16 [issue_cmd ] Kernel Arch : x86_64 12/16/12 02:22:16 [issue_cmd ] Main Memory (MB) : 0 12/16/12 02:22:16 [issue_cmd ] Swap space (MB) : 0 12/16/12 02:22:1
Dec 16 02:22:17 6 [issue_cmd ] Supported DS : 12/16/12 02:22:16 [issue_cmd ] IP Address(es) : 10.0.0.215 12/16/12 02:22:16 [issue_cmd ] Node Attributes : name=ConfigurationStatus value=complete 12/16/12 02:22:16 [issue_cmd ] name=heartbeat_config value=heartbeat_primary 12/16/12 02:22:16 [issue_cmd ] LAAM Version : 5.1.2 12/16/12 02:22:16 [issue_cmd ] Installed Patches : 0 12/16/12 02:22:16 [issue_cmd ] LAAM Version Info : Version 5.1.2 12/16/12 02:22:16 [issue_cmd ] Build Date : 12/16/12 02:22:16 [issue_cmd ] State : Agent Running 12/16/12 02:22:16 [issue_cmd ] 12/16/12 02:22:16 [wait_agent_startup ] ConfigurationStatus=complete, heartbeat_config complete 12/16/12 02:22:16 [elapsed_time ] elapsed time 0 minute(s) and 0 second(s) VMwareprogress=29 VMwareprogress=59 12/16/12 02:22:16 [get_rule_var ] 12/16/12 02:22:16
Dec 16 02:22:17 [issue_cli_cmd ] command is '/opt/vmware/aam/bin/ftcli -cmd "getRuleVars VMWareClusterManager"' 12/16/12 02:22:17 [issue_cmd ] CMD: /opt/vmware/aam/bin/ftcli -cmd "getRuleVars VMWareClusterManager" 12/16/12 02:22:17 [issue_cmd ] STATUS: 0 12/16/12 02:22:17 [issue_cmd ] RESULT: 12/16/12 02:22:17 [issue_cmd ] CHK_VMSTATE_AFTER_RESTART_DELAY = 120 12/16/12 02:22:17 [issue_cmd ] ft_Trace = ON 12/16/12 02:22:17 [issue_cmd ] ft_TraceLevel = 3 12/16/12 02:22:17 [issue_cmd ] ft_TraceMbs = 2 12/16/12 02:22:17 [issue_cmd ] ft_TraceOutput = File 12/16/12 02:22:17 [issue_cmd ] HOST_MONITORING = enabled 12/16/12 02:22:17 [issue_cmd ] MAX_FT_VM_RESTART_COUNT = 5 12/16/12 02:22:17 [issue_cmd ] MAX_PRIMARY_COUNT = 5 12/16/12 02:22:17 [issue_cmd ] MAX_VM_RESTART_COUNT = 5 12/16/12 02:22:17 [issue_cmd ] SIGNAL_LIVENESS_INTERVAL = 60 12/16/12 02:22:17 [issue_cmd ]
Dec 16 02:22:17 CHK_VMSTATE_AFTER_RESTART_DELAY = 120 ft_Trace = ON ft_TraceLevel = 3 ft_TraceMbs = 2 ft_TraceOutput = File HOST_MONITORING = enabled MAX_FT_VM_RESTART_COUNT = 5 MAX_PRIMARY_COUNT = 5 MAX_VM_RESTART_COUNT = 5 SIGNAL_LIVENESS_INTERVAL = 60 12/16/12 02:22:17 [manage_primaries ] manage_primaries CHK_VMSTATE_AFTER_RESTART_DELAY = 120 12/16/12 02:22:17 [manage_primaries ] ft_Trace = ON 12/16/12 02:22:17 [manage_primaries ] ft_TraceLevel = 3 12/16/12 02:22:17 [manage_primaries ] ft_TraceMbs = 2 12/16/12 02:22:17 [manage_primaries ] ft_TraceOutput = File 12/16/12 02:22:17 [manage_primaries ] HOST_MONITORING = enabled 12/16/12 02:22:17 [manage_primaries ] MAX_FT_VM_RESTART_COUNT = 5 12/16/12 02:22:17 [manage_primaries ] MAX_PRIMARY_COUNT = 5 12/16/12 02:22:17 [manage_primaries ] MAX_VM_RESTART_COUNT = 5 12/16/12 02:22:17 [manage_primaries ] SIGNAL_LIVENESS_INTERVAL = 6 manage_primaries: No preferredPrimaries specified VMwareprogress=59 VMwareprogress=59 12/16/12
Dec 16 02:22:17 02:22:17 [myexit ] VMwareresult=success 12/16/12 02:22:17 [elapsed_time ] Total time for script to complete: 0 minute(s) and 24 second(s)
Dec 16 02:22:23 CONFIG_DIR=/etc/opt/vmware/aam 12/16/12 02:22:22 [print_args ] PWD=/var/log/vmware/vpx 12/16/12 02:22:22 [print_args ] PS_OPTIONS= 12/16/12 02:22:22 [print_args ] FT_NO_CONSOLE_TRACE=1 12/16/12 02:22:22 [print_args ] PATH=/sbin:/usr/sbin:/bin:/usr/bin:/opt/vmware/aam/bin:/bin 12/16/12 02:22:22 [print_args ] FT_LOG_DIR=/var/log/vmware/aam 12/16/12 02:22:22 [print_args ] FT_DOMAIN=vmware 12/16/12 02:22:22 [print_args ] Parsed arguments: 12/16/12 02:22:22 [print_args ] cmd=listprimaries 12/16/12 02:22:22 [print_args ] -z=1 12/16/12 02:22:22 [print_args ] uname=VMkernel 12/16/12 02:22:22 [print_args ] shortname=typ-esx05 12/16/12 02:22:22 [print_args ] domain=vmware 12/16/12 02:22:22 [issue_cmd ] CMD: /opt/vmware/aam/bin/ft_gethostbyname typ-esx05 |grep FAILED 12/16/12 02:22:22 [issue_cmd ] STATUS: 1 12/16/12 02:22
Dec 16 02:22:23 :22 [issue_cmd ] RESULT: 12/16/12 02:22:22 [issue_cmd ] 12/16/12 02:22:22 [list_primary_nodes ] 12/16/12 02:22:22 [issue_cli_cmd ] command is '/opt/vmware/aam/bin/ftcli -domain vmware -timeout 15 -cmd listnodes' 12/16/12 02:22:22 [issue_cmd ] CMD: /opt/vmware/aam/bin/ftcli -domain vmware -timeout 15 -cmd listnodes 12/16/12 02:22:22 [issue_cmd ] STATUS: 0 12/16/12 02:22:22 [issue_cmd ] RESULT: 12/16/12 02:22:22 [issue_cmd ] *** Node typ-esx05 is the master primary *** 12/16/12 02:22:22 [issue_cmd ] Node Type State 12/16/12 02:22:22 [issue_cmd ] ----------------------- ------------ -------------- 12/16/12 02:22:22 [issue_cmd ] typ-esx05 Primary Agent Running 12/16/12 02:22:22 [issue_cmd ] VMwarelistprimaries=typ-esx05 12/16/12 02:22:22 [myexit ] VMwareresult=success 12/16/12 02:22:22 [elapsed_time ] Total tim
Dec 16 02:22:23 e for script to complete: 0 minute(s) and 0 second(s)
Dec 16 02:22:47 m 12/16/12 02:22:46 [print_args ] PWD=/var/log/vmware/vpx 12/16/12 02:22:46 [print_args ] PS_OPTIONS= 12/16/12 02:22:46 [print_args ] FT_NO_CONSOLE_TRACE=1 12/16/12 02:22:46 [print_args ] PATH=/sbin:/usr/sbin:/bin:/usr/bin:/opt/vmware/aam/bin:/bin 12/16/12 02:22:46 [print_args ] FT_LOG_DIR=/var/log/vmware/aam 12/16/12 02:22:46 [print_args ] FT_DOMAIN=vmware 12/16/12 02:22:46 [print_args ] Parsed arguments: 12/16/12 02:22:46 [print_args ] cmd=monitornodes 12/16/12 02:22:46 [print_args ] -z=1 12/16/12 02:22:46 [print_args ] uname=VMkernel 12/16/12 02:22:46 [print_args ] shortname=typ-esx05 12/16/12 02:22:46 [print_args ] domain=vmware 12/16/12 02:22:46 [issue_cmd ] CMD: /opt/vmware/aam/bin/ft_gethostbyname typ-esx05 |grep FAILED 12/16/12 02:22:46 [issue_cmd ] STATUS: 1 12/16/12 02:22:46 [issue_cmd ] RES
Dec 16 02:22:47 ULT: 12/16/12 02:22:46 [issue_cmd ] 12/16/12 02:22:46 [list_nodes ] 12/16/12 02:22:46 [issue_cli_cmd ] command is '/opt/vmware/aam/bin/ftcli -domain vmware -port 8042 -timeout 5 -cmd listnodes' 12/16/12 02:22:46 [issue_cmd ] CMD: /opt/vmware/aam/bin/ftcli -domain vmware -port 8042 -timeout 5 -cmd listnodes 12/16/12 02:22:46 [issue_cmd ] STATUS: 0 12/16/12 02:22:46 [issue_cmd ] RESULT: 12/16/12 02:22:46 [issue_cmd ] *** Node typ-esx05 is the master primary *** 12/16/12 02:22:46 [issue_cmd ] Node Type State 12/16/12 02:22:46 [issue_cmd ] ----------------------- ------------ -------------- 12/16/12 02:22:46 [issue_cmd ] typ-esx05 Primary Agent Running 12/16/12 02:22:46 [issue_cmd ] VMwarenodestatus=typ-esx05,Primary,AgentRunning 12/16/12 02:22:46 [issue_cmd ] CMD: /bin/ping -c 1 10.0.0.10 12/16/12 02:22:46 [issue_cmd
Dec 16 02:22:47 ] STATUS: 0 12/16/12 02:22:46 [issue_cmd ] RESULT: 12/16/12 02:22:46 [issue_cmd ] PING 10.0.0.10 (10.0.0.10): 56 data bytes 12/16/12 02:22:46 [issue_cmd ] 64 bytes from 10.0.0.10: icmp_seq=0 ttl=255 time=0.209 ms 12/16/12 02:22:46 [issue_cmd ] 12/16/12 02:22:46 [issue_cmd ] --- 10.0.0.10 ping statistics --- 12/16/12 02:22:46 [issue_cmd ] 1 packets transmitted, 1 packets received, 0% packet loss 12/16/12 02:22:46 [issue_cmd ] round-trip min/avg/max = 0.209/0.209/0.209 ms 12/16/12 02:22:46 [issue_cmd ] 12/16/12 02:22:46 [myexit ] VMwareresult=success 12/16/12 02:22:46 [elapsed_time ] Total time for script to complete: 0 minute(s) and 0 second(s)
Dec 16 02:23:09 authd[18064]: login from 10.0.0.126 as 5250d0bd-b1ed-94c3-4a32-dec4f11cbbb6
Dec 16 02:23:09 authd[18064]: Local connection for mks established.
Dec 16 02:23:22 authd[18417]: login from 10.0.0.126 as 5224eeb2-d51c-5fe8-1ace-12765e52f600
Dec 16 02:23:22 authd[18417]: Local connection for mks established.
Dec 16 02:23:22 m 12/16/12 02:23:17 [print_args ] PWD=/var/log/vmware/vpx 12/16/12 02:23:17 [print_args ] PS_OPTIONS= 12/16/12 02:23:17 [print_args ] FT_NO_CONSOLE_TRACE=1 12/16/12 02:23:17 [print_args ] PATH=/sbin:/usr/sbin:/bin:/usr/bin:/opt/vmware/aam/bin:/bin 12/16/12 02:23:17 [print_args ] FT_LOG_DIR=/var/log/vmware/aam 12/16/12 02:23:17 [print_args ] FT_DOMAIN=vmware 12/16/12 02:23:17 [print_args ] Parsed arguments: 12/16/12 02:23:17 [print_args ] cmd=monitornodes 12/16/12 02:23:17 [print_args ] -z=1 12/16/12 02:23:17 [print_args ] uname=VMkernel 12/16/12 02:23:17 [print_args ] shortname=typ-esx05 12/16/12 02:23:17 [print_args ] domain=vmware 12/16/12 02:23:17 [issue_cmd ] CMD: /opt/vmware/aam/bin/ft_gethostbyname typ-esx05 |grep FAILED 12/16/12 02:23:17 [issue_cmd ] STATUS: 1 12/16/12 02:23:17 [issue_cmd ] RES
Dec 16 02:23:22 ULT: 12/16/12 02:23:17 [issue_cmd ] 12/16/12 02:23:17 [list_nodes ] 12/16/12 02:23:17 [issue_cli_cmd ] command is '/opt/vmware/aam/bin/ftcli -domain vmware -port 8042 -timeout 5 -cmd listnodes' 12/16/12 02:23:22 [issue_cmd ] CMD: /opt/vmware/aam/bin/ftcli -domain vmware -port 8042 -timeout 5 -cmd listnodes 12/16/12 02:23:22 [issue_cmd ] STATUS: 0 12/16/12 02:23:22 [issue_cmd ] RESULT: 12/16/12 02:23:22 [issue_cmd ] *** Node typ-esx05 is the master primary *** 12/16/12 02:23:22 [issue_cmd ] Node Type State 12/16/12 02:23:22 [issue_cmd ] ----------------------- ------------ -------------- 12/16/12 02:23:22 [issue_cmd ] typ-esx05 Primary Agent Running 12/16/12 02:23:22 [issue_cmd ] typ-esx06 Secondary No Agent Run 12/16/12 02:23:22 [issue_cmd ] VMwarenodestatus=typ-esx05,Primary,AgentRunning,typ-esx06,Sec
Dec 16 02:23:22 ondary,NoAgentRun 12/16/12 02:23:22 [issue_cmd ] CMD: /bin/ping -c 1 10.0.0.10 12/16/12 02:23:22 [issue_cmd ] STATUS: 0 12/16/12 02:23:22 [issue_cmd ] RESULT: 12/16/12 02:23:22 [issue_cmd ] PING 10.0.0.10 (10.0.0.10): 56 data bytes 12/16/12 02:23:22 [issue_cmd ] 64 bytes from 10.0.0.10: icmp_seq=0 ttl=255 time=0.235 ms 12/16/12 02:23:22 [issue_cmd ] 12/16/12 02:23:22 [issue_cmd ] --- 10.0.0.10 ping statistics --- 12/16/12 02:23:22 [issue_cmd ] 1 packets transmitted, 1 packets received, 0% packet loss 12/16/12 02:23:22 [issue_cmd ] round-trip min/avg/max = 0.235/0.235/0.235 ms 12/16/12 02:23:22 [issue_cmd ] 12/16/12 02:23:22 [myexit ] VMwareresult=success 12/16/12 02:23:22 [elapsed_time ] Total time for script to complete: 0 minute(s) and 5 second(s)
Dec 16 02:23:22 ED_CONFIG_DIR=/etc/opt/vmware/aam 12/16/12 02:23:10 [print_args ] PWD=/var/log/vmware/vpx 12/16/12 02:23:10 [print_args ] PS_OPTIONS= 12/16/12 02:23:10 [print_args ] FT_NO_CONSOLE_TRACE=1 12/16/12 02:23:10 [print_args ] PATH=/sbin:/usr/sbin:/bin:/usr/bin:/opt/vmware/aam/bin:/bin 12/16/12 02:23:10 [print_args ] FT_LOG_DIR=/var/log/vmware/aam 12/16/12 02:23:10 [print_args ] FT_DOMAIN=vmware 12/16/12 02:23:10 [print_args ] Parsed arguments: 12/16/12 02:23:10 [print_args ] cmd=admitnode 12/16/12 02:23:10 [print_args ] u=root 12/16/12 02:23:10 [print_args ] -z=1 12/16/12 02:23:10 [print_args ] n=typ-esx06 12/16/12 02:23:10 [print_args ] uname=VMkernel 12/16/12 02:23:10 [print_args ] shortname=typ-esx05 12/16/12 02:23:10 [print_args ] domain=vmware 12/16/12 02:23:10 [issue_cmd ] CMD: /opt/vmware/aam
Dec 16 02:23:22 /bin/ft_gethostbyname typ-esx05 |grep FAILED 12/16/12 02:23:10 [issue_cmd ] STATUS: 1 12/16/12 02:23:10 [issue_cmd ] RESULT: 12/16/12 02:23:10 [issue_cmd ] 12/16/12 02:23:10 [issue_cli_cmd ] command is '/opt/vmware/aam/bin/ftcli -cmd "createUser root typ-esx06 Administrator PERM_ALL"' 12/16/12 02:23:10 [issue_cmd ] CMD: /opt/vmware/aam/bin/ftcli -cmd "createUser root typ-esx06 Administrator PERM_ALL" 12/16/12 02:23:10 [issue_cmd ] STATUS: 0 12/16/12 02:23:10 [issue_cmd ] RESULT: 12/16/12 02:23:10 [issue_cmd ] OK 12/16/12 02:23:10 [issue_cmd ] 12/16/12 02:23:10 [issue_cli_cmd ] command is '/opt/vmware/aam/bin/ftcli -domain vmware -cmd "addnode typ-esx06"' 12/16/12 02:23:21 [issue_cmd ] CMD: /opt/vmware/aam/bin/ftcli -domain vmware -cmd "addnode typ-esx06" 12/16/12 02:23:21 [issue_cmd ] STATUS: 0 12/16/12 02:23:21 [issue_cmd ] RESULT: 12/16/12 02:23:21 [issue_cmd
Dec 16 02:23:22 ] OK 12/16/12 02:23:21 [issue_cmd ] 12/16/12 02:23:21 [myexit ] VMwareresult=success 12/16/12 02:23:21 [elapsed_time ] Total time for script to complete: 0 minute(s) and 11 second(s)
Dec 16 02:23:34 etc/opt/vmware/aam 12/16/12 02:23:33 [print_args ] PWD=/var/log/vmware/vpx 12/16/12 02:23:33 [print_args ] PS_OPTIONS= 12/16/12 02:23:33 [print_args ] FT_NO_CONSOLE_TRACE=1 12/16/12 02:23:33 [print_args ] PATH=/sbin:/usr/sbin:/bin:/usr/bin:/opt/vmware/aam/bin:/bin 12/16/12 02:23:33 [print_args ] FT_LOG_DIR=/var/log/vmware/aam 12/16/12 02:23:33 [print_args ] FT_DOMAIN=vmware 12/16/12 02:23:33 [print_args ] Parsed arguments: 12/16/12 02:23:33 [print_args ] cmd=listprimaries 12/16/12 02:23:33 [print_args ] -z=1 12/16/12 02:23:33 [print_args ] uname=VMkernel 12/16/12 02:23:33 [print_args ] shortname=typ-esx05 12/16/12 02:23:33 [print_args ] domain=vmware 12/16/12 02:23:33 [issue_cmd ] CMD: /opt/vmware/aam/bin/ft_gethostbyname typ-esx05 |grep FAILED 12/16/12 02:23:33 [issue_cmd ] STATUS: 1 12/16/12 02:23:33 [issue_c
Dec 16 02:23:34 md ] RESULT: 12/16/12 02:23:33 [issue_cmd ] 12/16/12 02:23:33 [list_primary_nodes ] 12/16/12 02:23:33 [issue_cli_cmd ] command is '/opt/vmware/aam/bin/ftcli -domain vmware -timeout 15 -cmd listnodes' 12/16/12 02:23:33 [issue_cmd ] CMD: /opt/vmware/aam/bin/ftcli -domain vmware -timeout 15 -cmd listnodes 12/16/12 02:23:33 [issue_cmd ] STATUS: 0 12/16/12 02:23:33 [issue_cmd ] RESULT: 12/16/12 02:23:33 [issue_cmd ] *** Node typ-esx05 is the master primary *** 12/16/12 02:23:33 [issue_cmd ] Node Type State 12/16/12 02:23:33 [issue_cmd ] ----------------------- ------------ -------------- 12/16/12 02:23:33 [issue_cmd ] typ-esx05 Primary Agent Running 12/16/12 02:23:33 [issue_cmd ] typ-esx06 Primary Agent Shutdown 12/16/12 02:23:33 [issue_cmd ] VMwarelistprimaries=typ-esx05,typ-esx06 12/16/12 02:23:33 [m
Dec 16 02:23:34 yexit ] VMwareresult=success 12/16/12 02:23:33 [elapsed_time ] Total time for script to complete: 0 minute(s) and 0 second(s)
Dec 16 02:23:53 dropbear[18844]: Child connection from 10.0.0.235:34285
Dec 16 02:23:53 m 12/16/12 02:23:52 [print_args ] PWD=/var/log/vmware/vpx 12/16/12 02:23:52 [print_args ] PS_OPTIONS= 12/16/12 02:23:52 [print_args ] FT_NO_CONSOLE_TRACE=1 12/16/12 02:23:52 [print_args ] PATH=/sbin:/usr/sbin:/bin:/usr/bin:/opt/vmware/aam/bin:/bin 12/16/12 02:23:52 [print_args ] FT_LOG_DIR=/var/log/vmware/aam 12/16/12 02:23:52 [print_args ] FT_DOMAIN=vmware 12/16/12 02:23:52 [print_args ] Parsed arguments: 12/16/12 02:23:52 [print_args ] cmd=monitornodes 12/16/12 02:23:52 [print_args ] -z=1 12/16/12 02:23:52 [print_args ] uname=VMkernel 12/16/12 02:23:52 [print_args ] shortname=typ-esx05 12/16/12 02:23:52 [print_args ] domain=vmware 12/16/12 02:23:52 [issue_cmd ] CMD: /opt/vmware/aam/bin/ft_gethostbyname typ-esx05 |grep FAILED 12/16/12 02:23:52 [issue_cmd ] STATUS: 1 12/16/12 02:23:52 [issue_cmd ] RES
Dec 16 02:23:53 ULT: 12/16/12 02:23:52 [issue_cmd ] 12/16/12 02:23:52 [list_nodes ] 12/16/12 02:23:52 [issue_cli_cmd ] command is '/opt/vmware/aam/bin/ftcli -domain vmware -port 8042 -timeout 5 -cmd listnodes' 12/16/12 02:23:52 [issue_cmd ] CMD: /opt/vmware/aam/bin/ftcli -domain vmware -port 8042 -timeout 5 -cmd listnodes 12/16/12 02:23:52 [issue_cmd ] STATUS: 0 12/16/12 02:23:52 [issue_cmd ] RESULT: 12/16/12 02:23:52 [issue_cmd ] *** Node typ-esx05 is the master primary *** 12/16/12 02:23:52 [issue_cmd ] Node Type State 12/16/12 02:23:52 [issue_cmd ] ----------------------- ------------ -------------- 12/16/12 02:23:52 [issue_cmd ] typ-esx05 Primary Agent Running 12/16/12 02:23:52 [issue_cmd ] typ-esx06 Primary Agent Running 12/16/12 02:23:52 [issue_cmd ] VMwarenodestatus=typ-esx05,Primary,AgentRunning,typ-esx06,Pr
Dec 16 02:23:53 imary,AgentRunning 12/16/12 02:23:52 [issue_cmd ] CMD: /bin/ping -c 1 10.0.0.10 12/16/12 02:23:52 [issue_cmd ] STATUS: 0 12/16/12 02:23:52 [issue_cmd ] RESULT: 12/16/12 02:23:52 [issue_cmd ] PING 10.0.0.10 (10.0.0.10): 56 data bytes 12/16/12 02:23:52 [issue_cmd ] 64 bytes from 10.0.0.10: icmp_seq=0 ttl=255 time=0.178 ms 12/16/12 02:23:52 [issue_cmd ] 12/16/12 02:23:52 [issue_cmd ] --- 10.0.0.10 ping statistics --- 12/16/12 02:23:52 [issue_cmd ] 1 packets transmitted, 1 packets received, 0% packet loss 12/16/12 02:23:52 [issue_cmd ] round-trip min/avg/max = 0.178/0.178/0.178 ms 12/16/12 02:23:52 [issue_cmd ] 12/16/12 02:23:52 [myexit ] VMwareresult=success 12/16/12 02:23:52 [elapsed_time ] Total time for script to complete: 0 minute(s) and 0 second(s)
Dec 16 02:23:58 dropbear[18844]: exit before auth: Exited normally
Dec 16 02:24:24 m 12/16/12 02:24:23 [print_args ] PWD=/var/log/vmware/vpx 12/16/12 02:24:23 [print_args ] PS_OPTIONS= 12/16/12 02:24:23 [print_args ] FT_NO_CONSOLE_TRACE=1 12/16/12 02:24:23 [print_args ] PATH=/sbin:/usr/sbin:/bin:/usr/bin:/opt/vmware/aam/bin:/bin 12/16/12 02:24:23 [print_args ] FT_LOG_DIR=/var/log/vmware/aam 12/16/12 02:24:23 [print_args ] FT_DOMAIN=vmware 12/16/12 02:24:23 [print_args ] Parsed arguments: 12/16/12 02:24:23 [print_args ] cmd=monitornodes 12/16/12 02:24:23 [print_args ] -z=1 12/16/12 02:24:23 [print_args ] uname=VMkernel 12/16/12 02:24:23 [print_args ] shortname=typ-esx05 12/16/12 02:24:23 [print_args ] domain=vmware 12/16/12 02:24:23 [issue_cmd ] CMD: /opt/vmware/aam/bin/ft_gethostbyname typ-esx05 |grep FAILED 12/16/12 02:24:23 [issue_cmd ] STATUS: 1 12/16/12 02:24:23 [issue_cmd ] RES
Dec 16 02:24:24 ULT: 12/16/12 02:24:23 [issue_cmd ] 12/16/12 02:24:23 [list_nodes ] 12/16/12 02:24:23 [issue_cli_cmd ] command is '/opt/vmware/aam/bin/ftcli -domain vmware -port 8042 -timeout 5 -cmd listnodes' 12/16/12 02:24:23 [issue_cmd ] CMD: /opt/vmware/aam/bin/ftcli -domain vmware -port 8042 -timeout 5 -cmd listnodes 12/16/12 02:24:23 [issue_cmd ] STATUS: 0 12/16/12 02:24:23 [issue_cmd ] RESULT: 12/16/12 02:24:23 [issue_cmd ] *** Node typ-esx05 is the master primary *** 12/16/12 02:24:23 [issue_cmd ] Node Type State 12/16/12 02:24:23 [issue_cmd ] ----------------------- ------------ -------------- 12/16/12 02:24:23 [issue_cmd ] typ-esx05 Primary Agent Running 12/16/12 02:24:23 [issue_cmd ] typ-esx06 Primary Agent Running 12/16/12 02:24:23 [issue_cmd ] VMwarenodestatus=typ-esx05,Primary,AgentRunning,typ-esx06,Pr
Dec 16 02:24:24 imary,AgentRunning 12/16/12 02:24:23 [issue_cmd ] CMD: /bin/ping -c 1 10.0.0.10 12/16/12 02:24:23 [issue_cmd ] STATUS: 0 12/16/12 02:24:23 [issue_cmd ] RESULT: 12/16/12 02:24:23 [issue_cmd ] PING 10.0.0.10 (10.0.0.10): 56 data bytes 12/16/12 02:24:23 [issue_cmd ] 64 bytes from 10.0.0.10: icmp_seq=0 ttl=255 time=0.182 ms 12/16/12 02:24:23 [issue_cmd ] 12/16/12 02:24:23 [issue_cmd ] --- 10.0.0.10 ping statistics --- 12/16/12 02:24:23 [issue_cmd ] 1 packets transmitted, 1 packets received, 0% packet loss 12/16/12 02:24:23 [issue_cmd ] round-trip min/avg/max = 0.182/0.182/0.182 ms 12/16/12 02:24:23 [issue_cmd ] 12/16/12 02:24:23 [myexit ] VMwareresult=success 12/16/12 02:24:23 [elapsed_time ] Total time for script to complete: 0 minute(s) and 0 second(s)
Dec 16 02:25:00 D_CONFIG_DIR=/etc/opt/vmware/aam 12/16/12 02:24:48 [print_args ] PWD=/var/log/vmware/vpx 12/16/12 02:24:48 [print_args ] PS_OPTIONS= 12/16/12 02:24:48 [print_args ] FT_NO_CONSOLE_TRACE=1 12/16/12 02:24:48 [print_args ] PATH=/sbin:/usr/sbin:/bin:/usr/bin:/opt/vmware/aam/bin:/bin 12/16/12 02:24:48 [print_args ] FT_LOG_DIR=/var/log/vmware/aam 12/16/12 02:24:48 [print_args ] FT_DOMAIN=vmware 12/16/12 02:24:48 [print_args ] Parsed arguments: 12/16/12 02:24:48 [print_args ] cmd=admitnode 12/16/12 02:24:48 [print_args ] u=root 12/16/12 02:24:48 [print_args ] -z=1 12/16/12 02:24:48 [print_args ] n=ty-esx01 12/16/12 02:24:48 [print_args ] uname=VMkernel 12/16/12 02:24:48 [print_args ] shortname=typ-esx05 12/16/12 02:24:48 [print_args ] domain=vmware 12/16/12 02:24:48 [issue_cmd ] CMD: /opt/vmware/aam/b
Dec 16 02:25:00 in/ft_gethostbyname typ-esx05 |grep FAILED 12/16/12 02:24:48 [issue_cmd ] STATUS: 1 12/16/12 02:24:48 [issue_cmd ] RESULT: 12/16/12 02:24:48 [issue_cmd ] 12/16/12 02:24:48 [issue_cli_cmd ] command is '/opt/vmware/aam/bin/ftcli -cmd "createUser root ty-esx01 Administrator PERM_ALL"' 12/16/12 02:24:49 [issue_cmd ] CMD: /opt/vmware/aam/bin/ftcli -cmd "createUser root ty-esx01 Administrator PERM_ALL" 12/16/12 02:24:49 [issue_cmd ] STATUS: 0 12/16/12 02:24:49 [issue_cmd ] RESULT: 12/16/12 02:24:49 [issue_cmd ] OK 12/16/12 02:24:49 [issue_cmd ] 12/16/12 02:24:49 [issue_cli_cmd ] command is '/opt/vmware/aam/bin/ftcli -domain vmware -cmd "addnode ty-esx01"' 12/16/12 02:24:59 [issue_cmd ] CMD: /opt/vmware/aam/bin/ftcli -domain vmware -cmd "addnode ty-esx01" 12/16/12 02:24:59 [issue_cmd ] STATUS: 0 12/16/12 02:24:59 [issue_cmd ] RESULT: 12/16/12 02:24:59 [issue_cmd ]
Dec 16 02:25:00 OK 12/16/12 02:24:59 [issue_cmd ] 12/16/12 02:24:59 [myexit ] VMwareresult=success 12/16/12 02:24:59 [elapsed_time ] Total time for script to complete: 0 minute(s) and 11 second(s)
Dec 16 02:25:10 m 12/16/12 02:24:54 [print_args ] PWD=/var/log/vmware/vpx 12/16/12 02:24:54 [print_args ] PS_OPTIONS= 12/16/12 02:24:54 [print_args ] FT_NO_CONSOLE_TRACE=1 12/16/12 02:24:54 [print_args ] PATH=/sbin:/usr/sbin:/bin:/usr/bin:/opt/vmware/aam/bin:/bin 12/16/12 02:24:54 [print_args ] FT_LOG_DIR=/var/log/vmware/aam 12/16/12 02:24:54 [print_args ] FT_DOMAIN=vmware 12/16/12 02:24:54 [print_args ] Parsed arguments: 12/16/12 02:24:54 [print_args ] cmd=monitornodes 12/16/12 02:24:54 [print_args ] -z=1 12/16/12 02:24:54 [print_args ] uname=VMkernel 12/16/12 02:24:54 [print_args ] shortname=typ-esx05 12/16/12 02:24:54 [print_args ] domain=vmware 12/16/12 02:24:54 [issue_cmd ] CMD: /opt/vmware/aam/bin/ft_gethostbyname typ-esx05 |grep FAILED 12/16/12 02:24:54 [issue_cmd ] STATUS: 1 12/16/12 02:24:54 [issue_cmd ] RES
Dec 16 02:25:10 ULT: 12/16/12 02:24:54 [issue_cmd ] 12/16/12 02:24:54 [list_nodes ] 12/16/12 02:24:54 [issue_cli_cmd ] command is '/opt/vmware/aam/bin/ftcli -domain vmware -port 8042 -timeout 5 -cmd listnodes' 12/16/12 02:25:10 [issue_cmd ] CMD: /opt/vmware/aam/bin/ftcli -domain vmware -port 8042 -timeout 5 -cmd listnodes 12/16/12 02:25:10 [issue_cmd ] STATUS: 0 12/16/12 02:25:10 [issue_cmd ] RESULT: 12/16/12 02:25:10 [issue_cmd ] Connection with AAM Agent typ-esx05 has been lost. 12/16/12 02:25:10 [issue_cmd ] Reconnecting to AAM Agent... 12/16/12 02:25:10 [issue_cmd ] Connected. 12/16/12 02:25:10 [issue_cmd ] *** Node typ-esx05 is the master primary *** 12/16/12 02:25:10 [issue_cmd ] Node Type State 12/16/12 02:25:10 [issue_cmd ] ----------------------- ------------ -------------- 12/16/12 02:25:10 [issue_cmd ] ty-esx01 Secondar
Dec 16 02:25:10 y No Agent Run 12/16/12 02:25:10 [issue_cmd ] typ-esx05 Primary Agent Running 12/16/12 02:25:10 [issue_cmd ] typ-esx06 Primary Agent Running 12/16/12 02:25:10 [issue_cmd ] VMwarenodestatus=ty-esx01,Secondary,NoAgentRun,typ-esx05,Primary,AgentRunning,typ-esx06,Primary,AgentRunning 12/16/12 02:25:10 [issue_cmd ] CMD: /bin/ping -c 1 10.0.0.10 12/16/12 02:25:10 [issue_cmd ] STATUS: 0 12/16/12 02:25:10 [issue_cmd ] RESULT: 12/16/12 02:25:10 [issue_cmd ] PING 10.0.0.10 (10.0.0.10): 56 data bytes 12/16/12 02:25:10 [issue_cmd ] 64 bytes from 10.0.0.10: icmp_seq=0 ttl=255 time=0.193 ms 12/16/12 02:25:10 [issue_cmd ] 12/16/12 02:25:10 [issue_cmd ] --- 10.0.0.10 ping statistics --- 12/16/12 02:25:10 [issue_cmd ] 1 packets transmitted, 1 packets received, 0% packet loss 12/16/12 02:25:10 [issue_cmd ] round-trip min/avg/max = 0.193/0.193/0.193
Dec 16 02:25:10 ms 12/16/12 02:25:10 [issue_cmd ] 12/16/12 02:25:10 [myexit ] VMwareresult=success 12/16/12 02:25:10 [elapsed_time ] Total time for script to complete: 0 minute(s) and 16 second(s)
Dec 16 02:25:41 m 12/16/12 02:25:40 [print_args ] PWD=/var/log/vmware/vpx 12/16/12 02:25:40 [print_args ] PS_OPTIONS= 12/16/12 02:25:40 [print_args ] FT_NO_CONSOLE_TRACE=1 12/16/12 02:25:40 [print_args ] PATH=/sbin:/usr/sbin:/bin:/usr/bin:/opt/vmware/aam/bin:/bin 12/16/12 02:25:40 [print_args ] FT_LOG_DIR=/var/log/vmware/aam 12/16/12 02:25:40 [print_args ] FT_DOMAIN=vmware 12/16/12 02:25:40 [print_args ] Parsed arguments: 12/16/12 02:25:40 [print_args ] cmd=monitornodes 12/16/12 02:25:40 [print_args ] -z=1 12/16/12 02:25:40 [print_args ] uname=VMkernel 12/16/12 02:25:40 [print_args ] shortname=typ-esx05 12/16/12 02:25:40 [print_args ] domain=vmware 12/16/12 02:25:40 [issue_cmd ] CMD: /opt/vmware/aam/bin/ft_gethostbyname typ-esx05 |grep FAILED 12/16/12 02:25:40 [issue_cmd ] STATUS: 1 12/16/12 02:25:40 [issue_cmd ] RES
Dec 16 02:25:41 ULT: 12/16/12 02:25:40 [issue_cmd ] 12/16/12 02:25:40 [list_nodes ] 12/16/12 02:25:40 [issue_cli_cmd ] command is '/opt/vmware/aam/bin/ftcli -domain vmware -port 8042 -timeout 5 -cmd listnodes' 12/16/12 02:25:40 [issue_cmd ] CMD: /opt/vmware/aam/bin/ftcli -domain vmware -port 8042 -timeout 5 -cmd listnodes 12/16/12 02:25:40 [issue_cmd ] STATUS: 0 12/16/12 02:25:40 [issue_cmd ] RESULT: 12/16/12 02:25:40 [issue_cmd ] *** Node typ-esx05 is the master primary *** 12/16/12 02:25:40 [issue_cmd ] Node Type State 12/16/12 02:25:40 [issue_cmd ] ----------------------- ------------ -------------- 12/16/12 02:25:40 [issue_cmd ] ty-esx01 Secondary Agent Running 12/16/12 02:25:40 [issue_cmd ] typ-esx05 Primary Agent Running 12/16/12 02:25:40 [issue_cmd ] typ-esx06 Primary Agent Running 12/16/12
Dec 16 02:25:41 02:25:40 [issue_cmd ] VMwarenodestatus=ty-esx01,Secondary,AgentRunning,typ-esx05,Primary,AgentRunning,typ-esx06,Primary,AgentRunning 12/16/12 02:25:41 [issue_cmd ] CMD: /bin/ping -c 1 10.0.0.10 12/16/12 02:25:41 [issue_cmd ] STATUS: 0 12/16/12 02:25:41 [issue_cmd ] RESULT: 12/16/12 02:25:41 [issue_cmd ] PING 10.0.0.10 (10.0.0.10): 56 data bytes 12/16/12 02:25:41 [issue_cmd ] 64 bytes from 10.0.0.10: icmp_seq=0 ttl=255 time=0.211 ms 12/16/12 02:25:41 [issue_cmd ] 12/16/12 02:25:41 [issue_cmd ] --- 10.0.0.10 ping statistics --- 12/16/12 02:25:41 [issue_cmd ] 1 packets transmitted, 1 packets received, 0% packet loss 12/16/12 02:25:41 [issue_cmd ] round-trip min/avg/max = 0.211/0.211/0.211 ms 12/16/12 02:25:41 [issue_cmd ] 12/16/12 02:25:41 [myexit ] VMwareresult=success 12/16/12 02:25:41 [elapsed_time ] Total time for script to complete: 0 minute(s) and 1 second(s
Dec 16 02:25:41 )
Dec 16 02:25:59 etc/opt/vmware/aam 12/16/12 02:25:58 [print_args ] PWD=/var/log/vmware/vpx 12/16/12 02:25:58 [print_args ] PS_OPTIONS= 12/16/12 02:25:58 [print_args ] FT_NO_CONSOLE_TRACE=1 12/16/12 02:25:58 [print_args ] PATH=/sbin:/usr/sbin:/bin:/usr/bin:/opt/vmware/aam/bin:/bin 12/16/12 02:25:58 [print_args ] FT_LOG_DIR=/var/log/vmware/aam 12/16/12 02:25:58 [print_args ] FT_DOMAIN=vmware 12/16/12 02:25:58 [print_args ] Parsed arguments: 12/16/12 02:25:58 [print_args ] cmd=listprimaries 12/16/12 02:25:58 [print_args ] -z=1 12/16/12 02:25:58 [print_args ] uname=VMkernel 12/16/12 02:25:58 [print_args ] shortname=typ-esx05 12/16/12 02:25:58 [print_args ] domain=vmware 12/16/12 02:25:58 [issue_cmd ] CMD: /opt/vmware/aam/bin/ft_gethostbyname typ-esx05 |grep FAILED 12/16/12 02:25:58 [issue_cmd ] STATUS: 1 12/16/12 02:25:58 [issue_c
Dec 16 02:25:59 md ] RESULT: 12/16/12 02:25:58 [issue_cmd ] 12/16/12 02:25:58 [list_primary_nodes ] 12/16/12 02:25:58 [issue_cli_cmd ] command is '/opt/vmware/aam/bin/ftcli -domain vmware -timeout 15 -cmd listnodes' 12/16/12 02:25:59 [issue_cmd ] CMD: /opt/vmware/aam/bin/ftcli -domain vmware -timeout 15 -cmd listnodes 12/16/12 02:25:59 [issue_cmd ] STATUS: 0 12/16/12 02:25:59 [issue_cmd ] RESULT: 12/16/12 02:25:59 [issue_cmd ] *** Node typ-esx05 is the master primary *** 12/16/12 02:25:59 [issue_cmd ] Node Type State 12/16/12 02:25:59 [issue_cmd ] ----------------------- ------------ -------------- 12/16/12 02:25:59 [issue_cmd ] ty-esx01 Primary Agent Shutdown 12/16/12 02:25:59 [issue_cmd ] typ-esx05 Primary Agent Running 12/16/12 02:25:59 [issue_cmd ] typ-esx06 Primary Agent Running 12/16/12 0
Dec 16 02:25:59 2:25:59 [issue_cmd ] VMwarelistprimaries=ty-esx01,typ-esx05,typ-esx06 12/16/12 02:25:59 [myexit ] VMwareresult=success 12/16/12 02:25:59 [elapsed_time ] Total time for script to complete: 0 minute(s) and 1 second(s)
Dec 16 02:26:12 m 12/16/12 02:26:11 [print_args ] PWD=/var/log/vmware/vpx 12/16/12 02:26:11 [print_args ] PS_OPTIONS= 12/16/12 02:26:11 [print_args ] FT_NO_CONSOLE_TRACE=1 12/16/12 02:26:11 [print_args ] PATH=/sbin:/usr/sbin:/bin:/usr/bin:/opt/vmware/aam/bin:/bin 12/16/12 02:26:11 [print_args ] FT_LOG_DIR=/var/log/vmware/aam 12/16/12 02:26:11 [print_args ] FT_DOMAIN=vmware 12/16/12 02:26:11 [print_args ] Parsed arguments: 12/16/12 02:26:11 [print_args ] cmd=monitornodes 12/16/12 02:26:11 [print_args ] -z=1 12/16/12 02:26:11 [print_args ] uname=VMkernel 12/16/12 02:26:11 [print_args ] shortname=typ-esx05 12/16/12 02:26:11 [print_args ] domain=vmware 12/16/12 02:26:11 [issue_cmd ] CMD: /opt/vmware/aam/bin/ft_gethostbyname typ-esx05 |grep FAILED 12/16/12 02:26:11 [issue_cmd ] STATUS: 1 12/16/12 02:26:11 [issue_cmd ] RES
Dec 16 02:26:12 ULT: 12/16/12 02:26:11 [issue_cmd ] 12/16/12 02:26:11 [list_nodes ] 12/16/12 02:26:11 [issue_cli_cmd ] command is '/opt/vmware/aam/bin/ftcli -domain vmware -port 8042 -timeout 5 -cmd listnodes' 12/16/12 02:26:11 [issue_cmd ] CMD: /opt/vmware/aam/bin/ftcli -domain vmware -port 8042 -timeout 5 -cmd listnodes 12/16/12 02:26:11 [issue_cmd ] STATUS: 0 12/16/12 02:26:11 [issue_cmd ] RESULT: 12/16/12 02:26:11 [issue_cmd ] *** Node typ-esx05 is the master primary *** 12/16/12 02:26:11 [issue_cmd ] Node Type State 12/16/12 02:26:11 [issue_cmd ] ----------------------- ------------ -------------- 12/16/12 02:26:11 [issue_cmd ] ty-esx01 Primary Agent Running 12/16/12 02:26:11 [issue_cmd ] typ-esx05 Primary Agent Running 12/16/12 02:26:11 [issue_cmd ] typ-esx06 Primary Agent Running 12/16/12
Dec 16 02:26:12 02:26:11 [issue_cmd ] VMwarenodestatus=ty-esx01,Primary,AgentRunning,typ-esx05,Primary,AgentRunning,typ-esx06,Primary,AgentRunning 12/16/12 02:26:11 [issue_cmd ] CMD: /bin/ping -c 1 10.0.0.10 12/16/12 02:26:11 [issue_cmd ] STATUS: 0 12/16/12 02:26:11 [issue_cmd ] RESULT: 12/16/12 02:26:11 [issue_cmd ] PING 10.0.0.10 (10.0.0.10): 56 data bytes 12/16/12 02:26:11 [issue_cmd ] 64 bytes from 10.0.0.10: icmp_seq=0 ttl=255 time=0.205 ms 12/16/12 02:26:11 [issue_cmd ] 12/16/12 02:26:11 [issue_cmd ] --- 10.0.0.10 ping statistics --- 12/16/12 02:26:11 [issue_cmd ] 1 packets transmitted, 1 packets received, 0% packet loss 12/16/12 02:26:11 [issue_cmd ] round-trip min/avg/max = 0.205/0.205/0.205 ms 12/16/12 02:26:11 [issue_cmd ] 12/16/12 02:26:11 [myexit ] VMwareresult=success 12/16/12 02:26:11 [elapsed_time ] Total time for script to complete: 0 minute(s) and 0 second(s)
Dec 16 02:26:43 m 12/16/12 02:26:42 [print_args ] PWD=/var/log/vmware/vpx 12/16/12 02:26:42 [print_args ] PS_OPTIONS= 12/16/12 02:26:42 [print_args ] FT_NO_CONSOLE_TRACE=1 12/16/12 02:26:42 [print_args ] PATH=/sbin:/usr/sbin:/bin:/usr/bin:/opt/vmware/aam/bin:/bin 12/16/12 02:26:42 [print_args ] FT_LOG_DIR=/var/log/vmware/aam 12/16/12 02:26:42 [print_args ] FT_DOMAIN=vmware 12/16/12 02:26:42 [print_args ] Parsed arguments: 12/16/12 02:26:42 [print_args ] cmd=monitornodes 12/16/12 02:26:42 [print_args ] -z=1 12/16/12 02:26:42 [print_args ] uname=VMkernel 12/16/12 02:26:42 [print_args ] shortname=typ-esx05 12/16/12 02:26:42 [print_args ] domain=vmware 12/16/12 02:26:42 [issue_cmd ] CMD: /opt/vmware/aam/bin/ft_gethostbyname typ-esx05 |grep FAILED 12/16/12 02:26:42 [issue_cmd ] STATUS: 1 12/16/12 02:26:42 [issue_cmd ] RES
Dec 16 02:26:43 ULT: 12/16/12 02:26:42 [issue_cmd ] 12/16/12 02:26:42 [list_nodes ] 12/16/12 02:26:42 [issue_cli_cmd ] command is '/opt/vmware/aam/bin/ftcli -domain vmware -port 8042 -timeout 5 -cmd listnodes' 12/16/12 02:26:42 [issue_cmd ] CMD: /opt/vmware/aam/bin/ftcli -domain vmware -port 8042 -timeout 5 -cmd listnodes 12/16/12 02:26:42 [issue_cmd ] STATUS: 0 12/16/12 02:26:42 [issue_cmd ] RESULT: 12/16/12 02:26:42 [issue_cmd ] *** Node typ-esx05 is the master primary *** 12/16/12 02:26:42 [issue_cmd ] Node Type State 12/16/12 02:26:42 [issue_cmd ] ----------------------- ------------ -------------- 12/16/12 02:26:42 [issue_cmd ] ty-esx01 Primary Agent Running 12/16/12 02:26:42 [issue_cmd ] typ-esx05 Primary Agent Running 12/16/12 02:26:42 [issue_cmd ] typ-esx06 Primary Agent Running 12/16/12
Dec 16 02:26:43 02:26:42 [issue_cmd ] VMwarenodestatus=ty-esx01,Primary,AgentRunning,typ-esx05,Primary,AgentRunning,typ-esx06,Primary,AgentRunning 12/16/12 02:26:42 [issue_cmd ] CMD: /bin/ping -c 1 10.0.0.10 12/16/12 02:26:42 [issue_cmd ] STATUS: 0 12/16/12 02:26:42 [issue_cmd ] RESULT: 12/16/12 02:26:42 [issue_cmd ] PING 10.0.0.10 (10.0.0.10): 56 data bytes 12/16/12 02:26:42 [issue_cmd ] 64 bytes from 10.0.0.10: icmp_seq=0 ttl=255 time=0.183 ms 12/16/12 02:26:42 [issue_cmd ] 12/16/12 02:26:42 [issue_cmd ] --- 10.0.0.10 ping statistics --- 12/16/12 02:26:42 [issue_cmd ] 1 packets transmitted, 1 packets received, 0% packet loss 12/16/12 02:26:42 [issue_cmd ] round-trip min/avg/max = 0.183/0.183/0.183 ms 12/16/12 02:26:42 [issue_cmd ] 12/16/12 02:26:42 [myexit ] VMwareresult=success 12/16/12 02:26:42 [elapsed_time ] Total time for script to complete: 0 minute(s) and 0 second(s)
Dec 16 02:27:14 m 12/16/12 02:27:13 [print_args ] PWD=/var/log/vmware/vpx 12/16/12 02:27:13 [print_args ] PS_OPTIONS= 12/16/12 02:27:13 [print_args ] FT_NO_CONSOLE_TRACE=1 12/16/12 02:27:13 [print_args ] PATH=/sbin:/usr/sbin:/bin:/usr/bin:/opt/vmware/aam/bin:/bin 12/16/12 02:27:13 [print_args ] FT_LOG_DIR=/var/log/vmware/aam 12/16/12 02:27:13 [print_args ] FT_DOMAIN=vmware 12/16/12 02:27:13 [print_args ] Parsed arguments: 12/16/12 02:27:13 [print_args ] cmd=monitornodes 12/16/12 02:27:13 [print_args ] -z=1 12/16/12 02:27:13 [print_args ] uname=VMkernel 12/16/12 02:27:13 [print_args ] shortname=typ-esx05 12/16/12 02:27:13 [print_args ] domain=vmware 12/16/12 02:27:13 [issue_cmd ] CMD: /opt/vmware/aam/bin/ft_gethostbyname typ-esx05 |grep FAILED 12/16/12 02:27:13 [issue_cmd ] STATUS: 1 12/16/12 02:27:13 [issue_cmd ] RES
Dec 16 02:27:14 ULT: 12/16/12 02:27:13 [issue_cmd ] 12/16/12 02:27:13 [list_nodes ] 12/16/12 02:27:13 [issue_cli_cmd ] command is '/opt/vmware/aam/bin/ftcli -domain vmware -port 8042 -timeout 5 -cmd listnodes' 12/16/12 02:27:13 [issue_cmd ] CMD: /opt/vmware/aam/bin/ftcli -domain vmware -port 8042 -timeout 5 -cmd listnodes 12/16/12 02:27:13 [issue_cmd ] STATUS: 0 12/16/12 02:27:13 [issue_cmd ] RESULT: 12/16/12 02:27:13 [issue_cmd ] *** Node typ-esx05 is the master primary *** 12/16/12 02:27:13 [issue_cmd ] Node Type State 12/16/12 02:27:13 [issue_cmd ] ----------------------- ------------ -------------- 12/16/12 02:27:13 [issue_cmd ] ty-esx01 Primary Agent Running 12/16/12 02:27:13 [issue_cmd ] typ-esx05 Primary Agent Running 12/16/12 02:27:13 [issue_cmd ] typ-esx06 Primary Agent Running 12/16/12
Dec 16 02:27:14 02:27:13 [issue_cmd ] VMwarenodestatus=ty-esx01,Primary,AgentRunning,typ-esx05,Primary,AgentRunning,typ-esx06,Primary,AgentRunning 12/16/12 02:27:13 [issue_cmd ] CMD: /bin/ping -c 1 10.0.0.10 12/16/12 02:27:13 [issue_cmd ] STATUS: 0 12/16/12 02:27:13 [issue_cmd ] RESULT: 12/16/12 02:27:13 [issue_cmd ] PING 10.0.0.10 (10.0.0.10): 56 data bytes 12/16/12 02:27:13 [issue_cmd ] 64 bytes from 10.0.0.10: icmp_seq=0 ttl=255 time=0.205 ms 12/16/12 02:27:13 [issue_cmd ] 12/16/12 02:27:13 [issue_cmd ] --- 10.0.0.10 ping statistics --- 12/16/12 02:27:13 [issue_cmd ] 1 packets transmitted, 1 packets received, 0% packet loss 12/16/12 02:27:13 [issue_cmd ] round-trip min/avg/max = 0.205/0.205/0.205 ms 12/16/12 02:27:13 [issue_cmd ] 12/16/12 02:27:13 [myexit ] VMwareresult=success 12/16/12 02:27:13 [elapsed_time ] Total time for script to complete: 0 minute(s) and 0 second(s)
Dec 16 02:27:45 m 12/16/12 02:27:44 [print_args ] PWD=/var/log/vmware/vpx 12/16/12 02:27:44 [print_args ] PS_OPTIONS= 12/16/12 02:27:44 [print_args ] FT_NO_CONSOLE_TRACE=1 12/16/12 02:27:44 [print_args ] PATH=/sbin:/usr/sbin:/bin:/usr/bin:/opt/vmware/aam/bin:/bin 12/16/12 02:27:44 [print_args ] FT_LOG_DIR=/var/log/vmware/aam 12/16/12 02:27:44 [print_args ] FT_DOMAIN=vmware 12/16/12 02:27:44 [print_args ] Parsed arguments: 12/16/12 02:27:44 [print_args ] cmd=monitornodes 12/16/12 02:27:44 [print_args ] -z=1 12/16/12 02:27:44 [print_args ] uname=VMkernel 12/16/12 02:27:44 [print_args ] shortname=typ-esx05 12/16/12 02:27:44 [print_args ] domain=vmware 12/16/12 02:27:44 [issue_cmd ] CMD: /opt/vmware/aam/bin/ft_gethostbyname typ-esx05 |grep FAILED 12/16/12 02:27:44 [issue_cmd ] STATUS: 1 12/16/12 02:27:44 [issue_cmd ] RES
Dec 16 02:27:45 ULT: 12/16/12 02:27:44 [issue_cmd ] 12/16/12 02:27:44 [list_nodes ] 12/16/12 02:27:44 [issue_cli_cmd ] command is '/opt/vmware/aam/bin/ftcli -domain vmware -port 8042 -timeout 5 -cmd listnodes' 12/16/12 02:27:45 [issue_cmd ] CMD: /opt/vmware/aam/bin/ftcli -domain vmware -port 8042 -timeout 5 -cmd listnodes 12/16/12 02:27:45 [issue_cmd ] STATUS: 0 12/16/12 02:27:45 [issue_cmd ] RESULT: 12/16/12 02:27:45 [issue_cmd ] *** Node typ-esx05 is the master primary *** 12/16/12 02:27:45 [issue_cmd ] Node Type State 12/16/12 02:27:45 [issue_cmd ] ----------------------- ------------ -------------- 12/16/12 02:27:45 [issue_cmd ] ty-esx01 Primary Agent Running 12/16/12 02:27:45 [issue_cmd ] typ-esx05 Primary Agent Running 12/16/12 02:27:45 [issue_cmd ] typ-esx06 Primary Agent Running 12/16/12
Dec 16 02:27:45 02:27:45 [issue_cmd ] VMwarenodestatus=ty-esx01,Primary,AgentRunning,typ-esx05,Primary,AgentRunning,typ-esx06,Primary,AgentRunning 12/16/12 02:27:45 [issue_cmd ] CMD: /bin/ping -c 1 10.0.0.10 12/16/12 02:27:45 [issue_cmd ] STATUS: 0 12/16/12 02:27:45 [issue_cmd ] RESULT: 12/16/12 02:27:45 [issue_cmd ] PING 10.0.0.10 (10.0.0.10): 56 data bytes 12/16/12 02:27:45 [issue_cmd ] 64 bytes from 10.0.0.10: icmp_seq=0 ttl=255 time=0.208 ms 12/16/12 02:27:45 [issue_cmd ] 12/16/12 02:27:45 [issue_cmd ] --- 10.0.0.10 ping statistics --- 12/16/12 02:27:45 [issue_cmd ] 1 packets transmitted, 1 packets received, 0% packet loss 12/16/12 02:27:45 [issue_cmd ] round-trip min/avg/max = 0.208/0.208/0.208 ms 12/16/12 02:27:45 [issue_cmd ] 12/16/12 02:27:45 [myexit ] VMwareresult=success 12/16/12 02:27:45 [elapsed_time ] Total time for script to complete: 0 minute(s) and 1 second(s)
Dec 16 02:28:16 m 12/16/12 02:28:15 [print_args ] PWD=/var/log/vmware/vpx 12/16/12 02:28:15 [print_args ] PS_OPTIONS= 12/16/12 02:28:15 [print_args ] FT_NO_CONSOLE_TRACE=1 12/16/12 02:28:15 [print_args ] PATH=/sbin:/usr/sbin:/bin:/usr/bin:/opt/vmware/aam/bin:/bin 12/16/12 02:28:15 [print_args ] FT_LOG_DIR=/var/log/vmware/aam 12/16/12 02:28:15 [print_args ] FT_DOMAIN=vmware 12/16/12 02:28:15 [print_args ] Parsed arguments: 12/16/12 02:28:15 [print_args ] cmd=monitornodes 12/16/12 02:28:15 [print_args ] -z=1 12/16/12 02:28:15 [print_args ] uname=VMkernel 12/16/12 02:28:15 [print_args ] shortname=typ-esx05 12/16/12 02:28:15 [print_args ] domain=vmware 12/16/12 02:28:15 [issue_cmd ] CMD: /opt/vmware/aam/bin/ft_gethostbyname typ-esx05 |grep FAILED 12/16/12 02:28:15 [issue_cmd ] STATUS: 1 12/16/12 02:28:15 [issue_cmd ] RES
Dec 16 02:28:16 ULT: 12/16/12 02:28:15 [issue_cmd ] 12/16/12 02:28:15 [list_nodes ] 12/16/12 02:28:15 [issue_cli_cmd ] command is '/opt/vmware/aam/bin/ftcli -domain vmware -port 8042 -timeout 5 -cmd listnodes' 12/16/12 02:28:16 [issue_cmd ] CMD: /opt/vmware/aam/bin/ftcli -domain vmware -port 8042 -timeout 5 -cmd listnodes 12/16/12 02:28:16 [issue_cmd ] STATUS: 0 12/16/12 02:28:16 [issue_cmd ] RESULT: 12/16/12 02:28:16 [issue_cmd ] *** Node typ-esx05 is the master primary *** 12/16/12 02:28:16 [issue_cmd ] Node Type State 12/16/12 02:28:16 [issue_cmd ] ----------------------- ------------ -------------- 12/16/12 02:28:16 [issue_cmd ] ty-esx01 Primary Agent Running 12/16/12 02:28:16 [issue_cmd ] typ-esx05 Primary Agent Running 12/16/12 02:28:16 [issue_cmd ] typ-esx06 Primary Agent Running 12/16/12
Dec 16 02:28:16 02:28:16 [issue_cmd ] VMwarenodestatus=ty-esx01,Primary,AgentRunning,typ-esx05,Primary,AgentRunning,typ-esx06,Primary,AgentRunning 12/16/12 02:28:16 [issue_cmd ] CMD: /bin/ping -c 1 10.0.0.10 12/16/12 02:28:16 [issue_cmd ] STATUS: 0 12/16/12 02:28:16 [issue_cmd ] RESULT: 12/16/12 02:28:16 [issue_cmd ] PING 10.0.0.10 (10.0.0.10): 56 data bytes 12/16/12 02:28:16 [issue_cmd ] 64 bytes from 10.0.0.10: icmp_seq=0 ttl=255 time=1.368 ms 12/16/12 02:28:16 [issue_cmd ] 12/16/12 02:28:16 [issue_cmd ] --- 10.0.0.10 ping statistics --- 12/16/12 02:28:16 [issue_cmd ] 1 packets transmitted, 1 packets received, 0% packet loss 12/16/12 02:28:16 [issue_cmd ] round-trip min/avg/max = 1.368/1.368/1.368 ms 12/16/12 02:28:16 [issue_cmd ] 12/16/12 02:28:16 [myexit ] VMwareresult=success 12/16/12 02:28:16 [elapsed_time ] Total time for script to complete: 0 minute(s) and 1 second(s)
Dec 16 02:28:47 m 12/16/12 02:28:46 [print_args ] PWD=/var/log/vmware/vpx 12/16/12 02:28:46 [print_args ] PS_OPTIONS= 12/16/12 02:28:46 [print_args ] FT_NO_CONSOLE_TRACE=1 12/16/12 02:28:46 [print_args ] PATH=/sbin:/usr/sbin:/bin:/usr/bin:/opt/vmware/aam/bin:/bin 12/16/12 02:28:46 [print_args ] FT_LOG_DIR=/var/log/vmware/aam 12/16/12 02:28:46 [print_args ] FT_DOMAIN=vmware 12/16/12 02:28:46 [print_args ] Parsed arguments: 12/16/12 02:28:46 [print_args ] cmd=monitornodes 12/16/12 02:28:46 [print_args ] -z=1 12/16/12 02:28:46 [print_args ] uname=VMkernel 12/16/12 02:28:46 [print_args ] shortname=typ-esx05 12/16/12 02:28:46 [print_args ] domain=vmware 12/16/12 02:28:46 [issue_cmd ] CMD: /opt/vmware/aam/bin/ft_gethostbyname typ-esx05 |grep FAILED 12/16/12 02:28:46 [issue_cmd ] STATUS: 1 12/16/12 02:28:46 [issue_cmd ] RES
Dec 16 02:28:47 ULT: 12/16/12 02:28:46 [issue_cmd ] 12/16/12 02:28:46 [list_nodes ] 12/16/12 02:28:46 [issue_cli_cmd ] command is '/opt/vmware/aam/bin/ftcli -domain vmware -port 8042 -timeout 5 -cmd listnodes' 12/16/12 02:28:47 [issue_cmd ] CMD: /opt/vmware/aam/bin/ftcli -domain vmware -port 8042 -timeout 5 -cmd listnodes 12/16/12 02:28:47 [issue_cmd ] STATUS: 0 12/16/12 02:28:47 [issue_cmd ] RESULT: 12/16/12 02:28:47 [issue_cmd ] *** Node typ-esx05 is the master primary *** 12/16/12 02:28:47 [issue_cmd ] Node Type State 12/16/12 02:28:47 [issue_cmd ] ----------------------- ------------ -------------- 12/16/12 02:28:47 [issue_cmd ] ty-esx01 Primary Agent Running 12/16/12 02:28:47 [issue_cmd ] typ-esx05 Primary Agent Running 12/16/12 02:28:47 [issue_cmd ] typ-esx06 Primary Agent Running 12/16/12
Dec 16 02:28:47 02:28:47 [issue_cmd ] VMwarenodestatus=ty-esx01,Primary,AgentRunning,typ-esx05,Primary,AgentRunning,typ-esx06,Primary,AgentRunning 12/16/12 02:28:47 [issue_cmd ] CMD: /bin/ping -c 1 10.0.0.10 12/16/12 02:28:47 [issue_cmd ] STATUS: 0 12/16/12 02:28:47 [issue_cmd ] RESULT: 12/16/12 02:28:47 [issue_cmd ] PING 10.0.0.10 (10.0.0.10): 56 data bytes 12/16/12 02:28:47 [issue_cmd ] 64 bytes from 10.0.0.10: icmp_seq=0 ttl=255 time=0.190 ms 12/16/12 02:28:47 [issue_cmd ] 12/16/12 02:28:47 [issue_cmd ] --- 10.0.0.10 ping statistics --- 12/16/12 02:28:47 [issue_cmd ] 1 packets transmitted, 1 packets received, 0% packet loss 12/16/12 02:28:47 [issue_cmd ] round-trip min/avg/max = 0.190/0.190/0.190 ms 12/16/12 02:28:47 [issue_cmd ] 12/16/12 02:28:47 [myexit ] VMwareresult=success 12/16/12 02:28:47 [elapsed_time ] Total time for script to complete: 0 minute(s) and 1 second(s)
Dec 16 02:29:15 vmkernel: 0:00:53:10.720 cpu41:19957)ScsiNpiv: 1500: GetInfo for adapter vmhba2, [0x4100b80309c0], max_vports=255, vports_inuse=0, linktype=0, state=1, failreason=0, rv=0, sts=0
Dec 16 02:29:15 vmkernel: 0:00:53:10.720 cpu41:19957)ScsiNpiv: 1500: GetInfo for adapter vmhba3, [0x4100b8031300], max_vports=255, vports_inuse=0, linktype=0, state=1, failreason=0, rv=0, sts=0
Dec 16 02:29:15 vmkernel: 0:00:53:10.833 cpu34:4782)NMP: nmp_CompleteCommandForPath: Command 0x12 (0x41033f3ec440) to NMP device "mpx.vmhba0:C0:T0:L0" failed on physical path "vmhba0:C0:T0:L0" H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
Dec 16 02:29:15 vmkernel: 0:00:53:10.833 cpu34:4782)ScsiDeviceIO: 1688: Command 0x12 to device "mpx.vmhba0:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: NMP module ID: a.
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: NMP heap ID: 0x4100a2000000.
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: NMP slab ID: 0x4100a2002900.
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: NMP is managing 34 devices:
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: naa.600508b1001c5649118ef34ae6a0d2cc (SATP='VMW_SATP_LOCAL', PSP='VMW_PSP_FIXED'):
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: Device state = normal, Failover state = none
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: 1 path:
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba1:C0:T0:L0 (on, active)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: naa.60060160e3703000f6fec54c4ae8e111 (SATP='VMW_SATP_ALUA_CX', PSP='VMW_PSP_FIXED_AP'):
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: Device state = normal, Failover state = none
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: 5 paths:
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba2:C0:T1:L19 (on, active)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba2:C0:T0:L19 (on)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba3:C0:T1:L19 (on)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba3:C0:T0:L19 (on)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: naa.60060160e3703000e4368e369be7e111 (SATP='VMW_SATP_ALUA_CX', PSP='VMW_PSP_FIXED_AP'):
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: Device state = normal, Failover state = none
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: 9 paths:
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba2:C0:T1:L18 (on)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba2:C0:T0:L18 (on)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba3:C0:T1:L18 (on)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba3:C0:T0:L18 (on, active)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: naa.60060160e3703000d28d7775b6e6e111 (SATP='VMW_SATP_ALUA_CX', PSP='VMW_PSP_FIXED_AP'):
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: Device state = normal, Failover state = none
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: 13 paths:
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba2:C0:T1:L17 (on)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba2:C0:T0:L17 (on)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba3:C0:T1:L17 (on)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba3:C0:T0:L17 (on, active)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: naa.60060160e37030003afe898598e6e111 (SATP='VMW_SATP_ALUA_CX', PSP='VMW_PSP_FIXED_AP'):
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: Device state = normal, Failover state = none
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: 17 paths:
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba2:C0:T1:L16 (on, active)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba2:C0:T0:L16 (on)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba3:C0:T1:L16 (on)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba3:C0:T0:L16 (on)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: naa.60060160e3703000aa66b07298e6e111 (SATP='VMW_SATP_ALUA_CX', PSP='VMW_PSP_FIXED_AP'):
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: Device state = normal, Failover state = none
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: 21 paths:
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba2:C0:T1:L15 (on)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba2:C0:T0:L15 (on)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba3:C0:T1:L15 (on)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba3:C0:T0:L15 (on, active)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: naa.60060160e370300014e7d676d0dfe111 (SATP='VMW_SATP_ALUA_CX', PSP='VMW_PSP_FIXED_AP'):
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: Device state = normal, Failover state = none
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: 25 paths:
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba2:C0:T1:L14 (on, active)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba2:C0:T0:L14 (on)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba3:C0:T1:L14 (on)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba3:C0:T0:L14 (on)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: naa.60060160e37030000255701bb5d4e111 (SATP='VMW_SATP_ALUA_CX', PSP='VMW_PSP_FIXED_AP'):
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: Device state = normal, Failover state = none
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: 29 paths:
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba2:C0:T1:L13 (on)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba2:C0:T0:L13 (on)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba3:C0:T1:L13 (on)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba3:C0:T0:L13 (on, active)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: naa.60060160e37030005ed07e8d863ce211 (SATP='VMW_SATP_ALUA_CX', PSP='VMW_PSP_FIXED_AP'):
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: Device state = normal, Failover state = none
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: 33 paths:
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba2:C0:T1:L12 (on)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba2:C0:T0:L12 (on)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba3:C0:T1:L12 (on)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba3:C0:T0:L12 (on, active)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: naa.6006016077f030002ec596b8caafe111 (SATP='VMW_SATP_ALUA_CX', PSP='VMW_PSP_FIXED_AP'):
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: Device state = normal, Failover state = none
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: 37 paths:
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba2:C0:T1:L11 (on, active)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba2:C0:T0:L11 (on)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba3:C0:T1:L11 (on)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba3:C0:T0:L11 (on)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: naa.6006016077f03000007d039acaafe111 (SATP='VMW_SATP_ALUA_CX', PSP='VMW_PSP_FIXED_AP'):
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: Device state = normal, Failover state = none
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: 41 paths:
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba2:C0:T1:L10 (on)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba2:C0:T0:L10 (on)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba3:C0:T1:L10 (on)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba3:C0:T0:L10 (on, active)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: mpx.vmhba0:C0:T0:L0 (SATP='VMW_SATP_LOCAL', PSP='VMW_PSP_FIXED'):
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: Device state = normal, Failover state = none
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: 42 paths:
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba0:C0:T0:L0 (on, active)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: naa.6006016077f030003c3e245ac9afe111 (SATP='VMW_SATP_ALUA_CX', PSP='VMW_PSP_FIXED_AP'):
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: Device state = normal, Failover state = none
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: 46 paths:
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba2:C0:T0:L9 (on)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba2:C0:T1:L9 (on)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba3:C0:T0:L9 (on)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba3:C0:T1:L9 (on, active)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: naa.6006016077f03000f002b47a87aae111 (SATP='VMW_SATP_ALUA_CX', PSP='VMW_PSP_FIXED_AP'):
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: Device state = normal, Failover state = none
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: 50 paths:
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba2:C0:T0:L8 (on)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba2:C0:T1:L8 (on)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba3:C0:T0:L8 (on)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba3:C0:T1:L8 (on, active)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: naa.6006016077f030004474675fa7a5e111 (SATP='VMW_SATP_ALUA_CX', PSP='VMW_PSP_FIXED_AP'):
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: Device state = normal, Failover state = none
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: 54 paths:
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba2:C0:T0:L7 (on, active)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba2:C0:T1:L7 (on)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba3:C0:T0:L7 (on)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba3:C0:T1:L7 (on)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: naa.6006016077f03000f6500b46a7a5e111 (SATP='VMW_SATP_ALUA_CX', PSP='VMW_PSP_FIXED_AP'):
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: Device state = normal, Failover state = none
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: 58 paths:
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba2:C0:T0:L6 (on)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba2:C0:T1:L6 (on)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba3:C0:T0:L6 (on)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba3:C0:T1:L6 (on, active)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: naa.6006016077f03000d0672bada7a5e111 (SATP='VMW_SATP_ALUA_CX', PSP='VMW_PSP_FIXED_AP'):
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: Device state = normal, Failover state = none
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: 62 paths:
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba2:C0:T0:L5 (on)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba2:C0:T1:L5 (on)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba3:C0:T0:L5 (on)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba3:C0:T1:L5 (on, active)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: naa.6006016077f03000b0e8cedba6a5e111 (SATP='VMW_SATP_ALUA_CX', PSP='VMW_PSP_FIXED_AP'):
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: Device state = normal, Failover state = none
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: 66 paths:
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba2:C0:T0:L4 (on, active)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba2:C0:T1:L4 (on)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba3:C0:T0:L4 (on)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba3:C0:T1:L4 (on)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: naa.6006016077f030002c64ef41a6a5e111 (SATP='VMW_SATP_ALUA_CX', PSP='VMW_PSP_FIXED_AP'):
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: Device state = normal, Failover state = none
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: 70 paths:
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba2:C0:T0:L3 (on)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba2:C0:T1:L3 (on)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba3:C0:T0:L3 (on)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba3:C0:T1:L3 (on, active)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: naa.6006016077f0300052febe6183a5e111 (SATP='VMW_SATP_ALUA_CX', PSP='VMW_PSP_FIXED_AP'):
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: Device state = normal, Failover state = none
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: 74 paths:
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba2:C0:T0:L2 (on, active)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba2:C0:T1:L2 (on)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba3:C0:T0:L2 (on)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba3:C0:T1:L2 (on)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: naa.6006016077f030009e5dcf7fe1a4e111 (SATP='VMW_SATP_ALUA_CX', PSP='VMW_PSP_FIXED_AP'):
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: Device state = normal, Failover state = none
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: 78 paths:
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba2:C0:T0:L1 (on)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba2:C0:T1:L1 (on)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba3:C0:T0:L1 (on)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba3:C0:T1:L1 (on, active)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: naa.6006016077f03000a0701c1bdaa4e111 (SATP='VMW_SATP_ALUA_CX', PSP='VMW_PSP_FIXED_AP'):
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: Device state = normal, Failover state = none
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: 82 paths:
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba2:C0:T0:L0 (on, active)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba2:C0:T1:L0 (on)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba3:C0:T0:L0 (on)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba3:C0:T1:L0 (on)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: naa.60060160e37030003c62a271a03ce211 (SATP='VMW_SATP_ALUA_CX', PSP='VMW_PSP_FIXED_AP'):
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: Device state = normal, Failover state = none
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: 86 paths:
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba3:C0:T0:L31 (on)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba2:C0:T1:L31 (on)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba2:C0:T0:L31 (on)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba3:C0:T1:L31 (on, active)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: naa.60060160e37030003c346805893ce211 (SATP='VMW_SATP_ALUA_CX', PSP='VMW_PSP_FIXED_AP'):
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: Device state = normal, Failover state = none
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: 90 paths:
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba3:C0:T0:L30 (on, active)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba2:C0:T1:L30 (on)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba2:C0:T0:L30 (on)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba3:C0:T1:L30 (on)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: naa.60060160e37030003e508f017f3ce211 (SATP='VMW_SATP_ALUA_CX', PSP='VMW_PSP_FIXED_AP'):
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: Device state = normal, Failover state = none
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: 94 paths:
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba2:C0:T0:L29 (on, active)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba3:C0:T1:L29 (on)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba3:C0:T0:L29 (on)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba2:C0:T1:L29 (on)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: naa.60060160e370300000a85763773ce211 (SATP='VMW_SATP_ALUA_CX', PSP='VMW_PSP_FIXED_AP'):
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: Device state = normal, Failover state = none
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: 98 paths:
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba2:C0:T0:L28 (on)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba3:C0:T1:L28 (on)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba3:C0:T0:L28 (on)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba2:C0:T1:L28 (on, active)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: naa.60060160e3703000e454ee956d3ce211 (SATP='VMW_SATP_ALUA_CX', PSP='VMW_PSP_FIXED_AP'):
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: Device state = normal, Failover state = none
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: 102 paths:
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba2:C0:T0:L27 (on, active)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba3:C0:T1:L27 (on)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba3:C0:T0:L27 (on)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba2:C0:T1:L27 (on)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: naa.60060160e3703000c694025c6c3ce211 (SATP='VMW_SATP_ALUA_CX', PSP='VMW_PSP_FIXED_AP'):
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: Device state = normal, Failover state = none
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: 106 paths:
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba2:C0:T0:L26 (on)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba3:C0:T1:L26 (on)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba3:C0:T0:L26 (on)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba2:C0:T1:L26 (on, active)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: naa.60060160e3703000d6322615693ce211 (SATP='VMW_SATP_ALUA_CX', PSP='VMW_PSP_FIXED_AP'):
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: Device state = normal, Failover state = none
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: 110 paths:
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba2:C0:T0:L25 (on, active)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba3:C0:T1:L25 (on)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba3:C0:T0:L25 (on)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba2:C0:T1:L25 (on)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: naa.60060160e370300094af75ed653ce211 (SATP='VMW_SATP_ALUA_CX', PSP='VMW_PSP_FIXED_AP'):
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: Device state = normal, Failover state = none
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: 114 paths:
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba2:C0:T0:L24 (on)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba3:C0:T1:L24 (on)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba3:C0:T0:L24 (on)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba2:C0:T1:L24 (on, active)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: naa.60060160e37030001490506a0134e211 (SATP='VMW_SATP_ALUA_CX', PSP='VMW_PSP_FIXED_AP'):
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: Device state = normal, Failover state = none
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: 118 paths:
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba2:C0:T0:L23 (on, active)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba3:C0:T1:L23 (on)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba3:C0:T0:L23 (on)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba2:C0:T1:L23 (on)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: naa.60060160e37030005c829419ff1ce211 (SATP='VMW_SATP_ALUA_CX', PSP='VMW_PSP_FIXED_AP'):
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: Device state = normal, Failover state = none
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: 122 paths:
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba2:C0:T0:L22 (on, active)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba3:C0:T1:L22 (on)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba3:C0:T0:L22 (on)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba2:C0:T1:L22 (on)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: naa.60060160e3703000383c6a633c07e211 (SATP='VMW_SATP_ALUA_CX', PSP='VMW_PSP_FIXED_AP'):
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: Device state = normal, Failover state = none
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: 126 paths:
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba2:C0:T0:L21 (on)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba3:C0:T1:L21 (on)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba3:C0:T0:L21 (on)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba2:C0:T1:L21 (on, active)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: naa.60060160e37030006a9f6d32b5fee111 (SATP='VMW_SATP_ALUA_CX', PSP='VMW_PSP_FIXED_AP'):
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: Device state = normal, Failover state = none
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: 130 paths:
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba2:C0:T0:L20 (on, active)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba3:C0:T1:L20 (on)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba3:C0:T0:L20 (on)
Dec 16 02:29:15 vmkernel: 0:00:53:10.851 cpu23:19957)NMP: nmpStateLogger: vmhba2:C0:T1:L20 (on)
Dec 16 02:29:17 vmkernel: ACPI Warning (nspredef-0955): \_SB_._OSC: Return type mismatch - found Integer, expected Buffer0:00:53:13.042 cpu56:19957) [20080926]
Dec 16 02:29:17 vmkernel: ACPI Error (nsxfeval-0218): Incorrect return type [Integer] requested [Buffer]0:00:53:13.042 cpu56:19957) [20080926]
Dec 16 02:29:17 vmkernel: 0:00:53:13.042 cpu56:19957)VMKAcpi: 3463: APEI not supported: Could not evaluate OSC method, Error = 0x0
Dec 16 02:29:18 m 12/16/12 02:29:17 [print_args ] PWD=/var/log/vmware/vpx 12/16/12 02:29:17 [print_args ] PS_OPTIONS= 12/16/12 02:29:17 [print_args ] FT_NO_CONSOLE_TRACE=1 12/16/12 02:29:17 [print_args ] PATH=/sbin:/usr/sbin:/bin:/usr/bin:/opt/vmware/aam/bin:/bin 12/16/12 02:29:17 [print_args ] FT_LOG_DIR=/var/log/vmware/aam 12/16/12 02:29:17 [print_args ] FT_DOMAIN=vmware 12/16/12 02:29:17 [print_args ] Parsed arguments: 12/16/12 02:29:17 [print_args ] cmd=monitornodes 12/16/12 02:29:17 [print_args ] -z=1 12/16/12 02:29:17 [print_args ] uname=VMkernel 12/16/12 02:29:17 [print_args ] shortname=typ-esx05 12/16/12 02:29:17 [print_args ] domain=vmware 12/16/12 02:29:17 [issue_cmd ] CMD: /opt/vmware/aam/bin/ft_gethostbyname typ-esx05 |grep FAILED 12/16/12 02:29:17 [issue_cmd ] STATUS: 1 12/16/12 02:29:17 [issue_cmd ] RES
Dec 16 02:29:18 ULT: 12/16/12 02:29:17 [issue_cmd ] 12/16/12 02:29:17 [list_nodes ] 12/16/12 02:29:17 [issue_cli_cmd ] command is '/opt/vmware/aam/bin/ftcli -domain vmware -port 8042 -timeout 5 -cmd listnodes' 12/16/12 02:29:17 [issue_cmd ] CMD: /opt/vmware/aam/bin/ftcli -domain vmware -port 8042 -timeout 5 -cmd listnodes 12/16/12 02:29:17 [issue_cmd ] STATUS: 0 12/16/12 02:29:17 [issue_cmd ] RESULT: 12/16/12 02:29:17 [issue_cmd ] *** Node typ-esx05 is the master primary *** 12/16/12 02:29:17 [issue_cmd ] Node Type State 12/16/12 02:29:17 [issue_cmd ] ----------------------- ------------ -------------- 12/16/12 02:29:17 [issue_cmd ] ty-esx01 Primary Agent Running 12/16/12 02:29:17 [issue_cmd ] typ-esx05 Primary Agent Running 12/16/12 02:29:17 [issue_cmd ] typ-esx06 Primary Agent Running 12/16/12
Dec 16 02:29:18 02:29:17 [issue_cmd ] VMwarenodestatus=ty-esx01,Primary,AgentRunning,typ-esx05,Primary,AgentRunning,typ-esx06,Primary,AgentRunning 12/16/12 02:29:17 [issue_cmd ] CMD: /bin/ping -c 1 10.0.0.10 12/16/12 02:29:17 [issue_cmd ] STATUS: 0 12/16/12 02:29:17 [issue_cmd ] RESULT: 12/16/12 02:29:17 [issue_cmd ] PING 10.0.0.10 (10.0.0.10): 56 data bytes 12/16/12 02:29:17 [issue_cmd ] 64 bytes from 10.0.0.10: icmp_seq=0 ttl=255 time=0.202 ms 12/16/12 02:29:17 [issue_cmd ] 12/16/12 02:29:17 [issue_cmd ] --- 10.0.0.10 ping statistics --- 12/16/12 02:29:17 [issue_cmd ] 1 packets transmitted, 1 packets received, 0% packet loss 12/16/12 02:29:17 [issue_cmd ] round-trip min/avg/max = 0.202/0.202/0.202 ms 12/16/12 02:29:17 [issue_cmd ] 12/16/12 02:29:17 [myexit ] VMwareresult=success 12/16/12 02:29:17 [elapsed_time ] Total time for script to complete: 0 minute(s) and 0 second(s)
Dec 16 02:29:36 vmkernel: 0:00:53:32.331 cpu14:20036)ScsiNpiv: 1500: GetInfo for adapter vmhba2, [0x4100b80309c0], max_vports=255, vports_inuse=0, linktype=0, state=1, failreason=0, rv=0, sts=0
Dec 16 02:29:36 vmkernel: 0:00:53:32.331 cpu14:20036)ScsiNpiv: 1500: GetInfo for adapter vmhba3, [0x4100b8031300], max_vports=255, vports_inuse=0, linktype=0, state=1, failreason=0, rv=0, sts=0
Dec 16 02:29:37 vmkernel: 0:00:53:32.452 cpu37:4782)NMP: nmp_CompleteCommandForPath: Command 0x12 (0x41033f422340) to NMP device "mpx.vmhba0:C0:T0:L0" failed on physical path "vmhba0:C0:T0:L0" H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
Dec 16 02:29:37 vmkernel: 0:00:53:32.452 cpu37:4782)ScsiDeviceIO: 1688: Command 0x12 to device "mpx.vmhba0:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0.
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: NMP module ID: a.
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: NMP heap ID: 0x4100a2000000.
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: NMP slab ID: 0x4100a2002900.
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: NMP is managing 34 devices:
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: naa.600508b1001c5649118ef34ae6a0d2cc (SATP='VMW_SATP_LOCAL', PSP='VMW_PSP_FIXED'):
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: Device state = normal, Failover state = none
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: 1 path:
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba1:C0:T0:L0 (on, active)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: naa.60060160e3703000f6fec54c4ae8e111 (SATP='VMW_SATP_ALUA_CX', PSP='VMW_PSP_FIXED_AP'):
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: Device state = normal, Failover state = none
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: 5 paths:
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba2:C0:T1:L19 (on, active)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba2:C0:T0:L19 (on)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba3:C0:T1:L19 (on)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba3:C0:T0:L19 (on)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: naa.60060160e3703000e4368e369be7e111 (SATP='VMW_SATP_ALUA_CX', PSP='VMW_PSP_FIXED_AP'):
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: Device state = normal, Failover state = none
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: 9 paths:
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba2:C0:T1:L18 (on)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba2:C0:T0:L18 (on)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba3:C0:T1:L18 (on)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba3:C0:T0:L18 (on, active)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: naa.60060160e3703000d28d7775b6e6e111 (SATP='VMW_SATP_ALUA_CX', PSP='VMW_PSP_FIXED_AP'):
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: Device state = normal, Failover state = none
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: 13 paths:
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba2:C0:T1:L17 (on)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba2:C0:T0:L17 (on)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba3:C0:T1:L17 (on)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba3:C0:T0:L17 (on, active)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: naa.60060160e37030003afe898598e6e111 (SATP='VMW_SATP_ALUA_CX', PSP='VMW_PSP_FIXED_AP'):
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: Device state = normal, Failover state = none
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: 17 paths:
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba2:C0:T1:L16 (on, active)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba2:C0:T0:L16 (on)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba3:C0:T1:L16 (on)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba3:C0:T0:L16 (on)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: naa.60060160e3703000aa66b07298e6e111 (SATP='VMW_SATP_ALUA_CX', PSP='VMW_PSP_FIXED_AP'):
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: Device state = normal, Failover state = none
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: 21 paths:
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba2:C0:T1:L15 (on)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba2:C0:T0:L15 (on)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba3:C0:T1:L15 (on)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba3:C0:T0:L15 (on, active)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: naa.60060160e370300014e7d676d0dfe111 (SATP='VMW_SATP_ALUA_CX', PSP='VMW_PSP_FIXED_AP'):
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: Device state = normal, Failover state = none
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: 25 paths:
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba2:C0:T1:L14 (on, active)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba2:C0:T0:L14 (on)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba3:C0:T1:L14 (on)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba3:C0:T0:L14 (on)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: naa.60060160e37030000255701bb5d4e111 (SATP='VMW_SATP_ALUA_CX', PSP='VMW_PSP_FIXED_AP'):
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: Device state = normal, Failover state = none
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: 29 paths:
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba2:C0:T1:L13 (on)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba2:C0:T0:L13 (on)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba3:C0:T1:L13 (on)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba3:C0:T0:L13 (on, active)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: naa.60060160e37030005ed07e8d863ce211 (SATP='VMW_SATP_ALUA_CX', PSP='VMW_PSP_FIXED_AP'):
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: Device state = normal, Failover state = none
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: 33 paths:
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba2:C0:T1:L12 (on)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba2:C0:T0:L12 (on)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba3:C0:T1:L12 (on)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba3:C0:T0:L12 (on, active)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: naa.6006016077f030002ec596b8caafe111 (SATP='VMW_SATP_ALUA_CX', PSP='VMW_PSP_FIXED_AP'):
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: Device state = normal, Failover state = none
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: 37 paths:
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba2:C0:T1:L11 (on, active)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba2:C0:T0:L11 (on)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba3:C0:T1:L11 (on)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba3:C0:T0:L11 (on)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: naa.6006016077f03000007d039acaafe111 (SATP='VMW_SATP_ALUA_CX', PSP='VMW_PSP_FIXED_AP'):
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: Device state = normal, Failover state = none
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: 41 paths:
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba2:C0:T1:L10 (on)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba2:C0:T0:L10 (on)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba3:C0:T1:L10 (on)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba3:C0:T0:L10 (on, active)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: mpx.vmhba0:C0:T0:L0 (SATP='VMW_SATP_LOCAL', PSP='VMW_PSP_FIXED'):
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: Device state = normal, Failover state = none
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: 42 paths:
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba0:C0:T0:L0 (on, active)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: naa.6006016077f030003c3e245ac9afe111 (SATP='VMW_SATP_ALUA_CX', PSP='VMW_PSP_FIXED_AP'):
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: Device state = normal, Failover state = none
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: 46 paths:
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba2:C0:T0:L9 (on)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba2:C0:T1:L9 (on)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba3:C0:T0:L9 (on)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba3:C0:T1:L9 (on, active)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: naa.6006016077f03000f002b47a87aae111 (SATP='VMW_SATP_ALUA_CX', PSP='VMW_PSP_FIXED_AP'):
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: Device state = normal, Failover state = none
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: 50 paths:
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba2:C0:T0:L8 (on)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba2:C0:T1:L8 (on)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba3:C0:T0:L8 (on)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba3:C0:T1:L8 (on, active)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: naa.6006016077f030004474675fa7a5e111 (SATP='VMW_SATP_ALUA_CX', PSP='VMW_PSP_FIXED_AP'):
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: Device state = normal, Failover state = none
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: 54 paths:
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba2:C0:T0:L7 (on, active)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba2:C0:T1:L7 (on)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba3:C0:T0:L7 (on)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba3:C0:T1:L7 (on)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: naa.6006016077f03000f6500b46a7a5e111 (SATP='VMW_SATP_ALUA_CX', PSP='VMW_PSP_FIXED_AP'):
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: Device state = normal, Failover state = none
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: 58 paths:
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba2:C0:T0:L6 (on)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba2:C0:T1:L6 (on)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba3:C0:T0:L6 (on)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba3:C0:T1:L6 (on, active)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: naa.6006016077f03000d0672bada7a5e111 (SATP='VMW_SATP_ALUA_CX', PSP='VMW_PSP_FIXED_AP'):
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: Device state = normal, Failover state = none
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: 62 paths:
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba2:C0:T0:L5 (on)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba2:C0:T1:L5 (on)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba3:C0:T0:L5 (on)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba3:C0:T1:L5 (on, active)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: naa.6006016077f03000b0e8cedba6a5e111 (SATP='VMW_SATP_ALUA_CX', PSP='VMW_PSP_FIXED_AP'):
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: Device state = normal, Failover state = none
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: 66 paths:
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba2:C0:T0:L4 (on, active)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba2:C0:T1:L4 (on)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba3:C0:T0:L4 (on)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba3:C0:T1:L4 (on)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: naa.6006016077f030002c64ef41a6a5e111 (SATP='VMW_SATP_ALUA_CX', PSP='VMW_PSP_FIXED_AP'):
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: Device state = normal, Failover state = none
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: 70 paths:
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba2:C0:T0:L3 (on)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba2:C0:T1:L3 (on)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba3:C0:T0:L3 (on)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba3:C0:T1:L3 (on, active)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: naa.6006016077f0300052febe6183a5e111 (SATP='VMW_SATP_ALUA_CX', PSP='VMW_PSP_FIXED_AP'):
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: Device state = normal, Failover state = none
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: 74 paths:
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba2:C0:T0:L2 (on, active)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba2:C0:T1:L2 (on)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba3:C0:T0:L2 (on)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba3:C0:T1:L2 (on)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: naa.6006016077f030009e5dcf7fe1a4e111 (SATP='VMW_SATP_ALUA_CX', PSP='VMW_PSP_FIXED_AP'):
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: Device state = normal, Failover state = none
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: 78 paths:
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba2:C0:T0:L1 (on)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba2:C0:T1:L1 (on)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba3:C0:T0:L1 (on)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba3:C0:T1:L1 (on, active)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: naa.6006016077f03000a0701c1bdaa4e111 (SATP='VMW_SATP_ALUA_CX', PSP='VMW_PSP_FIXED_AP'):
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: Device state = normal, Failover state = none
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: 82 paths:
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba2:C0:T0:L0 (on, active)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba2:C0:T1:L0 (on)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba3:C0:T0:L0 (on)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba3:C0:T1:L0 (on)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: naa.60060160e37030003c62a271a03ce211 (SATP='VMW_SATP_ALUA_CX', PSP='VMW_PSP_FIXED_AP'):
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: Device state = normal, Failover state = none
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: 86 paths:
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba3:C0:T0:L31 (on)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba2:C0:T1:L31 (on)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba2:C0:T0:L31 (on)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba3:C0:T1:L31 (on, active)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: naa.60060160e37030003c346805893ce211 (SATP='VMW_SATP_ALUA_CX', PSP='VMW_PSP_FIXED_AP'):
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: Device state = normal, Failover state = none
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: 90 paths:
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba3:C0:T0:L30 (on, active)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba2:C0:T1:L30 (on)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba2:C0:T0:L30 (on)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba3:C0:T1:L30 (on)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: naa.60060160e37030003e508f017f3ce211 (SATP='VMW_SATP_ALUA_CX', PSP='VMW_PSP_FIXED_AP'):
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: Device state = normal, Failover state = none
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: 94 paths:
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba2:C0:T0:L29 (on, active)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba3:C0:T1:L29 (on)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba3:C0:T0:L29 (on)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba2:C0:T1:L29 (on)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: naa.60060160e370300000a85763773ce211 (SATP='VMW_SATP_ALUA_CX', PSP='VMW_PSP_FIXED_AP'):
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: Device state = normal, Failover state = none
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: 98 paths:
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba2:C0:T0:L28 (on)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba3:C0:T1:L28 (on)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba3:C0:T0:L28 (on)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba2:C0:T1:L28 (on, active)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: naa.60060160e3703000e454ee956d3ce211 (SATP='VMW_SATP_ALUA_CX', PSP='VMW_PSP_FIXED_AP'):
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: Device state = normal, Failover state = none
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: 102 paths:
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba2:C0:T0:L27 (on, active)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba3:C0:T1:L27 (on)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba3:C0:T0:L27 (on)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba2:C0:T1:L27 (on)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: naa.60060160e3703000c694025c6c3ce211 (SATP='VMW_SATP_ALUA_CX', PSP='VMW_PSP_FIXED_AP'):
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: Device state = normal, Failover state = none
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: 106 paths:
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba2:C0:T0:L26 (on)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba3:C0:T1:L26 (on)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba3:C0:T0:L26 (on)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba2:C0:T1:L26 (on, active)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: naa.60060160e3703000d6322615693ce211 (SATP='VMW_SATP_ALUA_CX', PSP='VMW_PSP_FIXED_AP'):
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: Device state = normal, Failover state = none
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: 110 paths:
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba2:C0:T0:L25 (on, active)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba3:C0:T1:L25 (on)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba3:C0:T0:L25 (on)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba2:C0:T1:L25 (on)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: naa.60060160e370300094af75ed653ce211 (SATP='VMW_SATP_ALUA_CX', PSP='VMW_PSP_FIXED_AP'):
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: Device state = normal, Failover state = none
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: 114 paths:
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba2:C0:T0:L24 (on)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba3:C0:T1:L24 (on)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba3:C0:T0:L24 (on)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba2:C0:T1:L24 (on, active)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: naa.60060160e37030001490506a0134e211 (SATP='VMW_SATP_ALUA_CX', PSP='VMW_PSP_FIXED_AP'):
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: Device state = normal, Failover state = none
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: 118 paths:
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba2:C0:T0:L23 (on, active)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba3:C0:T1:L23 (on)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba3:C0:T0:L23 (on)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba2:C0:T1:L23 (on)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: naa.60060160e37030005c829419ff1ce211 (SATP='VMW_SATP_ALUA_CX', PSP='VMW_PSP_FIXED_AP'):
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: Device state = normal, Failover state = none
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: 122 paths:
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba2:C0:T0:L22 (on, active)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba3:C0:T1:L22 (on)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba3:C0:T0:L22 (on)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba2:C0:T1:L22 (on)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: naa.60060160e3703000383c6a633c07e211 (SATP='VMW_SATP_ALUA_CX', PSP='VMW_PSP_FIXED_AP'):
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: Device state = normal, Failover state = none
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: 126 paths:
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba2:C0:T0:L21 (on)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba3:C0:T1:L21 (on)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba3:C0:T0:L21 (on)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba2:C0:T1:L21 (on, active)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: naa.60060160e37030006a9f6d32b5fee111 (SATP='VMW_SATP_ALUA_CX', PSP='VMW_PSP_FIXED_AP'):
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: Device state = normal, Failover state = none
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: 130 paths:
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba2:C0:T0:L20 (on, active)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba3:C0:T1:L20 (on)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba3:C0:T0:L20 (on)
Dec 16 02:29:37 vmkernel: 0:00:53:32.472 cpu29:20036)NMP: nmpStateLogger: vmhba2:C0:T1:L20 (on)
Dec 16 02:29:39 vmkernel: ACPI Error (nsxfeval-0218): Incorrect return type [Integer] requested [Buffer]0:00:53:34.683 cpu12:20036) [20080926]
Dec 16 02:29:39 vmkernel: 0:00:53:34.683 cpu12:20036)VMKAcpi: 3463: APEI not supported: Could not evaluate OSC method, Error = 0x0
Dec 16 02:29:49 m 12/16/12 02:29:48 [print_args ] PWD=/var/log/vmware/vpx 12/16/12 02:29:48 [print_args ] PS_OPTIONS= 12/16/12 02:29:48 [print_args ] FT_NO_CONSOLE_TRACE=1 12/16/12 02:29:48 [print_args ] PATH=/sbin:/usr/sbin:/bin:/usr/bin:/opt/vmware/aam/bin:/bin 12/16/12 02:29:48 [print_args ] FT_LOG_DIR=/var/log/vmware/aam 12/16/12 02:29:48 [print_args ] FT_DOMAIN=vmware 12/16/12 02:29:48 [print_args ] Parsed arguments: 12/16/12 02:29:48 [print_args ] cmd=monitornodes 12/16/12 02:29:48 [print_args ] -z=1 12/16/12 02:29:48 [print_args ] uname=VMkernel 12/16/12 02:29:48 [print_args ] shortname=typ-esx05 12/16/12 02:29:48 [print_args ] domain=vmware 12/16/12 02:29:48 [issue_cmd ] CMD: /opt/vmware/aam/bin/ft_gethostbyname typ-esx05 |grep FAILED 12/16/12 02:29:48 [issue_cmd ] STATUS: 1 12/16/12 02:29:48 [issue_cmd ] RES
Dec 16 02:29:49 ULT: 12/16/12 02:29:48 [issue_cmd ] 12/16/12 02:29:48 [list_nodes ] 12/16/12 02:29:48 [issue_cli_cmd ] command is '/opt/vmware/aam/bin/ftcli -domain vmware -port 8042 -timeout 5 -cmd listnodes' 12/16/12 02:29:49 [issue_cmd ] CMD: /opt/vmware/aam/bin/ftcli -domain vmware -port 8042 -timeout 5 -cmd listnodes 12/16/12 02:29:49 [issue_cmd ] STATUS: 0 12/16/12 02:29:49 [issue_cmd ] RESULT: 12/16/12 02:29:49 [issue_cmd ] *** Node typ-esx05 is the master primary *** 12/16/12 02:29:49 [issue_cmd ] Node Type State 12/16/12 02:29:49 [issue_cmd ] ----------------------- ------------ -------------- 12/16/12 02:29:49 [issue_cmd ] ty-esx01 Primary Agent Running 12/16/12 02:29:49 [issue_cmd ] typ-esx05 Primary Agent Running 12/16/12 02:29:49 [issue_cmd ] typ-esx06 Primary Agent Running 12/16/12
Dec 16 02:29:49 02:29:49 [issue_cmd ] VMwarenodestatus=ty-esx01,Primary,AgentRunning,typ-esx05,Primary,AgentRunning,typ-esx06,Primary,AgentRunning 12/16/12 02:29:49 [issue_cmd ] CMD: /bin/ping -c 1 10.0.0.10 12/16/12 02:29:49 [issue_cmd ] STATUS: 0 12/16/12 02:29:49 [issue_cmd ] RESULT: 12/16/12 02:29:49 [issue_cmd ] PING 10.0.0.10 (10.0.0.10): 56 data bytes 12/16/12 02:29:49 [issue_cmd ] 64 bytes from 10.0.0.10: icmp_seq=0 ttl=255 time=0.211 ms 12/16/12 02:29:49 [issue_cmd ] 12/16/12 02:29:49 [issue_cmd ] --- 10.0.0.10 ping statistics --- 12/16/12 02:29:49 [issue_cmd ] 1 packets transmitted, 1 packets received, 0% packet loss 12/16/12 02:29:49 [issue_cmd ] round-trip min/avg/max = 0.211/0.211/0.211 ms 12/16/12 02:29:49 [issue_cmd ] 12/16/12 02:29:49 [myexit ] VMwareresult=success 12/16/12 02:29:49 [elapsed_time ] Total time for script to complete: 0 minute(s) and 1 second(s)
Dec 16 02:30:20 m 12/16/12 02:30:19 [print_args ] PWD=/var/log/vmware/vpx 12/16/12 02:30:19 [print_args ] PS_OPTIONS= 12/16/12 02:30:19 [print_args ] FT_NO_CONSOLE_TRACE=1 12/16/12 02:30:19 [print_args ] PATH=/sbin:/usr/sbin:/bin:/usr/bin:/opt/vmware/aam/bin:/bin 12/16/12 02:30:19 [print_args ] FT_LOG_DIR=/var/log/vmware/aam 12/16/12 02:30:19 [print_args ] FT_DOMAIN=vmware 12/16/12 02:30:19 [print_args ] Parsed arguments: 12/16/12 02:30:19 [print_args ] cmd=monitornodes 12/16/12 02:30:19 [print_args ] -z=1 12/16/12 02:30:19 [print_args ] uname=VMkernel 12/16/12 02:30:19 [print_args ] shortname=typ-esx05 12/16/12 02:30:19 [print_args ] domain=vmware 12/16/12 02:30:19 [issue_cmd ] CMD: /opt/vmware/aam/bin/ft_gethostbyname typ-esx05 |grep FAILED 12/16/12 02:30:19 [issue_cmd ] STATUS: 1 12/16/12 02:30:19 [issue_cmd ] RES
Dec 16 02:30:20 ULT: 12/16/12 02:30:19 [issue_cmd ] 12/16/12 02:30:19 [list_nodes ] 12/16/12 02:30:19 [issue_cli_cmd ] command is '/opt/vmware/aam/bin/ftcli -domain vmware -port 8042 -timeout 5 -cmd listnodes' 12/16/12 02:30:20 [issue_cmd ] CMD: /opt/vmware/aam/bin/ftcli -domain vmware -port 8042 -timeout 5 -cmd listnodes 12/16/12 02:30:20 [issue_cmd ] STATUS: 0 12/16/12 02:30:20 [issue_cmd ] RESULT: 12/16/12 02:30:20 [issue_cmd ] *** Node typ-esx05 is the master primary *** 12/16/12 02:30:20 [issue_cmd ] Node Type State 12/16/12 02:30:20 [issue_cmd ] ----------------------- ------------ -------------- 12/16/12 02:30:20 [issue_cmd ] ty-esx01 Primary Agent Running 12/16/12 02:30:20 [issue_cmd ] typ-esx05 Primary Agent Running 12/16/12 02:30:20 [issue_cmd ] typ-esx06 Primary Agent Running 12/16/12
Dec 16 02:30:20 02:30:20 [issue_cmd ] VMwarenodestatus=ty-esx01,Primary,AgentRunning,typ-esx05,Primary,AgentRunning,typ-esx06,Primary,AgentRunning 12/16/12 02:30:20 [issue_cmd ] CMD: /bin/ping -c 1 10.0.0.10 12/16/12 02:30:20 [issue_cmd ] STATUS: 0 12/16/12 02:30:20 [issue_cmd ] RESULT: 12/16/12 02:30:20 [issue_cmd ] PING 10.0.0.10 (10.0.0.10): 56 data bytes 12/16/12 02:30:20 [issue_cmd ] 64 bytes from 10.0.0.10: icmp_seq=0 ttl=255 time=0.223 ms 12/16/12 02:30:20 [issue_cmd ] 12/16/12 02:30:20 [issue_cmd ] --- 10.0.0.10 ping statistics --- 12/16/12 02:30:20 [issue_cmd ] 1 packets transmitted, 1 packets received, 0% packet loss 12/16/12 02:30:20 [issue_cmd ] round-trip min/avg/max = 0.223/0.223/0.223 ms 12/16/12 02:30:20 [issue_cmd ] 12/16/12 02:30:20 [myexit ] VMwareresult=success 12/16/12 02:30:20 [elapsed_time ] Total time for script to complete: 0 minute(s) and 1 second(s)
Dec 16 02:30:51 m 12/16/12 02:30:50 [print_args ] PWD=/var/log/vmware/vpx 12/16/12 02:30:50 [print_args ] PS_OPTIONS= 12/16/12 02:30:50 [print_args ] FT_NO_CONSOLE_TRACE=1 12/16/12 02:30:50 [print_args ] PATH=/sbin:/usr/sbin:/bin:/usr/bin:/opt/vmware/aam/bin:/bin 12/16/12 02:30:50 [print_args ] FT_LOG_DIR=/var/log/vmware/aam 12/16/12 02:30:50 [print_args ] FT_DOMAIN=vmware 12/16/12 02:30:50 [print_args ] Parsed arguments: 12/16/12 02:30:50 [print_args ] cmd=monitornodes 12/16/12 02:30:50 [print_args ] -z=1 12/16/12 02:30:50 [print_args ] uname=VMkernel 12/16/12 02:30:50 [print_args ] shortname=typ-esx05 12/16/12 02:30:50 [print_args ] domain=vmware 12/16/12 02:30:50 [issue_cmd ] CMD: /opt/vmware/aam/bin/ft_gethostbyname typ-esx05 |grep FAILED 12/16/12 02:30:50 [issue_cmd ] STATUS: 1 12/16/12 02:30:50 [issue_cmd ] RES
Dec 16 02:30:51 ULT: 12/16/12 02:30:50 [issue_cmd ] 12/16/12 02:30:50 [list_nodes ] 12/16/12 02:30:50 [issue_cli_cmd ] command is '/opt/vmware/aam/bin/ftcli -domain vmware -port 8042 -timeout 5 -cmd listnodes' 12/16/12 02:30:51 [issue_cmd ] CMD: /opt/vmware/aam/bin/ftcli -domain vmware -port 8042 -timeout 5 -cmd listnodes 12/16/12 02:30:51 [issue_cmd ] STATUS: 0 12/16/12 02:30:51 [issue_cmd ] RESULT: 12/16/12 02:30:51 [issue_cmd ] *** Node typ-esx05 is the master primary *** 12/16/12 02:30:51 [issue_cmd ] Node Type State 12/16/12 02:30:51 [issue_cmd ] ----------------------- ------------ -------------- 12/16/12 02:30:51 [issue_cmd ] ty-esx01 Primary Agent Running 12/16/12 02:30:51 [issue_cmd ] typ-esx05 Primary Agent Running 12/16/12 02:30:51 [issue_cmd ] typ-esx06 Primary Agent Running 12/16/12
Dec 16 02:30:51 02:30:51 [issue_cmd ] VMwarenodestatus=ty-esx01,Primary,AgentRunning,typ-esx05,Primary,AgentRunning,typ-esx06,Primary,AgentRunning 12/16/12 02:30:51 [issue_cmd ] CMD: /bin/ping -c 1 10.0.0.10 12/16/12 02:30:51 [issue_cmd ] STATUS: 0 12/16/12 02:30:51 [issue_cmd ] RESULT: 12/16/12 02:30:51 [issue_cmd ] PING 10.0.0.10 (10.0.0.10): 56 data bytes 12/16/12 02:30:51 [issue_cmd ] 64 bytes from 10.0.0.10: icmp_seq=0 ttl=255 time=0.210 ms 12/16/12 02:30:51 [issue_cmd ] 12/16/12 02:30:51 [issue_cmd ] --- 10.0.0.10 ping statistics --- 12/16/12 02:30:51 [issue_cmd ] 1 packets transmitted, 1 packets received, 0% packet loss 12/16/12 02:30:51 [issue_cmd ] round-trip min/avg/max = 0.210/0.210/0.210 ms 12/16/12 02:30:51 [issue_cmd ] 12/16/12 02:30:51 [myexit ] VMwareresult=success 12/16/12 02:30:51 [elapsed_time ] Total time for script to complete: 0 minute(s) and 1 second(s)