VMware vSphere

 View Only

  • 1.  Is there a storage issue or not?

    Posted Feb 12, 2015 11:48 AM

    Hi

    I'm having a strange issue in my environment of which I'm not sure if it really is an issue or just cosmetic.

    Successfully restored access to volume 53e9d67e-12eabc5b-3ce2-0025b5300008 (DYN-VMFS-007-FAST-VNX-5400-04-PRI) following connectivity issues.

    Lost access to volume 53e9d67e-12eabc5b-3ce2-0025b5300008 (DYN-VMFS-007-FAST-VNX-5400-04-PRI) due to connectivity issues. Recovery attempt is in progress and outcome will be reported shortly.


    Successfully restored access to volume 53e9d67e-12eabc5b-3ce2-0025b5300008 (DYN-VMFS-007-FAST-VNX-5400-04-PRI) following connectivity issues.

    Lost access to volume 53e9d67e-12eabc5b-3ce2-0025b5300008 (DYN-VMFS-007-FAST-VNX-5400-04-PRI) due to connectivity issues. Recovery attempt is in progress and outcome will be reported shortly.


    Even on an empty vCenter / ESXi host I get these message. The lost access is between 0.1sec and 0.5sec, after that connection is restored again. This then happens on all hosts connected to the LUN at the same time. My storage and SAN admin can't see anything. Been talking with VMware Support, upgraded firmware, upgrade drivers but no change in behaviour. VMware Support can't help me anymore then tell me that the issue is outside the host.


    We have quite a number of storage arrays connected in our environment. Hosts are connected to three arrays max. The error message I have can been seen on all hosts throughout the environment. All connected in two SAN Fabrics and a number of fabrics.


    I'm not sure if I should dive in deeper, since we're not getting any performance complaints, but maybe it is a warning of bigger problems to come.


    Any tips?



  • 2.  RE: Is there a storage issue or not?



  • 3.  RE: Is there a storage issue or not?

    Posted Feb 12, 2015 02:24 PM

    Unfortunately no Brocade, but it has some pointers that will make me investigate SAN settings on our Cisco environment.



  • 4.  RE: Is there a storage issue or not?

    Posted Feb 14, 2015 12:55 AM

    One question:
    do you get these messages for all LUNs that are shared in the cluster or just for a single one ?

    If it is just for a single LUN I would immediatly start to evaucate it and rebuild it from scratch.
    If you get this for all LUNs I would try to get answers from the storage vendor.

    Anyway I would like to have a look at a few vmkernel logs to see the messages that accompany the disconnects.

    Ulli



  • 5.  RE: Is there a storage issue or not?

    Posted Feb 19, 2015 12:18 PM

    continuum wrote:

    One question:
    do you get these messages for all LUNs that are shared in the cluster or just for a single one ?

    If it is just for a single LUN I would immediatly start to evaucate it and rebuild it from scratch.
    If you get this for all LUNs I would try to get answers from the storage vendor.

    Anyway I would like to have a look at a few vmkernel logs to see the messages that accompany the disconnects.

    Ulli

    Hi Ulli,

    There are multiple LUNs having the issue on multiple hosts. But seldom all LUNs at the same time. I will start a big inventory of the problem with exact events next week. This will take a lot of time.

    See here the logs I copied from LogInsight for one host. The events you should look for are these:

    2015-02-19T10:28:02.311Z vcdvm280.mydomain.com Hostd: [271FFB90 info 'Vimsvc.ha-eventmgr'] Event 3157 : Lost access to volume 4f7428d8-6fd44884-440e-0025b500004d (VCD-VMFS-003-R5-Tier2-VNX-5500-02) due to connectivity issues. Recovery attempt is in progress and outcome will be reported shortly.

    2015-02-19T10:28:02.320Z vcdvm280.mydomain.com Hostd: [27401B90 info 'Vimsvc.ha-eventmgr'] Event 3158 : Successfully restored access to volume 4f7428d8-6fd44884-440e-0025b500004d (VCD-VMFS-003-R5-Tier2-VNX-5500-02) following connectivity issues.

    Domain name and IP addresses have been changed as far as I could find them :-)

    ***** Full log :

    2015-02-19T10:29:00Z vcdvm280.mydomain.com dhclient-uw[12559]: send_packet: errno=101, Network is unreachable

    2015-02-19T10:29:00Z vcdvm280.mydomain.com dhclient-uw[12559]: DHCPREQUEST on vmk1 to xxx.yyy.254.1 port 67

    2015-02-19T10:28:58.930Z vcdvm280.mydomain.com Vpxa: [FFFE7B90 info 'commonvpxLro' opID=4716a548-6a] [VpxLRO] -- FINISH task-internal-438347 --  -- vpxapi.VpxaService.queryBatchPerformanceStatistics --

    2015-02-19T10:28:58.930Z vcdvm280.mydomain.com Vpxa: [FFFE7B90 verbose 'vpxavpxaVmomi' opID=4716a548-6a] [VpxaVmomi] SetStatusDone was called with syncGenNo (19447)

    2015-02-19T10:28:58.844Z vcdvm280.mydomain.com Vpxa: [FFFE7B90 verbose 'vpxavpxaMoService' opID=4716a548-6a] Adding querySpec. Had=44, has=43

    2015-02-19T10:28:58.844Z vcdvm280.mydomain.com Vpxa: [FFFE7B90 verbose 'vpxavpxaStatsMetadata' opID=4716a548-6a] [TranslateVpxdMetrics] Could not translate vpxd counter 156, metric omitted.

    2015-02-19T10:28:58.843Z vcdvm280.mydomain.com Vpxa: [FFFE7B90 info 'commonvpxLro' opID=4716a548-6a] [VpxLRO] -- BEGIN task-internal-438347 --  -- vpxapi.VpxaService.queryBatchPerformanceStatistics -- 7cd92923-b655-7ea8-6ebe-b6988408ea22

    2015-02-19T10:28:48.813Z vcdvm280.mydomain.com Vpxa: [2D767B90 info 'commonvpxLro' opID=SWI-68f8c78a-60] [VpxLRO] -- FINISH task-internal-438346 --  -- vpxapi.VpxaService.fetchQuickStats --

    2015-02-19T10:28:48.812Z vcdvm280.mydomain.com Vpxa: [2D767B90 verbose 'vpxavpxaVmomi' opID=SWI-68f8c78a-60] [VpxaVmomi] SetStatusDone was called with syncGenNo (19447)

    2015-02-19T10:28:48.812Z vcdvm280.mydomain.com Vpxa: [2D767B90 info 'commonvpxLro' opID=SWI-68f8c78a-60] [VpxLRO] -- BEGIN task-internal-438346 --  -- vpxapi.VpxaService.fetchQuickStats -- 7cd92923-b655-7ea8-6ebe-b6988408ea22

    2015-02-19T10:28:48.578Z vcdvm280.mydomain.com Rhttpproxy: [FF945430 verbose 'Proxy Req 37515'] The client closed the stream, not unexpectedly.

    2015-02-19T10:28:48.574Z vcdvm280.mydomain.com Vpxa: [FFF84B90 info 'commonvpxLro' opID=HB-host-1705@19447-77e19142-ec] [VpxLRO] -- FINISH task-internal-438345 --  -- vmodl.query.PropertyCollector.Filter.destroy --

    2015-02-19T10:28:48.574Z vcdvm280.mydomain.com Vpxa: [FFF84B90 verbose 'vpxavpxaVmomi' opID=HB-host-1705@19447-77e19142-ec] [VpxaVmomi] SetStatusDone was called with syncGenNo (19447)

    2015-02-19T10:28:48.573Z vcdvm280.mydomain.com Vpxa: [FFF84B90 info 'commonvpxLro' opID=HB-host-1705@19447-77e19142-ec] [VpxLRO] -- BEGIN task-internal-438345 --  -- vmodl.query.PropertyCollector.Filter.destroy -- 7cd92923-b655-7ea8-6ebe-b6988408ea22

    2015-02-19T10:28:48.563Z vcdvm280.mydomain.com Vpxa: [2D80DB90 info 'commonvpxLro' opID=HB-host-1705@19447-77e19142-1d] [VpxLRO] -- FINISH session[7cd92923-b655-7ea8-6ebe-b6988408ea22]522f2f35-02d0-a63a-c879-8304e156e57d --  -- vpxapi.VpxaService.retrieveChanges --

    2015-02-19T10:28:48.562Z vcdvm280.mydomain.com Vpxa: [2D80DB90 verbose 'PropertyProvider' opID=HB-host-1705@19447-77e19142-1d] RecordOp ASSIGN: info.result, session[7cd92923-b655-7ea8-6ebe-b6988408ea22]522f2f35-02d0-a63a-c879-8304e156e57d

    2015-02-19T10:28:48.563Z vcdvm280.mydomain.com Vpxa: [2D80DB90 verbose 'PropertyProvider'] RecordOp ASSIGN: info.completeTime, session[7cd92923-b655-7ea8-6ebe-b6988408ea22]522f2f35-02d0-a63a-c879-8304e156e57d

    2015-02-19T10:28:48.562Z vcdvm280.mydomain.com Vpxa: [2D80DB90 verbose 'PropertyProvider' opID=HB-host-1705@19447-77e19142-1d] RecordOp ASSIGN: info.cancelable, session[7cd92923-b655-7ea8-6ebe-b6988408ea22]522f2f35-02d0-a63a-c879-8304e156e57d

    2015-02-19T10:28:48.562Z vcdvm280.mydomain.com Vpxa: [2D80DB90 verbose 'PropertyProvider' opID=HB-host-1705@19447-77e19142-1d] RecordOp ASSIGN: info, session[7cd92923-b655-7ea8-6ebe-b6988408ea22]522f2f35-02d0-a63a-c879-8304e156e57d

    2015-02-19T10:28:48.562Z vcdvm280.mydomain.com Vpxa: [2D80DB90 verbose 'vpxavpxaVmomi' opID=HB-host-1705@19447-77e19142-1d] [VpxaVmomi] SetStatusDone was called with syncGenNo (19447)

    2015-02-19T10:28:48.562Z vcdvm280.mydomain.com Vpxa: [2D80DB90 verbose 'PropertyProvider' opID=HB-host-1705@19447-77e19142-1d] RecordOp ASSIGN: info.state, session[7cd92923-b655-7ea8-6ebe-b6988408ea22]522f2f35-02d0-a63a-c879-8304e156e57d

    2015-02-19T10:28:48.562Z vcdvm280.mydomain.com Vpxa: [2D80DB90 verbose 'VpxProfiler' opID=HB-host-1705@19447-77e19142-1d] [1-] VpxaMoService::GetChanges (took 86 ms)

    2015-02-19T10:28:48.562Z vcdvm280.mydomain.com Vpxa: [2D80DB90 verbose 'VpxProfiler' opID=HB-host-1705@19447-77e19142-1d] [2-] VpxaDrs::PrepareChanges (took 0 ms)

    2015-02-19T10:28:48.562Z vcdvm280.mydomain.com Vpxa: [2D80DB90 verbose 'VpxProfiler' opID=HB-host-1705@19447-77e19142-1d] [2+] VpxaDrs::PrepareChanges

    2015-02-19T10:28:48.562Z vcdvm280.mydomain.com Vpxa: [2D80DB90 verbose 'VpxaHalResourcePool' opID=HB-host-1705@19447-77e19142-1d] NeedFullSync() = false

    2015-02-19T10:28:48.561Z vcdvm280.mydomain.com Vpxa: [2D80DB90 verbose 'VpxProfiler' opID=HB-host-1705@19447-77e19142-1d] [2-] VpxaMoService::PrepareVmChanges (took 0 ms)

    2015-02-19T10:28:48.561Z vcdvm280.mydomain.com Vpxa: [2D80DB90 verbose 'VpxProfiler' opID=HB-host-1705@19447-77e19142-1d] [2+] VpxaMoService::PrepareVmChanges

    2015-02-19T10:28:48.561Z vcdvm280.mydomain.com Vpxa: [2D80DB90 verbose 'VpxProfiler' opID=HB-host-1705@19447-77e19142-1d] [2-] VpxaMoService::PrepareHostChanges (took 35 ms)

    2015-02-19T10:28:48.561Z vcdvm280.mydomain.com Vpxa: [2D80DB90 verbose 'vpxavpxaMoService' opID=HB-host-1705@19447-77e19142-1d] [VpxaMoService] host changes include changed properties []

    2015-02-19T10:28:48.547Z vcdvm280.mydomain.com Hostd: [270BBB90 verbose 'Default' opID=HB-host-1705@19447-77e19142-1d] ha-license-manager:Validate -> Valid license detected for "VMware ESX Server 5.0" (lastError=0, desc.IsValid:Yes)

    2015-02-19T10:28:48.528Z vcdvm280.mydomain.com Hostd: [270BBB90 verbose 'Vimsvc.ha-license-manager' opID=HB-host-1705@19447-77e19142-1d] Load: Loading existing file: /etc/vmware/license.cfg

    2015-02-19T10:28:48.526Z vcdvm280.mydomain.com Vpxa: [2D80DB90 verbose 'VpxProfiler' opID=HB-host-1705@19447-77e19142-1d] [2+] VpxaMoService::PrepareHostChanges

    2015-02-19T10:28:48.526Z vcdvm280.mydomain.com Vpxa: [2D80DB90 verbose 'VpxProfiler' opID=HB-host-1705@19447-77e19142-1d] [2-] VpxdDatastore::PrepareChanges (took 0 ms)

    2015-02-19T10:28:48.526Z vcdvm280.mydomain.com Vpxa: [2D80DB90 verbose 'VpxProfiler' opID=HB-host-1705@19447-77e19142-1d] [2+] VpxdDatastore::PrepareChanges

    2015-02-19T10:28:48.526Z vcdvm280.mydomain.com Vpxa: [2D80DB90 verbose 'VpxProfiler' opID=HB-host-1705@19447-77e19142-1d] [2-] VpxaStatsMetadata::PrepareStatsChanges (took 50 ms)

    2015-02-19T10:28:48.525Z vcdvm280.mydomain.com Vpxa: [2D80DB90 verbose 'vpxavpxaStatsMetadata' opID=HB-host-1705@19447-77e19142-1d] [VpxaStatsMetadata] Refilled translations (344, 344)

    2015-02-19T10:28:48.524Z vcdvm280.mydomain.com Vpxa: [2D80DB90 verbose 'vpxavpxaStatsMetadata' opID=HB-host-1705@19447-77e19142-1d] [VpxaStatsMetadata] Prior to refilling translations (344, 344)

    2015-02-19T10:28:48.488Z vcdvm280.mydomain.com Hostd: [2713CB90 verbose 'Locale' opID=HB-host-1705@19447-77e19142-1d] Default resource used for 'counter.sys.resourceMemConsumed.summary' expected in module 'perf'.

    2015-02-19T10:28:48.476Z vcdvm280.mydomain.com Vpxa: [2D7CBB90 info 'commonvpxLro' opID=HB-host-1705@19447-77e19142-1d-SWI-fbd4684d] [VpxLroList::ForgetTask] Unregistering vim.Task:session[7cd92923-b655-7ea8-6ebe-b6988408ea22]5292e7c0-14f9-18d4-086a-fa4040f4c44d

    2015-02-19T10:28:48.475Z vcdvm280.mydomain.com Vpxa: [2D80DB90 verbose 'VpxProfiler' opID=HB-host-1705@19447-77e19142-1d] [2+] VpxaStatsMetadata::PrepareStatsChanges

    2015-02-19T10:28:48.475Z vcdvm280.mydomain.com Vpxa: [2D7CBB90 verbose 'commonvpxLro' opID=HB-host-1705@19447-77e19142-1d-SWI-fbd4684d] [VpxLroList::ForgetTask] Removed vim.Task:session[7cd92923-b655-7ea8-6ebe-b6988408ea22]5292e7c0-14f9-18d4-086a-fa4040f4c44d from _completed

    2015-02-19T10:28:48.475Z vcdvm280.mydomain.com Vpxa: [2D80DB90 info 'commonvpxLro' opID=HB-host-1705@19447-77e19142-1d] [VpxLRO] -- BEGIN session[7cd92923-b655-7ea8-6ebe-b6988408ea22]522f2f35-02d0-a63a-c879-8304e156e57d --  -- vpxapi.VpxaService.retrieveChanges -- 7cd92923-b655-7ea8-6ebe-b6988408ea22

    2015-02-19T10:28:48.475Z vcdvm280.mydomain.com Vpxa: [2D80DB90 verbose 'vpxavpxaMoService' opID=HB-host-1705@19447-77e19142-1d] [VpxaMoService] GetChanges: 19446 -> 19447

    2015-02-19T10:28:48.475Z vcdvm280.mydomain.com Vpxa: [2D80DB90 verbose 'VpxProfiler' opID=HB-host-1705@19447-77e19142-1d] [1+] VpxaMoService::GetChanges

    2015-02-19T10:28:48.475Z vcdvm280.mydomain.com Vpxa: [2D80DB90 verbose 'PropertyProvider' opID=HB-host-1705@19447-77e19142-1d] RecordOp ASSIGN: info.cancelable, session[7cd92923-b655-7ea8-6ebe-b6988408ea22]522f2f35-02d0-a63a-c879-8304e156e57d

    2015-02-19T10:28:48.474Z vcdvm280.mydomain.com Vpxa: [2D80DB90 verbose 'PropertyProvider' opID=HB-host-1705@19447-77e19142-1d] RecordOp ASSIGN: info.state, session[7cd92923-b655-7ea8-6ebe-b6988408ea22]522f2f35-02d0-a63a-c879-8304e156e57d

    2015-02-19T10:28:48.474Z vcdvm280.mydomain.com Vpxa: [2D80DB90 verbose 'PropertyProvider'] RecordOp ASSIGN: info.startTime, session[7cd92923-b655-7ea8-6ebe-b6988408ea22]522f2f35-02d0-a63a-c879-8304e156e57d

    2015-02-19T10:28:48.474Z vcdvm280.mydomain.com Vpxa: [FFFA5B90 verbose 'PropertyProvider' opID=HB-host-1705@19447-77e19142-1d] RecordOp ASSIGN: info.reason, session[7cd92923-b655-7ea8-6ebe-b6988408ea22]522f2f35-02d0-a63a-c879-8304e156e57d

    2015-02-19T10:28:48.472Z vcdvm280.mydomain.com Rhttpproxy: [6FEC2B90 verbose 'Proxy Req 37515'] Connected to localhost:8089

    2015-02-19T10:28:48.470Z vcdvm280.mydomain.com Rhttpproxy: [FFE40B90 verbose 'Proxy Req 37515'] New proxy client SSL(TCP(local=xxx.yyy.162.180:443, peer=xxx.yyy.201.33:59850))

    2015-02-19T10:28:48.452Z vcdvm280.mydomain.com Vpxa: [FFB00B90 verbose 'SoapAdapter'] Responded to service state request

    2015-02-19T10:28:48Z vcdvm280.mydomain.com dhclient-uw[12559]: send_packet: errno=101, Network is unreachable

    2015-02-19T10:28:48Z vcdvm280.mydomain.com dhclient-uw[12559]: DHCPREQUEST on vmk1 to xxx.yyy.254.1 port 67

    2015-02-19T10:28:47.205Z vcdvm280.mydomain.com storageRM:  Skipping device naa.6006016040002f00ccea56bdc257e111 either due to VSI read error or abnormal state

    2015-02-19T10:28:44.451Z vcdvm280.mydomain.com vmkernel: cpu0:8752)<7>fnic : 1 :: fnic_rport_exch_reset called portid 0x321400

    2015-02-19T10:28:44.353Z vcdvm280.mydomain.com vmkernel: cpu6:8748)<7>fnic : 1 :: fnic_rport_exch_reset called portid 0x321400

    2015-02-19T10:28:44.238Z vcdvm280.mydomain.com vmkernel: cpu12:8754)<7>fnic : 1 :: fnic_rport_exch_reset called portid 0x321400

    2015-02-19T10:28:44.123Z vcdvm280.mydomain.com vmkernel: cpu2:8762)<7>fnic : 1 :: fnic_rport_exch_reset called portid 0x321400

    2015-02-19T10:28:44.008Z vcdvm280.mydomain.com vmkernel: cpu6:8761)<7>fnic : 1 :: fnic_rport_exch_reset called portid 0x321400

    2015-02-19T10:28:43.890Z vcdvm280.mydomain.com vmkernel: cpu12:8751)<7>fnic : 1 :: fnic_rport_exch_reset called portid 0x321400

    2015-02-19T10:28:43.775Z vcdvm280.mydomain.com vmkernel: cpu14:8743)<7>fnic : 1 :: fnic_rport_exch_reset called portid 0x321400

    2015-02-19T10:28:43.656Z vcdvm280.mydomain.com vmkernel: cpu2:8756)<7>fnic : 1 :: fnic_rport_exch_reset called portid 0x321400

    2015-02-19T10:28:43.540Z vcdvm280.mydomain.com vmkernel: cpu4:8753)<7>fnic : 1 :: fnic_rport_exch_reset called portid 0x321400

    2015-02-19T10:28:43.410Z vcdvm280.mydomain.com vmkernel: cpu2:8750)<7>fnic : 1 :: fnic_rport_exch_reset called portid 0x321400

    2015-02-19T10:28:43.295Z vcdvm280.mydomain.com vmkernel: cpu4:8747)<7>fnic : 1 :: fnic_rport_exch_reset called portid 0x321400

    2015-02-19T10:28:43.179Z vcdvm280.mydomain.com vmkernel: cpu12:8759)<7>fnic : 1 :: fnic_rport_exch_reset called portid 0x321400

    2015-02-19T10:28:43.064Z vcdvm280.mydomain.com vmkernel: cpu2:8766)<7>fnic : 1 :: fnic_rport_exch_reset called portid 0x321400

    2015-02-19T10:28:42.948Z vcdvm280.mydomain.com vmkernel: cpu18:8744)<7>fnic : 1 :: fnic_rport_exch_reset called portid 0x321400

    2015-02-19T10:28:42.829Z vcdvm280.mydomain.com vmkernel: cpu0:8745)<7>fnic : 1 :: fnic_rport_exch_reset called portid 0x321400

    2015-02-19T10:28:42.714Z vcdvm280.mydomain.com vmkernel: cpu0:8763)<7>fnic : 1 :: fnic_rport_exch_reset called portid 0x321400

    2015-02-19T10:28:42.598Z vcdvm280.mydomain.com vmkernel: cpu12:8754)<7>fnic : 1 :: fnic_rport_exch_reset called portid 0x321400

    2015-02-19T10:28:42.467Z vcdvm280.mydomain.com vmkernel: cpu3:8760)<7>fnic : 1 :: fnic_rport_exch_reset called portid 0x321400

    2015-02-19T10:28:42.352Z vcdvm280.mydomain.com vmkernel: cpu0:8757)<7>fnic : 1 :: fnic_rport_exch_reset called portid 0x321400

    2015-02-19T10:28:42.236Z vcdvm280.mydomain.com vmkernel: cpu4:8764)<7>fnic : 1 :: fnic_rport_exch_reset called portid 0x321400

    2015-02-19T10:28:42.122Z vcdvm280.mydomain.com vmkernel: cpu12:8751)<7>fnic : 1 :: fnic_rport_exch_reset called portid 0x321400

    2015-02-19T10:28:42.006Z vcdvm280.mydomain.com vmkernel: cpu4:8746)<7>fnic : 1 :: fnic_rport_exch_reset called portid 0x321400

    2015-02-19T10:28:41.889Z vcdvm280.mydomain.com vmkernel: cpu6:8748)<7>fnic : 1 :: fnic_rport_exch_reset called portid 0x321400

    2015-02-19T10:28:41.774Z vcdvm280.mydomain.com vmkernel: cpu4:8758)<7>fnic : 1 :: fnic_rport_exch_reset called portid 0x321400

    2015-02-19T10:28:41.719Z vcdvm280.mydomain.com vmkernel: cpu6:8373)WARNING: vmw_psp_rr: psp_rrSelectPathToActivate:1049:Could not select path for device "naa.6006016040002f00ccea56bdc257e111".

    2015-02-19T10:28:41.719Z vcdvm280.mydomain.com vmkwarning: cpu6:8373)WARNING: vmw_psp_rr: psp_rrSelectPathToActivate:1049:Could not select path for device "naa.6006016040002f00ccea56bdc257e111".

    2015-02-19T10:28:41.659Z vcdvm280.mydomain.com vmkernel: cpu4:8747)<7>fnic : 1 :: fnic_rport_exch_reset called portid 0x321400

    2015-02-19T10:28:41.527Z vcdvm280.mydomain.com vmkernel: cpu0:8757)<7>fnic : 1 :: fnic_rport_exch_reset called portid 0x321400

    2015-02-19T10:28:41.397Z vcdvm280.mydomain.com vmkernel: cpu4:8764)<7>fnic : 1 :: fnic_rport_exch_reset called portid 0x321400

    2015-02-19T10:28:41.282Z vcdvm280.mydomain.com vmkernel: cpu12:8751)<7>fnic : 1 :: fnic_rport_exch_reset called portid 0x321400

    2015-02-19T10:28:41.166Z vcdvm280.mydomain.com vmkernel: cpu0:8756)<7>fnic : 1 :: fnic_rport_exch_reset called portid 0x321400

    2015-02-19T10:28:41.050Z vcdvm280.mydomain.com vmkernel: cpu0:8763)<7>fnic : 1 :: fnic_rport_exch_reset called portid 0x321400

    2015-02-19T10:28:41Z vcdvm280.mydomain.com dhclient-uw[12559]: send_packet: errno=101, Network is unreachable

    2015-02-19T10:28:41Z vcdvm280.mydomain.com dhclient-uw[12559]: DHCPREQUEST on vmk1 to xxx.yyy.254.1 port 67

    2015-02-19T10:28:40.932Z vcdvm280.mydomain.com vmkernel: cpu2:8750)<7>fnic : 1 :: fnic_rport_exch_reset called portid 0x321400

    2015-02-19T10:28:40.817Z vcdvm280.mydomain.com vmkernel: cpu3:8760)<7>fnic : 1 :: fnic_rport_exch_reset called portid 0x321400

    2015-02-19T10:28:40.701Z vcdvm280.mydomain.com vmkernel: cpu12:8759)<7>fnic : 1 :: fnic_rport_exch_reset called portid 0x321400

    2015-02-19T10:28:40.567Z vcdvm280.mydomain.com vmkernel: cpu2:8766)<7>fnic : 1 :: fnic_rport_exch_reset called portid 0x321400

    2015-02-19T10:28:40.436Z vcdvm280.mydomain.com vmkernel: cpu18:8744)<7>fnic : 1 :: fnic_rport_exch_reset called portid 0x321400

    2015-02-19T10:28:40.321Z vcdvm280.mydomain.com vmkernel: cpu4:8746)<7>fnic : 1 :: fnic_rport_exch_reset called portid 0x321400

    2015-02-19T10:28:40.206Z vcdvm280.mydomain.com vmkernel: cpu0:8752)<7>fnic : 1 :: fnic_rport_exch_reset called portid 0x321400

    2015-02-19T10:28:40.091Z vcdvm280.mydomain.com vmkernel: cpu4:8758)<7>fnic : 1 :: fnic_rport_exch_reset called portid 0x321400

    2015-02-19T10:28:39.956Z vcdvm280.mydomain.com vmkernel: cpu1:8765)<7>fnic : 1 :: fnic_rport_exch_reset called portid 0x321400

    2015-02-19T10:28:39.839Z vcdvm280.mydomain.com vmkernel: cpu6:8761)<7>fnic : 1 :: fnic_rport_exch_reset called portid 0x321400

    2015-02-19T10:28:39.724Z vcdvm280.mydomain.com vmkernel: cpu3:8749)<7>fnic : 1 :: fnic_rport_exch_reset called portid 0x321400

    2015-02-19T10:28:39.605Z vcdvm280.mydomain.com vmkernel: cpu7:8755)<7>fnic : 1 :: fnic_rport_exch_reset called portid 0x321400

    2015-02-19T10:28:39.490Z vcdvm280.mydomain.com vmkernel: cpu0:8756)<7>fnic : 1 :: fnic_rport_exch_reset called portid 0x321400

    2015-02-19T10:28:39.350Z vcdvm280.mydomain.com vmkernel: cpu6:8748)<7>fnic : 1 :: fnic_rport_exch_reset called portid 0x321400

    2015-02-19T10:28:39.234Z vcdvm280.mydomain.com vmkernel: cpu4:8753)<7>fnic : 1 :: fnic_rport_exch_reset called portid 0x321400

    2015-02-19T10:28:39.119Z vcdvm280.mydomain.com vmkernel: cpu12:8754)<7>fnic : 1 :: fnic_rport_exch_reset called portid 0x321400

    2015-02-19T10:28:39.065Z vcdvm280.mydomain.com Vpxa: [2D788B90 verbose 'vpxavpxaInvtHost'] [VpxaInvtHost] Increment master gen. no to (19447): Event:VpxaHalEvent::CheckQueuedEvents

    2015-02-19T10:28:39.003Z vcdvm280.mydomain.com vmkernel: cpu2:8762)<7>fnic : 1 :: fnic_rport_exch_reset called portid 0x321400

    2015-02-19T10:28:38.887Z vcdvm280.mydomain.com vmkernel: cpu4:8764)<7>fnic : 1 :: fnic_rport_exch_reset called portid 0x321400

    2015-02-19T10:28:38.771Z vcdvm280.mydomain.com vmkernel: cpu12:8751)<7>fnic : 1 :: fnic_rport_exch_reset called portid 0x321400

    2015-02-19T10:28:38.655Z vcdvm280.mydomain.com vmkernel: cpu14:8743)<7>fnic : 1 :: fnic_rport_exch_reset called portid 0x321400

    2015-02-19T10:28:38.539Z vcdvm280.mydomain.com vmkernel: cpu0:8752)<7>fnic : 1 :: fnic_rport_exch_reset called portid 0x321400

    2015-02-19T10:28:38.408Z vcdvm280.mydomain.com vmkernel: cpu2:8750)<7>fnic : 1 :: fnic_rport_exch_reset called portid 0x321400

    2015-02-19T10:28:38.293Z vcdvm280.mydomain.com vmkernel: cpu4:8747)<7>fnic : 1 :: fnic_rport_exch_reset called portid 0x321400

    2015-02-19T10:28:38.177Z vcdvm280.mydomain.com vmkernel: cpu3:8760)<7>fnic : 1 :: fnic_rport_exch_reset called portid 0x321400

    2015-02-19T10:28:38.062Z vcdvm280.mydomain.com vmkernel: cpu0:8757)<7>fnic : 1 :: fnic_rport_exch_reset called portid 0x321400

    2015-02-19T10:28:37.946Z vcdvm280.mydomain.com vmkernel: cpu2:8766)<7>fnic : 1 :: fnic_rport_exch_reset called portid 0x321400

    2015-02-19T10:28:37.830Z vcdvm280.mydomain.com vmkernel: cpu4:8746)<7>fnic : 1 :: fnic_rport_exch_reset called portid 0x321400

    2015-02-19T10:28:37.714Z vcdvm280.mydomain.com vmkernel: cpu0:8745)<7>fnic : 1 :: fnic_rport_exch_reset called portid 0x321400

    2015-02-19T10:28:37.598Z vcdvm280.mydomain.com vmkernel: cpu0:8763)<7>fnic : 1 :: fnic_rport_exch_reset called portid 0x321400

    2015-02-19T10:28:37.483Z vcdvm280.mydomain.com vmkernel: cpu12:8754)<7>fnic : 1 :: fnic_rport_exch_reset called portid 0x321400

    2015-02-19T10:28:37.353Z vcdvm280.mydomain.com vmkernel: cpu12:8759)<7>fnic : 1 :: fnic_rport_exch_reset called portid 0x321400

    2015-02-19T10:28:37.237Z vcdvm280.mydomain.com vmkernel: cpu6:8761)<7>fnic : 1 :: fnic_rport_exch_reset called portid 0x321400

    2015-02-19T10:28:37.122Z vcdvm280.mydomain.com vmkernel: cpu4:8764)<7>fnic : 1 :: fnic_rport_exch_reset called portid 0x321400

    2015-02-19T10:28:37.072Z vcdvm280.mydomain.com Hostd: [2717DB90 verbose 'SoapAdapter'] Responded to service state request

    2015-02-19T10:28:37.006Z vcdvm280.mydomain.com vmkernel: cpu18:8744)<7>fnic : 1 :: fnic_rport_exch_reset called portid 0x321400

    2015-02-19T10:28:36.890Z vcdvm280.mydomain.com vmkernel: cpu0:8756)<7>fnic : 1 :: fnic_rport_exch_reset called portid 0x321400

    2015-02-19T10:28:36.775Z vcdvm280.mydomain.com vmkernel: cpu4:8753)<7>fnic : 1 :: fnic_rport_exch_reset called portid 0x321400

    2015-02-19T10:28:36.737Z vcdvm280.mydomain.com Vpxa: [FFFA5B90 verbose 'VpxaHalCnxHostagent' opID=WFU-792cfedd] [WaitForUpdatesDone] Completed callback

    2015-02-19T10:28:36.737Z vcdvm280.mydomain.com Vpxa: [FFFA5B90 verbose 'VpxaHalCnxHostagent' opID=WFU-792cfedd] [WaitForUpdatesDone] Starting next WaitForUpdates() call to hostd

    2015-02-19T10:28:36.737Z vcdvm280.mydomain.com Vpxa: [FFFA5B90 verbose 'vpxavpxaTaskInfo' opID=WFU-792cfedd] [VpxaTaskInfoPublisher::PropertyChanged] Number of deferred task updates: 0

    2015-02-19T10:28:36.736Z vcdvm280.mydomain.com Vpxa: [FFFA5B90 verbose 'VpxaHalCnxHostagent' opID=WFU-792cfedd] [VpxaHalCnxHostagent::ProcessUpdate] Applying updates from 230230 to 230231 (at 230230)

    2015-02-19T10:28:36.736Z vcdvm280.mydomain.com Vpxa: [FFFA5B90 verbose 'VpxaHalCnxHostagent' opID=WFU-792cfedd] [WaitForUpdatesDone] Received callback

    2015-02-19T10:28:36.659Z vcdvm280.mydomain.com vmkernel: cpu4:8747)<7>fnic : 1 :: fnic_rport_exch_reset called portid 0x321400

    2015-02-19T10:28:36.547Z vcdvm280.mydomain.com vmkernel: cpu3:8760)<7>fnic : 1 :: fnic_rport_exch_reset called portid 0x321400

    2015-02-19T10:28:36.449Z vcdvm280.mydomain.com vmkernel: cpu3:8749)<7>fnic : 1 :: fnic_rport_exch_reset called portid 0x321400

    2015-02-19T10:28:36.346Z vcdvm280.mydomain.com vmkernel: cpu7:8755)<7>fnic : 1 :: fnic_rport_exch_reset called portid 0x321400

    2015-02-19T10:28:36.258Z vcdvm280.mydomain.com vmkernel: cpu12:8751)<7>fnic : 1 :: fnic_rport_exch_reset called portid 0x321400

    2015-02-19T10:28:36.170Z vcdvm280.mydomain.com vmkernel: cpu4:8746)<7>fnic : 1 :: fnic_rport_exch_reset called portid 0x321400

    2015-02-19T10:28:36.082Z vcdvm280.mydomain.com vmkernel: cpu0:8752)<7>fnic : 1 :: fnic_rport_exch_reset called portid 0x321400

    2015-02-19T10:28:35.994Z vcdvm280.mydomain.com vmkernel: cpu2:8750)<7>fnic : 1 :: fnic_rport_exch_reset called portid 0x321400

    2015-02-19T10:28:35.905Z vcdvm280.mydomain.com vmkernel: cpu12:8754)<7>fnic : 1 :: fnic_rport_exch_reset called portid 0x321400

    2015-02-19T10:28:35.816Z vcdvm280.mydomain.com vmkernel: cpu2:8762)<7>fnic : 1 :: fnic_rport_exch_reset called portid 0x321400

    2015-02-19T10:28:35.728Z vcdvm280.mydomain.com vmkernel: cpu0:8757)<7>fnic : 1 :: fnic_rport_exch_reset called portid 0x321400

    2015-02-19T10:28:35.638Z vcdvm280.mydomain.com vmkernel: cpu2:8766)<7>fnic : 1 :: fnic_rport_exch_reset called portid 0x321400

    2015-02-19T10:28:35.550Z vcdvm280.mydomain.com vmkernel: cpu6:8748)<7>fnic : 1 :: fnic_rport_exch_reset called portid 0x321400

    2015-02-19T10:28:35.461Z vcdvm280.mydomain.com vmkernel: cpu0:8745)<7>fnic : 1 :: fnic_rport_exch_reset called portid 0x321400

    2015-02-19T10:28:35.342Z vcdvm280.mydomain.com vmkernel: cpu0:8763)<7>fnic : 1 :: fnic_rport_exch_reset called portid 0x321400

    2015-02-19T10:28:35.254Z vcdvm280.mydomain.com vmkernel: cpu4:8747)<7>fnic : 1 :: fnic_rport_exch_reset called portid 0x321400

    2015-02-19T10:28:35.166Z vcdvm280.mydomain.com vmkernel: cpu12:8759)<7>fnic : 1 :: fnic_rport_exch_reset called portid 0x321400

    2015-02-19T10:28:35.079Z vcdvm280.mydomain.com vmkernel: cpu3:8749)<7>fnic : 1 :: fnic_rport_exch_reset called portid 0x321400

    2015-02-19T10:28:34.991Z vcdvm280.mydomain.com vmkernel: cpu7:8755)<7>fnic : 1 :: fnic_rport_exch_reset called portid 0x321400

    2015-02-19T10:28:34.899Z vcdvm280.mydomain.com vmkernel: cpu12:8751)<7>fnic : 1 :: fnic_rport_exch_reset called portid 0x321400

    2015-02-19T10:28:34.810Z vcdvm280.mydomain.com vmkernel: cpu0:8756)<7>fnic : 1 :: fnic_rport_exch_reset called portid 0x321400

    2015-02-19T10:28:34.722Z vcdvm280.mydomain.com vmkernel: cpu0:8752)<7>fnic : 1 :: fnic_rport_exch_reset called portid 0x321400

    2015-02-19T10:28:34.624Z vcdvm280.mydomain.com vmkernel: cpu2:8750)<7>fnic : 1 :: fnic_rport_exch_reset called portid 0x321400

    2015-02-19T10:28:34.528Z vcdvm280.mydomain.com vmkernel: cpu3:8760)<7>fnic : 1 :: fnic_rport_exch_reset called portid 0x321400

    2015-02-19T10:28:34.445Z vcdvm280.mydomain.com Hostd: [27380B90 verbose 'SoapAdapter'] Responded to service state request

    2015-02-19T10:28:34.440Z vcdvm280.mydomain.com vmkernel: cpu2:8762)<7>fnic : 1 :: fnic_rport_exch_reset called portid 0x321400

    2015-02-19T10:28:34.321Z vcdvm280.mydomain.com vmkernel: cpu0:8757)<7>fnic : 1 :: fnic_rport_exch_reset called portid 0x321400

    2015-02-19T10:28:34.234Z vcdvm280.mydomain.com vmkernel: cpu2:8766)<7>fnic : 1 :: fnic_rport_exch_reset called portid 0x321400

    2015-02-19T10:28:34.146Z vcdvm280.mydomain.com vmkernel: cpu4:8746)<7>fnic : 1 :: fnic_rport_exch_reset called portid 0x321400

    2015-02-19T10:28:34.058Z vcdvm280.mydomain.com vmkernel: cpu0:8745)<7>fnic : 1 :: fnic_rport_exch_reset called portid 0x321400

    2015-02-19T10:28:34Z vcdvm280.mydomain.com dhclient-uw[12559]: send_packet: errno=101, Network is unreachable

    2015-02-19T10:28:34Z vcdvm280.mydomain.com dhclient-uw[12559]: DHCPREQUEST on vmk1 to xxx.yyy.254.1 port 67

    2015-02-19T10:28:33.970Z vcdvm280.mydomain.com vmkernel: cpu0:8763)<7>fnic : 1 :: fnic_rport_exch_reset called portid 0x321400

    2015-02-19T10:28:33.889Z vcdvm280.mydomain.com vmkernel: cpu4:8747)<7>fnic : 1 :: fnic_rport_exch_reset called portid 0x321400

    2015-02-19T10:28:33.548Z vcdvm280.mydomain.com Vpxa: [FFB00B90 verbose 'VpxaHalResourcePool' opID=SWI-c4e634b2] [ProcessResourceNotification] No syncs pending, exiting loop...

    2015-02-19T10:28:33.548Z vcdvm280.mydomain.com Vpxa: [FFB00B90 verbose 'VpxaHalResourcePool' opID=SWI-c4e634b2] [VpxaHostdSpecSync] Copied root values ...

    2015-02-19T10:28:33.545Z vcdvm280.mydomain.com Vpxa: [FFF63B90 verbose 'VpxaHalCnxHostagent' opID=WFU-6f28ac1c] [WaitForUpdatesDone] Completed callback

    2015-02-19T10:28:33.544Z vcdvm280.mydomain.com Vpxa: [FFF63B90 verbose 'VpxaHalCnxHostagent' opID=WFU-6f28ac1c] [WaitForUpdatesDone] Starting next WaitForUpdates() call to hostd

    2015-02-19T10:28:33.544Z vcdvm280.mydomain.com Vpxa: [FFB00B90 info 'VpxaHalResourcePool' opID=SWI-c4e634b2] GetOverrideFailureCount() = 0MAX_CONSECUTIVE_OVERRIDE_FAILURES = 3

    2015-02-19T10:28:33.544Z vcdvm280.mydomain.com Vpxa: [FFF63B90 verbose 'VpxaHalResourcePool' opID=WFU-6f28ac1c] [VpxaHostdSpecSync] Launching ProcessResourceNotification in new thread...

    2015-02-19T10:28:33.544Z vcdvm280.mydomain.com Vpxa: [FFF63B90 verbose 'VpxaHalResourcePool' opID=WFU-6f28ac1c] [VpxaHostdSpecSync] Setting _syncPending = true ...

    2015-02-19T10:28:33.544Z vcdvm280.mydomain.com Vpxa: [FFF63B90 verbose 'VpxaHalResourcePool' opID=WFU-6f28ac1c] [VpxaHostdSpecSync] [IsVmListChangeUpdate] returning false

    2015-02-19T10:28:33.543Z vcdvm280.mydomain.com Vpxa: [FFF63B90 verbose 'VpxaHalResourcePool' opID=WFU-6f28ac1c] [VpxaHostdSpecSync] Received change notification from hostd ...

    2015-02-19T10:28:33.543Z vcdvm280.mydomain.com Vpxa: [FFF63B90 verbose 'VpxaHalCnxHostagent' opID=WFU-6f28ac1c] [VpxaHalCnxHostagent::ProcessUpdate] Applying updates from 230229 to 230230 (at 230229)

    2015-02-19T10:28:33.543Z vcdvm280.mydomain.com Vpxa: [FFF63B90 verbose 'VpxaHalCnxHostagent' opID=WFU-6f28ac1c] [WaitForUpdatesDone] Received callback

    2015-02-19T10:28:33.543Z vcdvm280.mydomain.com Vpxa: [2D7CBB90 verbose 'VpxaHalResourcePool' opID=SWI-e9f8c37b] [ProcessResourceNotification] No syncs pending, exiting loop...

    2015-02-19T10:28:33.542Z vcdvm280.mydomain.com Vpxa: [2D7CBB90 verbose 'VpxaHalResourcePool' opID=SWI-e9f8c37b] [VpxaHostdSpecSync] Copied root values ...

    2015-02-19T10:28:33.538Z vcdvm280.mydomain.com Vpxa: [FFF42B90 verbose 'VpxaHalCnxHostagent' opID=WFU-ec6cc86b] [WaitForUpdatesDone] Completed callback

    2015-02-19T10:28:33.538Z vcdvm280.mydomain.com Vpxa: [2D7CBB90 info 'VpxaHalResourcePool' opID=SWI-e9f8c37b] GetOverrideFailureCount() = 0MAX_CONSECUTIVE_OVERRIDE_FAILURES = 3

    2015-02-19T10:28:33.538Z vcdvm280.mydomain.com Vpxa: [FFF42B90 verbose 'VpxaHalCnxHostagent' opID=WFU-ec6cc86b] [WaitForUpdatesDone] Starting next WaitForUpdates() call to hostd

    2015-02-19T10:28:33.538Z vcdvm280.mydomain.com Vpxa: [FFF42B90 verbose 'VpxaHalResourcePool' opID=WFU-ec6cc86b] [VpxaHostdSpecSync] Launching ProcessResourceNotification in new thread...

    2015-02-19T10:28:33.538Z vcdvm280.mydomain.com Vpxa: [FFF42B90 verbose 'VpxaHalResourcePool' opID=WFU-ec6cc86b] [VpxaHostdSpecSync] Setting _syncPending = true ...

    2015-02-19T10:28:33.538Z vcdvm280.mydomain.com Vpxa: [FFF42B90 verbose 'VpxaHalResourcePool' opID=WFU-ec6cc86b] [VpxaHostdSpecSync] Received change notification from hostd ...

    2015-02-19T10:28:33.538Z vcdvm280.mydomain.com Vpxa: [FFF42B90 verbose 'VpxaHalResourcePool' opID=WFU-ec6cc86b] [VpxaHostdSpecSync] [IsVmListChangeUpdate] returning false

    2015-02-19T10:28:33.538Z vcdvm280.mydomain.com Vpxa: [FFF42B90 verbose 'VpxaHalCnxHostagent' opID=WFU-ec6cc86b] [WaitForUpdatesDone] Received callback

    2015-02-19T10:28:33.538Z vcdvm280.mydomain.com Vpxa: [FFF42B90 verbose 'VpxaHalCnxHostagent' opID=WFU-ec6cc86b] [VpxaHalCnxHostagent::ProcessUpdate] Applying updates from 230228 to 230229 (at 230228)

    2015-02-19T10:28:33.535Z vcdvm280.mydomain.com Hostd: [275C5B90 verbose 'Hostsvc.ResourcePool ha-root-pool'] Root pool capacity changed from 22596MHz/253021MB to 22596MHz/253020MB

    2015-02-19T10:28:33.490Z vcdvm280.mydomain.com Vpxa: [2D82EB90 verbose 'VpxaHalCnxHostagent' opID=WFU-428c63] [WaitForUpdatesDone] Completed callback

    2015-02-19T10:28:33.490Z vcdvm280.mydomain.com Vpxa: [2D82EB90 verbose 'VpxaHalCnxHostagent' opID=WFU-428c63] [WaitForUpdatesDone] Starting next WaitForUpdates() call to hostd

    2015-02-19T10:28:33.489Z vcdvm280.mydomain.com Vpxa: [2D82EB90 verbose 'vpxavpxaInvtHost' opID=WFU-428c63] [HostChanged] Found update for tracked MoRef vim.HostSystem:ha-host

    2015-02-19T10:28:33.489Z vcdvm280.mydomain.com Vpxa: [2D82EB90 verbose 'halservices' opID=WFU-428c63] [VpxaHalServices] HostChanged Event Fired, properties changed [runtime.healthSystemRuntime]

    2015-02-19T10:28:33.488Z vcdvm280.mydomain.com Vpxa: [2D82EB90 verbose 'VpxaHalCnxHostagent' opID=WFU-428c63] [VpxaHalCnxHostagent::ProcessUpdate] Applying updates from 230227 to 230228 (at 230227)

    2015-02-19T10:28:33.488Z vcdvm280.mydomain.com Vpxa: [2D82EB90 verbose 'VpxaHalCnxHostagent' opID=WFU-428c63] [WaitForUpdatesDone] Received callback

    2015-02-19T10:28:31.762Z vcdvm280.mydomain.com Hostd: [FFBFBD20 verbose 'Cimsvc'] Ticket issued for CIMOM version 1.0, user root

    2015-02-19T10:28:27.194Z vcdvm280.mydomain.com storageRM:  Skipping device naa.6006016040002f00ccea56bdc257e111 either due to VSI read error or abnormal state

    2015-02-19T10:28:27Z vcdvm280.mydomain.com dhclient-uw[12559]: send_packet: errno=101, Network is unreachable

    2015-02-19T10:28:27Z vcdvm280.mydomain.com dhclient-uw[12559]: DHCPREQUEST on vmk1 to xxx.yyy.254.1 port 67

    2015-02-19T10:28:18.075Z vcdvm280.mydomain.com Vpxa: [2D788B90 verbose 'VpxaHalCnxHostagent' opID=WFU-c439f5c1] [WaitForUpdatesDone] Completed callback

    2015-02-19T10:28:18.075Z vcdvm280.mydomain.com Vpxa: [2D788B90 verbose 'VpxaHalCnxHostagent' opID=WFU-c439f5c1] [WaitForUpdatesDone] Starting next WaitForUpdates() call to hostd

    2015-02-19T10:28:18.075Z vcdvm280.mydomain.com Vpxa: [2D788B90 verbose 'vpxavpxaTaskInfo' opID=WFU-c439f5c1] [VpxaTaskInfoPublisher::PropertyChanged] Number of deferred task updates: 0

    2015-02-19T10:28:18.075Z vcdvm280.mydomain.com Vpxa: [2D788B90 verbose 'VpxaHalCnxHostagent' opID=WFU-c439f5c1] [VpxaHalCnxHostagent::ProcessUpdate] Applying updates from 230226 to 230227 (at 230226)

    2015-02-19T10:28:18.074Z vcdvm280.mydomain.com Vpxa: [2D788B90 verbose 'VpxaHalCnxHostagent' opID=WFU-c439f5c1] [WaitForUpdatesDone] Received callback

    2015-02-19T10:28:18.074Z vcdvm280.mydomain.com Vpxa: [FFFE7B90 info 'commonvpxLro' opID=485a7be-bb] [VpxLRO] -- FINISH task-internal-438344 -- host -- vim.HostSystem.acquireCimServicesTicket --

    2015-02-19T10:28:18.073Z vcdvm280.mydomain.com Vpxa: [FFFE7B90 verbose 'vpxavpxaVmomi' opID=485a7be-bb] [VpxaVmomi] SetStatusDone was called with syncGenNo (19446)

    2015-02-19T10:28:18.072Z vcdvm280.mydomain.com Hostd: [2713CB90 info 'Vimsvc.TaskManager' opID=485a7be-bb] Task Completed : haTask-ha-host-vim.HostSystem.acquireCimServicesTicket-534552495 Status success

    2015-02-19T10:28:18.070Z vcdvm280.mydomain.com Hostd: [2713CB90 verbose 'Cimsvc' opID=485a7be-bb] Ticket issued for CIMOM version 1.0, user vpxuser

    2015-02-19T10:28:18.069Z vcdvm280.mydomain.com Hostd: [271BEB90 info 'Vimsvc.TaskManager' opID=485a7be-bb] Task Created : haTask-ha-host-vim.HostSystem.acquireCimServicesTicket-534552495

    2015-02-19T10:28:18.067Z vcdvm280.mydomain.com Vpxa: [FFFE7B90 info 'commonvpxLro' opID=485a7be-bb] [VpxLRO] -- BEGIN task-internal-438344 -- host -- vim.HostSystem.acquireCimServicesTicket -- 7cd92923-b655-7ea8-6ebe-b6988408ea22

    2015-02-19T10:28:17Z vcdvm280.mydomain.com dhclient-uw[12559]: send_packet: errno=101, Network is unreachable

    2015-02-19T10:28:17Z vcdvm280.mydomain.com dhclient-uw[12559]: DHCPREQUEST on vmk1 to xxx.yyy.254.1 port 67

    2015-02-19T10:28:07.184Z vcdvm280.mydomain.com storageRM:  Skipping device naa.6006016040002f00ccea56bdc257e111 either due to VSI read error or abnormal state

    2015-02-19T10:28:07.069Z vcdvm280.mydomain.com Hostd: [27442B90 verbose 'SoapAdapter'] Responded to service state request

    2015-02-19T10:28:05Z vcdvm280.mydomain.com vmkernel: num 0 gblnum 0 gblgen 0 gblbrk 0]

    2015-02-19T10:28:05.111Z vcdvm280.mydomain.com vmkernel: cpu23:8291)[type 10c00002 offset 25694208 v 15206, hb offset 3661824

    2015-02-19T10:28:05Z vcdvm280.mydomain.com vmkernel: gen 3523, mode 1, owner 542c4158-167bde59-32fa-0025b550003d mtime 1277858

    2015-02-19T10:28:05.111Z vcdvm280.mydomain.com vmkernel: cpu23:8291)FS3: 1250: vol 'VCD-VMFS-031-R10-Tier1-VNX-5500-02', lock at 25694208: [Req mode: 1] Checking liveness:

    2015-02-19T10:28:04.443Z vcdvm280.mydomain.com Hostd: [275C5B90 verbose 'SoapAdapter'] Responded to service state request

    2015-02-19T10:28:02.323Z vcdvm280.mydomain.com Vpxa: [2D713B90 verbose 'VpxaHalCnxHostagent' opID=WFU-23e725df] [WaitForUpdatesDone] Starting next WaitForUpdates() call to hostd

    2015-02-19T10:28:02.322Z vcdvm280.mydomain.com Vpxa: [2D713B90 verbose 'halservices' opID=WFU-23e725df] [VpxaHalServices] EventsRecorded Event Fired

    2015-02-19T10:28:02.322Z vcdvm280.mydomain.com Vpxa: [2D713B90 verbose 'VpxaHalCnxHostagent' opID=WFU-23e725df] [VpxaHalCnxHostagent::ProcessUpdate] Applying updates from 230225 to 230226 (at 230225)

    2015-02-19T10:28:02.322Z vcdvm280.mydomain.com Vpxa: [2D713B90 verbose 'VpxaHalCnxHostagent' opID=WFU-23e725df] [WaitForUpdatesDone] Received callback

    2015-02-19T10:28:02.320Z vcdvm280.mydomain.com Hostd: [27401B90 info 'Vimsvc.ha-eventmgr'] Event 3158 : Successfully restored access to volume 4f7428d8-6fd44884-440e-0025b500004d (VCD-VMFS-003-R5-Tier2-VNX-5500-02) following connectivity issues.

    2015-02-19T10:28:02.317Z vcdvm280.mydomain.com Hostd: --> }

    2015-02-19T10:28:02.323Z vcdvm280.mydomain.com Vpxa: [2D713B90 verbose 'VpxaHalCnxHostagent' opID=WFU-23e725df] [WaitForUpdatesDone] Completed callback

    2015-02-19T10:28:02.317Z vcdvm280.mydomain.com Hostd: -->    fault = (vmodl.MethodFault) null,

    2015-02-19T10:28:02.317Z vcdvm280.mydomain.com Hostd: -->    objectName = <unset>,

    2015-02-19T10:28:02.317Z vcdvm280.mydomain.com Hostd: -->    objectType = "vim.HostSystem",

    2015-02-19T10:28:02.317Z vcdvm280.mydomain.com Hostd: -->       }

    2015-02-19T10:28:02.317Z vcdvm280.mydomain.com Hostd: -->          value = "VCD-VMFS-003-R5-Tier2-VNX-5500-02",

    2015-02-19T10:28:02.317Z vcdvm280.mydomain.com Hostd: -->          key = "2",

    2015-02-19T10:28:02.317Z vcdvm280.mydomain.com Hostd: -->          dynamicType = <unset>,

    2015-02-19T10:28:02.317Z vcdvm280.mydomain.com Hostd: -->       (vmodl.KeyAnyValue) {

    2015-02-19T10:28:02.317Z vcdvm280.mydomain.com Hostd: -->       },

    2015-02-19T10:28:02.317Z vcdvm280.mydomain.com Hostd: -->          value = "4f7428d8-6fd44884-440e-0025b500004d",

    2015-02-19T10:28:02.317Z vcdvm280.mydomain.com Hostd: -->          key = "1",

    2015-02-19T10:28:02.317Z vcdvm280.mydomain.com Hostd: -->          dynamicType = <unset>,

    2015-02-19T10:28:02.317Z vcdvm280.mydomain.com Hostd: -->    objectId = "ha-eventmgr",

    2015-02-19T10:28:02.317Z vcdvm280.mydomain.com Hostd: -->    ],

    2015-02-19T10:28:02.317Z vcdvm280.mydomain.com Hostd: -->       (vmodl.KeyAnyValue) {

    2015-02-19T10:28:02.317Z vcdvm280.mydomain.com Hostd: -->    arguments = (vmodl.KeyAnyValue) [

    2015-02-19T10:28:02.317Z vcdvm280.mydomain.com Hostd: -->    severity = <unset>,

    2015-02-19T10:28:02.317Z vcdvm280.mydomain.com Hostd: -->    eventTypeId = "esx.problem.vmfs.heartbeat.recovered",

    2015-02-19T10:28:02.317Z vcdvm280.mydomain.com Hostd: -->    changeTag = <unset>,

    2015-02-19T10:28:02.317Z vcdvm280.mydomain.com Hostd: -->    fullFormattedMessage = <unset>,

    2015-02-19T10:28:02.317Z vcdvm280.mydomain.com Hostd: -->    message = <unset>,

    2015-02-19T10:28:02.317Z vcdvm280.mydomain.com Hostd: -->    dvs = (vim.event.DvsEventArgument) null,

    2015-02-19T10:28:02.317Z vcdvm280.mydomain.com Hostd: -->    net = (vim.event.NetworkEventArgument) null,

    2015-02-19T10:28:02.317Z vcdvm280.mydomain.com Hostd: -->    ds = (vim.event.DatastoreEventArgument) null,

    2015-02-19T10:28:02.317Z vcdvm280.mydomain.com Hostd: -->    vm = (vim.event.VmEventArgument) null,

    2015-02-19T10:28:02.317Z vcdvm280.mydomain.com Hostd: -->    },

    2015-02-19T10:28:02.317Z vcdvm280.mydomain.com Hostd: -->       host = 'vim.HostSystem:ha-host',

    2015-02-19T10:28:02.317Z vcdvm280.mydomain.com Hostd: -->       name = "vcdvm280.mydomain.com",

    2015-02-19T10:28:02.317Z vcdvm280.mydomain.com Hostd: -->       dynamicType = <unset>,

    2015-02-19T10:28:02.317Z vcdvm280.mydomain.com Hostd: -->    computeResource = (vim.event.ComputeResourceEventArgument) null,

    2015-02-19T10:28:02.317Z vcdvm280.mydomain.com Hostd: -->    host = (vim.event.HostEventArgument) {

    2015-02-19T10:28:02.317Z vcdvm280.mydomain.com Hostd: -->    datacenter = (vim.event.DatacenterEventArgument) null,

    2015-02-19T10:28:02.317Z vcdvm280.mydomain.com Hostd: -->    userName = "",

    2015-02-19T10:28:02.317Z vcdvm280.mydomain.com Hostd: -->    createdTime = "1970-01-01T00:00:00Z",

    2015-02-19T10:28:02.317Z vcdvm280.mydomain.com Hostd: -->    chainId = 808857904,

    2015-02-19T10:28:02.317Z vcdvm280.mydomain.com Hostd: -->    key = 909127733,

    2015-02-19T10:28:02.317Z vcdvm280.mydomain.com Hostd: -->    dynamicType = <unset>,

    2015-02-19T10:28:02.317Z vcdvm280.mydomain.com vobd:  [vmfsCorrelator] 13014614002238us: [esx.problem.vmfs.heartbeat.recovered] 4f7428d8-6fd44884-440e-0025b500004d VCD-VMFS-003-R5-Tier2-VNX-5500-02

    2015-02-19T10:28:02.317Z vcdvm280.mydomain.com Hostd: [27401B90 info 'Hostsvc.VmkVprobSource'] VmkVprobSource::Post event: (vim.event.EventEx) {

    2015-02-19T10:28:02.317Z vcdvm280.mydomain.com vmkernel: cpu0:8229)HBX: 255: Reclaimed heartbeat for volume 4f7428d8-6fd44884-440e-0025b500004d (VCD-VMFS-003-R5-Tier2-VNX-5500-02): [Timeout] [HB state abcdef02 offset 3502080 gen 559 stampUS 13014260946523 uuid 541f249e-493df21d-7436-0$

    2015-02-19T10:28:02.317Z vcdvm280.mydomain.com vobd:  No correlator for vob.vmfs.heartbeat.recovered

    2015-02-19T10:28:02.316Z vcdvm280.mydomain.com Vpxa: [FFF84B90 verbose 'VpxaHalCnxHostagent' opID=WFU-6197c370] [WaitForUpdatesDone] Completed callback

    2015-02-19T10:28:02.315Z vcdvm280.mydomain.com Vpxa: [FFF84B90 verbose 'halservices' opID=WFU-6197c370] [VpxaHalServices] EventsRecorded Event Fired

    2015-02-19T10:28:02.315Z vcdvm280.mydomain.com Vpxa: [FFF84B90 verbose 'VpxaHalCnxHostagent' opID=WFU-6197c370] [WaitForUpdatesDone] Starting next WaitForUpdates() call to hostd

    2015-02-19T10:28:02.315Z vcdvm280.mydomain.com Vpxa: [FFF84B90 verbose 'hostdevent' opID=WFU-6197c370] [VpxaHalEventHostAgent::NormalizeDsArgument] Transated a DatastoreEventArgument MoRef '4f7428d8-6fd44884-440e-0025b500004d' to 'ds:///vmfs/volumes/4f7428d8-6fd44884-440e-0025b500004d/'

    2015-02-19T10:28:02.315Z vcdvm280.mydomain.com Vpxa: [FFF84B90 verbose 'hostdevent' opID=WFU-6197c370] [VpxaHalEventHostAgent::NormalizeDsArgument] Transated a DatastoreEventArgument MoRef '4f7428d8-6fd44884-440e-0025b500004d' to 'ds:///vmfs/volumes/4f7428d8-6fd44884-440e-0025b500004d/'

    2015-02-19T10:28:02.315Z vcdvm280.mydomain.com Vpxa: [FFF84B90 verbose 'hostdevent' opID=WFU-6197c370] [VpxaHalEventHostAgent::NormalizeDsArgument] Transated a DatastoreEventArgument MoRef '4f7428d8-6fd44884-440e-0025b500004d' to 'ds:///vmfs/volumes/4f7428d8-6fd44884-440e-0025b500004d/'

    2015-02-19T10:28:02.315Z vcdvm280.mydomain.com Vpxa: [FFF84B90 verbose 'VpxaHalCnxHostagent' opID=WFU-6197c370] [VpxaHalCnxHostagent::ProcessUpdate] Applying updates from 230224 to 230225 (at 230224)

    2015-02-19T10:28:02.315Z vcdvm280.mydomain.com Vpxa: [FFF84B90 verbose 'VpxaHalCnxHostagent' opID=WFU-6197c370] [WaitForUpdatesDone] Received callback

    2015-02-19T10:28:02.311Z vcdvm280.mydomain.com Hostd: [271FFB90 info 'Vimsvc.ha-eventmgr'] Event 3157 : Lost access to volume 4f7428d8-6fd44884-440e-0025b500004d (VCD-VMFS-003-R5-Tier2-VNX-5500-02) due to connectivity issues. Recovery attempt is in progress and outcome will be reported shortly.

    2015-02-19T10:28:02.309Z vcdvm280.mydomain.com Hostd: --> }

    2015-02-19T10:28:02.309Z vcdvm280.mydomain.com Hostd: -->    fault = (vmodl.MethodFault) null,

    2015-02-19T10:28:02.309Z vcdvm280.mydomain.com Hostd: -->    objectName = "VCD-VMFS-003-R5-Tier2-VNX-5500-02",

    2015-02-19T10:28:02.309Z vcdvm280.mydomain.com Hostd: -->    objectType = "vim.Datastore",

    2015-02-19T10:28:02.309Z vcdvm280.mydomain.com Hostd: -->    objectId = "4f7428d8-6fd44884-440e-0025b500004d",

    2015-02-19T10:28:02.309Z vcdvm280.mydomain.com Hostd: -->    ],

    2015-02-19T10:28:02.309Z vcdvm280.mydomain.com Hostd: -->             datastore = 'vim.Datastore:4f7428d8-6fd44884-440e-0025b500004d',

    2015-02-19T10:28:02.309Z vcdvm280.mydomain.com Hostd: -->             name = "VCD-VMFS-003-R5-Tier2-VNX-5500-02",

    2015-02-19T10:28:02.309Z vcdvm280.mydomain.com Hostd: -->       }

    2015-02-19T10:28:02.309Z vcdvm280.mydomain.com Hostd: -->          },

    2015-02-19T10:28:02.309Z vcdvm280.mydomain.com Hostd: -->             dynamicType = <unset>,

    2015-02-19T10:28:02.309Z vcdvm280.mydomain.com Hostd: -->          value = (vim.event.DatastoreEventArgument) {

    2015-02-19T10:28:02.309Z vcdvm280.mydomain.com Hostd: -->          key = "2",

    2015-02-19T10:28:02.309Z vcdvm280.mydomain.com Hostd: -->          dynamicType = <unset>,

    2015-02-19T10:28:02.309Z vcdvm280.mydomain.com Hostd: -->       (vmodl.KeyAnyValue) {

    2015-02-19T10:28:02.309Z vcdvm280.mydomain.com Hostd: -->       },

    2015-02-19T10:28:02.309Z vcdvm280.mydomain.com Hostd: -->          value = "4f7428d8-6fd44884-440e-0025b500004d",

    2015-02-19T10:28:02.309Z vcdvm280.mydomain.com Hostd: -->          key = "1",

    2015-02-19T10:28:02.309Z vcdvm280.mydomain.com Hostd: -->          dynamicType = <unset>,

    2015-02-19T10:28:02.309Z vcdvm280.mydomain.com Hostd: -->       (vmodl.KeyAnyValue) {

    2015-02-19T10:28:02.309Z vcdvm280.mydomain.com Hostd: -->    arguments = (vmodl.KeyAnyValue) [

    2015-02-19T10:28:02.309Z vcdvm280.mydomain.com Hostd: -->    eventTypeId = "esx.problem.vmfs.heartbeat.timedout",

    2015-02-19T10:28:02.309Z vcdvm280.mydomain.com Hostd: -->    changeTag = <unset>,

    2015-02-19T10:28:02.309Z vcdvm280.mydomain.com Hostd: -->    message = <unset>,

    2015-02-19T10:28:02.309Z vcdvm280.mydomain.com Hostd: -->    severity = <unset>,

    2015-02-19T10:28:02.309Z vcdvm280.mydomain.com Hostd: -->    fullFormattedMessage = <unset>,

    2015-02-19T10:28:02.309Z vcdvm280.mydomain.com Hostd: -->    dvs = (vim.event.DvsEventArgument) null,

    2015-02-19T10:28:02.309Z vcdvm280.mydomain.com Hostd: -->    net = (vim.event.NetworkEventArgument) null,

    2015-02-19T10:28:02.309Z vcdvm280.mydomain.com Hostd: -->    },

    2015-02-19T10:28:02.309Z vcdvm280.mydomain.com Hostd: -->       datastore = 'vim.Datastore:4f7428d8-6fd44884-440e-0025b500004d',

    2015-02-19T10:28:02.309Z vcdvm280.mydomain.com Hostd: -->       name = "VCD-VMFS-003-R5-Tier2-VNX-5500-02",

    2015-02-19T10:28:02.309Z vcdvm280.mydomain.com Hostd: -->       dynamicType = <unset>,

    2015-02-19T10:28:02.309Z vcdvm280.mydomain.com Hostd: -->    ds = (vim.event.DatastoreEventArgument) {

    2015-02-19T10:28:02.309Z vcdvm280.mydomain.com Hostd: -->    vm = (vim.event.VmEventArgument) null,

    2015-02-19T10:28:02.309Z vcdvm280.mydomain.com Hostd: -->    },

    2015-02-19T10:28:02.309Z vcdvm280.mydomain.com Hostd: -->       host = 'vim.HostSystem:ha-host',

    2015-02-19T10:28:02.309Z vcdvm280.mydomain.com Hostd: -->       name = "vcdvm280.mydomain.com",

    2015-02-19T10:28:02.309Z vcdvm280.mydomain.com Hostd: -->       dynamicType = <unset>,

    2015-02-19T10:28:02.309Z vcdvm280.mydomain.com Hostd: -->    host = (vim.event.HostEventArgument) {

    2015-02-19T10:28:02.309Z vcdvm280.mydomain.com Hostd: -->    computeResource = (vim.event.ComputeResourceEventArgument) null,

    2015-02-19T10:28:02.309Z vcdvm280.mydomain.com Hostd: -->    datacenter = (vim.event.DatacenterEventArgument) null,

    2015-02-19T10:28:02.309Z vcdvm280.mydomain.com Hostd: -->    userName = "",

    2015-02-19T10:28:02.309Z vcdvm280.mydomain.com Hostd: -->    createdTime = "1970-01-01T00:00:00Z",

    2015-02-19T10:28:02.309Z vcdvm280.mydomain.com Hostd: -->    chainId = 808464928,

    2015-02-19T10:28:02.309Z vcdvm280.mydomain.com Hostd: -->    key = 825110831,

    2015-02-19T10:28:02.309Z vcdvm280.mydomain.com Hostd: -->    dynamicType = <unset>,

    2015-02-19T10:28:02.309Z vcdvm280.mydomain.com Hostd: [271FFB90 info 'Hostsvc.VmkVprobSource'] VmkVprobSource::Post event: (vim.event.EventEx) {

    2015-02-19T10:28:02.308Z vcdvm280.mydomain.com vobd:  [vmfsCorrelator] 13014613992938us: [esx.problem.vmfs.heartbeat.timedout] 4f7428d8-6fd44884-440e-0025b500004d VCD-VMFS-003-R5-Tier2-VNX-5500-02

    2015-02-19T10:28:02.308Z vcdvm280.mydomain.com vobd:  No correlator for vob.vmfs.heartbeat.timedout

    2015-02-19T10:28:01Z vcdvm280.mydomain.com vmkernel: num 0 gblnum 0 gblgen 0 gblbrk 0]

    2015-02-19T10:28:01.108Z vcdvm280.mydomain.com vmkernel: cpu23:8291)[type 10c00002 offset 25694208 v 15204, hb offset 3604480

    2015-02-19T10:28:01.108Z vcdvm280.mydomain.com vmkernel: cpu23:8291)FS3: 1250: vol 'VCD-VMFS-031-R10-Tier1-VNX-5500-02', lock at 25694208: [Req mode: 1] Checking liveness:

    2015-02-19T10:28:01Z vcdvm280.mydomain.com vmkernel: gen 28331, mode 1, owner 542c4cd9-3575085f-012b-0025b5500045 mtime 906594

    2015-02-19T10:28:01Z vcdvm280.mydomain.com dhclient-uw[12559]: send_packet: errno=101, Network is unreachable

    2015-02-19T10:28:01Z vcdvm280.mydomain.com dhclient-uw[12559]: DHCPREQUEST on vmk1 to xxx.yyy.254.1 port 67

    2015-02-19T10:28:00.817Z vcdvm280.mydomain.com vmkernel: cpu4:8196)ScsiDeviceIO: 2331: Cmd(0x4124424e5c40) 0x2a, CmdSN 0xaf915 from world 8216 to dev "naa.60060160bb303400909606d1f167e311" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.

    2015-02-19T10:28:00.817Z vcdvm280.mydomain.com vmkernel: cpu4:8196)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60060160bb303400909606d1f167e311" state in doubt; requested fast path state update...

    2015-02-19T10:28:00.817Z vcdvm280.mydomain.com vmkwarning: cpu4:8196)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.60060160bb303400909606d1f167e311" state in doubt; requested fast path state update...

    2015-02-19T10:28:00.817Z vcdvm280.mydomain.com vmkernel: cpu4:8196)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x2a (0x4124424e5c40, 8216) to dev "naa.60060160bb303400909606d1f167e311" on path "vmhba2:C0:T4:L54" Failed: H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL

    2015-02-19T10:28:00.817Z vcdvm280.mydomain.com vmkernel: cpu7:8742786)<7>fnic : 2 :: Returning from abort cmd type 2 SUCCESS

    2015-02-19T10:28:00.817Z vcdvm280.mydomain.com vmkernel: cpu4:8196)<7>fnic : 2 :: abts cmpl recd. id 133 status FCPIO_SUCCESS

    2015-02-19T10:28:00.815Z vcdvm280.mydomain.com Hostd: [274C3B90 info 'Snmpsvc'] UpdateStats: report cimom converter stats completed

    2015-02-19T10:28:00.815Z vcdvm280.mydomain.com Hostd: [274C3B90 info 'Snmpsvc'] NotifyAgent: write(40, /var/run/snmp.ctl, N) 1 bytes to snmpd

    2015-02-19T10:28:00.815Z vcdvm280.mydomain.com Hostd: [274C3B90 info 'Snmpsvc'] PublishReport: file /tmp/.cvtcimsnmp.xml published as /tmp/cvtcimsnmp.xml

    2015-02-19T10:28:00.815Z vcdvm280.mydomain.com Hostd: [274C3B90 info 'Snmpsvc'] DumpStats: cimom stats file /tmp/.cvtcimsnmp.xml generated, size 302 bytes.

    2015-02-19T10:28:00.815Z vcdvm280.mydomain.com Hostd: [274C3B90 info 'Snmpsvc'] DumpStats: cimom stats file published

    2015-02-19T10:28:00.814Z vcdvm280.mydomain.com Hostd: [274C3B90 info 'Snmpsvc'] UpdateStats: report cimom converter stats started

    2015-02-19T10:28:00.557Z vcdvm280.mydomain.com vmkernel: cpu4:8196)ScsiDeviceIO: 2331: Cmd(0x4124440613c0) 0x2a, CmdSN 0x884c from world 8216 to dev "naa.6006016040002f007ae012e9efcbe111" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.

    2015-02-19T10:28:00.556Z vcdvm280.mydomain.com vmkernel: cpu4:8196)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.6006016040002f007ae012e9efcbe111" state in doubt; requested fast path state update...

    2015-02-19T10:28:00.556Z vcdvm280.mydomain.com vmkwarning: cpu4:8196)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.6006016040002f007ae012e9efcbe111" state in doubt; requested fast path state update...

    2015-02-19T10:28:00.556Z vcdvm280.mydomain.com vmkernel: cpu7:8742786)<7>fnic : 2 :: Abort Cmd called FCID 0x321400, LUN 0x36 TAG 85 flags 3

    2015-02-19T10:28:00.556Z vcdvm280.mydomain.com vmkernel: cpu4:8196)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x2a (0x4124440613c0, 8216) to dev "naa.6006016040002f007ae012e9efcbe111" on path "vmhba2:C0:T3:L20" Failed: H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL

    2015-02-19T10:28:00.556Z vcdvm280.mydomain.com vmkernel: cpu7:8742786)<7>fnic : 2 :: Returning from abort cmd type 2 SUCCESS

    2015-02-19T10:28:00.556Z vcdvm280.mydomain.com vmkernel: cpu4:8196)<7>fnic : 2 :: abts cmpl recd. id 125 status FCPIO_SUCCESS

    2015-02-19T10:28:00.289Z vcdvm280.mydomain.com vmkernel: cpu4:8196)ScsiDeviceIO: 2331: Cmd(0x4124403ce7c0) 0x2a, CmdSN 0x4a from world 8216 to dev "naa.6006016040002f009977092d5f66e311" failed H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.

    2015-02-19T10:28:00.289Z vcdvm280.mydomain.com vmkernel: cpu4:8196)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.6006016040002f009977092d5f66e311" state in doubt; requested fast path state update...

    2015-02-19T10:28:00.289Z vcdvm280.mydomain.com vmkwarning: cpu4:8196)WARNING: NMP: nmp_DeviceRequestFastDeviceProbe:237:NMP device "naa.6006016040002f009977092d5f66e311" state in doubt; requested fast path state update...

    2015-02-19T10:28:00.289Z vcdvm280.mydomain.com vmkernel: cpu4:8196)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x2a (0x4124403ce7c0, 8216) to dev "naa.6006016040002f009977092d5f66e311" on path "vmhba1:C0:T4:L57" Failed: H:0x8 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0. Act:EVAL

    2015-02-19T10:28:00.289Z vcdvm280.mydomain.com vmkernel: cpu7:8742786)<7>fnic : 2 :: Abort Cmd called FCID 0x321300, LUN 0x14 TAG 7d flags 3

    2015-02-19T10:28:00.289Z vcdvm280.mydomain.com vmkernel: cpu7:8742786)<7>fnic : 1 :: Returning from abort cmd type 2 FAILED

    2015-02-19T10:28:00.289Z vcdvm280.mydomain.com vmkernel: cpu4:8196)<7>fnic : 1 :: abts cmpl recd. id 137 status FCPIO_TIMEOUT

    2015-02-19T10:27:58.292Z vcdvm280.mydomain.com vmkernel: cpu7:8742786)<7>fnic : 1 :: Abort Cmd called FCID 0x321300, LUN 0x39 TAG 89 flags 3

    2015-02-19T10:27:56Z vcdvm280.mydomain.com snmpd: lookup_vswitch: fetch VSI_MODULE_NODE_PortCfgs failed Not found

    2015-02-19T10:27:56Z vcdvm280.mydomain.com snmpd: lookup_vswitch: fetch VSI_MODULE_NODE_PortCfgs failed Not found

    2015-02-19T10:27:56.845Z vcdvm280.mydomain.com vmkernel: cpu14:11233)etherswitch: ESVsiGetParamPortCfg:392: : not found

    2015-02-19T10:27:56.846Z vcdvm280.mydomain.com vmkernel: cpu14:11233)etherswitch: ESVsiGetParamPortCfg:392: : not found

    2015-02-19T10:27:52Z vcdvm280.mydomain.com sfcb-CIMXML-Processor[8750136]: --- [vpxuser] GetVMToDeviceMapping on root/cimv2:IODM_Service.CreationClassName="IODM_Service",Name="IODM_Service",SystemCreationClassName="OMC_UnitaryComputerSystem",SystemName="00000000-0000-0000-0000-0025b5500050"

    2015-02-19T10:27:48.727Z vcdvm280.mydomain.com Vpxa: [FFF63B90 info 'commonvpxLro' opID=SWI-68f8c78a-64] [VpxLRO] -- FINISH task-internal-438343 --  -- vpxapi.VpxaService.fetchQuickStats --

    2015-02-19T10:27:48.727Z vcdvm280.mydomain.com Vpxa: [FFF63B90 verbose 'vpxavpxaVmomi' opID=SWI-68f8c78a-64] [VpxaVmomi] SetStatusDone was called with syncGenNo (19446)

    2015-02-19T10:27:48.726Z vcdvm280.mydomain.com Vpxa: [FFF63B90 info 'commonvpxLro' opID=SWI-68f8c78a-64] [VpxLRO] -- BEGIN task-internal-438343 --  -- vpxapi.VpxaService.fetchQuickStats -- 7cd92923-b655-7ea8-6ebe-b6988408ea22

    2015-02-19T10:27:47.174Z vcdvm280.mydomain.com storageRM:  Skipping device naa.6006016040002f00ccea56bdc257e111 either due to VSI read error or abnormal state

    2015-02-19T10:27:42Z vcdvm280.mydomain.com dhclient-uw[12559]: send_packet: errno=101, Network is unreachable

    2015-02-19T10:27:42Z vcdvm280.mydomain.com dhclient-uw[12559]: DHCPREQUEST on vmk1 to xxx.yyy.254.1 port 67

    2015-02-19T10:27:38.537Z vcdvm280.mydomain.com Rhttpproxy: [FF945430 verbose 'Proxy Req 37514'] The client closed the stream, not unexpectedly.

    2015-02-19T10:27:38.532Z vcdvm280.mydomain.com Vpxa: [FFF63B90 info 'commonvpxLro' opID=HB-host-1705@19446-57a5e7a1-f2] [VpxLRO] -- BEGIN task-internal-438342 --  -- vmodl.query.PropertyCollector.Filter.destroy -- 7cd92923-b655-7ea8-6ebe-b6988408ea22

    2015-02-19T10:27:38.532Z vcdvm280.mydomain.com Vpxa: [FFF63B90 info 'commonvpxLro' opID=HB-host-1705@19446-57a5e7a1-f2] [VpxLRO] -- FINISH task-internal-438342 --  -- vmodl.query.PropertyCollector.Filter.destroy --

    2015-02-19T10:27:38.532Z vcdvm280.mydomain.com Vpxa: [FFF63B90 verbose 'vpxavpxaVmomi' opID=HB-host-1705@19446-57a5e7a1-f2] [VpxaVmomi] SetStatusDone was called with syncGenNo (19446)

    2015-02-19T10:27:38.522Z vcdvm280.mydomain.com Vpxa: [2D80DB90 verbose 'PropertyProvider'] RecordOp ASSIGN: info.completeTime, session[7cd92923-b655-7ea8-6ebe-b6988408ea22]5292e7c0-14f9-18d4-086a-fa4040f4c44d

    2015-02-19T10:27:38.521Z vcdvm280.mydomain.com Vpxa: [2D80DB90 info 'commonvpxLro' opID=HB-host-1705@19446-57a5e7a1-b5] [VpxLRO] -- FINISH session[7cd92923-b655-7ea8-6ebe-b6988408ea22]5292e7c0-14f9-18d4-086a-fa4040f4c44d --  -- vpxapi.VpxaService.retrieveChanges --

    2015-02-19T10:27:38.521Z vcdvm280.mydomain.com Vpxa: [2D80DB90 verbose 'PropertyProvider' opID=HB-host-1705@19446-57a5e7a1-b5] RecordOp ASSIGN: info.result, session[7cd92923-b655-7ea8-6ebe-b6988408ea22]5292e7c0-14f9-18d4-086a-fa4040f4c44d

    2015-02-19T10:27:38.521Z vcdvm280.mydomain.com Vpxa: [2D80DB90 verbose 'PropertyProvider' opID=HB-host-1705@19446-57a5e7a1-b5] RecordOp ASSIGN: info.cancelable, session[7cd92923-b655-7ea8-6ebe-b6988408ea22]5292e7c0-14f9-18d4-086a-fa4040f4c44d

    2015-02-19T10:27:38.521Z vcdvm280.mydomain.com Vpxa: [2D80DB90 verbose 'PropertyProvider' opID=HB-host-1705@19446-57a5e7a1-b5] RecordOp ASSIGN: info.state, session[7cd92923-b655-7ea8-6ebe-b6988408ea22]5292e7c0-14f9-18d4-086a-fa4040f4c44d

    2015-02-19T10:27:38.521Z vcdvm280.mydomain.com Vpxa: [2D80DB90 verbose 'PropertyProvider' opID=HB-host-1705@19446-57a5e7a1-b5] RecordOp ASSIGN: info, session[7cd92923-b655-7ea8-6ebe-b6988408ea22]5292e7c0-14f9-18d4-086a-fa4040f4c44d

    2015-02-19T10:27:38.521Z vcdvm280.mydomain.com Vpxa: [2D80DB90 verbose 'vpxavpxaVmomi' opID=HB-host-1705@19446-57a5e7a1-b5] [VpxaVmomi] SetStatusDone was called with syncGenNo (19446)

    2015-02-19T10:27:38.521Z vcdvm280.mydomain.com Vpxa: [2D80DB90 verbose 'VpxProfiler' opID=HB-host-1705@19446-57a5e7a1-b5] [1-] VpxaMoService::GetChanges (took 84 ms)

    2015-02-19T10:27:38.521Z vcdvm280.mydomain.com Vpxa: [2D80DB90 verbose 'VpxProfiler' opID=HB-host-1705@19446-57a5e7a1-b5] [2-] VpxaDrs::PrepareChanges (took 0 ms)

    2015-02-19T10:27:38.520Z vcdvm280.mydomain.com Vpxa: [2D80DB90 verbose 'VpxProfiler' opID=HB-host-1705@19446-57a5e7a1-b5] [2+] VpxaDrs::PrepareChanges

    2015-02-19T10:27:38.520Z vcdvm280.mydomain.com Vpxa: [2D80DB90 verbose 'VpxProfiler' opID=HB-host-1705@19446-57a5e7a1-b5] [2-] VpxaMoService::PrepareResGroupChanges (took 0 ms)

    2015-02-19T10:27:38.520Z vcdvm280.mydomain.com Vpxa: [2D80DB90 verbose 'VpxaHalResourcePool' opID=HB-host-1705@19446-57a5e7a1-b5] [VpxaHalResourcePoolCurrentImpl::GetCachedRootResources] done

    2015-02-19T10:27:38.520Z vcdvm280.mydomain.com Vpxa: [2D80DB90 verbose 'VpxaHalResourcePool' opID=HB-host-1705@19446-57a5e7a1-b5] NeedFullSync() = false

    2015-02-19T10:27:38.520Z vcdvm280.mydomain.com Vpxa: [2D80DB90 verbose 'VpxProfiler' opID=HB-host-1705@19446-57a5e7a1-b5] [2+] VpxaMoService::PrepareResGroupChanges

    2015-02-19T10:27:38.520Z vcdvm280.mydomain.com Vpxa: [2D80DB90 verbose 'VpxProfiler' opID=HB-host-1705@19446-57a5e7a1-b5] [2-] VpxaMoService::PrepareVmChanges (took 0 ms)

    2015-02-19T10:27:38.520Z vcdvm280.mydomain.com Vpxa: [2D80DB90 verbose 'VpxProfiler' opID=HB-host-1705@19446-57a5e7a1-b5] [2+] VpxaMoService::PrepareVmChanges

    2015-02-19T10:27:38.520Z vcdvm280.mydomain.com Vpxa: [2D80DB90 verbose 'VpxProfiler' opID=HB-host-1705@19446-57a5e7a1-b5] [2-] VpxaMoService::PrepareHostChanges (took 34 ms)

    2015-02-19T10:27:38.520Z vcdvm280.mydomain.com Vpxa: [2D80DB90 verbose 'vpxavpxaMoService' opID=HB-host-1705@19446-57a5e7a1-b5] [VpxaMoService] host changes include changed properties []

    2015-02-19T10:27:38.506Z vcdvm280.mydomain.com Hostd: [2713CB90 verbose 'Default' opID=HB-host-1705@19446-57a5e7a1-b5] ha-license-manager:Validate -> Valid license detected for "VMware ESX Server 5.0" (lastError=0, desc.IsValid:Yes)

    2015-02-19T10:27:38.487Z vcdvm280.mydomain.com Hostd: [2713CB90 verbose 'Vimsvc.ha-license-manager' opID=HB-host-1705@19446-57a5e7a1-b5] Load: Loading existing file: /etc/vmware/license.cfg

    2015-02-19T10:27:38.485Z vcdvm280.mydomain.com Vpxa: [2D80DB90 verbose 'VpxProfiler' opID=HB-host-1705@19446-57a5e7a1-b5] [2+] VpxaMoService::PrepareHostChanges

    2015-02-19T10:27:38.485Z vcdvm280.mydomain.com Vpxa: [2D80DB90 verbose 'VpxProfiler' opID=HB-host-1705@19446-57a5e7a1-b5] [2+] VpxdDatastore::PrepareChanges

    2015-02-19T10:27:38.485Z vcdvm280.mydomain.com Vpxa: [2D80DB90 verbose 'VpxProfiler' opID=HB-host-1705@19446-57a5e7a1-b5] [2-] VpxdDatastore::PrepareChanges (took 0 ms)

    2015-02-19T10:27:38.485Z vcdvm280.mydomain.com Vpxa: [2D80DB90 verbose 'VpxProfiler' opID=HB-host-1705@19446-57a5e7a1-b5] [2-] VpxaStatsMetadata::PrepareStatsChanges (took 47 ms)

    2015-02-19T10:27:38.484Z vcdvm280.mydomain.com Vpxa: [2D80DB90 verbose 'vpxavpxaStatsMetadata' opID=HB-host-1705@19446-57a5e7a1-b5] [VpxaStatsMetadata] Refilled translations (344, 344)

    2015-02-19T10:27:38.483Z vcdvm280.mydomain.com Vpxa: [2D80DB90 verbose 'vpxavpxaStatsMetadata' opID=HB-host-1705@19446-57a5e7a1-b5] [VpxaStatsMetadata] Prior to refilling translations (344, 344)

    2015-02-19T10:27:38.458Z vcdvm280.mydomain.com Rhttpproxy: [FFC40B90 verbose 'Proxy Req 37514'] Connected to localhost:8089

    2015-02-19T10:27:38.456Z vcdvm280.mydomain.com Rhttpproxy: [6FE81B90 verbose 'Proxy Req 37514'] New proxy client SSL(TCP(local=xxx.yyy.162.180:443, peer=xxx.yyy.201.33:59437))

    2015-02-19T10:27:38.447Z vcdvm280.mydomain.com Hostd: [271BEB90 verbose 'Locale' opID=HB-host-1705@19446-57a5e7a1-b5] Default resource used for 'counter.sys.resourceMemConsumed.summary' expected in module 'perf'.

    2015-02-19T10:27:38.437Z vcdvm280.mydomain.com Vpxa: [2D80DB90 verbose 'VpxProfiler' opID=HB-host-1705@19446-57a5e7a1-b5] [2+] VpxaStatsMetadata::PrepareStatsChanges

    2015-02-19T10:27:38.436Z vcdvm280.mydomain.com Vpxa: [2D80DB90 info 'commonvpxLro' opID=HB-host-1705@19446-57a5e7a1-b5] [VpxLRO] -- BEGIN session[7cd92923-b655-7ea8-6ebe-b6988408ea22]5292e7c0-14f9-18d4-086a-fa4040f4c44d --  -- vpxapi.VpxaService.retrieveChanges -- 7cd92923-b655-7ea8-6ebe-b6988408ea22

    2015-02-19T10:27:38.437Z vcdvm280.mydomain.com Vpxa: [2D80DB90 verbose 'vpxavpxaMoService' opID=HB-host-1705@19446-57a5e7a1-b5] [VpxaMoService] GetChanges: 19445 -> 19446

    2015-02-19T10:27:38.437Z vcdvm280.mydomain.com Vpxa: [2D80DB90 verbose 'VpxProfiler' opID=HB-host-1705@19446-57a5e7a1-b5] [1+] VpxaMoService::GetChanges

    2015-02-19T10:27:38.436Z vcdvm280.mydomain.com Vpxa: [2D80DB90 verbose 'PropertyProvider' opID=HB-host-1705@19446-57a5e7a1-b5] RecordOp ASSIGN: info.cancelable, session[7cd92923-b655-7ea8-6ebe-b6988408ea22]5292e7c0-14f9-18d4-086a-fa4040f4c44d

    2015-02-19T10:27:38.436Z vcdvm280.mydomain.com Vpxa: [2D80DB90 verbose 'PropertyProvider' opID=HB-host-1705@19446-57a5e7a1-b5] RecordOp ASSIGN: info.state, session[7cd92923-b655-7ea8-6ebe-b6988408ea22]5292e7c0-14f9-18d4-086a-fa4040f4c44d

    2015-02-19T10:27:38.436Z vcdvm280.mydomain.com Vpxa: [2D80DB90 verbose 'PropertyProvider'] RecordOp ASSIGN: info.startTime, session[7cd92923-b655-7ea8-6ebe-b6988408ea22]5292e7c0-14f9-18d4-086a-fa4040f4c44d

    2015-02-19T10:27:38.436Z vcdvm280.mydomain.com Vpxa: [FFFC6B90 verbose 'PropertyProvider' opID=HB-host-1705@19446-57a5e7a1-b5] RecordOp ASSIGN: info.reason, session[7cd92923-b655-7ea8-6ebe-b6988408ea22]5292e7c0-14f9-18d4-086a-fa4040f4c44d

    2015-02-19T10:27:38.434Z vcdvm280.mydomain.com Vpxa: [FFFA5B90 verbose 'SoapAdapter'] Responded to service state request

    2015-02-19T10:27:37.066Z vcdvm280.mydomain.com Hostd: [27401B90 verbose 'SoapAdapter'] Responded to service state request

    2015-02-19T10:27:35.122Z vcdvm280.mydomain.com Vpxa: [2D767B90 info 'commonvpxLro' opID=2d035a32-1a] [VpxLRO] -- FINISH task-internal-438341 --  -- vpxapi.VpxaService.queryBatchPerformanceStatistics --

    2015-02-19T10:27:35.120Z vcdvm280.mydomain.com Vpxa: [2D767B90 verbose 'vpxavpxaVmomi' opID=2d035a32-1a] [VpxaVmomi] SetStatusDone was called with syncGenNo (19446)

    2015-02-19T10:27:34.498Z vcdvm280.mydomain.com Vpxa: [2D767B90 verbose 'vpxavpxaMoService' opID=2d035a32-1a] Adding querySpec. Had=14, has=14

    2015-02-19T10:27:34.497Z vcdvm280.mydomain.com Vpxa: [2D767B90 info 'commonvpxLro' opID=2d035a32-1a] [VpxLRO] -- BEGIN task-internal-438341 --  -- vpxapi.VpxaService.queryBatchPerformanceStatistics -- 7cd92923-b655-7ea8-6ebe-b6988408ea22

    2015-02-19T10:27:34.497Z vcdvm280.mydomain.com Vpxa: [2D767B90 verbose 'vpxavpxaMoService' opID=2d035a32-1a] Adding querySpec. Had=0, has=0

    2015-02-19T10:27:34.440Z vcdvm280.mydomain.com Hostd: [27B41B90 verbose 'SoapAdapter'] Responded to service state request

    2015-02-19T10:27:33.545Z vcdvm280.mydomain.com Vpxa: [2D7CBB90 verbose 'VpxaHalResourcePool' opID=SWI-97d1d972] [ProcessResourceNotification] No syncs pending, exiting loop...

    2015-02-19T10:27:33.545Z vcdvm280.mydomain.com Vpxa: [2D7CBB90 verbose 'VpxaHalResourcePool' opID=SWI-97d1d972] [VpxaHostdSpecSync] Copied root values ...

    2015-02-19T10:27:33.543Z vcdvm280.mydomain.com Vpxa: [2D7CBB90 verbose 'VpxaHalResourcePool' opID=SWI-97d1d972] [ProcessResourceNotification] Sync is pending, running loop once more...

    2015-02-19T10:27:33.543Z vcdvm280.mydomain.com Vpxa: [2D7CBB90 verbose 'vpxavpxaInvtHost' opID=SWI-97d1d972] [VpxaInvtHost] Increment master gen. no to (19446): ResourcePool:VpxaInvtHostResPoolListener::ConfigChanged

    2015-02-19T10:27:33.543Z vcdvm280.mydomain.com Vpxa: [2D7CBB90 verbose 'VpxaHalResourcePool' opID=SWI-97d1d972] [ProcessResourceNotification] Sync mutex lock released

    2015-02-19T10:27:33.542Z vcdvm280.mydomain.com Vpxa: [2D7CBB90 verbose 'halservices' opID=SWI-97d1d972] [VpxaHalServices] ResPoolConfigChange Event fired

    2015-02-19T10:27:33.542Z vcdvm280.mydomain.com Vpxa: [FFB00B90 verbose 'VpxaHalCnxHostagent' opID=WFU-cb6c0700] [WaitForUpdatesDone] Completed callback

    2015-02-19T10:27:33.542Z vcdvm280.mydomain.com Vpxa: [2D7CBB90 verbose 'VpxaHalResourcePool' opID=SWI-97d1d972] [HandleHostdPropertyChange] Firing res pool change event, rootChanged = true

    2015-02-19T10:27:33.542Z vcdvm280.mydomain.com Vpxa: [2D7CBB90 verbose 'VpxaHalResourcePool' opID=SWI-97d1d972] [VpxaHostdSpecSync] Copied root values ...

    2015-02-19T10:27:33.542Z vcdvm280.mydomain.com Vpxa: [FFB00B90 verbose 'VpxaHalCnxHostagent' opID=WFU-cb6c0700] [WaitForUpdatesDone] Starting next WaitForUpdates() call to hostd

    2015-02-19T10:27:33.542Z vcdvm280.mydomain.com Vpxa: [FFB00B90 verbose 'VpxaHalResourcePool' opID=WFU-cb6c0700] [VpxaHostdSpecSync] Sync is already running, so let it continue to run in that thread...

    2015-02-19T10:27:33.542Z vcdvm280.mydomain.com Vpxa: [FFB00B90 verbose 'VpxaHalResourcePool' opID=WFU-cb6c0700] [VpxaHostdSpecSync] Setting _syncPending = true ...

    2015-02-19T10:27:33.542Z vcdvm280.mydomain.com Vpxa: [FFB00B90 verbose 'VpxaHalResourcePool' opID=WFU-cb6c0700] [VpxaHostdSpecSync] [IsVmListChangeUpdate] returning false

    2015-02-19T10:27:33.541Z vcdvm280.mydomain.com Vpxa: [FFB00B90 verbose 'VpxaHalResourcePool' opID=WFU-cb6c0700] [VpxaHostdSpecSync] Received change notification from hostd ...

    2015-02-19T10:27:33.541Z vcdvm280.mydomain.com Vpxa: [FFB00B90 verbose 'VpxaHalCnxHostagent' opID=WFU-cb6c0700] [VpxaHalCnxHostagent::ProcessUpdate] Applying updates from 230223 to 230224 (at 230223)

    2015-02-19T10:27:33.541Z vcdvm280.mydomain.com Vpxa: [FFB00B90 verbose 'VpxaHalCnxHostagent' opID=WFU-cb6c0700] [WaitForUpdatesDone] Received callback

    2015-02-19T10:27:33.538Z vcdvm280.mydomain.com Vpxa: [FFF63B90 verbose 'VpxaHalCnxHostagent' opID=WFU-3baf0e75] [WaitForUpdatesDone] Completed callback

    2015-02-19T10:27:33.538Z vcdvm280.mydomain.com Vpxa: [2D7CBB90 info 'VpxaHalResourcePool' opID=SWI-97d1d972] GetOverrideFailureCount() = 0MAX_CONSECUTIVE_OVERRIDE_FAILURES = 3

    2015-02-19T10:27:33.537Z vcdvm280.mydomain.com Vpxa: [FFF63B90 verbose 'VpxaHalCnxHostagent' opID=WFU-3baf0e75] [WaitForUpdatesDone] Starting next WaitForUpdates() call to hostd

    2015-02-19T10:27:33.537Z vcdvm280.mydomain.com Vpxa: [FFF63B90 verbose 'VpxaHalResourcePool' opID=WFU-3baf0e75] [VpxaHostdSpecSync] Launching ProcessResourceNotification in new thread...

    2015-02-19T10:27:33.537Z vcdvm280.mydomain.com Vpxa: [FFF63B90 verbose 'VpxaHalResourcePool' opID=WFU-3baf0e75] [VpxaHostdSpecSync] Setting _syncPending = true ...

    2015-02-19T10:27:33.537Z vcdvm280.mydomain.com Vpxa: [FFF63B90 verbose 'VpxaHalResourcePool' opID=WFU-3baf0e75] [VpxaHostdSpecSync] [IsVmListChangeUpdate] returning false

    2015-02-19T10:27:33.537Z vcdvm280.mydomain.com Vpxa: [FFF63B90 verbose 'VpxaHalResourcePool' opID=WFU-3baf0e75] [VpxaHostdSpecSync] Received change notification from hostd ...

    2015-02-19T10:27:33.537Z vcdvm280.mydomain.com Vpxa: [FFF63B90 verbose 'VpxaHalCnxHostagent' opID=WFU-3baf0e75] [VpxaHalCnxHostagent::ProcessUpdate] Applying updates from 230222 to 230223 (at 230222)

    2015-02-19T10:27:33.537Z vcdvm280.mydomain.com Vpxa: [FFF63B90 verbose 'VpxaHalCnxHostagent' opID=WFU-3baf0e75] [WaitForUpdatesDone] Received callback

    2015-02-19T10:27:33.534Z vcdvm280.mydomain.com Hostd: [269B0B90 verbose 'Hostsvc.ResourcePool ha-root-pool'] Root pool capacity changed from 22596MHz/253017MB to 22596MHz/253021MB

    2015-02-19T10:27:27.163Z vcdvm280.mydomain.com storageRM:  Skipping device naa.6006016040002f00ccea56bdc257e111 either due to VSI read error or abnormal state

    2015-02-19T10:27:21Z vcdvm280.mydomain.com dhclient-uw[12559]: send_packet: errno=101, Network is unreachable

    2015-02-19T10:27:21Z vcdvm280.mydomain.com dhclient-uw[12559]: DHCPREQUEST on vmk1 to xxx.yyy.254.1 port 67

    2015-02-19T10:27:07.153Z vcdvm280.mydomain.com storageRM:  Skipping device naa.6006016040002f00ccea56bdc257e111 either due to VSI read error or abnormal state

    2015-02-19T10:27:07.064Z vcdvm280.mydomain.com Hostd: [274C3B90 verbose 'SoapAdapter'] Responded to service state request

    2015-02-19T10:27:06Z vcdvm280.mydomain.com dhclient-uw[12559]: send_packet: errno=101, Network is unreachable

    2015-02-19T10:27:06Z vcdvm280.mydomain.com dhclient-uw[12559]: DHCPREQUEST on vmk1 to xxx.yyy.254.1 port 67

    2015-02-19T10:27:04.438Z vcdvm280.mydomain.com Hostd: [271BEB90 verbose 'SoapAdapter'] Responded to service state request

    2015-02-19T10:27:01.778Z vcdvm280.mydomain.com Vpxa: [2D767B90 verbose 'VpxaHalCnxHostagent' opID=WFU-4d19079c] [WaitForUpdatesDone] Completed callback

    2015-02-19T10:27:01.778Z vcdvm280.mydomain.com Vpxa: [2D767B90 verbose 'VpxaHalCnxHostagent' opID=WFU-4d19079c] [WaitForUpdatesDone] Starting next WaitForUpdates() call to hostd

    2015-02-19T10:27:01.777Z vcdvm280.mydomain.com Vpxa: [2D767B90 verbose 'vpxavpxaInvtHost' opID=WFU-4d19079c] [HostChanged] Found update for tracked MoRef vim.HostSystem:ha-host

    2015-02-19T10:27:01.777Z vcdvm280.mydomain.com Vpxa: [2D767B90 verbose 'halservices' opID=WFU-4d19079c] [VpxaHalServices] HostChanged Event Fired, properties changed [runtime.healthSystemRuntime]

    2015-02-19T10:27:01.776Z vcdvm280.mydomain.com Vpxa: [2D767B90 verbose 'VpxaHalCnxHostagent' opID=WFU-4d19079c] [VpxaHalCnxHostagent::ProcessUpdate] Applying updates from 230221 to 230222 (at 230221)

    2015-02-19T10:27:01.776Z vcdvm280.mydomain.com Vpxa: [2D767B90 verbose 'VpxaHalCnxHostagent' opID=WFU-4d19079c] [WaitForUpdatesDone] Received callback

    2015-02-19T10:27:00.814Z vcdvm280.mydomain.com Hostd: [2717DB90 info 'Snmpsvc'] UpdateStats: report cimom converter stats completed

    2015-02-19T10:27:00.814Z vcdvm280.mydomain.com Hostd: [2717DB90 info 'Snmpsvc'] NotifyAgent: write(40, /var/run/snmp.ctl, N) 1 bytes to snmpd

    2015-02-19T10:27:00.813Z vcdvm280.mydomain.com Hostd: [2717DB90 info 'Snmpsvc'] DumpStats: cimom stats file published

    2015-02-19T10:27:00.813Z vcdvm280.mydomain.com Hostd: [2717DB90 info 'Snmpsvc'] PublishReport: file /tmp/.cvtcimsnmp.xml published as /tmp/cvtcimsnmp.xml

    2015-02-19T10:27:00.813Z vcdvm280.mydomain.com Hostd: [2717DB90 info 'Snmpsvc'] DumpStats: cimom stats file /tmp/.cvtcimsnmp.xml generated, size 302 bytes.

    2015-02-19T10:27:00.813Z vcdvm280.mydomain.com Hostd: [2717DB90 info 'Snmpsvc'] UpdateStats: report cimom converter stats started

    2015-02-19T10:27:00.149Z vcdvm280.mydomain.com Hostd: [26940B90 verbose 'Cimsvc'] Ticket issued for CIMOM version 1.0, user root



  • 6.  RE: Is there a storage issue or not?

    Posted Feb 16, 2015 03:22 AM

    this is big issue where if let say you have high IOs VMs and I'm also having this issue in my environment and my SAP apps VMs are affected.  up on checking with EMC storage team, they mentioned this bug has been identified and they yet to release the bug fix. so we are waiting for the fix.



  • 7.  RE: Is there a storage issue or not?

    Posted Feb 19, 2015 02:31 PM

    dhanarajramesh wrote:

    this is big issue where if let say you have high IOs VMs and I'm also having this issue in my environment and my SAP apps VMs are affected.  up on checking with EMC storage team, they mentioned this bug has been identified and they yet to release the bug fix. so we are waiting for the fix.

    Can you tell me what VNX system and which flair / OE code?



  • 8.  RE: Is there a storage issue or not?

    Broadcom Employee
    Posted Dec 20, 2015 10:04 AM

    This looks like definitly hba driver or firmware issue.


    See the logs "fnic_rport_exch_reset called portid" this means fnic crashed somehow and stopped responding. This will also cause host unexpected reboot if the firmware try to restart hba cards. Please open a support request to host vendor to investigate driver firmware issues. Also I suggest you to upgrade HBA drivers and firmware to latest version. You may read the release notes to determine if this problem resolved.



  • 9.  RE: Is there a storage issue or not?

    Posted Dec 22, 2015 08:53 PM

    I don't think it is firmware or drivers. We're running 4 flavours of ESXi, 6 different drivers, 3 sets of firmware. With VMware and EMC and Cisco Support we switched drivers, ESXi versions, firmwares on the blade, nothing seem to help. Then suddenly, after we replaced our SAN switches, the issues were gone for about two weeks. Now they're back again :-(