ESXi

 View Only
  • 1.  iscsi problems

    Posted Feb 11, 2014 08:44 AM

    Hi, we are having some iscsi problems with 2 vms. The 2 vms (SCCM servers) have both their dedicated host. And are in a dedicated cluster in vcenter. But since a couple of days the vm's freezes, most of the times it happens at night or early in the morning.

    When i look in the log file of one of the host i see allot of waring messages:

    the hosts are HP DL380G5 6x interfaces (2 Broadcom NC373i Intergrated Multifunction Gigabit adapter and quad port Intel 82571EB Gigabit adapater)

    And we use software iscsi 2 nic's on of each card. With round robin.The switch ports are configured with lacp active and jumbo frames

    We have ESXi 5.5 and vCenter 5.5

    2014-02-11T07:17:09.793Z cpu6:32795)NMP: nmp_ThrottleLogForDevice:2321: Cmd 0x1a (0x412e807fad40, 0) to dev "mpx.vmhba0:C0:T0:L0" on path "vmhba0:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE

    2014-02-11T07:17:09.793Z cpu6:32795)ScsiDeviceIO: 2337: Cmd(0x412e807fad40) 0x1a, CmdSN 0x125da from world 0 to dev "mpx.vmhba0:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.

    2014-02-11T07:17:14.540Z cpu7:32796)WARNING: ScsiDeviceIO: 1223: Device naa.203c001378ae0119 performance has deteriorated. I/O latency increased from average value of 44255 microseconds to 900981 microseconds.

    2014-02-11T07:17:15.957Z cpu2:974617)ScsiDeviceIO: 1203: Device naa.203c001378ae0119 performance has improved. I/O latency reduced from 900981 microseconds to 178170 microseconds.

    2014-02-11T07:17:17.429Z cpu3:32792)ScsiDeviceIO: 1203: Device naa.203c001378ae0119 performance has improved. I/O latency reduced from 178170 microseconds to 87376 microseconds.

    2014-02-11T07:18:53.757Z cpu4:32793)WARNING: ScsiDeviceIO: 1223: Device naa.203c001378ae0119 performance has deteriorated. I/O latency increased from average value of 44196 microseconds to 1140127 microseconds.

    2014-02-11T07:18:54.629Z cpu6:32795)ScsiDeviceIO: 1203: Device naa.203c001378ae0119 performance has improved. I/O latency reduced from 1140127 microseconds to 226372 microseconds.

    2014-02-11T07:19:23.457Z cpu0:32789)ScsiDeviceIO: 1203: Device naa.203c001378ae0119 performance has improved. I/O latency reduced from 226372 microseconds to 86809 microseconds.

    2014-02-11T07:20:35.057Z cpu5:32794)WARNING: ScsiDeviceIO: 1223: Device naa.203c001378ae0119 performance has deteriorated. I/O latency increased from average value of 44174 microseconds to 988170 microseconds.

    2014-02-11T07:20:35.988Z cpu3:32792)ScsiDeviceIO: 1203: Device naa.203c001378ae0119 performance has improved. I/O latency reduced from 988170 microseconds to 195084 microseconds.

    2014-02-11T07:20:41.225Z cpu2:974612)ScsiDeviceIO: 1203: Device naa.203c001378ae0119 performance has improved. I/O latency reduced from 195084 microseconds to 86788 microseconds.

    2014-02-11T07:20:54.754Z cpu2:32791)WARNING: ScsiDeviceIO: 1223: Device naa.203c001378ae0119 performance has deteriorated. I/O latency increased from average value of 44170 microseconds to 1059866 microseconds.

    2014-02-11T07:20:56.654Z cpu2:974616)ScsiDeviceIO: 1203: Device naa.203c001378ae0119 performance has improved. I/O latency reduced from 1059866 microseconds to 207657 microseconds.

    2014-02-11T07:20:59.739Z cpu2:32791)ScsiDeviceIO: 1203: Device naa.203c001378ae0119 performance has improved. I/O latency reduced from 207657 microseconds to 88011 microseconds.

    2014-02-11T07:21:45.652Z cpu5:32794)WARNING: ScsiDeviceIO: 1223: Device naa.203c001378ae0119 performance has deteriorated. I/O latency increased from average value of 44157 microseconds to 900933 microseconds.

    2014-02-11T07:21:48.533Z cpu2:974609)ScsiDeviceIO: 1203: Device naa.203c001378ae0119 performance has improved. I/O latency reduced from 900933 microseconds to 177646 microseconds.

    2014-02-11T07:21:51.843Z cpu2:32791)ScsiDeviceIO: 1203: Device naa.203c001378ae0119 performance has improved. I/O latency reduced from 177646 microseconds to 86080 microseconds.

    2014-02-11T07:22:09.804Z cpu4:32793)NMP: nmp_ThrottleLogForDevice:2321: Cmd 0x1a (0x412e80878900, 0) to dev "mpx.vmhba0:C0:T0:L0" on path "vmhba0:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE

    2014-02-11T07:22:09.804Z cpu4:32793)ScsiDeviceIO: 2337: Cmd(0x412e80878900) 0x1a, CmdSN 0x126bc from world 0 to dev "mpx.vmhba0:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.

    2014-02-11T07:22:26.543Z cpu7:32796)WARNING: ScsiDeviceIO: 1223: Device naa.203c001378ae0119 performance has deteriorated. I/O latency increased from average value of 44149 microseconds to 916920 microseconds.

    2014-02-11T07:22:27.729Z cpu7:974614)ScsiDeviceIO: 1203: Device naa.203c001378ae0119 performance has improved. I/O latency reduced from 916920 microseconds to 181764 microseconds.

    2014-02-11T07:22:30.592Z cpu3:32792)ScsiDeviceIO: 1203: Device naa.203c001378ae0119 performance has improved. I/O latency reduced from 181764 microseconds to 86366 microseconds.

    2014-02-11T07:23:25.682Z cpu7:1146088)FS3Misc: 1565: Long VMFS rsv time on 'THSLOW-DATA01' (held for 243 msecs). # R: 1, # W: 1 bytesXfer: 2 sectors

    2014-02-11T07:23:29.352Z cpu4:1146119)FS3Misc: 1565: Long VMFS rsv time on 'THSLOW-DATA01' (held for 245 msecs). # R: 1, # W: 1 bytesXfer: 2 sectors

    2014-02-11T07:23:51.612Z cpu5:1146119)FS3Misc: 1565: Long VMFS rsv time on 'THSLOW-DATA01' (held for 533 msecs). # R: 1, # W: 1 bytesXfer: 2 sectors

    2014-02-11T07:23:53.492Z cpu4:1146119)FS3Misc: 1565: Long VMFS rsv time on 'THSLOW-DATA01' (held for 246 msecs). # R: 1, # W: 1 bytesXfer: 2 sectors

    2014-02-11T07:25:29.229Z cpu5:32794)WARNING: ScsiDeviceIO: 1223: Device naa.203c001378ae0119 performance has deteriorated. I/O latency increased from average value of 44090 microseconds to 977489 microseconds.

    2014-02-11T07:25:31.187Z cpu0:32789)ScsiDeviceIO: 1203: Device naa.203c001378ae0119 performance has improved. I/O latency reduced from 977489 microseconds to 190612 microseconds.

    2014-02-11T07:25:39.470Z cpu3:974615)ScsiDeviceIO: 1203: Device naa.203c001378ae0119 performance has improved. I/O latency reduced from 190612 microseconds to 87346 microseconds.

    2014-02-11T07:26:26.384Z cpu1:1146119)FS3Misc: 1565: Long VMFS rsv time on 'THSLOW-DATA01' (held for 263 msecs). # R: 1, # W: 1 bytesXfer: 2 sectors

    2014-02-11T07:27:09.652Z cpu6:32795)NMP: nmp_ThrottleLogForDevice:2321: Cmd 0x1a (0x412e80868080, 0) to dev "mpx.vmhba0:C0:T0:L0" on path "vmhba0:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE

    2014-02-11T07:27:09.652Z cpu6:32795)ScsiDeviceIO: 2337: Cmd(0x412e80868080) 0x1a, CmdSN 0x127a0 from world 0 to dev "mpx.vmhba0:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.

    2014-02-11T07:32:09.405Z cpu7:32796)NMP: nmp_ThrottleLogForDevice:2321: Cmd 0x1a (0x412e8080d840, 0) to dev "mpx.vmhba0:C0:T0:L0" on path "vmhba0:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE

    2014-02-11T07:32:09.405Z cpu7:32796)ScsiDeviceIO: 2337: Cmd(0x412e8080d840) 0x1a, CmdSN 0x127e6 from world 0 to dev "mpx.vmhba0:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.

    2014-02-11T07:37:09.402Z cpu4:32793)NMP: nmp_ThrottleLogForDevice:2321: Cmd 0x1a (0x412e808787c0, 0) to dev "mpx.vmhba0:C0:T0:L0" on path "vmhba0:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE

    2014-02-11T07:37:09.402Z cpu4:32793)ScsiDeviceIO: 2337: Cmd(0x412e808787c0) 0x1a, CmdSN 0x12800 from world 0 to dev "mpx.vmhba0:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.

    2014-02-11T07:42:09.230Z cpu1:32790)NMP: nmp_ThrottleLogForDevice:2321: Cmd 0x1a (0x412e80821480, 0) to dev "mpx.vmhba0:C0:T0:L0" on path "vmhba0:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE

    2014-02-11T07:42:09.230Z cpu1:32790)ScsiDeviceIO: 2337: Cmd(0x412e80821480) 0x1a, CmdSN 0x1281a from world 0 to dev "mpx.vmhba0:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.

    2014-02-11T07:47:09.396Z cpu1:32790)NMP: nmp_ThrottleLogForDevice:2321: Cmd 0x1a (0x412e807a3880, 0) to dev "mpx.vmhba0:C0:T0:L0" on path "vmhba0:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE

    2014-02-11T07:47:09.396Z cpu1:32790)ScsiDeviceIO: 2337: Cmd(0x412e807a3880) 0x1a, CmdSN 0x12854 from world 0 to dev "mpx.vmhba0:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.

    2014-02-11T07:52:09.664Z cpu4:32793)NMP: nmp_ThrottleLogForDevice:2321: Cmd 0x1a (0x412e80819d00, 0) to dev "mpx.vmhba0:C0:T0:L0" on path "vmhba0:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE

    2014-02-11T07:52:09.664Z cpu4:32793)ScsiDeviceIO: 2337: Cmd(0x412e80819d00) 0x1a, CmdSN 0x1286e from world 0 to dev "mpx.vmhba0:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.

    2014-02-11T07:57:09.944Z cpu1:32790)NMP: nmp_ThrottleLogForDevice:2321: Cmd 0x1a (0x412e8084d300, 0) to dev "mpx.vmhba0:C0:T0:L0" on path "vmhba0:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE

    2014-02-11T07:57:09.944Z cpu1:32790)ScsiDeviceIO: 2337: Cmd(0x412e8084d300) 0x1a, CmdSN 0x12888 from world 0 to dev "mpx.vmhba0:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.

    2014-02-11T08:02:09.403Z cpu0:32789)NMP: nmp_ThrottleLogForDevice:2321: Cmd 0x1a (0x412e80888e40, 0) to dev "mpx.vmhba0:C0:T0:L0" on path "vmhba0:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE

    2014-02-11T08:02:09.403Z cpu0:32789)ScsiDeviceIO: 2337: Cmd(0x412e80888e40) 0x1a, CmdSN 0x128ad from world 0 to dev "mpx.vmhba0:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.

    2014-02-11T08:07:09.403Z cpu7:32796)NMP: nmp_ThrottleLogForDevice:2321: Cmd 0x1a (0x412e808410c0, 0) to dev "mpx.vmhba0:C0:T0:L0" on path "vmhba0:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE

    2014-02-11T08:07:09.403Z cpu7:32796)ScsiDeviceIO: 2337: Cmd(0x412e808410c0) 0x1a, CmdSN 0x128c7 from world 0 to dev "mpx.vmhba0:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.

    2014-02-11T08:12:09.394Z cpu0:32789)NMP: nmp_ThrottleLogForDevice:2321: Cmd 0x1a (0x412e8082df80, 0) to dev "mpx.vmhba0:C0:T0:L0" on path "vmhba0:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE

    2014-02-11T08:12:09.394Z cpu0:32789)ScsiDeviceIO: 2337: Cmd(0x412e8082df80) 0x1a, CmdSN 0x128e1 from world 0 to dev "mpx.vmhba0:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.

    2014-02-11T08:17:09.405Z cpu4:32793)NMP: nmp_ThrottleLogForDevice:2321: Cmd 0x1a (0x412e808623c0, 0) to dev "mpx.vmhba0:C0:T0:L0" on path "vmhba0:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE

    2014-02-11T08:17:09.405Z cpu4:32793)ScsiDeviceIO: 2337: Cmd(0x412e808623c0) 0x1a, CmdSN 0x1291b from world 0 to dev "mpx.vmhba0:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.

    2014-02-11T08:22:09.407Z cpu7:32796)NMP: nmp_ThrottleLogForDevice:2321: Cmd 0x1a (0x412e80856600, 0) to dev "mpx.vmhba0:C0:T0:L0" on path "vmhba0:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE

    2014-02-11T08:22:09.407Z cpu7:32796)ScsiDeviceIO: 2337: Cmd(0x412e80856600) 0x1a, CmdSN 0x12944 from world 0 to dev "mpx.vmhba0:C0:T0:L0" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.



  • 2.  RE: iscsi problems

    Posted Feb 11, 2014 02:42 PM

    Have you checked the logs on your SAN to see if some activity there is causing the latency?



  • 3.  RE: iscsi problems

    Posted Feb 11, 2014 03:28 PM

    yes that was the first thing that i did... and nothing is indicating the high latency.

    First thing that i thought of was the backup. But according to the logs, its finished 2 hours before i get the latency messages