how to check RDM iSCSI network traffic

[RussellJM]

I've recently followed vmware documentation to bind a single software iscsi vmhba to two vmkernal ports (one virtual switch) on the ESXi host side in two of my clusters (completely seperate setups).

On one cluster, I have 2 iSCSI attached VMFS datastores with a handful of iSCSI RDM's.  When I look at the performance tab for the NIC's that I have dedicated to iSCSI traffic, I see no traffic (also see no traffic in esxtop). 

In the other cluster, I have 2 NFS datastores and way too many iSCSI RDM's.  This cluster shows the same thing, the NICs that i have bound to the iSCSI hba do not show any data passing over them.  a single port on the management switch has veyr high R/W rates though.  this cluster has some significant latency that i'm trying to get rid of.

how can I can check to see what network ports the iSCSI RDM's are using?  Why don't I see network traffic on the iSCSI dedicated NICs and should I be able too?

[MKguy]

That sounds weird, you have to see traffic in esxtop. Are you sure the host(s) you checked actually had a VM with those RDMs running at the time?

First, make sure the ESXi host is actually issuing IO to the storage device in the esxtop device view ("u"). The RDM-LUN must be listed there and you should see IO if the attached VM is running on this host (and actually issuing some IO obviously).

If that's the case, the esxtop network view ("n") should display corresponding network traffic on the iSCSI-bound vmkernel ports as well as the physical vmnics linked to them.

Can you post screenshots of your iSCSI/vSwitch setup and esxtop/Client performance charts?

Or the output from:

esxcli iscsi adapter list

esxcli iscsi networkportal list

esxcli iscsi logicalnetworkportal list

esxcli iscsi physicalnetworkportal list

esxcli iscsi session list

[RussellJM]

I highlighted, somethign thsat looks odd to me.  if the vmhba is bound to only vmk2 and vmk3, why does vmk0 and vmk1 still show up?

~ # esxcli iscsi adapter list
Adapter  Driver     State   UID                                       Description
-------  ---------  ------  ----------------------------------------  ----------------------
vmhba37  iscsi_vmk  online  iqn.1998-01.com.vmware:TRY0VM02-39bdaaaf  iSCSI Software Adapter

~ # esxcli iscsi networkportal list
vmhba37
   Adapter: vmhba37
   Vmknic: vmk2
   MAC Address: 00:10:18:e4:31:7c
   MAC Address Valid: true
   IPv4: 10.32.96.81
   IPv4 Subnet Mask: 255.255.255.0
   IPv6:
   MTU: 1500
   Vlan Supported: true
   Vlan ID: 0
   Reserved Ports: 63488~65536
   TOE: false
   TSO: true
   TCP Checksum: false
   Link Up: true
   Current Speed: 1000
   Rx Packets: 4845294
   Tx Packets: 86356
   NIC Driver: tg3
   NIC Driver Version: 3.124c.v50.1
   NIC Firmware Version: FFV7.0.0 bc 5720-v1.17
   Compliant Status: compliant
   NonCompliant Message:
   NonCompliant Remedy:
   Vswitch: vSwitch5
   PortGroup: iSCSI1
   VswitchUuid:
   PortGroupKey:
   PortKey:
   Duplex:
   Path Status: active

vmhba37
   Adapter: vmhba37
   Vmknic: vmk3
   MAC Address: 00:10:18:e4:31:7d
   MAC Address Valid: true
   IPv4: 10.32.96.82
   IPv4 Subnet Mask: 255.255.255.0
   IPv6:
   MTU: 1500
   Vlan Supported: true
   Vlan ID: 0
   Reserved Ports: 63488~65536
   TOE: false
   TSO: true
   TCP Checksum: false
   Link Up: true
   Current Speed: 1000
   Rx Packets: 4529069
   Tx Packets: 87825
   NIC Driver: tg3
   NIC Driver Version: 3.124c.v50.1
   NIC Firmware Version: FFV7.0.0 bc 5720-v1.17
   Compliant Status: compliant
   NonCompliant Message:
   NonCompliant Remedy:
   Vswitch: vSwitch5
   PortGroup: iSCSI2
   VswitchUuid:
   PortGroupKey:
   PortKey:
   Duplex:
   Path Status: active

~ # esxcli iscsi logicalnetworkportal list
Adapter  Vmknic  MAC Address        MAC Address Valid  Compliant
-------  ------  -----------------  -----------------  ---------
vmhba37  vmk0    d4:ae:52:a2:fd:05               true      false
vmhba37  vmk1    00:50:56:7c:33:7f               true       true
vmhba37  vmk2    00:50:56:78:e6:0f               true       true
vmhba37  vmk3    00:50:56:7b:a0:41               true       true

~ # esxcli iscsi physicalnetworkportal list
Adapter  Vmnic   MAC Address        MAC Address Valid  Current Speed  Max Speed  Max Frame Size
-------  ------  -----------------  -----------------  -------------  ---------  --------------
vmhba37  vmnic6  00:10:18:d4:30:36               true              0          0            1500
vmhba37  vmnic0  d4:ae:52:a2:fd:05               true           1000       1000            1500
vmhba37  vmnic1  d4:ae:52:a2:fd:06               true           1000       1000            1500
vmhba37  vmnic2  d4:ae:52:a2:fd:07               true           1000       1000            1500
vmhba37  vmnic3  d4:ae:52:a2:fd:08               true           1000       1000            1500
vmhba37  vmnic4  00:10:18:d4:30:34               true           1000       1000            1500
vmhba37  vmnic5  00:10:18:d4:30:35               true           1000       1000            1500
vmhba37  vmnic7  00:10:18:d4:30:37               true           1000       1000            1500
vmhba37  vmnic8  00:10:18:e4:31:7c               true           1000       1000            1500
vmhba37  vmnic9  00:10:18:e4:31:7d               true           1000       1000            1500
~ # esxcli iscsi session list
vmhba37,iqn.1992-05.com.emc:apm000846013680000-1,00023d000001
   Adapter: vmhba37
   Target: iqn.1992-05.com.emc:apm000846013680000-1
   ISID: 00023d000001
   TargetPortalGroupTag: 1
   AuthenticationMethod: none
   DataPduInOrder: true
   DataSequenceInOrder: true
   DefaultTime2Retain: 0
   DefaultTime2Wait: 2
   ErrorRecoveryLevel: 0
   FirstBurstLength: 262144
   ImmediateData: true
   InitialR2T: false
   MaxBurstLength: 262144
   MaxConnections: 1
   MaxOutstandingR2T: 1
   TSIH: 145

vmhba37,iqn.1992-05.com.emc:apm000846013680000-1,00023d000002
   Adapter: vmhba37
   Target: iqn.1992-05.com.emc:apm000846013680000-1
   ISID: 00023d000002
   TargetPortalGroupTag: 1
   AuthenticationMethod: none
   DataPduInOrder: true
   DataSequenceInOrder: true
   DefaultTime2Retain: 0
   DefaultTime2Wait: 2
   ErrorRecoveryLevel: 0
   FirstBurstLength: 262144
   ImmediateData: true
   InitialR2T: false
   MaxBurstLength: 262144
   MaxConnections: 1
   MaxOutstandingR2T: 1
   TSIH: 148

vmhba37,iqn.1992-05.com.emc:apm000846013680000-1,00023d000003
   Adapter: vmhba37
   Target: iqn.1992-05.com.emc:apm000846013680000-1
   ISID: 00023d000003
   TargetPortalGroupTag: 1
   AuthenticationMethod: none
   DataPduInOrder: true
   DataSequenceInOrder: true
   DefaultTime2Retain: 0
   DefaultTime2Wait: 2
   ErrorRecoveryLevel: 0
   FirstBurstLength: 262144
   ImmediateData: true
   InitialR2T: false
   MaxBurstLength: 262144
   MaxConnections: 1
   MaxOutstandingR2T: 1
   TSIH: 149
~ #

[MKguy]

Don't mind that too much, the logicalnetworkportal option is just listing all vmkernel ports on the host (in my case too).

DId you make sure what I mentioned first, that the host is actively issuing IO to the iSCSI LUN in question?

What if you execute the following snippet of PowerCLI to query NIC statistics:

Get-VMHost esxhostname | Get-Stat -Realtime -Stat net.received.average | ? {$_.Instance -eq "vmnic4"}

Get-VMHost esxhostname | Get-Stat -Realtime -Stat net.transmitted.average | ? {$_.Instance -eq "vmnic4"}

(Adjust the vmnic number with your iSCSI-bound vmnics if necessary.)

[RussellJM]

yes.  there are 12 VM's on this host connected to the iSCSI VMFS datastore and that datastore had activity.  also there is a single VM doing minor activity to it's RDM.  together, I would expect the nics to do more than 0-1 KBps every few minutes, as indicated by the network real time performance chart.

I'll have to install and configure powerCLI, I dont think I have it installed/configured since I reimaged my laptop.

thanks for the help so far.

[RussellJM]

this is actually from the other cluster that has more RDMs.  I'll add that this cluster has a NFS datastore that is doing a good amount of data, so i expected the management nics to see some traffic, but with the VMHBA showing data too i did not expect the iscsi nics to be mostly idle.

vmhba35 is the software iscsi HBA.  highlighted in red showing a good amount of activity. 

vmnic 4, 5 also highlighted in red, showing barely any activity.  but the management ports (vmnic0, 1) show lots of activity. 

ADAPTR PATH                 NPTH   CMDS/s  READS/s WRITES/s MBREAD/s MBWRTN/s DAVG/cmd KAVG/cmd GAVG/cmd QAVG/cmd
vmhba0 -                       1     0.00     0.00     0.00     0.00     0.00     0.00     0.00     0.00     0.00
vmhba1 -                       2     2.76     1.58     1.18     0.05     0.05     0.10     0.01     0.11     0.00
vmhba32 -                       1     0.00     0.00     0.00     0.00     0.00     0.00     0.00     0.00     0.00
vmhba34 -                       0     0.00     0.00     0.00     0.00     0.00     0.00     0.00     0.00     0.00
vmhba35 -                      55   638.51   614.29    24.22     4.82     0.40     2.47     0.01     2.48     0.00

PORT-ID              USED-BY  TEAM-PNIC DNAME              PKTTX/s  MbTX/s    PKTRX/s  MbRX/s %DRPTX %DRPRX
  16777217           Management        n/a vSwitch0              0.00    0.00       0.00    0.00   0.00   0.00
  16777218               vmnic0          - vSwitch0           1098.63    7.88    2670.29   25.35   0.00   0.00
  16777219               vmnic1          - vSwitch0              0.00    0.00      15.26    0.02   0.00   0.00
  16777220                 vmk0     vmnic0 vSwitch0           1098.63    7.88     701.90   23.39   0.00   0.00
  33554433           Management        n/a vSwitch1              0.00    0.00       0.00    0.00   0.00   0.00
  33554434               vmnic2          - vSwitch1            183.11    0.16     213.62    0.24   0.00   0.00
  33554435               vmnic3          - vSwitch1             61.04    0.10     198.36    0.12   0.00   0.00
  33554439        6290:GEG0AP02     vmnic3 vSwitch1              0.00    0.00      15.26    0.02   0.00   0.00
  33554440        8543:GEG0SQ04     vmnic2 vSwitch1              0.00    0.00      15.26    0.02   0.00   0.00
  33554442        8700:GEG0SQ02     vmnic3 vSwitch1              0.00    0.00      15.26    0.02   0.00   0.00
  33554443       12383:GEG0RS03     vmnic3 vSwitch1              0.00    0.00      15.26    0.02   0.00   0.00
  33554444       36268:GEG0JS01     vmnic2 vSwitch1              0.00    0.00      15.26    0.02   0.00   0.00
  33554445      233318:GEG0VC01     vmnic2 vSwitch1              0.00    0.00      15.26    0.02   0.00   0.00
  33554447      235772:GEG0SP02     vmnic3 vSwitch1              0.00    0.00      15.26    0.02   0.00   0.00
  33554449      238926:GEG0SV01     vmnic2 vSwitch1              0.00    0.00      15.26    0.02   0.00   0.00
  33554451        4687:GEG0SQ01     vmnic3 vSwitch1            167.85    0.28     244.14    0.22   0.00   0.00
  33554453      234732:GEG0WL02     vmnic2 vSwitch1            152.59    0.15     137.33    0.21   0.00   0.00
  33554457      235772:GEG0SP02     vmnic2 vSwitch1              0.00    0.00      15.26    0.02   0.00   0.00
  33554458      302126:GEG0SC02     vmnic3 vSwitch1             15.26    0.01     106.81    0.07   0.00   0.00
  33554460      310462:GEG0WS02     vmnic2 vSwitch1             30.52    0.05      45.78    0.04   0.00   0.00
  33554461      310932:GEG0RS02     vmnic2 vSwitch1              0.00    0.00      15.26    0.02   0.00   0.00
  33554462      311075:GEG0WS04     vmnic3 vSwitch1              0.00    0.00      15.26    0.02   0.00   0.00
  33554464      323497:GEG0WS03     vmnic2 vSwitch1            137.33    0.09     183.11    0.22   0.00   0.00
  50331654           Management        n/a vSwitch2              0.00    0.00       0.00    0.00   0.00   0.00
  50331655               vmnic5          - vSwitch2              0.00    0.00      15.26    0.02   0.00   0.00
  50331656                 vmk1     vmnic4 vSwitch2              0.00    0.00       0.00    0.00   0.00   0.00
  50331657               vmnic4          - vSwitch2              0.00    0.00      15.26    0.02   0.00   0.00
  50331658                 vmk2     vmnic5 vSwitch2              0.00    0.00       0.00    0.00   0.00   0.00
  67108865           Management        n/a vSwitch4              0.00    0.00       0.00    0.00   0.00   0.00
  67108866               vmnic6          - vSwitch4              0.00    0.00       0.00    0.00   0.00   0.00
  83886081           Management        n/a vSwitch5              0.00    0.00       0.00    0.00   0.00   0.00
  83886082               vmnic7          - vSwitch5              0.00    0.00     213.62    0.29   0.00   0.00

[MKguy]

So that's your NFS traffic going through vmk0? You're wright, the vmkernel iSCSI interfaces MUST show activity in esxtop.

Is there no vmk3 on this host or did it just not fit into the esxtop screen (sort with "R" or "T")? If you're not using the Round-Robin PSP, traffic might only be routed through that one only.

Check which IPs are actively connected to your iSCSI target:

esxcli network ip connection list | grep "[IP of iSCSI vmk's or iSCSI target]"

[RussellJM]

On that box, there is no vmk3.  vmk0 is the managemnet interface.  vmk1, vmk2 are the nics bound to the vmhba

~ # esxcli network ip connection list | grep 10.34.96.15
tcp         0       0  10.34.96.104:58725  10.34.96.15:3260   ESTABLISHED      2717  vmkiscsid
tcp         0       0  10.34.96.104:49799  10.34.96.15:3260   ESTABLISHED      2717  vmkiscsid
tcp         0       0  10.34.96.103:54124  10.34.96.15:3260   ESTABLISHED      2717  vmkiscsid
tcp         0       0  10.34.96.103:56899  10.34.96.15:3260   ESTABLISHED      2717  vmkiscsid
tcp         0       0  10.34.96.21:53426   10.34.96.15:3260   ESTABLISHED      2717  vmkiscsid
tcp         0       0  10.34.96.21:56731   10.34.96.15:3260   ESTABLISHED      2064  idle0

.103 and .104 are the new iscsi bound vmk's.  .21 is vmk0, the management interface.

[MKguy]

It's definitely weird to see your management vmk NIC with active iSCSI connections. If iSCSI traffic really is flowing through this connection, it makes sense you don't see any traffic on the other vmks or vmnics. (grep again for "3260"  in case of multiple/different target IPs)

Can you check again on this hos that only the other vmnics are bound to the iSCSI software HBA of your host?

esxcli iscsi networkportal list

Also, in my case only the iSCSI bound vmks are listed with "Compliant true", in your earlier post it showed 3 vmks as compliant:

# esxcli iscsi logicalnetworkportal list
Adapter  Vmknic  MAC Address        MAC Address Valid  Compliant
-------  ------  -----------------  -----------------  ---------
vmhba36  vmk0    00:24:81:7d:17:b6               true      false
vmhba36  vmk1    00:50:56:7b:f2:9d               true      false
vmhba36  vmk2    00:50:56:79:42:f4               true       true
vmhba36  vmk3    00:50:56:73:55:3c               true       true
vmhba36  vmk4    00:50:56:7f:2a:42               true      false

Maybe you have to try to recreate/re-enable the ESXi iSCSI software HBA from scratch or something.

Or wait, now that I notice it:

You're running everything, the iSCSI targets/vmks and the management vmk ports in the same VLAN/subnet 10.34.96.0/24?

This might cause routing issues with ESXi trying to send everything over vmk0, which was only intended for management traffic only. You really should seperate management and iSCSI onto dedicated VLANs and subnets. You could even leave it in the same VLAN for the time being as long as you use different IP subnets, but it's definitely recommend to use seperate VLANs too.

To confirm this theory, post the output from:

esxcfg-route -l

and

esxcfg-route -n (or esxcli network ip neighbor list)

See which vmk resolved the MACs of your iSCSI target IPs.

If you want, you can also test the following to set a static route to your iSCSI target IP, I'd use it with caution though and on a host with test VMs only:

esxcfg-route -a [iSCSI target IP]/32 [IP of local iSCSI vmk]

esxcfg-route -a 10.34.96.15/32 10.34.96.103

But I think this won't work since esxcfg-route expects gateway IPs and won't "convert" it to local interfaces, it may even cause a local loop, so use it with caution.

[RussellJM]

I started checking the things above, and i was afriad i was goign to have to re-IP the iscsi target.  As a last ditch effort to avoid re-iping, I rebooted one host, just to check to see if a service needed to be restarted to start using those vmnics.  I double checked the vmware document I was referencing, and I do not see any "restart service" or "restart host" to apply changes.

But, I now see significant traffic on 1 of the 2 nics dedicated to the vmhba!  I'll need to find the setting to enable both nics, instead of using just the one path.

I'm going to reboot on the other 2 hosts in cluster 1 to make sure this wasn't a fluke, then i'll reboot the hosts on cluster 2.

[MKguy]

Sounds good.

The LUNs are probably running with the MRU (Most Recently Used) mutlipathing policy by default, which is using only one path. You need to enable the Round-Robin multipathing policy on each LUN to spread the traffic among the paths.

You may also want to adjust the path-switching value from 1000 to 1 or another value:

http://www.yellow-bricks.com/2011/03/21/changing-the-psp-from-fixed-to-rr/