Hi,
Just an update with the current situation, due to our current production environment we have purchased another SAN PS6010, 2 more 8024F switches and two more servers, Dell R610. Though this time I have ordered the Intel 520 10Gb cards instead of the Broadcom. The plan is to dedicate these switches for iSCSI network and move the production switch for the Data LAN only. (Currently sharing the iSCSI and Data)
So we setup the lab environment and configured the switches per some dell/equalogic support guidelines, we noticed that the speed issue still existed. After some troubleshooting we noticed that there was an mtu setting left out of the trunk. It was applied to the network ports but not the channel group. Immediately once we added this we notice a massive speed difference of up to 690MB/s up from the current of 200 to 300 MB running IOMeter 100% Read test.
The following is the switch configuration for the portchannel and interface configuration
interface ethernet 1/xg1
description 'san01-controller0-iscsi'
spanning-tree cost 2000
spanning-tree portfast
spanning-tree mst 0 external-cost 2000
mtu 9216
switchport access vlan 99
exit
!
interface ethernet 1/xg2
description 'san01-controller1-iscsi'
spanning-tree cost 2000
spanning-tree portfast
spanning-tree mst 0 external-cost 2000
mtu 9216
switchport access vlan 99
exit
!
interface ethernet 1/xg19
channel-group 1 mode auto
spanning-tree portfast
mtu 9216
switchport mode trunk
switchport trunk allowed vlan add 99
exit
!
interface ethernet 1/xg20
channel-group 1 mode auto
spanning-tree portfast
mtu 9216
switchport mode trunk
switchport trunk allowed vlan add 99
exit
!
interface port-channel 1
switchport mode trunk
switchport trunk allowed vlan add 99
mtu 9216
exit
From here we ran through various scenarios and tested the failover and logged the information through IOMeter readings to confirm no issues existed with the configuration for the following.
1. Controller Failure, - ran the restart command.
2. Switch Failure - removed the power from one switch at a time
3. iSCSI NIC Failure - Disabled the NIC port on the switch.
All of which passed without a single dropped packet to the VM and consistently running between 660MB/s - 670MB/s. The IOMeter slowed down during the testing, levelled off and then picked again.
Next test was to see if MEM was going to provide any better results than what we were currently seeing, after installing the plugin and running through the exactly same IOMeter tests the results indicated the following.It was slower than not running MEM altogether, I found this bizarre and ran through it several time with the same results.
Moving forward, the next step was to test the R710 servers we had put into production, the real difference between the lab servers we had setup and the production servers were the Broadcom NICs. We removed one of the servers from the cluster and moved into the lab environment. Immediately we saw a difference in performance compared to the Intel cards. We had noticed the IOMeter was running at approximately 220MB/s -250MBs We plugged the server directly into the SAN, this helped eliminate any switch configuration issues, we received the same results of around 250MBs. The VMWare iSCSI environment was setup exactly the same way, my colleague noticed VMware had released a new Broadcom driver specifically for the 10Gb cards. (Note we had tried all previous drivers from VMware and Broadcom and none were successful, we had spoken to tech support from equallgoic and dell and had escalated it, all with no luck.) We applied the latest one 1.62.... released 7 days ago and presto 662 MB/s. We have yet to go through all the scenarios of testing but it appears to be running slightly slower than Intel cards but faster than MEM.
I have attached a baseline reading running the IO Meter, for Intel, with and without MEM and Broadcom without MEM.
From this experience I will lean towards Intel over Broadcom more than ever,Broacom has caused a lot of headaches for my company and clients.
Joe