iSCSI connection failures during extreme high NBD load

[joergriether]

A little brainteaser

When doing an pure NBD Backup from a physical veeam server to a esxi 4.1 machine to backup a vm located on eql san (ALL equiped with 10 GB interfaces, the veeam 5.0 b+r server, the esxi 4.1 and also the equallogic system - sometimes THIS appears in the Equallogic Logs:

INFO 02.12.10 16:39:38 10eql2 iSCSI session to target '172.16.150.234:3260, iqn.2001-05.com.equallogic:0-8a0906-cd5e5a007-ed2000000524c8f7-10eql1esxsata1' from initiator '172.16.150.35:59312, iqn.1998-01.com.vmware:esx12-27bd5df6' was closed. iSCSI initiator connection failure. Connection was closed by peer.

Exactly four to six seconds later it reconnects.

INFO 02.12.10 16:39:43 10eql2 iSCSI login to target '172.16.150.234:3260, iqn.2001-05.com.equallogic:0-8a0906-cd5e5a007-ed2000000524c8f7-10eql1esxsata1' from initiator '172.16.150.35:60326, iqn.1998-01.com.vmware:esx12-27bd5df6' successful using standard-sized frames. NOTE: More than one initiator is now logged in to the target.

Now this only happens during extremely high bandwith operations, e.g. when about 40% of the 10 GB is used. It seems the ESXi 4.1 iSCSI Software initiator can´t take more and is failing for a very short period of time.

I wondered (as i have enterprise plus) if it would help if i´d create a dvswitch, map it to the vkernel and enable NIOC on it. Did it - BUT this problem persistet. The LUN is disconnected during the nbd backup job for a very short period of time and then reconnects. The 10 GB NIC in the ESXi 4.1 is an intel dualport using ixgbe esxi standard out of box driver.

Any thoughts? And please: This is research. Don´t tell me to use SAN mode - I am curious why that here is happening. And NBD mode via vstorage API (which veeam uses) should easily handle this kind of NBD traffic.

Best regards,

Joerg

[joergriether]

follow-up. i MAY have found the solution, not 100% verified (needs much more time), but it SEEMS the latest ixgbe driver (2.0.84.9) for intel 10 gb modules on ESXi 4.1 seems to solve the drop problem.

Now what i always find very frustrating with ESXi 4.1: These driver updates have to be installed manually via vsphere cli or vma, no way you get em installed via update manager - and they are also NOT inccluded in ESXi 4.1 firmware updates (not even in the very latest two days ago). Can´t understand that

But what is nice (we checked this): If you install the new ixgbe driver and then do a ESXi 4.1 firmware upgrade with update-manager, the new driver will stay in place.

BTW: If you don´t know how to get the driver version of your ixgbe on your ESXi 4.1 machine, just enable local or remote TSM, login and enter

vsish -e get /net/pNics/vmnic0/properties

(just replace the 0 with your nic number you are looking for). Scroll above (use putty, is much nicer) and check the driver version. Driver is too old or vanilla? You can update it with vsphere cli via

vihostupdate.pl –server –username root –install –bundle :\offline-bundle\INT-intel-lad-blablabla-offline_bundle-blablabla.blablabla

NOW If anyone has a solution how to read out the ixgbe driver version with vSphere Client or even UPDATE it, THAT would be more than great.

best regards,

Joerg