change MTU size in an iSCSI network without downtime?

[de2rfg]

Hi,

we have an vSphere 5 Cluster with 10 HP DL580G7 Hosts, 4 Procurve 5412 10ge Switches and 16 HP Lefthand nodes. We are using the vSphere software iSCSI adapter.

At the moment all devices are configured with a standard MTU size of 1500. Is it possible to change the MTU to 9000 without downtime of the whole cluster? I thought about changing the MTU of the pSwitches, then the Lefthands and then the ESXi hosts.

I asked HP about the right order for the changes and received the statement, that this is only possible with a complete downtime of the cluster. The change of the MTU size has to be done at the same time on all devices.

Is this true? If not, can anyone give me a link where the right approuch is described?

[rickardnobel]

I do not think you would have to shut down the whole cluster.

For me the order should be something like:

Physical switch: change to MTU on the iSCSI VLAN to 9000. This should not affect anything.

Virtual switch: change the MTU on the vSwitches connecting to the Vmkernels for iSCSI. Should also not disturb anything.

iSCSI SAN: Next change here. However, I do not know if the specific Lefthand SAN has to do anything for this change to apply. This you must check with HP. Perhaps the interface has to be shut down and up or something. This is SAN specific.

All this steps will not affect the actual MTU used. It is a common misunderstanding that all devices must have the same MTU settings or not the network will blindly sent large frames all over the place. The actual MTU being used is negotiated between the end nodes at the TCP session setup. This means that as long as the VMkernel iSCSI is not changed no frames will be larger than default.

So for the final step you should change the Vmkernel iSCSI adapters to 9000, and now all end-to-end devices must be configured with the higher MTU size. For the larger MTU to start apply a new TCP session to the iSCSI SAN must be setup and I does not know any real good way to easy verify this. If possible, it could be good to set a host to Maintance mode, migrate the VMs, change the Vmkernel MTU and then reboot.

[sparrowangelste]

make sure you're switch ports are jumbo framed enabled. your real switch not the vswtich,

sometimes they are turned off by policy depending on your network admin

[de2rfg]

Thanks for your reply. I will try the described way. Luckily we have 2 new ESXi hosts and a 4 not yet configured Lefthand nodes. So I can try this is an isolated network with a new temp. vlan on the iSCSI switches an only 2-3 test VMs. I guess I will notice if anything goes wrong even with only a couple of VMs if I let them write to their vDisks on the Lefthand LUNs.

Any testcases? vmkping? ping with bigger send size? Don't fragment bit?

[rickardnobel]

de2rfg wrote:

Any testcases? vmkping? ping with bigger send size? Don't fragment bit?

I have a blog post how to use vmkping to test the end-to-end jumbo frames functionality, see this link:

http://rickardnobel.se/archives/992

[de2rfg]

Support tested the whole setup in their lab and it didn't work well. The problems seems to be the communication between the Lefthand nodes when they have different MTU frame sizes. They can not commuicate anymore. They told us that we would need at least a 2h downtime of the whole cluster to change the MTU on all affected devices.

[rickardnobel]

Rickard Nobel wrote:

iSCSI SAN: Next change here. However, I do not know if the specific Lefthand SAN has to do anything for this change to apply. This you must check with HP. Perhaps the interface has to be shut down and up or something. This is SAN specific.

As mentioned before it could be something vendor specific in how (or in what order) the SAN interface should be changed.

However, it seems a bit strange that the Lefthand should not be able to communicate with each other. The frame sizes should (from the so called TCP MSS negotiation) always use the lowest common MTU size.