iSCSI network setup

[Saintly]

Hi,

I have been going through the iSCSI configuration on my new Dell hardware with the Dell tech today and he configured it differently from how i was picturing it.

This is my first SAN so i was relying on his expertise but it's bugging me so i thought i should ask.

I have 4 ports on my 3 R710 servers dedicated to iSCSI my MD3200i SAN has two cards with 4 ports each. they are connected via 2 seperate (not stacked) 6224 switches.

The way he set them up was each port on the server was a seperate vSwitch with a different subnet. i.e. port 1 is 10.10.10.10|255.255.255.0, port 2 is 10.10.11.10|255.255.255.0 etc. The SAN has card 1 port 1 set to 10.10.10.100|255.255.255.0, card 1 port 2 10.10.11.100 etc, card 2 port 1 10.10.10.101 etc...

port 1 and 3 go via switch 1, port 2 and 4 go via switch 2.

He said that this would provide the best redundancy as well as maximise the speed.

The way i thought it would work (from reading whatever i could find online) was that all 4 ports on the server would be in one vSwitch with one ip address and they would connect via the two switches (which would be stacked) which would then connect to the SAN cards via multiple ip addresses but all in the one subnet.

Was i compleatly wrong? Is his way the correct way or are both ways an option? Is there any advantage to either way or a different way that i should set up before i create my first VM on the SAN?

Thanks for any confirmation or alternatives.

ian

[rickardnobel]

The way he set them up was each port on the server was a seperate vSwitch with a different subnet.

i.e. port 1 is 10.10.10.10 | 255.255.255.0, port 2 is 10.10.11.10 | 255.255.255.0 etc.

Each port has its own vSwitch and the VMKNIC is then connected to one physical switch? You have four ports i.e. VMKNICs on the server for iSCSI?

I guess the reason for this would be to force all connections from e.g. 10.10.10.10 software iSCSI vmknic to use one physical NIC on your server to go directly to the IP on the same subnet on the SAN side.

It seems however like there could be a single point of failure here. If one of the switches would broke down / restart then you would lose all access to the LUNs on that IP network.

[FranckRookie]

Hi Saintly,

The way your infrastructure was configured looks to be the best one. It uses the maximum bandwidth available and insures redundancy. For a confirmation you can have a look at ESX Configuration Guide, page 68:

- "If your host has more than one physical NIC for iSCSI, for each physical NIC, create a separate iSCSI port using 1:1 mapping. To achieve the 1:1 mapping, designate a separate vSwitch for each network adapter and iSCSI port pair."

Your solution would work, but it wouldn't be optimal.

Just a note about Ricnob's comment: there is no single point of failure as every LUN is accessible through two paths, one on switch 1 using network 10.10.10.0 and second one on switch 2 with network 10.10.11.0.

So I would agree with Dell tech's design.

Hope it helps.

Regards

Franck

[rickardnobel]

Just a note about Ricnob's comment: there is no single point of failure as every LUN is accessible through two paths, one on switch 1 using network 10.10.10.0 and second one on switch 2 with network 10.10.11.0.

But do we have this information? Is is stated somewhere that each LUN is accessable through two different IP adresses on the storage side?

[Saintly]

Thanks for the replys / confirmation.

Each lun is appaerntly accessable via any of the 4 ports on the server, the 2 switches and the 8 SAN ports.

vSphere Client claims 8 paths to the lun under storage.

Under storage adapters it says 8 connected targets, 4 devices, 32 paths.

Thanks again

Ian

[JRink]

Are you able to take a picture of your vSwitch/VMKernel setup from the VIClient? Just want to make sure I understand how you set it up...

With (8) ports available on the MD3200i, is it possible to bind specific ports together to achieve greater than 1GB iSCSI speeds? I'm curious about this as I will be implementing a MD3200i in the next week or two. In the past, my iSCSI SANs have only had (4) ports available so I just had 4 available paths at 1GB speed - which is more than fine for my implementations... I'm just curious if there's a way to get 2GB iSCSI speeds but doing some kind of port binding/trunking and if Dell supports this? My thoughts would be this probably isn't supported from either Dell or VMware...

I also haven't found anything from Dell regarding support for iSCSI Round Robin on the MD3200i either so I was originally just planning MRU or Fixed Path. I'm still trying to find a GOOD document from Dell regarding iSCSI setup on the MD3200i. The MD3000i had a very nice tech document available... Does Dell have support forums available that people actually use (i heard something about Dell Tech Center forums)? I admit I haven't found them yet.

Thoughts?

J

[JRink]

Ian,

Was there a reason you opted to create (4) vSwitches each with a VMKernel each, as opposed to creating a (1) vSwitch that contained all 4 pNICs and 4 seperate VMKernels in that single vSwitch -- then binding the each VMKernel to a specific NIC?

Both options seem to work and I haven't found that one way works better than the other - yet. I've always setup a single vSwitch with multiple VMKernels inside the vSwitch asthey document it in the VMWare iSCSI configuration guide (page 39 or thereabouts in the 4.1 iSCSI guide).

J

[Saintly]

Hi,

If there are specific screens that you would like to see, let me know and i'll do my best to get them. I'm still learning and may have missed a step or two that he did.

In the Dell MD3200i management program there is a screen like this that alows you to set an IP address on each port.

I had the port list down so you can see the list.

Each port was on a different subnet, each module was on a different IP.

i.e. module 0, port 0 is 10.10.10.101

module 1, port 0 is 10.10.10.102

module 0, port 1 is 10.10.11.101

module 1, port 1 is 10.10.11.102

etc

In the client, here is most of my networking tab

I don't belive any of the ports are bound together. the binding thing was what i was expecting but wasn't the way he set it up.

I do remember seeing the guy set the ports to round robin but i can't seem to find that screen at the moment

Apparently the dell forums are not used that much. i found a lot of unanswered questions.

The tech is coming back tomorrow to update the switches firmware (delivered a few years out of date) and i'll be asking him some more questions for my docs, i hope i will have a better understanding after that. also, i'm not sure if jumbo frames was fully set up as we had a lot of issues setting the switches first and i think he missed a step.

Ian

[JRink]

I'd be curious to hear his answer as to why he setup 4 seperate vSwitches as opposed to a single vSwitch that included 4 VMKernels and 4 vmNics (and then assigning each VMKernel to a specific vmNic). Jumbo frames is pretty easy to do, just need to make sure your VMkernels, vSwitches, Physical Switches, and the SAN are ALL enabled for Jumbo.

The round robin option (or MRU, Fixed) is located under the storage/datastore properties itself... I believe there's a button somewhere in there called Managed Paths or something...? I don't have the screen up in front of me.

JR

[bmeadows]

JRink,

Here's the Dell Tech document that has been updated for the MD3200i and ESXi 4.1. This guide goes through each CLI command and its GUI counterparts. It is the same way the Dell tech set it up, except they use one vSwitch, with each vmkernel port being bound to one and only one pNic. They don't go into much detail about IP addressing scheme for the iSCSI ports in the controllers, but in the situation with two switches that aren't stacked, I agree with setting up two separate subnets. If the switches were stacked, I would probably just do one subnet.

http://www.dell.com/downloads/global/products/pvaul/en/powervault-md32xxi-deployment-guide-for-vmware-esx4.1.pdf

[bmeadows]

In the Dell iSCSI SANs (at least the ones I've configured) every LUN is available through both controllers by default. Each LUN has its preferred owner and you will get an alert if LUNs are being accessed through the non-preferred controller.