Issue with iSCSI setup

[Doders69]

Hi All,

Like most people I have having a play with the new vSphere stuff, but i have hit a problem with the setup of iSCSI.

The background here is, we have 3 Dell R805 servers with 12 Nics. Connected to 3 Dell PS5000 SANS. What im trying to do on our test server is Configure 4 of those Nic ports for conenctivity to our san.

I have followed the vSphere iSCSI guide to the letter up until page 35, where it talks about Jumbo Frames. This is where im having the issues.

Create a Jumbo Frames-Enabled vSwitch

Configure a vSwitch for Jumbo Frames by changing the MTU size for that vSwitch.

Procedure

1 To set the MTU size for the vSwitch, run the vicfg-vswitch -m <MTU> <vSwitch> command from the

vSphere CLI.

This command sets the MTU for all uplinks on that vSwitch. The MTU size should be set to the largest

MTU size among all the virtual network adapters connected to the vSwitch.

2 Run the vicfg-vswitch -l command to display a list of vSwitches on the host, and check that the

configuration of the vSwitch is correct.

This part I have done succesfully and vSwitch1 shows as being MTU 9000 enabled.

Its this next part that is proving difficult to work out.

Create a Jumbo Frames-Enabled VMkernel Interface

If you are using ESX host, you must use the command-line interface to create a VMkernel network interface

that is enabled with Jumbo Frames.

Procedure

1 Log in directly to the console of the ESX host.

2 Use the esxcfg-vmknic command to create a VMkernel connection with Jumbo Frame support.

esxcfg-vmknic -a -I <ip address> -n <netmask> -m <MTU> <port group name>

3 Run the esxcfg-vmknic -l command to display a list of VMkernel interfaces and check that the

configuration of the Jumbo Frame-enabled interface is correct.

4 Check that the VMkernel interface is connected to a vSwitch with Jumbo Frames enabled.

5 Configure all physical switches and any physical or virtual machines to which this VMkernel interface

connects to support Jumbo Frames.

If i run the esxcfg-vmknic -l command, it shows the 4 nics I have as MTU 1500. I am confused as to what I am actually doing here. what is the esxcfg-vmknic -a -I <ip address> -n <netmask> -m <MTU> <port group name> command actually doing ?.. No matter what I put as the Port Group Name it doesnt work.. All i get when I run the command is the list of options again.

So at this point my testing has come to a stop. How do I get my NICS to show as 9000, and why does the documentation provide commands that dont work ?

I should point out, that we have been running 3.5 succesfully for 12 months, so im not a total newbie to ESX.

Sorry for the long winded post, but i thought it best to explain it properly.

Cheers

Paull

[weinstein5]

You might have already done this but ave you enabled Jumbo frames on the physical side of your network? Jumbo Frames have to enabled on all devices end to end-

If you find this or any other answer useful please consider awarding points by marking the answer correct or helpful

[Doders69]

Hi.. Yes, Jumbo frames are enabled on our Physical switches ( i.e the ones on our storage network ), and also on the DELL SANS.. The only place that its not enabled is as above on the NICS in the ESX Host. The vSwitch that houses the 4 Physical NICS is also Jumbo enabled.

[jasoncllsystems]

Hope this can help! Similar setup -> http://malaysiavm.com/blog/performance-tuning-on-virtual-infrastructure-with-md3000i/

Regards,

MALAYSIA VMware Communities http://www.malaysiavm.com

CLL SYSTEMS http://www.cllsystems.com

#if you found this or any other answer useful please consider allocating points for helpful or correct answers ***

[Doders69]

Jason... I have done that and the MTU on the vSwitch shows as 9000... But there Physical NICS still show as 1500..

Ive now done a test, and have checked the Dell Logs and yes the NICS are connecting at 1500 not 9000, so having the vSwitches MTU set to 9000 makes no difference at all..

Andre, no I dont have licencing for Distributed switches..

Surely there is a way to set the MTU on the Nics...

[AndreTheGiant]

At physical NIC (esxcfg-nics command) there isn't a parameter to change MTU.

But what's the output of esxcfg-nics -l ?

Andre

**if you found this or any other answer useful please consider allocating points for helpful or correct answers

[AndreTheGiant]

So at this point my testing has come to a stop. How do I get my NICS to show as 9000, and why does the documentation provide commands that dont work ?

Do you have the license for distributed vSwitches?

If yes is better that your set the MTU on the DVS using VIC.

Andre

**if you found this or any other answer useful please consider allocating points for helpful or correct answers

[kunhuang]

there is one missing step between "create vswitch" and "add vmknic", that's "create a port group". that's why you are getting error, what ever "prt group" name you are passing to

esxcfg-vmknic -a -I <ip address> -n <netmask> -m <MTU> <port group name>

since you already has vswitch created, try these steps:

1) esxcfg-vswitch -A pg1 vSwitch?

// useyour vswitch name

2) esxcfg-vmknic -a -i DHCP -m 9000 "pg1" "vmkernel 4"

// use static ip if you what with -i ip -n netmask

3) use esxcfg-vmknic -l to verify vmknic MTU

e.g.

Interface Port Group/DVPort IP Family IP Address Netmask Broadcast MAC Address MTU TSO MSS Enabled Type

vmk4 pg1 IPv4 169.254.0.1 255.255.0.0 169.254.255.255 00:50:56:7d:f0:2f 9000 65535 true DHCP

4) use esxcfg-vswitch -l to verify vswitch MTU

e.g.

Switch Name Num Ports Used Ports Configured Ports MTU Uplinks

vSwitch3 64 4 64 9000 vmnic3

5) use esxcfg-nics -l to verify vmnic MTU

e.g.

Name PCI Driver Link Speed Duplex MAC Address MTU Description

vmnic3 43:00.01 e1000e Up 1000Mbps Full 00:1b:78:59:1a:df 9000 Intel Corporation NC360T PCI Express Dual Port Gigabit Server Adapter

6) rescan for sw iscsi adapter, you should be using jumbo now.

Cheers,

- Kun

[Doders69]

Hi All, I appreciate the help everyones given me so far, but im still stuck here... Ive completed the steps from Kun above... This is the current situation -

C:\Program Files\VMware\VMware vSphere CLI\bin>esxcfg-vmknic.pl -l -h 10.63.2.202

Interface Port Group/DVPort IP Family IP Address Netmask MAC Address MTU Type

vmk2 iSCSI2 IPv4 10.63.5.106 255.255.255.0 00:50:56:72:89:2e 1500 STATIC

vmk3 iSCSI3 IPv4 10.63.5.107 255.255.255.0 00:50:56:7d:22:8b 1500 STATIC

vmk4 iSCSI4 IPv4 10.63.5.108 255.255.255.0 00:50:56:71:84:f1 1500 STATIC

vmk0 iSCSI1 IPv4 10.63.5.105 255.255.255.0 00:50:56:70:85:ae 1500 STATIC

C:\Program Files\VMware\VMware vSphere CLI\bin>esxcfg-nics.pl -l -h 10.63.2.202

Name PCI Driver Link Speed Duplex MAC Address MTU Description

vmnic4 23:00.0 igb Up 1000Mbps Full 00:1b:21:20:3e:f8 9000 Intel Corporation 82575GB Gigabit Network Connection

vmnic5 23:00.1 igb Up 1000Mbps Full 00:1b:21:20:3e:f9 9000 Intel Corporation 82575GB Gigabit Network Connection

vmnic8 29:00.0 igb Up 1000Mbps Full 00:1b:21:20:40:08 9000 Intel Corporation 82575GB Gigabit Network Connection

vmnic9 29:00.1 igb Up 1000Mbps Full 00:1b:21:20:40:09 9000 Intel Corporation 82575GB Gigabit Network Connection

Switch Name Num Ports Used Ports Configured Ports MTU Uplinks

vSwitch1 64 9 64 9000 vmnic9,vmnic8,vmnic5,vmnic4

PortGroup Name VLAN ID Used Ports Uplinks

iSCSI1 0 1 vmnic4

iSCSI4 0 1 vmnic9

iSCSI3 0 1 vmnic8

iSCSI2 0 1 vmnic5

PG1 0 0 vmnic9,vmnic8,vmnic5,vmnic4

Any more clues as to where the issue is ??

Cheers

[Andy_Banta]

I think you misinterpretted Kun's recommendation. He wasn't recommending you recreate a new portgroup, just updating the ones you have.

2) esxcfg-vmknic -a -i DHCP -m 9000 "pg1" "vmkernel 4"

// use static ip if you what with -i ip -n netmask

should apply to the port groups you're using. You've created a new port group PG1, which is not what he was intending So,

1) esxcfg-vswitch -D PG1 <your vswitch>

to remove port group PG1

2) esxcfg-vswitch -m 9000 -p iSCSI1 <your vswitch>

esxcfg-vswitch -m 9000 -p iSCSI2 <your vswitch>

esxcfg-vswitch -m 9000 -p iSCSI3 <your vswitch>

esxcfg-vswitch -m 9000 -p iSCSI4 <your vswitch>

to change the MTU on your existing portgroups .

3) Remove and re-add your vmknics

esxcfg-vmknic -d iSCSI1

esxcfg-vmknic -d iSCSI2

esxcfg-vmknic -d iSCSI3

esxcfg-vmknic -d iSCSI4

esxcfg-vmknic -a -i 10.63.5.105 -n 255.255.255.0 -m 9000 iSCSI1

esxcfg-vmknic -a -i 10.63.5.106 -n 255.255.255.0 -m 9000 iSCSI2

esxcfg-vmknic -a -i 10.63.5.107 -n 255.255.255.0 -m 9000 iSCSI3

esxcfg-vmknic -a -i 10.63.5.108 -n 255.255.255.0 -m 9000 iSCSI4

Re-veirfy using the what Kun recommends in steps 3-6.

Let us know of any other problems.

Andy

[kunhuang]

C:\Program Files\VMware\VMware vSphere CLI\bin>esxcfg-vmknic.pl -l -h 10.63.2.202

Interface Port Group/DVPort IP Family IP Address Netmask MAC Address

MTU Type

vmk2 iSCSI2 IPv4 10.63.5.106 255.255.255.0 00:50:56:72:89:2e 1500 STATIC

vmk3 iSCSI3 IPv4 10.63.5.107 255.255.255.0 00:50:56:7d:22:8b 1500 STATIC

vmk4 iSCSI4 IPv4 10.63.5.108 255.255.255.0 00:50:56:71:84:f1 1500 STATIC

vmk0 iSCSI1 IPv4 10.63.5.105 255.255.255.0 00:50:56:70:85:ae 1500 STATIC

It looks like you still have vmknic as mtu = 1500. the only way to get ti to 9000 is to remvoe the port groups iscsi1~4, and readd them with "-m 9000" in the esxcfg-vmknic -a cmd.

try that, and let us know how it goes. also a esxcfg-vswitch output would be good to confirm your vswitch is 9000 mtu.

- Kun

[Andy_Banta]

<span class="jive-thread-reply-body-container">It looks like you still have vmknic as mtu = 1500. the only way to get ti to 9000 is to remvoe the port groups iscsi1~4, and readd them with "-m 9000" in the esxcfg-vmknic -a cmd.

No need to remove and recreate the portgroups. You can change the mtu on those (see my previous response).

You do need to remove and recreate the vmknics.

Andy

[Doders69]

OK Guys, Im now good on all 3 fronts.. Nics are showing as 9000 in both VMKNIC and ESXCFG-NICs as is the vSwitch... The only issue now is that its still only connecting at 1500 on the SAN.. Im sure its all enabled correctly on the physical switches as well as the SAN, so ill do some more investigation... But thanks to all that helped so far... Great community !