VMK Gateway Influences Management Gateway and Vice Versa

[bsdixoner]

I have noticed an odd issue with our ESXi 4.1.0 installation when using standard switches (this behavior happened on the distributed switch as well). Basically what happens is when I add a new virtual switch, I give it an IP on a different network with the corresponding gateway and then save it. I add my adapter and carry on. The problem is that the whole host can no longer get to the Internet. I take a look at the gateway on the management network and sure enough it is changed to the gateway I just set on my new vmkernel interface. I change it back and I am good to go, but then my vmkernel gateway is changed to the gateway of the management network. Clearly you can see how this is a problem and should not be happening.

To further illustrate the issue I can describe the configuration:

VMSwitch 1: Management Network

ip: 192.168.1.10

sub: 255.255.255.0

gw: 192.168.1.1

vlan: 110

VMSwitch 2: Test Network

ip: 192.168.2.10

sub: 255.255.255.0

gw: 192.168.2.1

vlan: 210

In the above configuration everything works fine until I add VMSwitch 2. After adding VMSwitch 2 I get the following results:

VMSwitch 1: Management Network

ip: 192.168.1.10

sub: 255.255.255.0

gw: 192.168.2.1 <---- Gateway is now changed and anything on that network can't talk out

vlan: 110

VMSwitch 2: Test Network

ip: 192.168.2.10

sub: 255.255.255.0

gw: 192.168.2.1

vlan: 210

[jcw248]

Try going into the consol through SSH or local tech support and add a static route to get to the network using the IP that you want for the default gateway.

esxcfg-route -a 192.168.1.0/24 192.168.1.1 as an example. I ran into this with my storage. ESXi uses 1 default gateway unlike what you could do in ESX.

[bsdixoner]

I attempted your suggestion, but ran into a few hours.

When I list my routing table I see something like the following:

NETWORK SUBNET GATEWAY INT

192.168.1.1 255.255.255.0 Local Subnet vmk0

I get duplicate route errors when I attempt to add the static route:

esxcfg-route -a 192.168.1.0/24 192.168.1.1

When I try to delete that route it says that it was created with the vmkernel interface and that it cannot be deleted.

Any thoughts?

[jcw248]

Ok what is happening is on ESXi you can only have 1 default gateway. Which is your management interface I would assume. So if you use a different network to connect say to your storage. It will try to go out the default gateway if your storage does not lie on the same segment as the IP you assigned for it. If you run esxcfg-route -l you will see what I mean. There should be 1 default route and the rest are local.

So from your OP I am gathering that 192.168.1.1 should be default gateway for vswitch1. So the 1.1 was just an example that I posted above. So for vswitch 2. If you want it to go out another gateway and not use the default route witch would send you out 1.1. You would need to create a route. For example if your storage was on 192.168.5.12 and you had an IP address for your NICs that connect to your storage of 192.168.2.5. So what you would want to do is put in a route for esxcfg-route -a 192.168.5.0/24 192.168.2.1

192.168.2.1 would be the GW for the 192.168.2.0/24 range.

Does this make sense

[jcw248]

Ok messed the last part up 192.168.2.1 would the gateway for anything trying to get to 192.168.5.0/24

[bsdixoner]

I think we are on the same page and storage is a great example. So I do want our storage traffic to go out a different default gateway from the management traffic.

To make things simple, I will stick with this address notation:

Management GW 192.168.1.1

Storage GW 192.168.2.1

The default route sends everything out through management which is fine. I already have a route (created by vmware with the vmkernel interface) that tells my storage network to use the local subnet as its gateway. I am not sure what VMware defines as the local subnet, so I tried to delete that, but had no luck. For each one of my networks there is a respective gateway that heads back up to a firewall. In total there are 10+ networks to segment traffic. This means that I should have 10+ different routes that route independent of each other.

After reading this thread: http://communities.vmware.com/thread/256089?start=15&amp;tstart=0

It does not look like this will be an easy thing to accomplish. Since VMware only has one network stack, everything routes through that management interface. In some cases (removing vlans) I could ping the vmkernel interface of lets say the storage network. However, since it did not have a valid gateway (it was forced to use the management one), I was not able to get out on to the network.

Does all that make sense?

[jcw248]

Sounds exactly what I ran into with my storage. If you put in an IP that is on the 1.0 for example your service console IP on your ACL on your storage (NFS guy here). We were then able to mount our volumes but my traffic was running with my Service console traffic.

So I had an IP on my storage nics just for example of 192.168.2.15 (192.168.2.1 is the gateway, but you cannot set it through the gui or it will hose up your management). It should be trying to hit my storage device on 192.168.5.22. With putting in the route for 192.168.5.0/24 to go out 192.168.2.1 it then showed up on the correct vmk port, and my ACL on my storage was happy. So basically saying any traffic the is going to 192.168.5.0/24 needs to go out 192.168.2.1.

If your storage was on the 192.168.2.0 network (using above example) it would be able to hit it by using the local route and not needing a gateway route.

[jcw248]

Sounds exactly what I ran into with my storage. If you put in an IP that is on the 1.0 for example your service console IP on your ACL on your storage (NFS guy here). We were then able to mount our volumes but my traffic was running with my Service console traffic.

So I had an IP on my storage nics just for example of 192.168.2.15 (192.168.2.1 is the gateway, but you cannot set it through the gui or it will hose up your management). It should be trying to hit my storage device on 192.168.5.22. With putting in the route for 192.168.5.0/24 to go out 192.168.2.1 it then showed up on the correct vmk port, and my ACL on my storage was happy. So basically saying any traffic the is going to 192.168.5.0/24 needs to go out 192.168.2.1.

If your storage was on the 192.168.2.0 network (using above example) it would be able to hit it by using the local route and not needing a gateway route.

[Phokay]

Hi

First of all you should not have any default gateway on the storage network unless your storage device is on different subnet from the host's vmkernel ports for storage access. Second, you don't want your storage traffic to go out of the local subnet if the host's vmkernel ports for storage access and the storage devices are on the same subnet. Looks like you have your host's vmkernel ports for storage access and the storage devices are on the same subnet. Therefore, you should not set the default gateway at all on the vmkernel ports for storage access.

The best practice is to put storage area network on different physical switches or VLAN on its own.

Hope this makes sense.

[bsdixoner]

So what if the machine is on a different segment. Storage was a poor choice example given you guys are correct, it will remain local. What about networks that extend past the local aspect?

[ejward]

Taking storage out of it, I get the same thing with Vmotion.  I want Vmotion to ride on the same physical NICs as the management console.  So there's no way through the GUI to do this really?