Multiple NICs on Multiple vLANs

[TheCableGuy96]

Hi,

 

I'm new to ESXi and have managed to get hold of an old Dell R720xd and put ESXi 8 on it.

 

It has 4 NICs and I'm really struggling to see how I can assign each one a different IP on different vLANs so I can assign different nodes t different vLANs.

 

Would anyone be able to offer me any help please?

 

Thank you in advance!

[Duncan Epping]

You don't need to assign IP addresses to the NICs itself. You assign an IP to the different types of interfaces you need, like for instance: management, vMotion, iSCSI etc.

You can create what VMware calls "portgroups" and set a VLAN ID on that level, if you then add VMs to that portgroup the traffic will be tagged with that VLAN. This is what is called "virtual switch tagging". Some documentation can be found here:

https://kb.vmware.com/s/article/1004074

https://kb.vmware.com/s/article/1003806 

[TheCableGuy96]

Hi depping,

 

Thank you for your reply. 

So from those articles I believe I need to setup "Virtual Switch Tagging". I have managed switches and know how to set tagged ports so I can manage that.

I have created the port groups now, but I don't see how to assign each one an IP address? When I try to edit the settings for it there are no IP options, please could you clarify where to do this?

Also with the port groups I am wanting to setup "Route based on IP hash" NIC teaming (from what I have read this is the best one for load balancing). I am worried though that when I turn this on I may lose access to the server if not setup correctly. I have managed switches but when I switch this on do I simply turn on LACP on the switch for each port or is that something different? If it said "LACP" in the options I would be more confident but it doesn't so I'm not 100% it's the same thing?

Thank you again for your help.

[Duncan Epping]

Why do you want to give the NIC an IP address? That is the whole thing, you don't need to give a NIC an IP address, you give the "vmkernel interface" an IP address, which then takes care of "management" traffic, or "vMotion traffic". VMs will get their IP assigned in the operating system etc.

[TheCableGuy96]

Okay sorry it's just all new to me.

So I've now created a new VMKernel NIC and set it to the new port group for the additional vLAN. I have also set the IP and subnet and assigned it to that to a new TCP/IP stack I created for this vLAN (had to do this in SSH).

However, when I go to set the gateway and DNS settings for the new TCP/IP stack it's greyed out. The gateway below this is not reyed out but fails to set. So I googled around and found someone else who had the same issue. I followed what they did:

esxcli network ip interface ipv4 set --interface-name=vmk1 --type=static --ipv4=192.168.10.40 --netmask=255.255.255.0 --gateway=192.168.10.254

This appears to have worked from what I see:

[root@esxi:~] esxcli network ip interface ipv4 get
Name IPv4 Address IPv4 Netmask IPv4 Broadcast Address Type Gateway DHCP DNS
---- ------------- ------------- -------------- ------------ -------------- --------
vmk0 192.168.1.40 255.255.255.0 192.168.1.255 STATIC 192.168.1.254 false
vmk1 192.168.10.40 255.255.255.0 192.168.10.255 STATIC 192.168.10.254 false

But it's not showing as being set in ESXi Host Client and I still don't see how to set the DNS for it?

192.168.10.40 is also not responding to ping even though I set all 4 NIC ports as part of that vLAN (I've tried tagged and untagged).

Thanks for your support.

 

[TheCableGuy96]

I keep trying to reply but the system keeps removing it and saying my reply has been marked as spam... not sure how to respond now?

[TheCableGuy96]

So I've now created a new VMKernel NIC and set it to the new port group for the additional vLAN. I have also set the IP and gateway and assigned it to that to a new TCP/IP stack I created for this vLAN (had to do this in SSH).

However, when I go to set the gateway and DNS settings for the new TCP/IP stack it's greyed out. The gateway below this is not reyed out but fails to set. So I googled around and found someone else who had the same issue. I followed what they did:

esxcli network ip interface ipv4 set --interface-name=vmk1 --type=static --ipv4=192.168.10.40 --netmask=255.255.255.0 --gateway=192.168.10.254

This appears to have worked from what I see:

[root@esxi:~] esxcli network ip interface ipv4 get
Name IPv4 Address IPv4 Netmask IPv4 Broadcast Address Type Gateway DHCP DNS
---- ------------- ------------- -------------- ------------ -------------- --------
vmk0 192.168.1.40 255.255.255.0 192.168.1.255 STATIC 192.168.1.254 false
vmk1 192.168.10.40 255.255.255.0 192.168.10.255 STATIC 192.168.10.254 false

But it's not showing as being set in ESXi Host Client and I still don't see how to set the DNS for it?

192.168.10.40 is also not responding to ping even though I set the ports as part of that vLAN (I've tried tagged and untagged).

Thanks for your support.

[TheCableGuy96]

So I've now created a new VMKernel NIC and set it to the new port group for the additional vLAN. I have also set the IP and gateway and assigned it to that to a new TCP/IP stack I created for this vLAN (had to do this in SSH).

However, when I go to set the gateway and DNS settings for the new TCP/IP stack it's greyed out. The gateway below this is not reyed out but fails to set. So I googled around and found someone else who had the same issue. I followed what they did:

esxcli network ip interface ipv4 set --interface-name=vmk1 --type=static --ipv4=192.168.10.40 --netmask=255.255.255.0 --gateway=192.168.10.254

This appears to have worked from what I see:

esxcli network ip interface ipv4 get
Name IPv4 Address IPv4 Netmask IPv4 Broadcast Address Type Gateway DHCP DNS
vmk0 192.168.1.40 255.255.255.0 192.168.1.255 STATIC 192.168.1.254 false
vmk1 192.168.10.40 255.255.255.0 192.168.10.255 STATIC 192.168.10.254 false

But it's not showing as being set in ESXi Host Client and I still don't see how to set the DNS for it?

192.168.10.40 is also not responding to ping even though I set the ports as part of that vLAN (I've tried tagged and untagged).

Thanks for your support.

[TheCableGuy96]

 I cannot reply with the information you need to continue my support because the automated system keeps removing it. I've tried several times editing it but it still deletes it as spam when it's not. 

[Duncan Epping]

I just unmarked them as spam. I am wondering what you are trying to achieve here? 

[TheCableGuy96]

I'm trying to achieve 2 things....

1) I want my virtual switch to have multiple VMKernel NICs, each one residing on a different vLAN assigned to a different gateway and specify the DNS servers (just like you would setup any NIC but each on is on a different vLAN).

2) Once that is done I want to setup LACP with the 4 NICs.

That is all but I'm struggling.

[Duncan Epping]

I don't understand why you need multiple vmkernel interfaces to configure LACP/portchannels? It feels like you are applying "OS" logic to ESXi, which doesn't work like that.

[TheCableGuy96]

The two are separate.

 

I need multiple vmkernel interfaces so I can assign different VMs to different vLANs.

 

The LACP is just so there's plenty of load balancing through the 4 NICs.

[Duncan Epping]

you don't need VMkernel interfaces to assign VMs to different VLANs, you need different Portgroups, and assign VLAN IDs to those portgroups and connect the VMs NIC to that portgroup and then configure IP / Gateway / DNS within the VM itself.

For Load Balancing you could simply go with "route based on virtual port id", which provides load balancing based on the virtual nics connected, much easier to configured and typically just as efficient.

[TheCableGuy96]

Ahh okay that makes sense now.

So on the switch the ports for all NICs will be tagged or untagged?

 

And will DHCP still work on different vLANS? For example, if I set a VM to vLAN10 will it pickup the DHCP server on that subnet automatically?

[Duncan Epping]

On the switch port for the NICs you typically only allow the VLANs which would be used by the system. (Management, vMotion, VM Portgroups etc) As described here: https://kb.vmware.com/s/article/1003806. 

Yes, DHCP still works.

[TheCableGuy96]

perfect, thank you for all your help!!!

[jc296]

BTW - you will need Enterprise licensing iand a vDistributed Switch f you want to utilise LACP or "Route Based upon Physical nic load".

depping is spot on: keep it simple. I would make all the physical switch ports as Trunk and the create 1 or more std vSwitches and add your physical adapters in esxi networking accordingly.  Add as many vSW Port-groups as you need for your individual VM VLANS/vnics.

I would set aside 1 physical adapter for your esxi mngt vmkernel port (with IP address) as a back door.

[TheCableGuy96]

I think I've done it correctly....

I've got a virtual switch which has all 4 NICs added as "uplinks".

The NIC Teaming is set to "Route based upon originating Port ID"

It's showing as vmnic0 is Active and all the others are on standby.

I've added all the vLANs as port groups and on the switch added each NIC port to trunked on each vLAN. 

Does that sound right?

[a_p_]

Except for the Active/Standby Configuration it looks ok.

Although Active/Standby will work, it will limit network traffic to the active vmnic.
I'd suggest that you set all vmnics to Active. ESXi will then assign the VMs to the 4 vmnics in a round-robin manner as the VMs are powered on.

André

[TheCableGuy96]

Okay that's done... I'm just having an issue with the vLANs.

I've created my vLANs, 1, 5, 10,etc...

I've booted from a vmdk and it wasn't responding. So I looked at the switch and set the ports for the NICs to "T" on vLAN1 and the VM came online, but then I lost access to the VMWare Web Interface.

I thought then maybe I don't need a to create a port group for vLAN1, but there's no option when creating the vLAN1 to select anything other than the created groups.

So how do I get some VMs on the default vLAN1?

[a_p_]

A VLAN ID on a virtual port group must only be set for VLANs other than the physical switch's native/default VLAN.

André

[TheCableGuy96]

Sorry I'm not totally understanding...

The management interface is set to vLAN0 by default but it's residing on vLAN1.

If I want to assign a VM to vLAN1 there's no option for the management vLAN when creating a VM, just all the other vLAN groups I created.

So to assign a VM to vLAN1 I have to create another group and assign it that that vLAN1.

Once it boots into it I can't access it unless I trunk the ports but then in doing so it stops access to the web interface on the management interface. 

 

[TheCableGuy96]

The only way around it I can think of is to change the management vLAN from 0 to 1 and trunk the ports then I think I should be able to access both the management web interface and the VM itself?

[a_p_]

hat you may do is to simply create a new Virtual Machine Portgroup with the desired name (can be "VLAN1") but let the VLAN-ID for that port group at its default "0".

The reason for this is that the physical switch sends network packages for the native/default VLAN as untagged packages. ESXi forwards such packages to port groups that have no VLAN-ID set (i.e. VLAN-ID 0).

André

[TheCableGuy96]

that's done the trick thank you!!!

One last question if you don't mind?

I installed an OVA/OVF which was Dell OpenManage Server Administrator. It's now responding and online but I cannot access the web interface. I was wondering if the ESXi firewall might have something to do with this? Or does the ESXi firewall only apply to the host?

 

Thanks again!

[a_p_]

ESXi itself does not restrict traffic from and to VMs.

André

[TheCableGuy96]

Okay issue is the VM then, thanks for all your help everyone