Help w/ ESXi VM W2016 Network Access

[ksdst1]

Hello, 

I have an ESXi host w/ one physical NIC where the VM W2016 server is using the default vSwitch0, in the default VM Network port group, with the same vVlan ID 0 as the Management Network.  The host's vmk0 is configured with a static IP/dns settings in the physical switch's assigned VLAN and the VM W2016 server also has a static IP (dns, wins settings) in that same VLAN IP range.  

I can access the ESXi host and work with it using its IP in a web browser from another PC in the same physical network VLAN.  I can see send/receive traffic on the W2016 configured E1000e NIC, can open a cmd prompt and ping its own IP, the IP of the ESXi host, but not the IP range's GTWY, or any other physical network host or resource.  

I am not sure why I can get outside traffic.  I have tried changing the vNic type from E1000e to VMXNET 3.  I've tried creating a new port group w/ vlan ID 1 using the same physical NIC and I get the same results.  Tried analyzing with WireShark in all previously mentioned scenarios, and only see network traffic from local netowrked printers, Dropbox traffic, and such (so it is seeing some traffic on the physical network), but the VM only continuously sends out an ARP broadcast from VMware_bd:dp:22 "Who has (the GTWY address) Tell the VM's static IP.

I was successful when I attached a 2nd physical UsbNIC and second wire, setup a new port group to use it, and used the same network IP/settings I have been using for the VM and I have no issues.  Can still access the host via one physical NIC/IP group, static IP, and have full outside network access when the host has the UsbNIC attached, etc.  Though, using two NICs and wires and static IPs is not a long term option.

Help letting me know how I might be configuring the VM network setup incorrectly would be highly appreciated.

[NickDaGeekUK]

An ESXi host with only one NIC is by default in the management network and using the default TCP/IP stack. That must also by default have a gateway on the management network.

Also, you mention VLAN but do not say how you are configuring the VLAN. On the VM in the vNIC or in ESXi in the vSwitch or are you not setting it in either VM or ESXi and letting the switch handle that.

I suspect you are letting the switch handle that and your switch is in Layer 2 mode.

If you are trying to route the VM network out to a different VLAN and gateway you must have interVLAN routing enabled on the switch. If you wish to enable interVLAN routing on the switch it will need to be in Layer 3 Mode and you will have to define gateway IP for the VLANs you wish to route between.

Hope that helps.
Cheers
Nick

[ksdst1]

Hi Nick...tnx for the reply. 

I may have over complicated things in my post.  VLAN is on the physical switch.  My general non-complicated question is how to configure things to allow the W2016 VM on the ESXi host PC to access the internet.  I assumed I could use E1000e vNIC (I also tried the VMXNET 3)  for the VM, put it in the default VM Network port group, and assign it only with a static IP/DNS/WINS/Gtwy of the physical switch VLAN and done.  The ESXi host's net-settings are setup with the same (though there isn't an option to enter WINS address as far as I can tell). 

And if this is not the correct configuration method then what is?

Thanks!

[NickDaGeekUK]

Hi,
My gut reaction is that it is impossible to overcomplicate a description of your network configuration if you want useful answers to the question so don’t worry about that.

I also think that a single NIC on the host providing both management and VM connectivity is the wrong way to do it in a production environment.

Do you have different IP ranges and VLAN for your Internet Gateway and your ESXi management NIC? You didn’t say. You said that the physical switch has an assigned VLAN but you don’t state if that VLAN is the only one on the switch or if the switch is trunking those out to other switches with VLANs and then out to the internet gateway router or if the gateway is plugged directly to the same switch but is in a different IP and VLAN or not. See what I mean about you haven’t overcomplicated it 😉

I am not sure why you state the second NIC is not an option long term, that seems very unusual for an ESXi host in production use. Normal best practice is for at least a failover NIC for the management network. A second NIC for VM use is better practice but, and this is important, it may be the only solution if there is more than one VLAN on the switch. On a layer 2 switch with more than one VLAN there is no route between them, the vSwitch is also Layer 2 only so it cannot route for you either. If there is only a single VLAN and IP range in use for the ESXi, VM and Gateway then I would assume with what you have told me so far it would already be working on the default vSwtich and Static IP.

I am therefore assuming that you have a direct physical connection (no routers or other trunks between the host and your switch and the internet gateway) and that it is in Layer 2 mode but with more than one VLAN in use on it and that you do not have any routers performing inter VLAN routing between your management VLAN and the VLAN you wish the VM to connect to. Given that the second NIC is your only option.


* Put a second physical NIC on the ESXi host, connect it to the switch on a port that has the correct VLAN for the gateway you wish to use assigned to it.
* Create a port group for the VM and a separate vSwitch for the VM to use.
* Connect that port group to the vSwitch and the vSwitch to the second physical NIC.
* Inside your guest OS put a static IP on the virtual NIC inside the VM with the correct subnet, DNS and gateway but do not tag it with a VLAN ID leave that blank on the NIC properties.

I don’t think you need a VMK1 with a static IP for the second vSwitch or a custom TCP/IP stack for this, let me know if that works for you please.

[ksdst1]

Tnx for all the other possible options mentioned, but I'd like to know how to get the setup I described to work as it is not a production setup. 

ESXi host, using one physical NIC, connected to a switch's network port configured with a VLAN and assigned an IP in the VLAN range.  I don't manage the switch so manipulating layer 2 or 3 settings is not possible for me.

In order to get outside internet traffic on the W2106 VM, can the VM use the default VM Network port group w/ that physical NIC, configuring the VM's E1000e NIC w/ an IP in the before mentioned VLAN range (and standard DNS/WINS/Gtwy settings), where both the VM Network port group and the Host's Management port group have default vVLAN id 0? 

If this will not work, how should I modify these factors to get it to work?

[NickDaGeekUK]

Ok so to sum up:
You want to use a single NIC and vSwitch for both ESXi management and the VM port group.
You don't manage the network switch, and you don't know anything about the Layer Mode the switch is in. (but strangely you do know it has a VLAN but haven't said if you know what it is)
You only have a single network port available for your host and, as it is currently configured, you can't get internet network traffic from the VM.

Before we go any further can I ask a serious question, you are not trying to make this work on a firewall secondary interface on a managed router are you? That is a nightmare scenario I have seen before, and I would advise you to call it quits now and call the tech support for your firewall or router.

If this really is just a managed network switch then the obvious conclusion is the ESXi host has a network problem, or it would already be working when you assigned the VM port group to the vSwitch. So let us examine what you have already done and the results.

"I was successful when I attached a 2nd physical UsbNIC and second wire, setup a new port group to use it, and used the same network IP/settings I have been using for the VM and I have no issues."

When you say "have no issues" does that mean you had internet connectivity from the VM and could ping the internal LAN devices too. If so, what does that tell us?

Sadly, not a lot. Because you changed at least two things at once we don't know which one fixed it. Was it the new NIC or the new port group? Also, you didn't say if you plugged the second wire into a different port on the switch or if you created a new vSwitch for this port group and UsbNIC. If you did either, or both, of those, then that is three or four things you changed.

So, let's go back to what you have done and the results and see if it tells us anything.

"I can access the ESXi host and work with it using its IP in a web browser from another PC in the same physical network VLAN."

OK you have LAN connectivity to the host. Doesn't say anything about if that PC also has internet access or most importantly if your ESXi host does.

"I can see send/receive traffic on the W2016 configured E1000e NIC, can open a cmd prompt and ping its own IP, the IP of the ESXi host, but not the IP range's GTWY, or any other physical network host or resource"

That is conclusive; you do not have any outbound connectivity to the LAN from the VM, via the ESXi host management interface. We know the ESXi host is accessible via the LAN, what we don't know is if the ESXi host has internet access or if DNS is working from the ESXi host.

"I've tried creating a new port group w/ vlan ID 1 using the same physical NIC"

Why did you use VLAN ID 1? Is that the VLAN ID on the switch that your host and other physical devices like the PC you mention are plugged into? Do you know what VLAN your PC, Host and Gateway are in? Let's not get bogged down on VLAN. leave ESXi on VLAN ID 0 and the VM host as untagged i.e. no VLAN ID in its NIC.

Going back to your statement that it worked with the UsbNIC and new port group. You need try that experiment again, but this time only change one thing at a time if you can please but first rule out problems in the existing host network configuration.
Can you ping from ESXi, not the VM but the ESXi host itself, out to the Gateway and then the Internet. If not, that is the problem. Check DNS, gateway IP, and subnet masks in every place you can think off VMK and vSwitch and settings in the Default TCP/IP Stack. Can you get internet access from the PC you use to access the ESXi host? Is that in the same VLAN and IP Range as the ESXi host? If not, why is it that way for the PC and not the ESXi host.

If they check out OK then its time to look at the VM connections internally, make sure that the 2016 Static IP, Subnet and Default Gateway are OK and match the external LAN settings.

If that doesn't answer the question of why it doesn't work, then go on to the port group and vSwitch issue.

List if you don't mind the answers to the questions I asked about that successful experiment.
Did you

* create a new vSwtich
* use the same physical port on the switch for your new cable

If the answer to both is yes, then we are back to the "what is wrong with your current ESXi Host setup and why our checks above didn't find anything wrong" question and I don't know what to tell you. The quick fix would be, if you are really are stuck with only one network port on your existing switch, then put in a small unmanaged switch next to your ESXi host, plug in both your NICs and the lead back to the existing switchport and leave both NIC's, vSwitches and port groups active.

If you used a different physical switch port, then what did you unplug to do that and have you tried plugging the existing Host NIC into that port?

If you added the UsbNIC to the existing vSwitch and then added a new VM port group then, unless you disabled the existing NIC the UsbNIC was in load balancing mode with it and you may have a physical NIC issue with the onboard NIC. i.e. you gave the vSwitch a choice of NIC and physical Switch Ports to use and one worked when the other didn't.

When connecting two NIC to a single vSwitch Be very careful as to which ports you plug the two cables into on the switch especially if you are using VLANs and never use separate external switches (unless they are stacked). See here https://serverfault.com/questions/977186/esxi-single-vswitch-with-2-physical-nics It can produce some weird results based on where those two network cables go. When in doubt always create two vSwitches and two Port groups to isolate the traffic.

Hope that helps you.

[ksdst1]

Hi,

You have gone above and beyond with your responses.  I have not made mention of many factors you do mention as I know they are not in play, i.e., i know what works on the managed enterprise network I'm using and our assigned workstation Vlan assigned to our physical ports.

So, yes to "You want to use a single NIC and vSwitch for both ESXi management and the VM port group......you can't get internet network traffic from the VM."  Your statement, "If this really is just a managed network switch then the obvious conclusion is the ESXi host has a network problem, or it would already be working when you assigned the VM port group to the vSwitch", is the most interesting to me.

I hoped it would "already be working", after having created the W2016 VM, on the ESXi host, using the default vNET Managment Network & VM Network port groups (default vlan id 0), assigning the VMs NIC to the VM Network Port group, and configuring the VMs NIC with our standard static IP/DNS/WINs settings that I would assign any of the other physical PCs within the same physical switch assigned VLAN. 

The host can ping itself, the VM's IP, its gateway, outside web addresses, etc., so it is working as I would expect, though the VM can only ping itself and the hosts IP, but nothing else as if vVLAN 0 is its own isolated VLAN that only communicates with the host and the VM. 

My original question was basically, should it be working (communicating w/ other PCs on our network's vlan and the outside world) with the above networking configuration and if not, should it be able to?   If not, are there some modifications to the above network configurations that would allow the VM to communicate outside of it's vVLAN 0. 

Cheers.

[NickDaGeekUK]

VLAN 0 in ESXi is confusing. It isn't a VLAN at all. As I understand it the use of VLAN 0 inside an ESXi host equals "untagged" in network switch terms. So once the VM sends its packet to the host, the host should pass it on as untagged to the external switch.

If you have assigned any VLAN on the external switch to the physical network port the host is plugged into that's another matter altogether.

Any packets exiting that port will be tagged by the switch. Also if the port is in access mode not trunk or if trunked and untagged packets are not allowed in the trunk things get more complicated. You need to know if an untagged packet would be rejected by that physical network port. If that is the case then your VM traffic will be blocked there at the switch port. You will be able to ping the host and no further.

This seems to exactly match the scenario you have at present. However, It doesn't answer the question raised by the USB NIC working unless you plugged that into a different switch port or you chose a different VLAN tag for that setup. Hence my question about the existing setup and VLAN and port configuration on your managed network.

Does that help?

Nicholas Kulkarni

[ksdst1]

I got dragged away from this subject for a few days but now am following up.

As the repeat test, I plugged the additional usbNIC back into the ESXi host PC so it is using two physical NICs. 

Then via the ESXi host's web-console, for the usbNIC I added a VMkernel NIC to link to this usbNIC, created a new vSwitch1, and then created a new VM port group (using default Vlan ID 0) for the VM W2016's NIC to use.  With this configuration, the VM W2016 does have outside internet access.

BTW, both physical adapters connect to ports on the same physical switch, that are programmed to be in our general workstation VLAN.   

In regard to the comment in the last post, "VLAN 0 in ESXi is confusing. It isn't a VLAN at all. As I understand it the use of VLAN 0 inside an ESXi host equals "untagged" in network switch terms. So once the VM sends its packet to the host, the host should pass it on as untagged to the external switch.", it seems that this is correct given that the test described above using two physical NICs, does allow the VM to send\receive traffic with through-put to the outside world w/o VM VLAN ID 0 imposed any VLAN packet tagging for the physical switch to interpret.

Also, I tested if the physical switch can properly interpret packets coming from the VM that have packets tagged with the same VLAN number assigned to the physical switch ports (e.g., the VMs port group is using VLAN ID 100 and the physical ports are also using VLAN 100 tags at the physical switch level.  The result was that the physical switch did not allow "through" traffic from the VM to the outside world.  This seems to indicate that a VM applied VLAN ID don't correspond to a physical switch port's assigned VLAN ID using the same number (100 and 100).

Unfortunately, I am still back at square one when using one physical NIC in the ESXi host PC.  It seems like packets from the VM are not passed on via its vNIC to the host's NIC, on to the switch.  The Hosts vmk0 VMKernel/defaultTcpipStack is configured with a static IP (in the physical switch imposed VLAN range)/Gtwy/DNS.WINS and the VM is also configured the same but with another static IP in the physical switch assigned VLAN.  The VM can communicate with the Host but can't get past the Host.

Not sure why.  Might just have to move on w/o understanding.

[NickDaGeekUK]

When you say you tagged the packets with VLAN ID 100 did you do this at the VM level i.e. with in physical NIC settings or did you do this at VM Port Group level?

------------------------------
IT Manager. That person just about managing to keep the IT going with no budget or staff despite the effects of Senior Management and End Users.

IT Worker. A biological system for turning caffeine into code.
------------------------------

[ksdst1]

Switch port is set to 100 (set by enterprise IT) and VM VLAN ID 100 to test what I thought you intimated earlier, that VM tags w/ VLAN # "x" are recognized as the same as physical port VLAN # "x" tags.  Doesn't seem to be the case.

[NickDaGeekUK]

Sorry, I was asking how you tagged the VM network as VLAN ID 100 inside the ESXI host. Did you use the setting in the property of the VMXNET3 to set a VLAN ID? Or did you set a VLAN ID at ESXi host level on the port group?

Nicholas Kulkarni
IT Manager

5 Star Cases Limited

Switchboard: 01945 427000 | Direct Dial: 01945 427001 | [e-mail]

This e-mail is intended for the addressee(s) named above and any other use is prohibited. It may contain confidential information. If you received this e-mail in error please contact the sender by return e-mail. 5 Star Cases Limited does not accept legal responsibility for the contents of this message if it has reached you via the Internet, as Internet communications are not secure. Any opinions expressed are those of the author and are not necessarily endorsed by 5 Star Cases Limited. Recipients are advised to apply their own virus checks to this message.

5 Star Cases Limited. Registered Office - Broadend Industrial Estate, Broadend Road, Wisbech, Cambridgeshire, PE14 7BQ, England. Reg. No.: 02757991

Please consider the environment before printing this email.

[ksdst1]

Right.  the vm port group vlan id.

[NickDaGeekUK]

I was told years ago to leave the VLAN 0 on the port group and put the VLAN ID you want windows to use in the VMXNET3 properties i.e. leave ESXi out of it and use Windows to Tag the VLAN packets straight out of the VM. At this point you have nothing to lose by trying it. i.e. leave the single NIC, default TCP Stack and VLAN 0 on the management but use the VMXNET3 properties on the WS2016 VM to tag the packets with VLAN ID 100. You need to make sure that VLAN & Priority is also enabled in the VMXNET settings.

Kind Regards,

Nick

[ksdst1]

Hmm.  I'm not sure how to make sure "VLAN & Priority is also enabled in the VMXNET settings".

[ksdst1]

Unfortunately, no go, unless "VLAN & Priority is also enabled in the VMXNET" are manual settings w/ the ESXi Host Client console, though I can't find them.  With one physical NIC on the ESXi Host PC and the VMs Nic in any scenario\config I've tried allow traffic passed the Host's physical NIC.  Oh well.  On to bigger and brighter things.  Cheers!

[NickDaGeekUK]

Hi the settings you are looking for are not in ESXi they are in the VMXNET3 adapter inside the Windows 2016 Network settings for the adapter itself, under properties of the NIC use the configure button and go to the advanced tab.

------------------------------
IT Manager. That person just about managing to keep the IT going with no budget or staff despite the effects of Senior Management and End Users.

IT Worker. A biological system for turning caffeine into code.
------------------------------

[ksdst1]

Sadly, no success after making those changes.  Cheers!

[NickDaGeekUK]

It was worth a try. I still do not understand why it didn't work the first time. There is something we are both missing.

Kind Regards,

Nick