vSwitch VLANs

[xlcor]

If I create two new VLANs on my vSwitch, which are completely isolated (ie. the VMs within each can only interact with other VMs within their respective VLANs, and no external VMs, physical servers or clients can interact with either VLAN "bubble"), is there any way to "connect" those VLANs such that they can fully interact only with each other? I realize the limitations in a physical switching/routing environment, but I'm not sure if a vSphere environment works the same way...

thanks in advance!!

[a_p_]

The only way to do this, is to setup another VM as a virtual router.

André

[xlcor]

the VM would exist in which environment? how would one set this up?

let's say I have VLAN 110 is my standard production environment, and VLAN 130 is my first isolated bubble and VLAN 131 is my second isolated bubble.

[a_p_]

You would create another portgroup for this VM, which allows all VLANs. The router software has to be able to do VGT (VLAN guest tagging)

for VGT see http://www.vmware.com/files/pdf/virtual_networking_concepts.pdf

André

[jpdicicco]

the VM would exist in which environment? how would one set this up?

let's say I have VLAN 110 is my standard production environment,

Which, as I understand it, you don't want to talk to either bubble...

and VLAN 130 is my first isolated bubble and VLAN 131 is my second isolated bubble.

You use a 2nd vSwitch with no pnics. Put both bubble VLANs on the new switch. Add a VM to both VLANs running the OS of your choice to act as the router between the VLANs.

Happy virtualizing!

JP

Please consider awarding points to helpful or correct replies.

[Texiwill]

Hello,

To allow the two 'bubbles' to talk to each other you need to bridge the portgroups on which those VLANs reside using some form of virtual appliance. virtual router, gateway, firewall all work. You can keep the two portgroups isolated from all else as well.

You can use a Private vSwitch with 2 portgroups or two private vswitches. But without some way to bridge between your VLAN bubbles (such as a virtual appliance) there is no way to communicate between them.

Which virtual appliance style you choose, firewall, gateway, router is up to you.

Best regards,
Edward L. Haletky VMware Communities User Moderator, VMware vExpert 2009, 2010

Now Available: 'VMware vSphere(TM) and Virtual Infrastructure Security'[/url]

Also available 'VMWare ESX Server in the Enterprise'[/url]

Blogging: The Virtualization Practice[/url]|Blue Gears[/url]|TechTarget[/url]|Network World[/url]

Podcast: Virtualization Security Round Table Podcast[/url]|Twitter: Texiwll[/url]

[xlcor]

So in following those details, I have setup a new vSwitch, with as of now, a single associated port group, attached to fake VLAN 130. When I try to change the port group association in the config settings of an existing VM to use this new port group / VLAN, it doesn't show up in the list. When I setup the new vSwitch I did NOT associate any adapters with it. Is this causing this inability to associate the port group with the existing VMs?

[a_p_]

I read it but I can't believe it :smileywink:

Are you sure you added a "Virtual Machine" port group on the new vSwitch and did not accidently add a VMkernel port group?

André

[xlcor]

LOL thanks! I'm admittedly a noob, but not THAT bad!! LOL

nope, just eyeballed it again in case it was my bad, but alas no. I created a new vSwitch (vSwitch2, as I already have two existing), and added a virtual machine port group called ocx-staging-test, associated with VLAN 200 (which is a fake, it doesn't exist on any physical switch). I added no adapters to vSwitch2, and when I try to select the new port group from the drop down list in the vNIC settings of the appropriate VMs, its not in the list.

any ideas?

[a_p_]

I never saw something like that.

Which version build of ESX do you use?

Can you add another virtual machine port group, just default without VLAN configuration, to see whether this one shows up in the list?

André

[xlcor]

sorry, should have stated that earlier. ESX 4.0, not sure of the build though.

I added another virtual machine port group with no VLAN association, and it does NOT show up in the list available to the VMs.

one thing I noticed though, I've been creating these networking components from the Networking section of the Configuration tab of one of my 6 hypervisors (randomly chosen). when I look in the same sections on the other 5, my networking additions (the vSwitch and the virtual machine port groups) don't show either. it just so happens that the VMs, to which I want to assign the isolated virtual machine port groups on the new vSwitch are on different hypervisors than the networking components which I had just created.

why are the new networking components not propagating to the other hypervisors? and is this my problem? have I been creating the networking components incorrectly? I tried creating them from the Inventory->Networking view (in the vSphere client), but there was no option to create anything other than a distributed vSwitch, which was not what I was looking for...

[a_p_]

why are the new networking components not propagating to the other hypervisors? and is this my problem? have I been creating the networking components incorrectly? I tried creating them from the Inventory->Networking view (in the vSphere client), but there was no option to create anything other than a distributed vSwitch, which was not what I was looking for...

You have to create standard vSwitches and port groups on each host separately. They are not replicated.

This is one of the pros for the new "Virtual Distributed Switch" in vSphere 4.x.

André

[xlcor]

argh!!!! are you kidding me??? so we should've setup two distributed vswitches for the environment instead of two on each hypervisor!!!!

although, wait a sec, when I went through the distributed vswitch setup process, I was stymied at the Add Hosts and Physical Adapters section, as there was no population in the list that I was to select from. Even when I chose the Add Later option, later there was still no population of hosts or physical adapters to choose from. How is that field populated? (its quite useless otherwise)

[a_p_]

argh!!!! are you kidding me??? so we should've setup two distributed vswitches for the environment instead of two on each hypervisor!!!!

No, what I said was that when using Standard vSwitches (which I assume you have in place) you have to create them on each host separately.

The use of a dvSwitch is an option you have with the Enterprise Plus version. However that's a completely different setup.

although, wait a sec, when I went through the distributed vswitch setup process, I was stymied at the Add Hosts and Physical Adapters section, as there was no population in the list that I was to select from. Even when I chose the Add Later option, later there was still no population of hosts or physical adapters to choose from. How is that field populated? (its quite useless otherwise)

If you want to create a dvSwitch you will need at least 1 free NIC. You should not start creating a dvSwitch unless you are familiar with all the pros and cons of this.

André

[xlcor]

ah! OK, so I'm good, I just need to replicate my process on all hypervisors (yuck), and not monkey around with distributed vswitches, which I don't understand anyways.

that sounds much better!!

[chadwickking]

Hi,

If you are wanting to be able to make changes to multiple ESX servers I would recommend looking into powerCLI, vCLI, or a Vma appliance. It essentially scripting to do the work for you. It helps us keep thing exactly the same across all our host when doing setups and configurations. Just thought I would throw that out for you. :smileywink:

Power CLI: (uses powershell)

http://communities.vmware.com/community/vmtn/vsphere/automationtools/powercli?rls=com.microsoft:en-us&ie=UTF-8&oe=UTF-8&startIndex=&startPage=1

vCLI:

http://www.vmware.com/support/developer/vcli/

vMA:

http://www.vmware.com/appliances/directory/178973

Cheers,

Chad King

VCP-410 | Server+

"If you find this post helpful in anyway please award points as necessary"

[xlcor]

so I have a few VMs in my isolated port group, on my new vswitch, but, eventhough they're addressed using consecutive IPs, they can't talk or even ping. I'm assuming this requires some sort of virtual router appliance? something to placehold for a default gateway?

[jpdicicco]

so I have a few VMs in my isolated port group, on my new vswitch, but, eventhough they're addressed using consecutive IPs, they can't talk or even ping. I'm assuming this requires some sort of virtual router appliance? something to placehold for a default gateway?

You shouldn't need one. Make sure it's not a private VLAN, you're not doing any tagging in the guests, and you have an appropriate subnet mask for them.

Also, make sure you have the NICs set to Connect at poweron in the VM settings, and it is enabled in the guest.

Happy virtualizing!

JP

Please consider awarding points to helpful or correct replies.

[xlcor]

OK, after all the postings and explanations, I know this sounds like super noob time, but none of the VMs in my isolated port group can talk to each other. I have 10 test VMs in this new, isolated stating environment. I did P2V migrations for some and created new VMs from templates for others. I even setup a VM router appliance to act as a fake default gateway, just in case. No go. I have static IPs setup for the Windows servers (the linux servers won't even let me define NICs, but I think that might be an issue with VMWare Tools, which I'll post about elsewhere) and the router, and none of them can see each other.

Here's the named environment: I created a new vswitch2, with no defined physical adapters. in it, I created a virtual machine port group called "ocx_staging_lan" to which I assigned a fake VLAN 200. this vswitch and port group I propogated to all 6 of my ESX 4 hypervisors. I created a test workstation (staging_workstation at IP 192.168.115.200) and downloaded and setup a router appliance (staging_router at IP 192.168.115.1) as a default gateway. I then successfully P2V migrated 8 physical servers into vsphere, and configured each of their vNICs to use the "ocx_staging_lan" port group (and, I presume, be members of the vVLAN 200). all of the P2V migrated servers have 192.168.115.x addresses. all the servers have their NICs connected and all are assigned to "ocx_staging_lan".

am I missing something fundamental here? everyone in the community thinks this ought to work, even without the router appliance, but its not. this is really getting aggravating... any and all suggestions welcome!

[a_p_]

Instead of creating 1 vSwitch with multiple VLAN'd port groups, create 1 vSwitch with only 1 port group (no VLAN configured) for each of your subnets.

It should actually work to remove the VLAN setting from your current port groups. In this case all VM's in the same IP subnet should be able to communicate with each other. However with only 1 vSwitch a VM could access the other VM's by just modifying the IP address. Therefore I would go with the 1 vSwitch per IP subnet method.

André

[xlcor]

Well it sounded really promising. I ensured that all hosts were in the same physical subnet (192.168.115.x), removed the extra virtual machine port group on the new vswitch (leaving only the one, "staging_ocx_lan), removed all VLAN references from each port group instance on each hypervisor, and even removed the router appliance from that port group (leaving only 11 VMs, all on the same subnet), and none of them can ping any other one, always Destination Host Unreachable (for the linux hosts) and Request Timed Out (for the Windows hosts).

[chadwickking]

Have you tried the following?

go to vsphere switch -

Properties > Select vswitch > Edit > Security > Promiscuous to accept > Ping away.....

Ensure your static IP's and Subnets are all the same - I AM SURE you checked like 50 times considering how this would drive me nuts!!! I am facing some issues as well - oh well it is what it is :smileyhappy:

Some information on the different switch modes:

http://pubs.vmware.com/vi35/wwhelp/wwhimpl/common/html/wwhelp.htm?context=server_config&file=sc_adv_netwk.6.4.html

Cheers,

Chad King

VCP-410 | Server+

Twitter: http://twitter.com/cwjking

If you find this or any other answer useful please consider awarding points by marking the answer correct or helpful

[rickardnobel]

Have you tried the following?

go to vsphere switch -

Properties > Select vswitch > Edit > Security > Promiscuous to accept > Ping away.....

I would not recommend modify that settings as it creates a very large security issue. It should only be done for VMs running certain software such as IDS or network analyzing.

To Xicor, these VMs that you are trying to make contact between, are they on the same ESX/ESXi host or on different hosts?

Have you actually got any VMNIC connections for the vSwitches your VMs are on?

[chadwickking]

Essentially go to the host service console and run esxcfg-vswitch -l .

Could you run it on the Host that has all the VM's?

Also run esxcfg-vswif -l and paste it as well so we can see your switching and swif setups.

As said before the promiscious mode is a "security risk" but we were doing a test.

Cheers,

Chad King

VCP-410 | Server+

Twitter:

If you find this or any other answer useful please consider awarding points by marking the answer correct or helpful

[jpdicicco]

Alright, here's what your config should look like from each VM out though each host and the pSwitch:

VM has NIC (VMXNET3, E1000, whatever) that has IP configuration in the guest for the local subnet, and a gateway if it needs to talk to systems that are not on the subnet.

The VM configuration should have the VM NIC set to the correct port group.

The portgroup is on the correct vSwitch and has the correct VLAN ID associated (if it will be connected to a port that is trunked/uses VLAN tagging).

The pNIC correct is associated with the vSwitch. You need pNICs in order for guests on different hosts to talk to each other, even if they're in the same VLAN or on a vSwitch with the same name.

The switch port configuration associated with the host NIC/pNIC is correct, allowing for your VLAN traffic either as the native VLAN or as a tagged VLAN.

Since you are using multiple hosts and therefore need to engage your physical network, you may want to have your network team setup routing, rather than using a multi-homed VM. It should be possible to configure security on a router interface for each VLAN to only allow traffic that is routed to the other VLAN that it is allowed to talk to. This prevents the need for a dedicated router VM.

Check the config at each layer and let us know. Start with 2 VMs on the same host, and verify that they can ping each other, then work your way out through the configurations from there.

Happy virtualizing!

JP

Please consider awarding points to helpful or correct replies.

[EshuunDara]

Wouldn't it be easier to configure the VLANs to talk just like you would if this were a physical environment? Just configure the link to your ESX Servers as trunk, and then set the ports on the virtual nics to whatever VLAN you need and call it good. This way your ESX hosts can have guests on any vlan, and you use your networking equipment that is already in place to define what vlans can talk to each other.

[xlcor]

Whew!! OK, the issue is finally resolved!!! Sorry for the delay in posting my findings... the result seems to be a combination of advice given by jpdicicco, ricnob and a.p. (although I do thank everyone who contributed, because I certainly learned alot from all you experts out there, and from the process in general!!).

The basic rule of thumb is, if you setup a vswitch with no adapters, and all associated VMs are using the same virtual machine port group (like was the case in this situation), then all VMs need to reside on the same, single hypervisor. Given that the vswitch has no physical adapters associated with it, for that port group, there can be no communication outside of the local host (ie. the local ESX server), not even via a dedicated vmotion VLAN (which I have, using a dedicated vmkernal port group).

Not knowing this, for load balancing, I had all the VMs spread across all of my ESX servers. I had to temporarily assign them to an actively connected port group, vmotion them to a single ESX server, change them back to using the isolated port group, and voila!! All the VMs could now ping each other! (I had preconfigured all their static IPs to be on the same flat subnet) and I didn't need the VM router appliance that I had configured!

Again, thanks so much for all the contributors, I have learned alot from this experience!!!