Ahhhhh okay,
Well if the VM's have to stay in the same port group and you can't split them out your best bet is Private VLANS assuming your physical switches have the capabilities.
Here is a pic that better explains PVLANs again:
Here is another with what you probably want to do:
As you can see you would put a set of your VM's in a community that you want to be able to communicate with each other say VLAN 17 as in this picture, then put another the VM's that you don't want to talk to anything in the Isolated PVLAN 155 as in the picture.
Now once you have your VM's in the proper groups you can either put a software firewall on the Promicuous group or have it just route out to your physical switches assuming they can do PVLAN's and have them route the traffic accordingly.
If you want to test this out in your VMware environment prior to production you can test everything out with test VM's on a test VDS and everything will work as long as all the VM's you are testing stay on the same host / VDS
To quickly go over PVLANs again here is how it breaks down:
- Promiscuous – A node attached to a port in a promiscuous secondary PVLAN may send and receive packets to any node in any others secondary VLAN associated to the same primary. Routers are typically attached to promiscuous ports.
- Isolated – A node attached to a port in an isolated secondary PVLAN may only send to and receive packets from the promiscuous PVLAN.
- Community – A node attached to a port in a community secondary PVLAN may send to and receive packets from other ports in the same secondary PVLAN, as well as send to and receive packets from the promiscuous PVLAN.
Here is some more information on the topic as well to help you along:
kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1010691
vSphere Private VLANs - Dev Environment Use Case
There is a free online lab / course for Distributed Switches in 5.5 but I don't remember if they do PVLANS or not:
VMware - NEE
Hope this helped clear things up,