Maximum number of nics?

[PaulB2]

I am trying to configure 4 nics in photon v4, 3 work but when I add the 4th all the networking stops.

What is the maximum number of network cards, eth0, eth1, eth2, etc. supported?

Can anyone supply a config with more than 3 nics?

I have been using a Windows VM for testing my vsphere cluster networks and am able to get 10 nics in a Windows server OS.

[Daniel Casota]

Did you try to add additional nics to the vm on the ui or programmatically? Here's a PowerCLI sample (see ref)

get-vm $VM | new-networkadapter -NetworkName "VM Network" -Type "VMXNET3" -StartConnected

Afaik tThe maximum network interfaces is hypervisor-related to the vHW and version.

https://kb.vmware.com/s/article/1003497

https://docs.microsoft.com/en-us/windows-server/virtualization/hyper-v/plan/plan-hyper-v-scalability-in-windows-server

https://cloud.google.com/vpc/docs/create-use-multiple-interfaces

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-eni.html

 

In addition, there might be a specific maximum network interface quota. As example, on Azure GenV2 virtual machine size Standard_B2ms it has a limitation of 3 nics.

New-AzVM : The number of network interfaces for virtual machine ph4 exceeds the maximum allowed for the virtual machine size Standard_B2ms. The number of network interfaces is 4 and the maximum allowed
is 3.
ErrorCode: NetworkInterfaceCountExceeded

 

Quickly checked: On Workstation 9 with a Photon OS 2.0 including 5 vmnics started successfully with 4 nics in operational carrier state = off (networkctl).
Edit /etc/systemd/network/10-eth0.network in section [Match] use Name=eth* instead of Name=eth0,restart then systemctl restart systemd-networkd.

 

[PaulB2]

Thanks for the reply. I spent quite a bit of time trying to get this to work without success. I did find this interesting link which discusses Linux and muti-home settings. https://netbeez.net/blog/avoiding-arp-flux-in-multi-interface-linux-hosts/

Adding the 4th adapter killed all the connections every time. I ended up going with w2019 core which works fine with 10 adapters.

[Daniel Casota]

Would you mind to share your findings? You say you switched to a Microsoft w2k19 core edition because of the fully supported multi-home feature up to 10 nics.

VCSA 7 includes multi-home feature support up with 4 nics. The commercial version of Photon is part of VCSA.

In which other scenarios is the full support of more than 4 nics highly desirable? Are the findings on Arm or on x86_64 cpu architecture?

[PaulB2]

This was on x86_64. My use case is unique. I have ESXi clusters with multiple vlans and when they are configured or when adding a new host I need to validate that all  vlans are functioning. I have one cluster with 19 different vlans. So I build 2 VMs and add all the vlans for testing. Those 2 VMs will remain present for testing future changes.

[Daniel Casota]

Hi , the text-based setup wizard of Photon actually does not support configuring multiple nics + Virtual Guest Tagging (VST).

For testing purposes a kickstart config with systemd network vlan interface setup might be a good starting point. Interesting use case.

[Anil Patil]

This thread gave lot of helpful info.

[ualonso]

I understand these are photon images for Tanzu... if that is the case I'm also interested in adding vmxnet3 NICs but i don't find any info on how to do that

In my case I'm using TKGS and using the VMClass mechanism when creating the cluster only allows to define NICs of type directpath.. but not VMXNET3

Any guidance would be appreciated

Thanks! in avance

[ualonso]

I have found https://docs.vmware.com/en/VMware-Tanzu-Kubernetes-Grid/1.5/vmware-tanzu-kubernetes-grid-15/GUID-build-images-index.html which will allow to create images with several vNICs but there are two questions remaining:

- once downloaded the TKG-Image-Builder from code.vmware.com the package doesn't indicate how to change the hardware spec that will be put in the OVF...

 

- how to bind the vNIC to the network (L2 segment) at cluster creation time?

 

Thanks!

[ualonso]

Answering to the first question myself:

Looking to the code of image-build-ova.py it seems this has to be specified with the following parameter: OVF_TEMPLATE and there is a sample XML in the git repository of the image builder

Any clue on the second question?