Bare metal Edge interfaces are not coming up

[christophn]

Hi community members,

we are setting up our NSX-T test environment.

We have installed the three NSX-T Managers and joined four Bare Metal Edges (two for T0 Gateway and two for Briding). After configuring the Edge Nodes the interfaces used for the Overlay (TEP) are coming up but neither the two interfaces used for routing to the outside nor the interfaces for bridging.
We compared the config (uplink profiles, transport zones, edge configuration) with the production put can't find any differences.
We double checked the cabling and configuration on the networking side but there everything looks fine as well.

That's the output of one of the interfaces / physical ports on one Edge Node with the T0 configured. 

edge101> get interface fp-eth1
Wed Dec 15 2021 UTC 13:10:22.675
Interface: fp-eth1
ID: 1
Link status: down
MAC address: 40:a6:b7:0f:2e:41
MTU: 9000
PCI: 0000:3b:00:01
Offload Capabilities: TX_VLAN_INSERT TX_IPV4_CKSUM TX_UDP_CKSUM TX_TCP_CKSUM TX_SCTP_CKSUM TX_TCP_TSO RX_VLAN_STRIP RX_IPV4_CKSUM RX_UDP_CKSUM RX_TCP_CKSUM
Polling Status: inactive
Driver: net_i40e
Rx queue: 12
Tx queue: 12
Socket: 0
RX packets: 0
RX bytes: 0
RX errors: 0
RX badcrc: unknown
RX badlen: unknown
RX misses: 0
RX nombufs: 0
RX pause xoff: unknown
RX pause xon: unknown
TX packets: 0
TX bytes: 0
TX errors: 0
TX pause xoff: unknown
TX pause xon: unknown

edge101> get physical-port fp-eth1
Wed Dec 15 2021 UTC 13:10:40.978
Physical Port
ADMIN_STATUS : up
DRIVER : net_i40e
DUPLEX : full
ESP_RSS_INFO : {'enabled': False}
FW_VER : 7.10 0x800075e6 19.5.12
ID : 1
LINK : down
MAC : 40:a6:b7:0f:2e:41
MAX_RX_PKTLEN : 9728
MAX_RX_QUEUES : 320
MAX_TX_QUEUES : 320
MBUFSIZE : 2048
MIN_RX_BUFSIZE : 1024
MTU : 9000
NAME : fp-eth1
OFFLOAD_CAPABILITIES : TX_VLAN_INSERT TX_IPV4_CKSUM TX_UDP_CKSUM TX_TCP_CKSUM TX_SCTP_CKSUM TX_TCP_TSO RX_VLAN_STRIP RX_IPV4_CKSUM RX_UDP_CKSUM RX_TCP_CKSUM
PCI : 0000:3b:00:01
POLLING_STATUS : inactive
REDIRECT-TO-KERNEL : True
RETA_SIZE : 512
RSS_HASH_FUNC : NONFRAG_IPV4_TCP NONFRAG_IPV4_UDP NONFRAG_IPV4_OTHER NONFRAG_IPV6_TCP NONFRAG_IPV6_UDP NONFRAG_IPV6_OTHER
RX_QUEUE : 12
RX_QUEUE_SIZE : 512
RXCONF : {'enable_scatter': 1, 'enable_lro': 0, 'jumbo_frame': 1, 'max_rx_pkt_len': 9026, 'scattered_rx': 1}
SOCKET : 0
SPEED : 100
TX_QUEUE : 12
TX_QUEUE_SIZE : 512

Kind regards

Christoph

[p0wertje]

Hi,

Are the interface nics the same? or do you have a mix of cards ?

Things I ran into when links did not come up:
1. My firmware on the nic was 'old'. upgraded the firmware and it worked.
2. Links do not tend to come up when they are not assigned in an uplink policy assigned to an edge node.
When assigning them to an edge, and you need to select the correct fp interface, does it show 'up' there?

 

In my case, it was one of these two.

 

 

[christophn]

Thank you for your answer.

Yes, the interfaces are all the same.
I attached a picture of the four interfaces.

Strange thing is that all TEP interfaces are coming up and all non TEP interfaces are not coming up.

[christophn]

I know checked the firmware of the system. 
It's the same version as in production where it works.

[p0wertje]

Hi,

 

And putting the cables from the working interface in the not working interface?
I am running a bit out of options. 
What nics do you have?
Does it need sfps in them?

You can try to login on the edge as root and see some information on the interface
in my case the interface corresponding with fp-eth8 is eth8-mlx

ethtool -m eth8-mlx

Gives info like

Laser wavelength : 850nm
Vendor name : FINISAR CORP.
Vendor OUI : 00:90:65
Vendor PN : FTLX8574D3BCL

Compare them to each other and production

 

[christophn]

Hi P0wertje,

thanks again for your help.
When running the command I got the error message "Cannot get module EEPROM information: Operation not supported".

Kind regards
Christoph

[christophn]

 

thank you for your support.

I solved this topic now.
There is a problem on the switch that the SFPs are only supported on the even ports but not on the odd ports.
Still investigating with the switch vendor.

Kind regards
Christoph

[p0wertje]

Good to hear that you solved it !