Packet transfer error when copy large data like SSH/HTTPS over OSPF North South Traffic NSX-T

[liibiiinnn]

Hi everyone

 

I'm using NSX and routing to external infrastructure with Cisco ACI Leaves using OSPF + ECMP. This is my topology

 

There is no problem, every connection is good, but when I copy a file or transfer payload larger than 2 MB over SSH or HTTPS from outside NSX, very low time of successes, ex copy 10 turns, only 2 or 3 turn is success, otherwise, then copy only 32k (HTTPS) or can not copy whole file 32MB/42MB (SSH). In topology above, I found problem come from OSPF ECMP connection, if I run only unique path, not Double-Path or Quad-path, traffic success 100% in all 10 turn are no packet dropped. I think ECMP in NSX are not compatible with ECMP in Cisco ACI.

 

My NSX version: VMware NSX 4.0.0.1

My Cisco ACI version: 4.2

 

Thank !

Attachment(s)

error-debug.drawio.pdf   72 KB 1 version

[Sree]

What is the End-End MTU configured in this setup? Are you seeing any packet drops at Edges/Host uplinks or ACI leaf? URPF is enabled in this setup? 

[liibiiinnn]

Hi Sreec

 

- URPF: I was tried to Enable/Disable on Interface Edge, but nothing happened

- MTU: 9000

- Dropped Packet: No packets are dropped, I was captured on all Edge DPDK Fast Path Interface and see a lot of ACK packets from TCP Application are sent duplicate to many like this and NSX Edge not forward ACK packets into VM in NSX segment, and so from ACI will resend the ACK packet many times. Only packet with Flag 0x010 (ACK) are faulty, the rest (not ACK packet) are not abnormal.

 

 

After a few days trying different scenarios, Problem In NSX with Cisco ACI. Has several bugs, I see that.

 

1. If we use VLAN Segment for Edge DPDK Fast Path with Uplink Teaming Dedicated, Only 1 Interface OSPF Path are not error, 2 Interface on Edge to Leaf 1 and Leaf 2 are Error copy file 32k (HTTPS)

 

2. If we use Distributed Port-Group like topology above, use for Edge DPDK Fast Path, 2 interface on single Edge are connected to Leaf 1 and Leaf 2 are good connection, no problem occurred, 100% packet application was pass. But if running Gateway Tier0 (Active/Active) with two Edge x 2 Interface on both Edge. The problem happened again. Then, I enter maintain mode either Edge -> no problem (success 100%)

[Perttu]

Why do you think those ACK packets are faulty (retransmissions)? If you look carefully you see the sequence number increasing, which means new data. TCP retransmissions can be identified with a repeating sequence number.

That trace seems perfectly okay, packets 295 and 308 are last segments of corresponding TLS records, Wireshark reassembles TCP segments belonging to same TLS record in last packet and this is also why they are marked differently. But they also piggyback ACKs like others, which you can confirm by navigating on them. 

A bidirectional capture would be more helpful and if this is ECMP capture it simultaneously it on both edges.

Also a distributed port group probably keeps a one connection on a one path. A HTTPS connection is a single connection and typically it takes a single path (all packets give same hash for hash based load balancing). This helps to avoid out of order transmissions.  

 

 

[liibiiinnn]

Thank for your reply

 

How VLAN Segment is different with Distributed-Port-Group, mechanism, algorithm. I want more document about it. Why are there so many tutorials on using VLAN segments for Edge DPDK Interface, by using VLAN segment I see the benefit of using same VLAN for both ESXi transport node and Edge VM, the internal connection does not error like this. But use VLAN segment for North-South traffic is not fantastic.

 

In CISCO ACI, current we can't capture packet on L3Out Interface, but we used the same connection to the respective Cisco device pair, it didn't fail, I would like to use the Physical Server for setup Bare-Metal Edge and test for further comparison.

[Sree]

May I know why you are using 4 unique VLANs(1011,1012,1013,1014) in this design for two leaf connectivity ( 2 VLANs in each Edge for N-S peering)? 

With 1 Edge with fp-eth2 and fp-eth3 (VLAN Segment) will cause of ACK packets are transmitted continuously as above, but when use fp-eth2 and fp-eth3 (Distributed vSwitch), the connection is good 100%. 

Can you please share the teaming policies you have set while using VLAN Segments? 

 

[liibiiinnn]

Hi Sreec

 

We are using 4 unique VLANs to configure OSPF P2P mode, VMware offer two topology, with Broadcast and P2P, we chose P2P, follow blog at here https://blogs.vmware.com/networkvirtualization/2021/06/introducing-ospf-support-in-nsx-t-3-1-1.html/

With each VLAN and a subnet /30 corresponding with Edge FP Interface, Two edge (two FP interfaces) = 4 VLAN. P2P is an easy way, and we not plan to scale up NSX Edge, thus, we don't use Broadcast mode

 

"Can you please share the teaming policies you have set while using VLAN Segments?"

My teeming policy here

1. First create Node Transport Node Profile and attach to ESXi 172.17.100.43 and 172.17.100.44

• uplink_01 (NSX) -> Uplink 01 (vDS) -> vmnic2
• uplink_02(NSX) -> Uplink 02 (vDS) -> vmnic4
• Transport Zone: MCR-Host-Overlay, MCR-Host-VLAN

2. I create two Trunk VLAN Segment for Edge FP Interface

 

3. Next, we create teaming policy for NSX Edge 

I'm having two edge VM

• NSX Edge VM-61 on 172.17.100.43
• NSX Edge VM-62 on 172.17.100.44

And apply uplink to both

• uplink_01 (fp-eth1) -> MCR-NSX-Edge-Fp_Edge_Alpha_Trunk (MCR-Host-VLAN)
• uplink_02(fp-eth2) -> MCR-NSX-Edge-Fp_Edge_Beta_Trunk (MCR-Host-VLAN)
• Transport zone:MCR-Host-Overlay, MCR-Edge-Uplink

4. And finish, create four VLAN Segment (MCR-Edge-Uplink)

Create OSPF Interface with IP/Subnet and OSPF

 

 

 

[Sree]

Apologies for the delayed response. I see for Edge trunk you have ACI leaf teaming policies. What is the use of the first set of Teaming policy Alpha/Beta? 

[liibiiinnn]

Hi  

After one day, I investigated many way relations about routing path or ECMP, and I even changed the topology to LACP with only one VLAN (two SVI each Leaf). But all of them, or anything OSPF/BGP topology, are not the cause of this error. The problem comes from Gateway Firewall Rule

 

My Gateway Tier-0 is Active/Active. In GFR (Gateway Firewall Rule), I have one Policy_default that I can't delete, and I leverage them to Deny. Then, I created a new Policy (stateful: on, tcp strict: off) & add a new rule "Allow All"

 

 

I have yet to investigate further, but the best way is not to use the same rule on different Policies, after disabling the rule "Allow All", they work as expected, and packet transfer success is 100% without any problem.

[liibiiinnn]

I also tried using VLAN Segment again to connect OSPF Routing with ACI, but the 32k error still occurs. There is no way to use for it. I was turned my system to use LACP to connect ESXi overlay & L3Out ACI instead NSX Teaming (follow Cisco guided here)

 

https://www.cisco.com/c/en/us/solutions/collateral/data-center-virtualization/application-centric-infrastructure/white-paper-c11-740124.html

 

Here, this is my config

 

Step 1. Create LACP in VDS, and also make corresponding LACP with Cisco ACI (vPC)

Assign vmnic from ESXi Host to corresponding Leaf Ethernet port

 

172.17.100.43

- vmnic2 -> Leaf 01: port E1/35

- vmnic4 -> Leaf 02: port E1/35

 

172.17.100.44

- vmnic2 -> Leaf 01: port E1/37

- vmnic4 -> Leaf 02: port E1/37

 

Step 2. Set transport node profile to new uplink profile as below

 

 

And assign LACP Port (nsx uplink) to ACI-Leaf-01-02 (vds lacp)

 

Step 3. On ACI, I set trunk VLAN 508 and VLAN 1011 on LACP port on both Leaf 1 and Leaf 2

Step 4. On ACI, I create SVI interface on both Leaf 1 and Leaf 2

 

SVI - Leaf 1 = 172.17.98.129

SVI - Leaf 2 = 172.17.98.130

 

Step 5. Create VLAN Segment for Edge FP (routing to ACI)

 

Step 6. Create Uplink profile for edge node as below

 

and Create VLAN Segment (trunk all) for Edge TEP connect

 

 

Now I will assign "Seg-VLAN-Trunk" to "MCR-NSX-Edge-Fp_Edge_Trunk"

And assign PG-VLAN-1011 to "MCR-NSX-Edge-Fp_Edge_Route".

I'm just naming it with the prefix PG and VDS-*, but I'll still assign it to the VLAN Segment to prove that the VLAN Segment isn't working properly

And create Gateway tier 0 interface from a vlan segment.

 

Now OSPF neighbor to ACI

 

Errors like post #1 will still occur. But after assign uplink "PG-VLAN-1011" to Port-Group from VDS, it's work 100% perfect without any packet loss. 

 

Summary

 

- Don't use VLAN Segment for North-South dynamic routing with Active/Active and ECMP, use VLAN port-group from VDS are good connection.

- Don't use same rule on different Policies in Gateway Firewall Rules, especially same rule in both stateful policy and stateless policy

 

[Sree]

Thanks for the detailed explanation. Can you please share how the Servers are connected to the ACI? Is it a blade server? When we have two edges please place both edges on the same ESXi node and let me know the results. Also, I would like to know if it's feasible to test by having Two Edges and a Single Leaf connection ( VLAN 1011 or VLAN 1013).

[liibiiinnn]

Hi Sreec

There is my physical connect of NSX and Cisco ACI, ESXI Server is Dell PowerEdge R750 with Broadcom OCP 3.0, and Cisco ACI is physical device model Cisco Nexus 93240YC-FX2. With vmnic[x] is port in the esxi and Ethernet1/x is port in the Leaf Switch.

 

Sorry for my first post is not clarity. With 1 Edge with fp-eth2 and fp-eth3 (VLAN Segment) will cause of ACK packets are transmitted continuously as above, but when use fp-eth2 and fp-eth3 (Distributed vSwitch), connection is good 100%.

 

Your Q: "When we have two edges please place both edges on the same ESXi node and let me know the results"

My Ans: I will test later

 

Your Q: "I would like to know if it's feasible to test by having Two Edges and a Single Leaf connection (VLAN 1011 or VLAN 1013)."

My Ans: Still error (DVS and VLAN Segment have same issue)

 

New result: I'm requested my team create four new Port on ESXi 172.17.100.43(vmnic6, vmnic7) and 172.17.100.44 (vmnic6, vmnic7)
corresponding to 4 ports on new two Leaf ACI called Leaf 03, Leaf 04, to avoid affecting Leaf on production env (Leaf 01, Leaf 02), them will be used for dedicated OSPF connect, separate with VLAN 508 (transport overlay). But I was created two new edges for staging environment and config new OSPF + L3Out + new Area, full mesh OSPF P2P with ECMP

=> and it's work as expected, no problem occurred, 100 copies are 100% successful, at tomorrow, I'll config new 4 port for production-environment. Hope the results will be as expected