nsx4 edge TCP performance issue

[gullio_hyperic]

hello Community,

i want to share you a problem in tcp performance with my nsx4 setup.

i have created all esx/nsx setup successfully and all it works as expected apart of performance. let me explain

when i try to testing tcp performance from 2 linux vm with iperf3 in the same nsx segment or different segment(tier1) i have 10G full throughput with 1 session no problem with DFW enable or disable.

when i try to do the same traversing TIER0(VRF) my speed cap at 100Mbits with DFW enable or disable.

all infrastructure is at 9000 MTU routing nsx infra is at  8900 MTU my network card are elxnet 2x 10G and nmlx5_core 2x10G all edge running in a single VDS with the mellanox card and th esx running with the other.

all my phys switches are jumbo frame (Cisco NEXUS) and my upstream router are at 8900 MTU. esx ver 7.0.3 vcenter 8

 

can someone point me to the right direction? if you need additional info please Reply!

 

thanks a lot

Giulio

 

[Evert Amssoms]

Your issue seems to be in getting traffic routed between the T0 and your uplink. 

There are a few easy things that will probably not fix it, but will exclude some reasons for the trouble:

- Move your edge VM's on a different physical node
- Make sure you haven't got any sort of rate limiting configured on your BGP peer

[gullio_hyperic]

hello Thanks for reply! for sure i manage all the device in this infrastructure bgp router etc.. i have already moved edge1 and 2 to another host but the result are the same, also there are no rate limiting on bgp peer and everything.

i will try today with a tier0 network also(testing from tier1 to tier0 network segment), to understand if only uplink as you say is the problem and come back with results

i want to share if i move vm from nsx segment to a DVS network i will obtain max speed 10Gbps.

any other clue?

BR

Giulio

[Evert Amssoms]

Not really at the moment. My gut feeling says maybe MTU, but you'd be seeing other issues then as well.

Is a hard/consistent cap at 100Mbps? Or is it fluctuating?

[gullio_hyperic]

hello,

it is not hard is fluctuating.

i have tested moving one vm to a tier0 direct segment and i got poor performance like from 50Mbps to 100 Mbps like with external iperf or download.

the MTU on linux server is at 8900 and i have tested no fragments with 8872 in both direction traversing tier1 and tier0(no internet)

also disable DFW with esclusion.

can you please point me in the right direction? maybe something on edge VM?

 

BR

Giulio

[Evert Amssoms]

I'm honestly not sure what else you can check. Maybe verify that your edge nodes aren't having issues with resources (either hitting their limits or having scheduling limits)?