Hello nickdd -- I believe we are making some progress here.
The answer isn't quite what I expected; MTU. If we look below here are the results:
1500 MTU:
File Copy within Windows VM: 50MB/s - 120MB/s
IOmeter Test (75% Read, 25% Write at 32k IO using a single thread): 250MB/s - 350MB/s
iperf Results:
[ ID] Interval Transfer Bandwidth
[ 3] 0.0- 1.0 sec 242 MBytes 2.03 Gbits/sec
[ 3] 1.0- 2.0 sec 223 MBytes 1.87 Gbits/sec
[ 3] 2.0- 3.0 sec 238 MBytes 1.99 Gbits/sec
[ 3] 3.0- 4.0 sec 240 MBytes 2.01 Gbits/sec
[ 3] 4.0- 5.0 sec 254 MBytes 2.13 Gbits/sec
[ 3] 5.0- 6.0 sec 259 MBytes 2.18 Gbits/sec
[ 3] 6.0- 7.0 sec 275 MBytes 2.31 Gbits/sec
[ 3] 7.0- 8.0 sec 256 MBytes 2.15 Gbits/sec
[ 3] 8.0- 9.0 sec 277 MBytes 2.32 Gbits/sec
[ 3] 9.0-10.0 sec 267 MBytes 2.24 Gbits/sec
[ 3] 0.0-10.0 sec 2.47 GBytes 2.12 Gbits/sec
9000 MTU:
File Copy within Windows VM: 300MB/s - 500MB/s
IOmeter Test (75% Read, 25% Write at 32k IO using a single thread): 800MB/s - 1.2GB/s
iperf Results:
[ ID] Interval Transfer Bandwidth
[ 3] 0.0- 1.0 sec 1.10 GBytes 9.48 Gbits/sec
[ 3] 1.0- 2.0 sec 1.12 GBytes 9.61 Gbits/sec
[ 3] 2.0- 3.0 sec 1.08 GBytes 9.29 Gbits/sec
[ 3] 3.0- 4.0 sec 1.12 GBytes 9.62 Gbits/sec
[ 3] 4.0- 5.0 sec 1.13 GBytes 9.72 Gbits/sec
[ 3] 5.0- 6.0 sec 1.12 GBytes 9.60 Gbits/sec
[ 3] 6.0- 7.0 sec 1.12 GBytes 9.64 Gbits/sec
[ 3] 7.0- 8.0 sec 1.12 GBytes 9.64 Gbits/sec
[ 3] 8.0- 9.0 sec 1.12 GBytes 9.63 Gbits/sec
[ 3] 0.0-10.0 sec 11.2 GBytes 9.58 Gbits/sec
I am not sure I have ever witnessed MTU reporting such a drastic difference in performance, in fact (depending on load) I have seen MTU increase decrease performance.
I am guessing it has to do something with how the hypervisor is assembling the packets and sending them to our FlashArray. I am continuing to investigate and will let you know once I have more information.
-jhop