Blogs

Welcome back to the second installment on Brocade 7840 IPEX. Today’s blog is all about improving performance of IP storage applications between data centers, a cornerstone of IPEX. There is A LOT to this pillar, so, I’ve devoted today’s entire blog to just to this subject. Here is a list of pertinent Brocade technologies implemented in IPEX that enhance native IP storage performance:

• Purpose built hardware and software
• Intolerant IP storage applications
• WAN Optimized TCP
• TCP Acceleration
• End-to-End Flow Control
• Data batching
• Jumbo Frames
• Adaptive Rate Limiting (ARL)
• QoS (Enforcement and Marking)
• Compression
• Scalability

Incrementally adding a bit of performance here, a little more performance there, and so on… via these various technologies ultimately add up to significant performance improvements. I’ll comment on each.

Purpose Built Hardware and Software

The 7840 is Brocade’s most recent incarnation of extension product incorporating the latest ASICs, processors, FPGAs and switches from Brocade, Cavium, Xilinx and Broadcom. Truly the best, purpose built extension technology Silicon Valley has to offer and all integrated by highly experienced Brocade engineers and leveraging the industry’s finest storage networking operating system, namely Fabric Operating System (fondly known as FOS). Brocade has demonstrated twice as many years of experience, innovation and stability in storage networking than its closest competitor.

What are Brocade 7840 customers getting for their money? They get 64 processors, 128 GB of RAM, two FPGAs and two ASICs and a decade worth of advanced TCP development. It should be obvious that the advanced level of electronics in the 7840 does not exist in native IP storage end-devices, hosts or appliances because it’s simply just not practical. Putting the electronics and engineering contained in the 7840 into every IP storage end-device would make those devices considerably more expensive and wouldn’t be competitive in the market. Does it make sense for every IP storage end-device company to hire a slew of engineers just to enhance transport across the WAN? Clearly, this is not the bailiwick of these companies.

Intolerant IP Storage Applications

Brocade has now tested a number of IP storage applications and there is one thing in common, they do not tolerate latency and packet loss. Metro distance latencies appear to be well tolerated without droop. Droop is the diminishing of throughput with more latency. 

Beyond metro distances throughput is lost due to the inability to maintain full utilization. Essentially, there is excessive idle time waiting for acknowledgments. As for packet loss, even the smallest number of lost packets results in dramatic throughput loss. IPEX solves this problem by requiring the IP storage end-devices to only communicate locally within the data center. The Brocade 7840 takes care of the WAN transport.

WAN Optimized TCP

Brocade Extension Engineering has spent significant resources developing advanced TCP technologies and associated features. This is not trivial! Brocade’s WAN Optimized TCP is a purpose built TCP stack with special features and functionality applied directly to extension solutions. WO-TCP is an advanced and aggressive TCP stack for quickly getting big data from point A to B over large long pipes. There is no better TCP stack in the storage industry.

TCP Acceleration

What is the fastest environment in which IP storage devices can communicate between each other? Answer: If you put them right next to each other connected by a fiber cable, they couldn’t go faster than that. Make that the maximum benchmark speed. If IPEX can maintain that speed over latency, dirty links and packet loss; then IPEX is doing its job.

Brocade IPEX terminates TCP streams locally and the 7840 supports 1024 IP storage TCP streams. This happens at both the local and remote data centers. WAN Optimized TCP does the heavy lifting of getting the data from data center A to data center B while mitigating the nasty effects of latency, packet loss and dirty links. Because the end-devices are talking locally via fiber in the data center, it sends at that device’s maximum capable speed. IPEX just provides “local” performance across the WAN. This is TCP Acceleration.

End-to-End Flow Control

What happens if one of those local TCP streams decides to close its Receive Window (Rwnd)? Closing the rwnd means – stop sending me (me=end-device) more data. How does that translate to WO-TCP across the WAN? We can’t just close the WO-TCP rwnd because that will slow or stop any other streams on the transport that have open rwnd. This problem is referred to as Head of Line Blocking (HoLB). Brocade prevents HoLB. There is no HoLB across the WAN! It doesn’t make sense to standup a different TCP session for each stream, although, we do this for each QoS priority. What made more sense was to logically have separate TCP sessions using TCP vWindows (Virtual Windows). Each local TCP stream terminated by IPEX gets its own WO-TCP vWindows so that end-to-end flow control can be maintained. This is significant in preventing packet loss in the IP network, which results in dramatically less throughput.

Data Batching

Brocade does not apply an extension header to each FC/FICON (FCIP) or IP datagram (IPEX) that enters the 7840. Think about that, WOW, how much overhead would that be! Instead, the 7840 forms batches that are about 32 KB in size. A batch is a compressed data stream (we compress as the data enters the box) and one extension header is placed on one batch. For IPEX, TCP/IP headers are stripped off to further reduce overhead. Those headers are reformed again at the remote side. Compressed byte streams do not care where frames or datagrams begin or end enabling the stuffing of TCP segments to their maximum size. This reduces overhead even more. The entire process is extremely efficient and superfast.

9216 Byte Jumbo Frames

First read Data Batching above. Now, know that jumbo frames make an already efficient and fast process even faster and more efficient. It’s that simple. Did I mention that the segments are filled to the maximum size, which in this case is 9 KB? Also, the Brocade 7840 supports PMTU (Path MTU) and can automatically figure out what your maximum WAN MTU is in the event you don’t know.

Adaptive Rate Limiting (ARL)

ARL is bandwidth management. You set a ceiling and the 7840 doesn’t push beyond that rate. Additionally, you set a floor and the 7840 pushes at least that much. Everything between the floor and ceiling is adaptive based on conditions in the IP network and WAN. What’s important here is the 7840 is a device that is seeing all the IP storage flows headed for the WAN and manages them properly. It is not possible for individual end-devices to manage BW properly because they do not know what other flows are doing and don’t have a comprehensive view. It is not possible for operations to set an optimal rate limit on end-devices because what’s optimal? The optimal rate limit is a constantly moving target. The only way to perform proper adaptive rate limiting is automatically and by a device that sees and manages the aggregate of those IP storage flows.

Please find more information about ARL in this White Paper, “Brocade Adaptive Rate Limiting."

QoS (Enforcement and Marking) and PTQ

QoS enforcement is a mechanism that improves performance of critical applications at the expense of less critical applications during times of contention for available WAN bandwidth. The Brocade 7840 has PTQ (PerPriority TCP QoS) in which there are separate autonomous WO-TCP sessions for each QoS priority. There are 7 priorities (4 for FCIP: class-F, High, Med, Low and 3 for IPEX High, Med, Low). If both FCIP and IPEX are being used simultaneously, the percentage of bandwidth that will be applied to each during contention is configurable, for example 50/50.

Compression

More compression means more data can traverse a fixed amount of WAN bandwidth, everything else being equal (like efficiency, protocol optimization…). The Brocade 7840 Extension Switch has all new compression algorithms: Fast Deflate (FCIP only), Deflate (FCIP & IPEX) and Aggressive Deflate (FCIP & IPEX) and powerful hardware to implement them.

When using Deflate with IPEX you could achieve 3:1 compression (typical), 15 Gbps per engine (there are 2 engines) for WAN bandwidth usage of 5 Gbps. Using both engines (referred to as DPs or Data Processors) gets you 30 Gbps of IP storage bandwidth between data centers across a single 10 Gbps connection.

Compelling! I have to interject this… Brocade makes no claims, promises or guarantees as to the compression ratio achievable for customer specific data. Your mileage will vary. Nonetheless, the “typical” algorithm ratios tend to be just that, typical.

Scalability

Why is scalability in this performance installment of IPEX? Well, if you cannot scale your IP storage flows across the WAN, you are going to end up with miserable performance. IPEX allows the maximum number of flows and flow sizes across your infrastructure while managing each flow and preventing poor performance due to dropped packets.

Please come back tomorrow to check out the last installment, part 3 of 3, and see how the remaining 4 pillars of IPEX make Storage and Mainframe Administrators’ jobs easier. The remaining pillars are: Operational Excellence, Superior Availability, Strong Security and Easy Deployment.

To do: Read this blog tomorrow, go to the Brocade booth again, have fun at the show and stop by my technical session:

Storage over Distance from SRDF to SyncIQ

Date: Wednesday, May 6, 2015

Time: 12 - 1 pm (Pacific)

Location: Lando 4205