Jumbo Frames in NetApp

This is just another one out of ten thousand posts talking about Jumbo frames, actually a week back while doing designing for my new NetApp environment was looking for Jumbo Frames. So here's some details which I have collected from different places, although still I have to do testing and see how much benefit I get in my environment however I am posting this thinking it might be useful for someone else.

Jumbo frames

Jumbo frames are TCP frames where MTU size is more than the IEEE standard of 1500 bytes. There are lots of variations in that and anything from 1500 to 12000 can be configured, be called as jumbo frames. However most of the industry uses MTU size of 9000 for jumbo frames due to support from most of the device manufacturers and memory page size limit of common protocols like NFS in which datagram size is 8400 bytes therefore a Ethernet frame size of 9018 can accommodate single NFS datagram in one Ethernet packet and stay comfortably within the standard Ethernet bit error rates.

As NetApp support maximum MTU size of 9192 hence in this paper I have taken 9000 as the MTU size.

Benefits:

• Less CPU overhead as system has to do less header processing because in VIF mode TOE on NetApp cards are disabled.
• 9000 bytes frames are six times higher then stock frames of 1500 MTU so larger frame size leads to higher throughput.
• Some tests in NetApp show upto 30% increase and other vendors have achieved more than 60% in network throughput.

Considerations:

• To use jumbo frames, client system, intermediate switches / routers and NetApp devices, all should be configured to process large frames.
• Any interface operating over 1000 Mbps is currently supported on NetApp systems for jumbo frame configuration.
• Client’s TCP window size should be two times the MTU size, minus 40, and the maximum value can be the highest value storage system support. Typically, the maximum value can be set for client's TCP window is 65,535.
• If storage system is configured to support jumbo frames and the client is not, the communication between them occurs at the client’s frame size
• UDP client’s MTU size and storage system’s MTU size should match as UDP clients do not communicate their MTU size.
• All the interfaces in a vif must have the same MTU size.

Suggestions:

Looking at different performance tests carried out from NetApp, it is clear that all of them do have Jumbo frames enable to achieve a higher throughput and even their different best practices call for using jumbo frames for better usage.

Although jumbo frames are good option for increasing network through however it needs proper testing and validation, some tests from different implementation have shown no performance increase or network errors also. Like in the case of intermediate IP network which doesn’t support extended frames in that case the IP device may have to fragment the frame which puts additional load and in worst case if the DON'T_FRAGMENT bit is set in the IP header of the packets, the router will drop the packets instead of fragmenting them and sending station will get a message “ICMP DESTINATION UNREACHABLE - FRAGMENTATION NEEDED”.

References for further reading

Optimizing Oracle on NFS - NetApp White Paper

CIFS Best Practices - NetApp Technical Report

iSCSI Performance Options - NetApp Technical Report

Oracle 10g Performance on Solaris 10 - NetApp Technical Report

Ethernet Jumbo Frames - Chelsio Communications White Paper

Gigabit Ethernet Jumbo Frames - WareOnEarth Communications

Extended Frame Sizes for Next Generation Ethernets - Alteon Networks White Paper

Boosting Server-to-Server Gigabit Throughput with Jumbo Frames- HP White Paper