PAM (Performance Accelerator Module) in NetApp

Notes: This is a multi part post and I have collected all the materials from various atricles published in Netapp and internet.

The PAM offers a way to optimize the performance of a NetApp storage system by improving Throughput and Latency while reducing the number of disk spindles/shelves required as well as power, cooling and rack space requirements.

It is a an array controller resident, intelligent 3/4 length PCIe card with 16GB of DDR2 SDRAM that is used as a read cache and is integrated with DataONTAP via FlexScale which is software that provides various tuning options and modes of operation.

Why the PAM?

In every disk array, the processors can access data from two areas: from disk or the Cache. If the data is not in the Cache, this results in a Cache miss which means the data will have to be fetched from disk. The time (latency) to fetch data directly from the disk is orders of magnitude greater than from cache. For read intensive random I/O applications that are latency sensitive this requires the configuration of a high number disk in order to provide user and application acceptable latencies. Most Microsoft applications fall in that category as well as Server virtualization platforms and File services which tend to generate, primarily, these types of workloads.

So the idea with deploying one or more PAMs is that a hit to the PAM's cache will considerably reduce the time it takes to fetch the data as compared to the same process occurring from disk. The additional benefit here is that it will also allow for a more reasonable amount of disks and shelves to be deployed to support the same application thus reducing costs.

In the simplest terms, the Performance Acceleration Module is a second-layer cache: a cache used to hold blocks evicted from the WAFL buffer cache. In a system without PAM, any attempt to read data that is not in system memory results in a disk read. With PAM, the storage system first checks to see whether a requested read has been cached in one of its installed modules before issuing a disk read. Data ONTAP maintains a set of cache tags in system memory and can determine whether or not a block resides in PAM without accessing the card. This reduces access latency because only one DMA operation is required on a cache hit. As with any cache, the key to success lies in the algorithms used to decide what goes into the cache. We’ll have more to say about that in the following section.

Hardware Architecture

PAM is a combination of both hardware and software (the PAM software is known as FlexScale.) A license is required to enable the hardware. The PAM hardware module is implemented as a ¾-length PCIe card offering dual-channel DMA access to 16GB of DDR2 memory per card and a custom-coded field-programmable gate array (FPGA) that provides the onboard intelligence necessary to accelerate caching tasks.

· PCI Express X4

· 16GB of RAM per module

· System interaction and cache managed by custom-coded FPGA

Power:	18W normal, 25W maximum at system initialization for approximately 10 seconds (For comparison, current disk shelves draw 340W continuous.)
Operating environment:	10ºC to 40ºC ambient temperature
Airflow:	150 LFM minimum
Heat:	90 BTU/hr
Form factor:	¾ length PCIe

Compatibility and Support

FAS/V-Series	Max # of Modules	Extended Cache Memory
FAS6080 / V6080 FAS6070 / V6070 SA600	5	80GB
FAS6040 / V6040 FAS6030 / V6030 FAS3170 / V3170	4	64GB
FAS3070 / V3070 FAS3140 / V3140 SA300	2	32GB
FAS3040 / V3040	1	16GB

Types of caching in PAM

PAM supports three modes of operation:

· Default mode, in which data and metadata are cached

· Metadata mode, in which only metadata is cached

· Low-priority mode, which enables caching of sequential reads and other low-priority data

Default Mode

The default mode caches both user data and metadata, similar to the caching policy that Data ONTAP implements for the WAFL buffer cache. For file service protocols such as NFS and CIFS, metadata includes the data required to maintain the file and directory structure. With SAN, the metadata includes the small number of blocks that are used for the bookkeeping of the data in a LUN.

This mode is best used when the working set size is equal to or less than the size of the PAM cache. It also helps when there are hot spots of frequently accessed data and ensures that the data will reside in cache.

flexscale.enable on

flexscale.lopri_blocks off

flexscale.normal_data_blocks on

Metadata Mode

In this mode, only storage system metadata is cached. In many random workloads, application data is seldom reused in a time frame in which caching would be beneficial. However, these workloads tend to reuse metadata, so caching it can improve performance. Caching just the metadata may also work well for data sets that are too large to be effectively cached (i.e., the active data set exceeds the size of the cache).

flexscale.enable on

flexscale.lopri_blocks off

flexscale.normal_data_blocks off

Low-Priority Mode

In low-priority mode, caching is enabled not only for normal data and metadata but also for low-priority data that would normally be excluded. Low-priority data in this category includes large sequential reads, plus data that has recently been written. Writes are normally excluded from the cache because the overall write workload can be high enough that writes overflow the cache and cause other more valuable data to be ejected. In addition, write workloads tend not to be read back after writing (they are often already cached locally on the system performing the write) so they’re generally not good candidates for caching.

The low-priority mode may be useful in applications that write data and read the same data after a time lag such that upstream caches evict the data. For example, this mode can avoid disk reads for a Web-based application that creates new data and distributes links that get accessed some time later by Web users. In some Web applications, we’ve found that the time lag for the first read is long enough that the data has to come from disk (even though subsequent data references are frequent enough to be handled by upstream caches). PAM in low-priority mode could accelerate these applications by turning such disk reads into cache hits.

flexscale.enable on

flexscale.lopri_blocks on

flexscale.normal_data_blocks on

In part 2 of this post I will tell you how to determine if PAM will improve the performance