Tuesday, June 16, 2009

How to check part number of installed adapter in Ontap?

There are number of situations when you want to check part number of installed PCI adapters in your NetApp FAS or V-series system, how do you do it?

Well whatever way you do there’s a simple way and undocumented also (atleast in their man pages) “sysconfig -ca”.
Just run the command and it will give you part number of all the pci adapters installed as well checks if they are in appropriate slot.

Here’s the sample output

Filer01> sysconfig -ca
sysconfig: slot 5 OK: X2054A: QLogic ISP 2432; PCI-E quad-port Fibre Channel (QLE2464)
sysconfig: slot 6 OK: X2054A: QLogic ISP 2432; PCI-E quad-port Fibre Channel (QLE2464)
sysconfig: slot 3 OK: X1005A: Chelsio T210 TOE 10G NIC
sysconfig: slot 1 OK: X3148: NetApp NVRAM6 2GB
sysconfig: slot 2 OK: X1936A: Performance Accelerator Module I
sysconfig: slot 9 OK: X1005A: Chelsio T210 TOE 10G NIC
sysconfig: slot 7 OK: X2054A: QLogic ISP 2432; PCI-E quad-port Fibre Channel (QLE2464)
sysconfig: slot 8 OK: X1049A: PCI-E Quad 10/100/1000 Ethernet G20
sysconfig: System configuration meets block protocol requirements.
sysconfig: There are no configuration errors.

Sunday, June 7, 2009

How to configure ontap to send specific event type on your pager or handheld?

Starting ontap 6.4 NetApp had added one more option in autosupport category “autosupport.noteto” which sends the alert generated by your ontap system to any pager or mobile address. This is useful when viewing urgent event e-mail that is read on a small screen, such as an alphanumeric pager screen. Autosupport short note e-mails will be received if the cell phone or text pager has an e-mail address that can receive messages. Up to five e-mail address can be specified using this option and an empty list will disable short note e-mails.

To send notification to an e-mail address, cell phone or text pager e-mail address, type:

options autosupport.noteto [pager1 mail address],[pager2 mail address],...

Ontap doesn’t sends all the messages in the specific event category through email to configured mail addresses however all autosupport type events are sent to NetApp, If you want to see what all the events trigger autosupport mail notification you can use this link from NetApp to check. By default ontap sends only critical and error messages types to autosupport.to and autosupport.noteto addresses however if you want to configure the ontap to trigger email notifications on your defined severity of alert you can use “autosupport.notify_threshold” option.

The “autosupport.notify_threshold” can be set to the following values:

(empty string) No messages will be sent to customer e-mail addresses.
Only critical messages will be sent to customer e-mail addresses.
Only critical and error messages are sent (default).
Warning messages and above are sent.
Notice messages and above are sent.
Info messages and above.
All messages will be sent to customer addresses.

Thursday, June 4, 2009

PCS - Predictive Cache Statistics for PAM cards

This is in continuation of last part where I have covered details of PAM, in this part I will cover PCS which you can use to determine if the PAM module will give you benifite before ordering your one from NetApp

PCS: Determining If PAM Will Improve Performance

To determine whether your storage systems can benefit from added cache, NetApp has developed its Predictive Cache Statistics software, which is currently available in Data ONTAP 7.3 and later releases. PCS allows you to predict the effects of adding the cache equivalent of two, four, and eight times system memory.

Using PCS, you can determine whether PAM will improve performance for your workloads and decide how many modules you will need. PCS also allows you to test the different modes of operation to determine whether the default, metadata, or low-priority mode is best.

To begin using PCS, you enable the feature with the command:

options flexscale.enable pcs

Don’t enable PCS if your storage system is consistently above 80% CPU utilization. Once PCS is enabled, you have to let the simulated cache “warm up” or gather data blocks. Once the cache is warmed up, you can review and analyze the data using the perfstat tool.

This procedure simulates caching using the default caching mode that includes both metadata and normal user data. You can also test the other operating modes.

To enable metadata mode:

options flexscale.normal_data_blocks off

To enable low-priority mode:

options flexscale.normal_data_blocks on
options flexscale.lopri_blocks on

Once you have completed testing, disable PCS:

options flexscale.enable off

With PCS enabled, you can find out what's happening using the following command:

stats show -p flexscale-pcs

Sample output is shown below

Example PCS output.

Use the following guidelines to help you interpret the data:

· If the hit/(invalidate+evict) ratio is small, then a lot of data is being discarded before it is used. The instance (ec0, ec1, ec2) may be too small.

· If the (hit+miss)/invalidate ratio is too small, it might indicate a workload with a large amount of updates; switch to metadata mode and check the hit% again.

· If the usage is stable and there are a small number of invalidates and evictions, then the working set fits well.

· The KB/s served by the cache is approximately equal to the hit/s × 4KB per block.

Note that the three caches simulated in PCS are cascading caches. In the example above, ec0 represents the first cache of size 8GB, ec1 represents the second cache of size 8GB, and ec3 represents the third cache of size 16GB. The hit per second for a 32GB cache is the sum of all the hits per second for all three caches. The key advantage of cascading caches is that in the process of measuring an accurate hit rate for a 32GB cache, we also obtain hit rate estimates of both 8GB and 16GB caches. This gives us three points on the hit rate curve and the ability to estimate hit rates for intermediate cache sizes.

PAM and FlexShare

FlexShare™ is a Data ONTAP option that allows you to set priorities for system resources (processors, memory, and I/O) at a volume level, thereby allocating more resources to workloads on particular volumes when the controller is under significant load. FlexShare is fully compatible with PAM, and settings made in FlexShare apply to the data kept in the PAM cache. With FlexShare, finer-grained control can be applied on top of the global policies you implement with PAM. For example, if an individual volume is given a higher priority with FlexShare, data from that volume will receive a higher priority in the cache.

Wednesday, June 3, 2009

PAM (Performance Accelerator Module) in NetApp

Notes: This is a multi part post and I have collected all the materials from various atricles published in Netapp and internet.

The PAM offers a way to optimize the performance of a NetApp storage system by improving Throughput and Latency while reducing the number of disk spindles/shelves required as well as power, cooling and rack space requirements.

It is a an array controller resident, intelligent 3/4 length PCIe card with 16GB of DDR2 SDRAM that is used as a read cache and is integrated with DataONTAP via FlexScale which is software that provides various tuning options and modes of operation.

Why the PAM?

In every disk array, the processors can access data from two areas: from disk or the Cache. If the data is not in the Cache, this results in a Cache miss which means the data will have to be fetched from disk. The time (latency) to fetch data directly from the disk is orders of magnitude greater than from cache. For read intensive random I/O applications that are latency sensitive this requires the configuration of a high number disk in order to provide user and application acceptable latencies. Most Microsoft applications fall in that category as well as Server virtualization platforms and File services which tend to generate, primarily, these types of workloads.

So the idea with deploying one or more PAMs is that a hit to the PAM's cache will considerably reduce the time it takes to fetch the data as compared to the same process occurring from disk. The additional benefit here is that it will also allow for a more reasonable amount of disks and shelves to be deployed to support the same application thus reducing costs.

In the simplest terms, the Performance Acceleration Module is a second-layer cache: a cache used to hold blocks evicted from the WAFL buffer cache. In a system without PAM, any attempt to read data that is not in system memory results in a disk read. With PAM, the storage system first checks to see whether a requested read has been cached in one of its installed modules before issuing a disk read. Data ONTAP maintains a set of cache tags in system memory and can determine whether or not a block resides in PAM without accessing the card. This reduces access latency because only one DMA operation is required on a cache hit. As with any cache, the key to success lies in the algorithms used to decide what goes into the cache. We’ll have more to say about that in the following section.

Hardware Architecture

PAM is a combination of both hardware and software (the PAM software is known as FlexScale.) A license is required to enable the hardware. The PAM hardware module is implemented as a ¾-length PCIe card offering dual-channel DMA access to 16GB of DDR2 memory per card and a custom-coded field-programmable gate array (FPGA) that provides the onboard intelligence necessary to accelerate caching tasks.

· PCI Express X4

· 16GB of RAM per module

· System interaction and cache managed by custom-coded FPGA


18W normal, 25W maximum at system initialization for approximately 10 seconds (For comparison, current disk shelves draw 340W continuous.)

Operating environment:

10ºC to 40ºC ambient temperature


150 LFM minimum


90 BTU/hr

Form factor:

¾ length PCIe

Compatibility and Support


Max # of Modules

Extended Cache Memory

FAS6080 / V6080 FAS6070 / V6070 SA600



FAS6040 / V6040 FAS6030 / V6030 FAS3170 / V3170



FAS3070 / V3070 FAS3140 / V3140 SA300



FAS3040 / V3040



Types of caching in PAM

PAM supports three modes of operation:

· Default mode, in which data and metadata are cached

· Metadata mode, in which only metadata is cached

· Low-priority mode, which enables caching of sequential reads and other low-priority data

Default Mode
The default mode caches both user data and metadata, similar to the caching policy that Data ONTAP implements for the WAFL buffer cache. For file service protocols such as NFS and CIFS, metadata includes the data required to maintain the file and directory structure. With SAN, the metadata includes the small number of blocks that are used for the bookkeeping of the data in a LUN.

This mode is best used when the working set size is equal to or less than the size of the PAM cache. It also helps when there are hot spots of frequently accessed data and ensures that the data will reside in cache.

flexscale.enable on

flexscale.lopri_blocks off

flexscale.normal_data_blocks on

Metadata Mode
In this mode, only storage system metadata is cached. In many random workloads, application data is seldom reused in a time frame in which caching would be beneficial. However, these workloads tend to reuse metadata, so caching it can improve performance. Caching just the metadata may also work well for data sets that are too large to be effectively cached (i.e., the active data set exceeds the size of the cache).

flexscale.enable on

flexscale.lopri_blocks off

flexscale.normal_data_blocks off

Low-Priority Mode
In low-priority mode, caching is enabled not only for normal data and metadata but also for low-priority data that would normally be excluded. Low-priority data in this category includes large sequential reads, plus data that has recently been written. Writes are normally excluded from the cache because the overall write workload can be high enough that writes overflow the cache and cause other more valuable data to be ejected. In addition, write workloads tend not to be read back after writing (they are often already cached locally on the system performing the write) so they’re generally not good candidates for caching.

The low-priority mode may be useful in applications that write data and read the same data after a time lag such that upstream caches evict the data. For example, this mode can avoid disk reads for a Web-based application that creates new data and distributes links that get accessed some time later by Web users. In some Web applications, we’ve found that the time lag for the first read is long enough that the data has to come from disk (even though subsequent data references are frequent enough to be handled by upstream caches). PAM in low-priority mode could accelerate these applications by turning such disk reads into cache hits.

flexscale.enable on

flexscale.lopri_blocks on

flexscale.normal_data_blocks on

In part 2 of this post I will tell you how to determine if PAM will improve the performance

How usable space in NetApp array LUN is calculated

The usable space of an array LUN is reduced by the elements shown in the following table.

If you are using block checksums

If you are using zoned checksums

10%–WAFL reserve

10%–WAFL reserve

1%– Core dump

1%– Core dump

20%–Volume-level Snapshot copy (default setting; optional)

20%–Volume-level Snapshot copy (default setting; optional)

5%–Aggregate-level Snapshot copy (default setting; optional)

5%–Aggregate-level Snapshot copy (default setting; optional)


Please note that the NetApp definition of a gigabyte (GB) is 1000 x 1024 x 1024 bytes.

Now you must be thinking why shouldn’t I go for zoned checksums to save 12.5% space? So what I would say is, if you are not much concerned about performance then you an easily go ahead and use zoned checksum over block checksum (specially in random-read intensive workloads) however please understand the difference between them before you make-up your mind to select one over another. There are a lot of references available on the internet to help you understand the concept even I will also try to cover them if I have some time.

Here’s couple of formulas from NetApp to calculate the space

Usable capacity is the actual space available to store the data and desired capacity is the exact array LUN size required to achieve a usable capacity.

Now if we denote usable capacity with Y and desired capacity with Z so we get the below formula here I have shown an example of 10GB in desired and usable space with Snapshot reserve set at default.



Y = Z x {0.875 x 0.9 x 0.99 x Snapshot reserve}

Y = Z x {0.9 x 0.99 x Snapshot reserve}

10 x {0.875 x 0.9 x 0.99 x 0.8} = 6.2 GB

10 x {0.9 x 0.99 x 0.8} = 7.1 GB

Z = Y / {0.875 x 0.9 x 0.99 x Snapshot reserve}

Z = Y / {0.9 x 0.99 x Snapshot reserve}

10 / {0.875 x 0.9 x 0.99 x 0.8} = 16 GB

10 / {0.9 x 0.99 x 0.8} = 14 GB

Please note that I have taken the disk size as identified by system not the marketing size given by manufacturers.