pNFS in NetApp

In my last post I discussed about pNFS features and architecture, and promised that in next post I will discuss about NetApp’s implementation of pNFS. So here it is.

Since 1992, the time when NetApp was formed it is always known for its solid NFS implementation and frontrunner of NFS design and standardization. Today when NetApp has gone unified, and support not only NFS but SMB, FC, iSCSI, FCOE and some other less known protocols also, still it is a major driving force to NFS design. The latest example is pNFS where NetApp has proved again by delivering pNFS to its ONTAP 8.1 Cluster Mode offering.

In my opinion they have done a wonderful job with their implementation of pNFS, though currently it’s limited to support of file only, but heck yeah it’s wonderful. In NetApp’s implementation of pNFS not only data is distributed across all the nodes but metadata also. So if you have got metadata intensive workload you can tackle that as well by using round robin from your DNS infrastructure and instantly you have a cluster which not only scales linearly with data nodes but metadata nodes also.

Another niche feature is live volume transition. It gives flexibility to move a live volume in any node of cluster if ever required, that too non-disruptively. Thanks to pNFS, it keeps same NFS handle while data being served from new node so you don’t need to unmounts and remount the filesystem and since we use cluster wide namespace no change is required at its directory location also. Isn’t this sweet, no more struggling to get downtime to move a volume, huh!

And the best part is that it’s built around industry standard set by IETF, so there is no requirement of special client software and you can use it along with NFS version 3, 4 or 4.1 without pNFS supported client.

The only setback from using it across your environment is limited client support. As of today, only Fedora 16 with kernel 2.6.39 and RHEL 6.2 can be used, however soon we will see its support in other distributions also as it’s already in kernel mainstream and it will not be long when other distributions start shipping it.

References:
http://media.netapp.com/documents/wp-7153.pdf
http://media.netapp.com/documents/tr-4063.pdf

pNFS

As my nature, I was lazing around over the weekend when I noticed the newly published best practice document on pNFS and ONTAP 8.1 Cluster Mode. Soon I realized this is what c-mode is made for; parallelism. How? Let’s see that.

First let’s start with what is Parallel NFS or in short ‘pNFS’?

pNFS is extension of NFS version 4.1 which adds support of parallelism to existing NFS version 4.

Well, that was the shortest answer I could write however here’s little more detail on it.

pNFS is part of NFS 4.1, the second minor version to NFS version 4 which adds support of session, and directory delegation, along-with parallelism. The idea to use SAN filesystem architecture and parallelism in NFS was originated from Gary Grider of Los Alamos National Lab and Lee Ward  of Sandia National Lab, later it was presented by Gearth Gibson, a professor at Carnegie Mellon University and founder and CTO of Panasas, Brent Welch of Panasas, and Peter Corbett of NetApp in a problem statement to Internet Engineering Task Force (IETF) in 2004. Later, in 2005 NFSv4 working group of IETF commended drafts and in 2006 it folded into the 4.1 minor version draft. It’s published under RFC 5661 describing NFS version 4.1 with parallel support and RFC 5662 detailing protocol definition codes. pNFS is not limited to file, support for block data (RFC 5663) and object based data (RFC 5664) are also added, so now it’s possible to access not only file but object (OSD) and block (FC/iSCSI) based storage also over NFS.

pNFS, being open source, does not require additional software or drivers on the client that are proprietary to enable. Therefore, the different varieties of NFS can coexist at the same time and supported NFS clients can mount the same file system over NFSv3, NFSv4, and NFSv4.1/pNFS.

It is widely accepted by industry and jointly developed by Panasas, NetApp, Sun, EMC, IBM, UMich/CITI and many more however at the time of writing only Fedora 16 with kernel 2.6.39 supports all three layout types (blocks, files and objects) whereas RHEL 6.2 support only files layout.

Now question comes, what was the need for it?

We all love NFS for its simplicity, and from the time it was designed by Sun in the era of 10Mb Ethernet, it scaled well to 100Mb and then gigabit Ethernet, however with the advent of 10Gb and 40Gb links, single stream designed protocol wasn’t enough to scale it further. Industry has already used TOE Ethernet cards, link aggregation and bigger boxes but that wasn’t sufficient to utilize the bandwidth and CPU powers we have available now. So what was left to deal with? Parallel NFS.

So, what’s so different from earlier version of NFS?

pNFS is not much different from its ancestors it just separates metadata from data. Unlike traditional NFS Versions 3, 4, and 4.1, where metadata and data are shared on the same I/O path, with pNFS, metadata and data travels on different I/O paths. It allows metadata server handles all the metadata activities from the client, while the data servers provide a direct path for data access.

      +-----------+

      |+-----------+                                 +-----------+

      ||+-----------+                                |           |

      |||           |       NFSv4.1 + pNFS           |  Metadata |

      +||  Clients  |<----------Metadata------------>|   Server  |

       +|           |                                |           |

        +-----------+                                |           |

             |||                                     +-----------+

            Data                                           |

             |||                                           |

             ||| Storage        +-----------+              |

             ||| Protocol       |+-----------+             |

             ||+----------------||+-----------+  Control   |

             |+-----------------|||           |    Protocol|

             +------------------+||   Data    |------------+

                                 +|  Server   |

                                  +-----------+

                         Figure 1: pNFS Architecture

As a result of this, in a clustered storage system with multiple nodes you mount only a directory or root namespace but you get direct access to data from each nodes. The metadata server (MDS) handles all nondata traffic such as GETATTRs, SETATTRs, ACCESS, LOOKUPs, and so on. Data servers (DSs) store file data and respond directly to client read and write requests. A control protocol is used to provide synchronization between the metadata server and data server.

Ok, but where does pNFS add value?

Large files and high number of concurrent users. With the advent of parallel computing with multi node cluster, single job gets divided amongst n nodes and when a job arrives at computational nodes they all try to access same data from one storage location which soon becomes bottleneck however with pNFS multiple storage system nodes responds with parts of file in parallel, increasing the aggregated bandwidth and lowering the latency. At the same time when many small files are accessed by large number of concurrent users single storage system can get chocked however with pNFS all nodes of storage system hosting the file, share user load.

Great, but how all this work?

In principal, pNFS uses parallelism used by RAID-0 however at different level. As in RAID-0 data is spread across multiple disks for faster response, same way in pNFS one file/filesystem is spread across multiple nodes in clustered storage array for faster response.

For example when client sends read requests for a file to storage system, storage system replies with file metadata along with layout details, detailing node address, data location, and striping information, after getting layout details client knows list of cluster nodes having parts of the file. Now client directly contacts to all the data nodes simultaneously for the file and nodes reply with the parts of files they have which clients later assembles.

I think It’s enough for now, next post will detail about pNFS implementation by NetApp

Read for scholars:

http://www.pnfs.com/

http://datatracker.ietf.org/wg/nfsv4/

http://www.pdl.cmu.edu/pNFS/index.shtml

http://www.citi.umich.edu/projects/asci/pnfs/linux/

http://www.pnfs.com/docs/LISA-11-pNFS-BoF-final.pdf

http://www.ietf.org/internet-drafts/draft-ietf-nfsv4-pnfs-00.txt

http://www.pdl.cmu.edu/pNFS/archive/gibson-pnfs-problem-statement.html

How to quit SSH session by typing 'exit' in ONTAP

Just a little trick to remove annoyance for new NetApp admin.

Usually when we want to terminate a SSH session on ONTAP you type Ctrl+D as typing 'exit' returns with message to press 'Ctrl+D' to exit. It's perfectly fine and there isn't any harm as that gracefully closes the session but as most of us are more use to of typing 'exit' to terminate the session, find it odd. 

So for those who prefer typing 'exit' to close the session can do so by enabling telnet.distinct.enable option.
Default setting for this option is off which in effect mirrors telnet and console session but if you don't use this feature, which I believe most of us don't, can enable it and feel more at home.

How to restore data from aggregate snapshot

Today one of our user found himself in wet pants when he noticed his robocopy job has overwritten a folder, rather than appending new data to it. Being panicked he run to me looking for any tape or snapshot backup of his original data, which unfortunately wasn’t there as previously he confirmed that they don’t need any kind of protection.

Now at this time I had only place left where I can recover the data, aggregate level snapshots; so I looked at aggregate snapshots and saw it goes back to time when he had data in place. Knowing that the data deleted from volume is still locked in aggregate’s snapshot I was feeling good that I have done a good job by having some space reserved for aggregate level snapshot, which no one ever advocated.

Now the next step is to recover the data, but problem was that if I revert aggregate using “snap restore –A” then all the volumes in that aggregate will be reverted which will be bigger problem. So had to go on a different way, use aggregate copy function to copy the aggregate’s snapshot to an empty aggregate and then restore the data from there.

Here’s the cookbook for this.

Pre-checks:

• The volume you lost data from is a flexible volume 
• Identify an aggregate which is empty so it can be used for destination (could be on another controller also) 
• Make sure the destination aggregate is either equal or larger than source aggregate 
• /etc/hosts.equiv has entry for the filer you want to copy data to and /etc/hosts has its IP address added, in case of copying on same controller loopback address (127.0.0.1) should be added in /etc/hosts file and local filername should be in hosts.equiv file 
• Name of aggregate’s snapshot which you want to copy 

Example:

Let’s say the volume we lost data was ‘vol1’, the aggregate which has this volume is ‘aggr_source’, the aggregate’s snapshot which has lost data is ‘hourly.1’ and empty aggregate where we will be storing data to is ‘aggr_destination’

Execution:

• Restrict the destination aggregate using  ‘aggr restrict aggr_destination’ 
• Start the aggregate data copy using ‘aggr copy start –s hourly.1 aggr_source aggr_destination’ 
• Once the copy is completed online the aggregate using ‘aggr online aggr_destination’ 
• If you have done copy on same controller, system will rename the volume ‘vol1’ of ‘aggr_destination’ to ‘vol1(1)’ 
• Now export the volume or lun and you have your all lost data available.

So here’s the answer to another popular question, why do I need to reserve space for aggregate level snapshot. Do you have the answer now?

Most destructive command in Ontap

There are some commands which shake me when I run or even when I am close to them, but never thought I could be make my filer so close to death by just mistyping a command.

Yes, indeed I did it by typing 'gbd' rather than 'dbg', these two are so close to each other that my buttery fingers didn't realize that I mistyped a command and by the time I could realize it was too late. Sigh!

Little background on this 'gbd' command.

This command is in diag mode and so debug, however whereas 'dbg' captures filer debug info on console or a file, 'gbd' sends kill signal to all the processors which stops all the work on filers and everything just hangs. The only way to recover your filer is by hard rebooting it, either by RLM or physically tipping the power button.

I don't understand why brilliant NetApp engineers have made it so simple, why couldn't they use a command like 'use_this_to_kill_your_system' or something like that and I swear no one would ever type it.

Anyway I did it and I admit, that I should have checked before hitting return which I didn't. But guess what I was lucky enough to not to do it on a prod system and this test/dev system was having only a bunch of NFS clients were connected to it which made it sort of invisible to client systems due to the nature of NFS protocol.

What is the other command which you think shouldn't be so easy along with this?

How to do host/user/group or netgroup from filer

Often we want to do nslookup for a host or NIS/LDAP lookup for a user or group for troubleshooting purpose. You have a unix system handy and you do it from there however what if you suspect results are not same as what your filer may be getting?

If you are troubleshooting CIFS issue, you are in luck with command 'cifs lookup' however, if you are dealing with DNS or NFS issue then you are out of luck, unless you go into advanced mode. Yes, you go inside advance mode and you get access to lot of other commands including one very nifty command 'getXXbyYY', which is incredibly useful but hidden from the view of admins for some strange reason, really I am not sure why NetApp thinks this shouldn't be available to end user as every time I do troubleshooting I feel the need of this and by no way I see this to be making any sort of changes on filer.

Anyway here's the command, though command says using "man na_getXXbyYY" for additional info however I couldn't locate it on systems therefore I use

test1*> getXXbyYY help
usage: getXXbyYY
Where sub-command is one of
gethostbyname_r
gethostbyaddr_r
netgrp
getspwbyname_r
getpwbyname_r
getpwbyuid_r
getgrbyname
getgrbygid
getgrlist

For more information, try 'man na_getXXbyYY'

Please remember this command is not available in admin mode and search order depends of your /etc/nsswitch.conf entry, so before you start thinking that this isn't working as expected please check these two things first.

Though all the subcommands are self explanatory however I have added small description for each of them.

gethostbyname_r - Resolves host name to IP address from configured DNS server, same as nslookup
gethostbyaddr_r - Retrieves IP address for host name from configured DNS server, same as reverse lookup
netgrp - Checks group membership for given host from LDAP/Files/NIS
getspwbyname_r - Displays user information using shadow file
getpwbyname_r - Displays user information including encrypted password from LDAP/Files/NIS
getpwbyuid_r - Same as above however you provide uid in this command rather than user name
getgrbyname - Displays group name and gid from LDAP/Files/NIS
getgrbygid - Same as above however you provide gid in this command rather than group name
getgrlist - Shows given user's gid from LDAP/Files/NIS

Examples:

test1*> getXXbyYY gethostbyname_r landinghost1
name: landinghost1
aliases:
addresses: 10.21.242.7

test1*> getXXbyYY gethostbyaddr_r 10.21.242.7
name: landinghost1
aliases:
addresses: 10.21.242.7

test1*> getXXbyYY netgrp support-group testhost1
client testhost1 is in netgroup support-group

test1*> getXXbyYY getpwbyname_r root
pw_name = root
pw_passwd = _J9..gsxiYTAHEtV3Qnk
pw_uid = 0, pw_gid = 1
pw_gecos =
pw_dir = /
pw_shell =

test1*> getXXbyYY getpwbyuid_r 0
pw_name = root
pw_passwd = _J9..gsxiYTAHEtV3Qnk
pw_uid = 0, pw_gid = 1
pw_gecos =
pw_dir = /
pw_shell =

test1*> getXXbyYY getgrbyname was
name = was
gid = 10826

test1*> getXXbyYY getgrbygid 10826
name = was
gid = 10826

test1*> getXXbyYY getgrlist wasadmin
pw_name = wasadmin
Groups: 10826

Execute command from a file on Ontap

Occasionally we want to quickly or periodically run a set of pre defined commands on filers, like when we are making little change in network configuration and want to minimize the network down time or when we are creating a volume and know snap reserve, snap schedule, de-dupe, autosize or anything else which needs to be changed after every volume creation. If you are executing commands from unix terminal then a better way would be to keep all the commands in a text file and would do something like

bash>for $i in vol1
>do
>snap sched $i 0
>snap reserve $i 0
>blah $i
>blah $i
>done

and everything would be done, but imagine how would you do if you are doing through console. Unfortunately Ontap doesn't supports any kind of script not even for loop so we have to run each and every line of command either by typing on console or doing a copy-paste from text file from our desktop.

However there is a better way, use notepad to create set of command as you would execute on console with correct order, copy it off to filer and use 'source' command to execute each and every line from it. I know it's not such a brilliant idea as still you have to copy and paste everything to a file on filer however it's wee better than executing each and every command on console.

Think about you have to re-run /etc/rc so either you can use 'rdfile /etc/rc' to print everything on console and copy all the the line and execute on console or just run 'source /etc/rc' and let it run all the commands for you. You can also use 'source -v /etc/rc' to print the commands on console but not to execute, just to get an idea if there are any junk characters or any unwanted command inside the file, just as a precaution you better be sure that all the commands are valid and correct as if a command fails source doesn't stops there and just goes to next command in list.

Use it and I am sure you will like it next time when you are making some changes on filer which needs ten different commands to be executed.

How to check unplanned downtime detail for a NetApp filer

Every now and then someone ask us what is uptime of system and we just type 'uptime' on system console to get the detail instantly.


This is really handy command to know when the system was last rebooted and how many operations per protocol it has served since then. Wouldn't our life be little easy if managers get satisfy with this detail? Alas! but that doesn't happen and they ask us to give all the details since we have acquired the system or 1st January and then we go back to our excel sheet or ppt we have created as part of monthly report to pull the data.


How about if we can get same information from system with just a command, wouldn't that be cool. Fortunate enough we have little known command 'availtime' right inside Ontap which just do the exact same function and specifically created after thinking about our bosses.


HOST02*> availtime full
Service statistics as of Sat Aug 28 18:07:33 BST 2010
 System  (UP). First recorded 68824252 secs ago on Mon Jun 23 04:16:41 BST 2008
         Planned   downs 31, downtime 6781737 secs, longest 6771328, Tue Sep  9 15:07:33 BST 2008
         Uptime counting unplanned downtime: 100.00%; counting total downtime:  90.14%
 NFS     (UP). First recorded 68824242 secs ago on Mon Jun 23 04:16:51 BST 2008
         Planned   downs 43, downtime 6849318 secs, longest 6839978, Wed Sep 10 10:11:43 BST 2008
         Uptime counting unplanned downtime: 100.00%; counting total downtime:  90.04%
 CIFS    (UP). First recorded 61969859 secs ago on Wed Sep 10 12:16:34 BST 2008
         Planned   downs 35, downtime 17166 secs, longest 7351, Thu Jul 30 13:52:25 BST 2009
         Uptime counting unplanned downtime: 100.00%; counting total downtime:  99.97%
 HTTP    (UP). First recorded 47876362 secs ago on Fri Feb 20 14:08:11 GMT 2009
         Planned   downs 8, downtime 235 secs, longest 53, Wed Jan 20 14:10:18 GMT 2010
         Unplanned downs 16, downtime 4915 secs, longest 3800, Mon Jul 27 16:01:02 BST 2009
         Uptime counting unplanned downtime:  99.98%; counting total downtime:  99.98%
 FCP     (DOWN). First recorded 68817797 secs ago on Mon Jun 23 06:04:16 BST 2008
         Planned   downs 17, downtime 44988443 secs, longest 38209631, Sat Aug 28 18:07:33 BST 2010
         Unplanned downs 6, downtime 78 secs, longest 21, Fri Feb 20 15:24:44 GMT 2009
         Uptime counting unplanned downtime:  99.99%; counting total downtime:  34.62%
 iSCSI   (DOWN). First recorded 61970687 secs ago on Wed Sep 10 12:02:46 BST 2008
         Planned   downs 21, downtime 38211244 secs, longest 36389556, Sat Aug 28 18:07:33 BST 2010
         Uptime counting unplanned downtime: 100.00%; counting total downtime:  38.33% 


I am not sure why NetApp has kept this command in Advanced mode but once you know this command I bet next time you will not refrain yourself going inside advance mode to see how many unscheduled downtime you had since last reset.


A shorter version of this command is just 'availtime' it also shows the same information as 'availtime full' however it truncates letters from output and denotes  Planned with P and Unplanned with U which is very good if you want to pass it in script. 

HOST04*> availtime
Service statistics as of Sat Aug 28 18:07:33 BST 2010
 System  (UP). First recorded (20667804) on Wed Sep 23 09:35:49 GMT 2009
         P  5, 496, 139, Fri Dec 11 15:58:19 GMT 2009
         U  1, 1605, 1605, Wed Mar 31 17:01:41 GMT 2010
 CIFS    (UP). First recorded (20666589) on Wed Sep 23 09:56:04 GMT 2009
         P  7, 825, 646, Thu Jan 21 19:08:03 GMT 2010
         U  1, 77, 77, Wed Mar 31 16:34:54 GMT 2010
 HTTP    (UP). First recorded (20664731) on Wed Sep 23 10:27:02 GMT 2009
         P  3, 51, 22, Thu Jan 21 19:17:25 GMT 2010
         U  4, 203, 96, Thu Jan 21 19:08:03 GMT 2010
 FCP     (UP). First recorded (20477735) on Fri Sep 25 14:23:38 GMT 2009
         P  3, 126, 92, Thu Jan 21 19:07:57 GMT 2010
         U  4, 108, 76, Wed Mar 31 16:34:53 GMT 2010

In order to reset the output use 'reset' switch and it will zero out all the counters, make sure you have recorded the statistics before you reset the counters as once you reset the counters you will not be able to get details of system uptime since system was built so you may like to do only after you acquire a new system, have done all the configuration and now it's the time for it to serve user requests.

Operations Manager Efficiency Plugin

After the release of DFM 3.8.1 NetApp has released a nice little plugin for DFM called as ‘Operations Manager Storage Efficiency Dashboard Plugin’. Though quite a long name but it’s good, it cleverly uses DFM database to pull storage utilization and presents the information in nice flash based webpage.

It’s useful when you have to show higher management current storage utilization and saving came from NetApp thin provisioning, dedupe, flexclone and other stuffs and goes very well with NetApp’s storage efficiency mantra. The best part is, after you install the plugin you don’t have to anything and you can access it from anywhere in network without installing any software, however there isn’t a simple way to reach the page even after you are right inside OM webpage as there is no link pointing to dashboard, so you have to remember the location to access it later or for people like me bookmark in your browser.

The most common problem arising from this is due to lack of foresight while creating the plugin. Here’s what I mean to say. Usually we install DFM server on c:\ and move all perfdata, DB, script folder and other bits and pieces to a different drive for easy backup or in the case of cluster, for clustering setup and here script falls apart. Script expects that it is sitting in its default location and web folder is sitting right next to it, so it acts accordingly whereas in real situation web folder is on c:\ and script is in some other volume.

Now there isn’t any way to rectify the behaviour of script or web server, as apache running on DFM can’t be configured to use any folder other then the one sitting inside the installation directory (AFAIK) and no switches are provided in script to tell him the location of original web folder where he needs to copy its content.

So in nutshell even though script executes and copies all the files required for showing the dashboard it’s useless unless you figure out by yourself what’s going wrong and why not the page is showing in your browser.

Overcoming this limitation is easy enough for folks those who are on Unix environment as creating an alias to original web folder makes everything working fine but for windows folks like me creating a shortcut doesn’t works.

So here’s the way to correct the problem.

Download the plugin from now toolchest. Extract the zip and edit file ‘package.xml’, change the string “dfmeff.exe” to “dfmeff.bat”, next you have to create a new batch file in called “dfmeff.bat” with below contents.

@echo off

D:\DFM\script-plugins\dfmeff\dfmeff.exe

xcopy D:\DFM\web\*.* "C:\Program Files\NetApp\DataFabric\DFM\web" /Q/I/Y/R

Obviously you have to change the path as per your installation however once you have created the batch file and added its reference in xml file you are good to go, just zip it again using any zip software and use the new zip file as plugin source for installation in DFM.

Update:
Just noticed a video showing features of plugin on netapp community site http://communities.netapp.com/videos/1209

Which is faster, NDMPcopy or vol copy?

After posting my last post I got few mails asking, amongst ndpmpcopy and vol copy, which one would be faster?

Only if I have to count speed then vol copy, because it copies blocks directly from disk without going through FS, however I think it’s well suitable if you want to migrate a volume.

Pros

• CPU usage can be throttled
• Source volume snapshot can be copied
• Simultaneously 4 copy operations can be started
• Once started it goes to background and you can use console for other purpose 

Cons

• Destination can’t be root volume
• Destination volume should be offline
• All data in destination volume will be over-written
• Destination volume size should be bigger or equal to source
• Single file or directory cannot be specified for copy operation
• Both the volumes should be of same type; traditional or flexible
• If data is copied between two filers both filer should have other filer’s entry in /etc/hosts.equiv file and loopback address for itself in /etc/hosts file 

However for copying data between two filers for test or any other purpose ndmpcopy is more suitable because it gives you additional control and less restrictions, which is very useful.

Pros

• Little or no CPU overhead
• Incremental copy is supported
• No limitation on volume size and type
• No need to take destination volume offline
• Single file or directory can also be specified
• No file fragmentation on destination volume as all data is copied sequentially from source volume so improved data layout
• No configuration is required between two filers and username and password is used for authentication

Cons

• Snapshots can’t be copied from source
• Console is not available till the time copy operation is running so no multiple ndmpcopy operations
• If lots of small files has to be copied then copy operation will be slower 

So as you have seen both are well however one can’t be replaced for other and both have their usage for different purposes.

How to copy files in Ontap

As soon as someone asks this question we all say ‘use ndmpcoyp’ but what if you don’t have any network adapters configured, will ndmpcoyp work?

No; ndmpcopy is very useful if you want to copy a file or a whole volume however one thing very few people know that it doesn’t work if you don’t have loopback adapter configured because ndmpcopy passes all the data through lo adapter so it’s not only dependent on lo’s availability, its speed also. So how do you copy the data if lo is not available?

The answer is simple, use dd, just an old fashioned unix command which does lot of thing, not only it can copy the file with full pathname you can even use block number and disk number and the best part, syntax is simple ‘if’ for from and ‘of’ for to.

It can be used not only for copying file around the system, in fact you can use it for testing I/O and copying file from snapshot also and this command can be used regardless of permission.

A little note, if you are afraid of going in advanced or diagnostic mode better keep use rdfile and wrfile because this command is not available in admin mode so you have to go in advanced mode to use this.

Here’s the syntax of this command.

dd [ [if= file ] | [ din= disknum bin= blocknum ] ] [ [of= file ] | [ dout= disknum bout= blocknum ] ] count= number_of_blocks

Another note, if you are using count make sure you are using in multiply of 4 because a WAFL block size is 4k.

Example:

sim1> priv set advanced
sim1*> dd if=/vol/vol0/.snapshot/hourly.2/etc/snapmirror.conf of=/vol/vol0/etc/snapmirror.conf1

I2P (inode to pathname) in Ontap

Few days before I became curios to know what is I2P and what will happen if you turn it off. So started hunting NetApp site and Google for information however I couldn’t find much on this. Now as obvious once you are off from vendor site and Google then start looking at your social network and then, bingo! I was able to get some information on this.

So the first question what the heck is this I2P?

Actually it’s a new feature in Ontap 7.1 and later versions that maps each inode number and file name with relative path to speed up certain operations. As we all know every file, directory, soft link, hard link or any other metadata file has one inode associated with it so each inode has to go through this process and each file/directory gets 8 bits added to its metadata by Ontap whereas every hard link gets 12 bytes penalty and this happen every time you create, delete or rename file, directory or link.

Ok, but why do I need this?

As far as its usage goes there are some well known application for this like fpolicy, virus scan and file auditing which needs to know full path of the requested file whereas some are Ontap specific feature like in mixed volume it informs NFS clients for any changes made by CIFS clients; one important place where you will see difference is dump command, as having full path available for each file makes it much faster and efficient in operation, some grey areas are also there which are used by Ontap for its internal work but that’s all covered deep under their IP protection policy so I couldn’t get any info on it.

Now how to you get information about this from your system?

If you look closely you will see in vol options there is an option ‘no_i2p’ (default is off and only in 7.1 and later) to enable or disable i2p feature as well if you go to advance mode you can see few more commands related to this like ‘inodepath’ which shows the i2p information stored in a given inode whereas ‘wafl scan status’ command shows you running i2p scans which can be aborted with 'wafl scan abort scan_id' or you can also change the scan speed with ‘wafl scan speed new_speed’ command after you list the current scan speed with ‘wafl scan speed’ command.

After having these information pushed me to think that ok so most of the volumes in my systems are NFS only and they don’t need any virus scan, neither we use dump, fpolicy or any other feature so why not to turn off and get extra juice out of system but speaking with chaps it turned out that turning off wouldn’t be a good idea though they were also not sure what will break if you turn it off and as it has very little performance impact so better to leave it untouched.

And yes it’s true it does have very less performance impact as in general we don’t do so much metadata modification that it may hurt the system with i2p workload however when you upgrade your system from an earlier release to 7.1 or later family they get very busy in creating i2p information for each and every file/directory/link etc and may run in high utilization for quite some time and at that time you may wish to use these commands to quickly pull your system back in normal state and let scan run with slow speed or one volume at a time or completely stop it if you want to revert back the system to pre 7.1 release.

I think if I get some time I would like to do an extensive testing and see what comes out however if anyone else knows please share your knowledge.

Defragement in NetApp

Usually we face this problem with our PC and then we defrag our volumes clear temp files and what not; most of the times that solves the problem, though not fully but yes it gets better.

In NetApp though we don’t have to deal with fragmented registry or temp files but due to nature of WAFL file system it gets fragmented very soon, soon after you start overwriting or start deleting and adding the data to volume. So what do you do then?

Well the answer is very simple use ‘reallocate’ command. Yes, this is the defrag tool of NetApp built right in the Ontap OS.

First you have to turn on the reallocation on system with ‘reallocate on’ command. This command turns on the reallocation on system and same way turns off with off switch.

This can be used not only on volumes, infact you can run this on a file, lun or aggregate itself. However I should warn you that optimization of lun may not give you any performance benefit or may get worse, as Ontap doesn’t have any clue what’s in the lun and it’s file system layout.

If you want to run the reallocation only one time you should use -f or -o switch however if you want Ontap to keep a track of your FS and optimize the data when if feels necessary you should control it with –i switch or schedule it with ‘reallocate schedule’ command.

To check current optimization level of volume, you can use ‘reallocate measure -o ’ or if you want to feel adventurous use ‘wafl scan measure_layout ’ through advanced mode, though I don’t suggest using wafl set of commands in general use but yes sometime you want to do something different.

This command is pretty straightforward and no harm (except extra load on CPU and disk) so you can play with this but you should always consider using -p switch for volumes having snapshot and/or snapmirror on to keep the snapshot size small.

How to get the list of domain users added to filer without fiddling with SID

There were numerous time when I wanted to see an AD user’s permission on filer however just to locate that user on system itself took me a lot of time. Why? Because Ontap shows domain users added to system in SID format rather than their names which is very much annoying as when it dumps the SIDs on screen then we have to use ‘cifs lookup’ command to hunt for the user I am looking for from that bunch of SIDs.

So here’s a little handy unix script to see the list of all AD users added on filers in their username format rather then SIDs

I have already setup a password less login to filer therefore I haven’t added the username and password fields however if you haven’t done that add your login credentials after name of the filer in below command.

rsh useradmin domainuser list -g ‘Administrators’ | sed 's/^S/rsh cifs lookup S/'

Now this command will display the AD users added in Administrator group however if you want to see users from any other group replace the Administrators word with group name on your screen.

Restoring data from snapshot through snaprestore in NetApp

Ok so now you have allocated correct snap reserve space, configured snap schedules, snap autodelete, users have access to their snapshots and they recover their data without any interference of backup team. Everyone is happy so you happy but all of sudden on a Friday evening get a call from VP marketing crying on phone that he lost all his data from his network drive and windows shows recovery time of 2 hrs but he wants his 1Gb pst to be accessible now as he is on VPN with a client and needs to pull some old mails from his pst. Well that’s nothing abnormal as he was having lots of data and to recover the data windows has to read all the data from snapshot and then write back on network drive which but obvious will take time. Now what would you say, will you tell him to navigate to his pst and recover it (which shouldn’t take much time on fast connection) then try to recover all the data or ok I have recovered all your data while talking on the phone and become hero.

Well I must say I would like to use the opportunity to become hero with a minute or less of work, but before we do a few things to note.

For volume snaprestore:

• The volume must be online and must not be a mirror.
• When reverting the root volume, filer will be rebooted.
• Non-root volumes do not require a reboot however when reverting a non-root volume, all ongoing access to the volume must be terminated, just as is done when a volume is brought offline.

For single-file snaprestore:

• The volume used for restoring the file must be online and must not be a mirror.
• If restore_as_path is specified, the path must be a full path to a filename, and must be in the same volume as the volume used for the restore.
• Files other than normal files and LUNs are not restored. This includes directories (and their contents), and files with NT streams.
• If there is not enough space in the volume, the single file snap restore will not start.
• If the file already exists (in the active file system), it will be overwritten with the version in the snapshot.

To restore data there are two ways, first system admins using “snap restore” command invoked by SMO, SMVI, Filer view or system console and second by end users where they can restore by copying file from .snapshot or ~snapshot directory or by using revert function in XP or newer system. However restoring data through snap restore command is very quick (seconds) even for TBs of data. Syntax for snap restore is as below.

“snap restore -t vol -s <snapshot_name> -r <restore-as-path> <volume_name>”

If you don’t want to restore the data at different place then remove the “-r <restore-as-path>” argument and filer will replace current file with the version in snapshot and if you don’t provide a snapshot name in syntax then system will show you all available snapshots and will prompt to select snapshot from which you want to restore the data.

Here’s the simplest form of this command as example to recover a file.

testfiler> snap restore -t file /vol/testvol/RootQtree/test.pst

WARNING! This will restore a file from a snapshot into the active filesystem. If the file already exists in the active filesystem, it will be overwritten with the contents from the snapshot.

Are you sure you want to do this? yes

The following snapshots are available for volume testvol:

date            name
------------    ---------
Nov 17 13:00    hourly.0
Nov 17 11:00    hourly.1
Nov 17 09:00    hourly.2
Nov 17 00:00    weekly.0
Nov 16 21:00    hourly.3
Nov 16 19:00    hourly.4
Nov 16 17:00    hourly.5
Nov 16 15:00    hourly.6
Nov 16 00:00    nightly.0
Nov 15 00:00    nightly.1
Nov 14 00:00    nightly.2
Nov 13 00:00    nightly.3
Nov 12 00:00    nightly.4
Nov 11 00:00    nightly.5
Nov 10 00:00    weekly.1
Nov 09 00:00    nightly.6
Nov 03 00:00    weekly.2
Oct 27 00:00    weekly.3

Which snapshot in volume testvol would you like to revert the file from? nightly.5

You have selected file /vol/testvol/RootQtree/test.pst, snapshot nightly.5

Proceed with restore? yes
testfiler>

Snapshot configuration in NetApp

Ok first of all let me admit that my last post sounded more as a sales pitch rather than something technical though I am not a NetApp employee or paid by anyone to do blogging. However I must agree that whatever I have tried to show there was petty much similar available from other vendors so it was more about general awareness of  technology rather than a particular vendor but in this post I will talk about Snapshot configuration and other functions in NetApp, so let’s start.

What is snapshot copy?

A Snapshot copy is a frozen, read-only image of a volume or an aggregate that captures the state of the file system at a point in time and each volume can hold maximum 255 Snapshot copies at one time.

Snapshots can be taken either by system at pre-defined schedule, Protection Manager Policies, SMO, SMVI, Filer view or manually running command at system console or through custom scripts.

How to disable client access to snapshot copy?

To disable client access of .snapshot volume you can give “vol options <volume_name> nosnapdir on” command.

Notes:

• Please DO NOT use any snap family of command without volume name as it may drive CPU processor to its peak for systems having lots of volume with a number of snapshots and it can hung the system which may result in system panic situation.
• Use “-A” if you want to run these command against any aggregate and replace volume name with aggregate name.

How to Configure Snapshots through system console?

It’s always recommended that when you provision a volume you should look at snapshot reserve and schedule as by default when a volume is created 20% of space is reserved for snapshots which most of the time you need to change for efficient usage of space and snapshots. Always ask requester what is the rate of change, how much snapshots he wants to have access to and when he wants to snapshots to be taken because if you take snapshot of some oracle data and database is not in hot-backup mode then it’s just utter waster and same goes for VM.

So once you have those details do a little calculation and then use these command to configure.

1. ‘snap reserve <volume name> <snapshot reserve size in % volume size, gb, mb or kb>’
   Example:
   ‘snap reserve testvol 10’
   This command will allocate 10% of space for snapshots on volume “testvol”
   
2. ‘snap sched <volume name> <week days hour@list>’
   Example:
   ‘snap sched testvol 4 7 7@9,11,13,15,17,19,21’
   
   This command will define the automatic snapshot schedule, and here you specify how much weekly, daily or hourly snapshot you want to retain as well at what time hourly snapshot will be taken. In given example volume testvol is having 4 weekly, 7 daily and 7 hourly available where hourly snapshots are taken at 9,11,13,15,17,19 and 21 hours of system local time. Please make sure that ‘nosnap’ is set to off in volume options.

How to take snapshots manually?

To take the snapshot manually you can run below command.

“snap create <volume name> <snapshot name>”

Here volume name is the name of volume you want to take snapshot of and snapshot name is the name you want to identify snapshot with.

How to list snapshots?

You can check the status of snapshots associated with any volume with command

“snap list <volume name>”

After issuing the above command you will get similar output

testfiler> snap list testvol

Volume testvol
working...

%/used %/total date name
---------- ---------- ------------ --------
36% (36%) 0% ( 0%) Dec 02 16:00 hourly.0
50% (30%) 0% ( 0%) Dec 02 12:00 hourly.1
61% (36%) 0% ( 0%) Dec 02 08:00 hourly.2
62% ( 5%) 0% ( 0%) Dec 02 00:01 nightly.0
69% (36%) 0% ( 0%) Dec 01 20:00 hourly.3
73% (36%) 0% ( 0%) Dec 01 16:00 hourly.4
77% (36%) 0% ( 0%) Dec 01 00:01 nightly.1

What if you are running low on snap reserve?

Sometimes due to excessive rate of change in data, very soon snapshot reserve gets full and they over spill on data area of volume, to remediate this you have to either extend volume or delete old snapshots.

To resize the volume use “vol size” command and to delete the old snapshots you can use “snap delete” command which I will cover in next section, however before deleting if you want to check how much free space you can gain from this snapshot use below command

“snap reclaimable <volume name> <snapshot name> | <snapshot name>…”

Running above command will give you output as below and you can add multiple snapshot names after one other if you are not getting required free space by deleting one snapshot. Please note that you should select snapshots for deletion only from oldest to latest order otherwise blocks freed by deleting any middle snapshot will still be locked in its following snapshot

testfiler> snap reclaimable testvol nightly.1 hourly.4
Processing (Press Ctrl-C to exit) ............
snap reclaimable: Approximately 9572 Kbytes would be freed.

How to delete snapshot?

To delete the snapshot use command snap delete with volume name and snap name in below fashion

“snap delete <volume name> <snapshot name>”

Running this command will print similar information on screen

testvol> snap delete testvol hourly.5

Wed Dec 2 16:58:29 GMT [testfiler: wafl.snap.delete:info]: Snapshot copy hourly.5 on volume testvol NetApp was deleted by the Data ONTAP function snapcmd_delete. The unique ID for this Snapshot copy is (67, 3876).

How to know what is the actual rate of change?

Sometime on a particular volume very often you will be running out of snap reserve space as snapshots fill them up much before old snaps gets expire and deleted by auto delete function (if you have configured) and you must be interested to resize the snap reserve accurately to avoid any issues. So in order to check how much is the actual rate of change KB per/hour calculated from all the snapshots or between two snap on given volume you can use snap delta command.

“snap delta <volume name> [<1st snapshot name> <2nd snapshot name>]”

testfiler> snap delta testvol

Volume testvol
working...

From Snapshot   To                   KB changed  Time         Rate (KB/hour)
--------------- -------------------- ----------- ------------ ---------------
hourly.0        Active File System   30044          0d 00:28  63176.635
hourly.1        hourly.0             552            0d 02:00  276.000
hourly.2        hourly.1             552            0d 01:59  276.115
weekly.0        hourly.2             628            0d 09:00  69.680
hourly.3        weekly.0             468            0d 03:00  155.956
hourly.4        hourly.3             552            0d 01:59  276.115
hourly.5        hourly.4             500            0d 02:00  249.895
hourly.6        hourly.5             548            0d 01:59  274.038
nightly.0       hourly.6             560            0d 14:59  37.334
nightly.1       nightly.0            700            0d 23:59  29.171
nightly.2       nightly.1            5392           1d 00:00  224.666
nightly.3       nightly.2            820            0d 23:59  34.172
nightly.4       nightly.3            2920           0d 23:59  121.687
nightly.5       nightly.4            880            1d 00:00  36.666
weekly.1        nightly.5            1111956        1d 00:00  46307.381
nightly.6       weekly.1             632            1d 00:00  26.333
weekly.2        nightly.6            42420          6d 00:00  294.583
weekly.3        weekly.2             8892           7d 00:00  52.928

Summary...

From Snapshot   To                   KB changed  Time         Rate (KB/hour)
--------------- -------------------- ----------- ------------ ---------------
weekly.3        Active File System   1209016       21d 13:29  2336.320

 

That was all about configuring creating and deleting snapshots but what it’s good if you don’t know how to restore the data from snapshots for which you have done so much things. So, in next post I will address how to restore data from snapshot through snap restore command

Snapshots in NetApp

Volumes and data:

Volume used for test was a flexible volume named ‘buffer_aggr12’ and “My Documents” folder from my laptop for data and sync tools from Microsoft to sync ‘My Documents’ folder with cifs share created from volume buffer_aggr12.

Snapshot configuration:

Scheduled snapshot were configured at 9,11,13,15,17,19,21 hours and retention period was 4 weekly, 7 daily and 7 hourly with 20% space reserve for snapshot.

The coolest part of the snapshot is flexibility, because as an administrator once you have configured it no more you have to look into this as it takes snapshot at defined schedule and if you have configured ‘snap autodelete’ then it will purge expired snapshots also as per your retention period. So effectively you don’t have to ever worry about managing hundreds of old snapshots lying in volume and eating up space (except when change rate of data overshoots and snapshots starts spilling on data area). As a end user you experience backups at your click away because snapshots integrates well with shadow copy services of windows 2000, XP or Vista and you can recover them whenever you need.

Here’s the configuration of snapshot for my test volume ‘buffer_aggr12’

AMSNAS02> snap sched buffer_aggr12
Volume buffer_aggr12: 4 7 7@9,11,13,15,17,19,21

AMSNAS02> snap reserve buffer_aggr12
Volume buffer_aggr12: current snapshot reserve is 20% or 157286400 k-bytes.

As I was running this test for months so there were enough snaps for me to play with and you can see below that these snapshots are going way back to 20^th July,  which is 4 week old snapshot and anytime I can recover that from just a right click.

How to recover files or folders from snapshot:

There are two ways to recover the data from snapshots.

As an end user you can recover your data from windows explorer by just right clicking in an empty space while you are in the share in which you lost your data. Here’s an example of this.

a) This is the snapshot of my share folder, in this as you can see my pst file is corrupted and showing 0 kb.

b) To recover this, right click on any empty area and go to properties>previous version it shows me all the snapshots taken for this folder, as shown in below screenshot.

c) Now at this point either I can revert the whole folder to previous state or just copy it to another location to recover a deleted file but at this place my point is to revert a corrupted file rather than recovering a deleted file. So I will just do a right click on that file and navigate to previous versions tab in properties dialogue box. Here in this it shows me the changes captured by snapshot at different times, so I can just select the date I want to revert with and click on restore.

d) Now it starts replacing the corrupted file with the one taken by snapshot. Its taking a long time because the file in question is >1Gb size and I am on WAN link so it’s slow but there is another way to do it and that’s recovering directly from filer console which recovers in seconds but unfortunately not available to end user.

e) Now here’ the screenshot of my before and after.

As an Administrator you can recover a file, folder or whole volume within second as while doing it from filer console, system doesn’t have to copy the old file from snapshot to temp location, delete old file and then change the recovered file’s metadata , instead it just changes the block pointers internally so it’s blazing fast . Here’s an example of this.

a) In this test again I will use same pst file which is corrupted but this time we will recover it from console. So first login to filer and do a snap list to see what all snapshots are available.
AMSNAS02> snap restore buffer_aggr12
Volume buffer_aggr12
working…
  %/used       %/total  date          name
----------  ----------  ------------  --------
  0% ( 0%)    0% ( 0%)  Aug 14 17:00  hourly.0
  0% ( 0%)    0% ( 0%)  Aug 14 15:00  hourly.1
40% (40%)    0% ( 0%)  Aug 14 13:00  hourly.2
40% ( 0%)    0% ( 0%)  Aug 14 11:00  hourly.3
40% ( 0%)    0% ( 0%)  Aug 14 09:00  hourly.4
40% ( 0%)    0% ( 0%)  Aug 14 00:00  nightly.0
40% ( 0%)    0% ( 0%)  Aug 13 21:00  hourly.5
40% ( 0%)    0% ( 0%)  Aug 13 19:00  hourly.6
40% ( 0%)    0% ( 0%)  Aug 13 00:00  nightly.1
41% ( 0%)    0% ( 0%)  Aug 12 00:00  nightly.2
57% (39%)    0% ( 0%)  Aug 11 00:00  nightly.3
57% ( 0%)    0% ( 0%)  Aug 10 00:00  weekly.0
57% ( 0%)    0% ( 0%)  Aug 09 00:00  nightly.4
57% ( 0%)    0% ( 0%)  Aug 08 00:00  nightly.5
57% ( 0%)    0% ( 0%)  Aug 07 00:00  nightly.6
57% ( 0%)    0% ( 0%)  Aug 03 00:00  weekly.1
65% (35%)    0% ( 0%)  Jul 27 00:00  weekly.2
65% ( 0%)    0% ( 0%)  Jul 20 00:00  weekly.3

b) Now to recover the file you give below command and it recovers that in just a second.
AMSNAS02> snap restore -t file -s nightly.5 /vol/buffer_aggr12/RootQtree/test.pst

WARNING! This will restore a file from a snapshot into the active filesystem.  If the file already exists in the active filesystem, it will be overwritten with the contents from the snapshot.

Are you sure you want to do this? yes

You have selected file /vol/buffer_aggr12/RootQtree/test.pst, snapshot nightly.5

Proceed with restore? yes

AMSNAS02>

c) Here’s the screenshot of my folder which confirm file back in previous state.

Now as you see it was quite easy to use and very useful also, but to have a snapshot you need some extra space reserved in volume specially if your data is changing very frequently as more changes means more space you need to store changed block and the condition goes more complicated if you are trying to take snapshot of a VM, Exchange or Database volume, because before the snapshot is taken application has to put itself in hot-backup mode so a consistent copy can be made. Most of the applications have this functionality available but you have to use some script or snapmanager so when application is prepared it can inform filer to take snapshot and once snapshot is taken filer can inform back the application to resume its normal activity.

Restrict snapmirror access by host and volume on NetApp

Recently one of my fellow NetApp admin friend asked me a very general question,

“How do you restrict your data to be copied through snapmirror?”

As like any other normal NetApp guy my answer was also same old vanilla type.

“Go to snapmirror.allow file and put the host name if your have set snapmirror.access to legacy or you can directly put hostname in host=host1,host2 format in snapmirror.access option.”

But he wanted more granular level of permission, so my another answer was,

“You can also use snapmirror.checkip.enable so any system reporting same hostname will not be able to access data.”

But even on that he wasn’t happy and was asking if there is any other way so he can restrict snapmirror access on volume basis. At this point I said “No, NetApp doesn’t provide this level of granular access.”

So the topic stopped there, but this question was there in my mind and always hunted me why there isn’t any such way.

Fast forward Past week when I had some extra time in my hand I started searching on net for this and fortunate enough I got a way on NOW site to get this work.

It was recorded under Bugs section with Bug ID # 80611 Which reads as.

“There is an unsupported undocumented feature of the /etc/snapmirror.allow file, such that if it is filled as follows:
    hostA:vol1
    hostA:vol29
    hostB:/vol/vol0/q42
    hostC
and "options snapmirror.access legacy" is issued, then the desired access policy will be implemented. Again note that this is unsupported and undocumented so use at your own risk.”

Yes, though NetApp says that there is a way to do that but they also say well sometimes it may break other functionality or may not work as expected.

Finding this I sent the details to my friend but unfortunately he don’t want to give it a try on his production systems and test systems are not available with him.

So if anyone of you want to try it or have tried it before please put your experience in comments field.

NetApp NFS agent for VCS - Part 3

In first post I wrote why I need this agent installed and what all are the features of this, and in last post I mentioned how to configure it on cluster node but that was incomplete because the post was going very big and I had to stop it, so here’s the remaining and very important part of that configuration.

How to configure different account name in NetApp NFS agent for VCS?

Hunting around in agent’s configuration guide from Veritas and NetApp didn’t reveal any result and even their KB search was not helpful. So I was left to choose my way and explore the stuff which I started with creating a new customized account on filer only for this purpose.

Here are the actual commands I used to create them starting from customized role to account.

‘useradmin role add exportfs -c "To manage NFS exports from CLI" -a cli-exportfs*,cli-lock*,cli-priv*,cli-sm_mon*’
‘useradmin group add cli-exportfs-group -r exportfs -c "Group to manage NFS exportfs from CLI"’
‘useradmin user add vcsagent -g cli-exportfs-group -c "To manage NFS exports from NetApp VCS Agent"’

And here’s the account after creation

testfiler1> useradmin user list vcsagent
Name: vcsagent
Info: To manage NFS exports from NetApp VCS Agent
Rid: 131090
Groups: cli-exportfs-group
Full Name:
Allowed Capabilities: cli-exportfs*,cli-lock*,cli-priv*,cli-sm_mon*
Password min/max age in days: 0/4294967295
Status: enabled

Now next thing was to give limited access to cluster node using vcsagent user and revoke its root access which was nothing more then removing dsa keys from /etc/sshd/root/.ssh/authorized_keys file and adding in /etc/sshd/vcsagent/.ssh/authorized_keys file.

After completing that I headed back to host and created a new file named config in .ssh directory of root with below content

Host testfiler1
User vcsagent
port 22
hostName testfiler1.lab.com

As a test I issued command “ssh testfiler1 version” on node terminal and I got access denied error which was perfectly fine because now when I do ‘ssh testfiler1’ system looks into config file in .ssh directory and uses vcsagent user which is not having access to run version command. Everything was looking good so I started running tests by moving resource from one node to another but to my surprise they were failing to make changes on filer and looking at filer audit logs it shown that they are still using root for ssh to filer.

Till the moment I didn’t run test I was thinking that agent is just relying to OS for ssh username as NetApp hasn’t set any username attribute in agent moreover as I haven’t configured in OS which account to use that’s why when agent executes command ‘ssh testfiler1 ’ OS directs the ssh connection to connect with root (cluster node’s local logged-in user).

But after going through my failed test it made me to believe that username is hardcoded in agent script so I started looking in script and soon found below line in file NetApp_VCS.pm

‘$cmd = "$main::ssh -n root\@$host '$remote_cmd'";’

After having this finding it was not a big brainer work to figure out what was going wrong and what I have to do. Just removed the word ‘root’ from script and it started working because now it is using config file from .ssh directory and uses vcsagent as username, alternatively I could have replaced word root with vcsagent directly in script also to make it simple and stay away from maintaining config file but I felt this to be much better.

Unfortunately till today there is no alternative apart from making changes in script as NetApp and Veritas both were not able to help us apart from a statement “we will raise a product enhancement request”.

Update: You need to give access "security-priv-advanced" also to user, so role should look like below.

testfiler01> useradmin role list exportfs

Name: exportfs

Info: To manage NFS exports from CLI

Allowed Capabilities: cli-exportfs*,cli-lock*,cli-priv*,cli-sm_mon*,security-priv-advanced