Monthly Archives: July 2015

Translating a NFS file handle to inode the long-ish way

There might be a better tool or easier way to do this, but the method works for me 🙂

I was looking into a large number of getfattr requests on one of our CentOS 6 NFS Servers, and was curious what files all the requests were for and where they were coming from. The where part just requires the always helpful tcpdump…


# tcpdump -i bond0 'tcp port nfs'
....
08:58:18.971667 IP xxxxxx.3434118295 > xxxxx.sdsc.edu.nfs: 128 getattr fh Unknown/0100060188D45B4900CC684F00000000000000000...

This alone gets us most of the information we are looking for with the exception of which file was being acted on. It does give us the NFS file handle though, and that is easy enough to translate. Using the chart on page 5 of the following pdf, we see the file handle has the format…

Length Bytes Field Name Meaning Typical Values
1 1 fb_version NFS version Always 1
1 2 fb_auth_type Authentication method Always 0
1 3 fb_fsid_type File system ID encoding method Always 0
1 4 fb_fileid_type File ID encoding method Always either 0,
1, or 2
4 5-8 xdev Major/Minor number of exported device Major
number 3 (IDE), 8 (SCSI)
4 9-12 xino Export inode number Almost always 2
4 13-16 ino Inode number 2 for /, 19 for
/home/foo
4 17-20 gen_no Generation number 0xFF16DDF1, 0x3F6AE3C0
4 21-24 par_ino_no Parent’s inode number 2 for /, 19
for /home
8 25-32 Padding for NFSv2 Always 0
32 33-64 Unused by Linux

 

…so now we just need to split up our file handle to get the inode.

0 fb_version
1 fb_auth_type
0 fb_fsid_type
0 fb_fileid_type
0601 xdev
88D4 xino
5B49 ino (what we want)
00CC gen_no
684F par_ino_no
00000000 Padding
000000000… ??

 

Then we just need to convert from hex to decimal and pass it to find…


$ echo $((0x5b49))
23369

find /path/to/export -inum 23369

And we get to take a bit of a shortcut here as this system only had a single export, so no need to figure out what filesystem it was coming from.

nfsd io bytes read/written counter lower then they were two seconds ago??

While looking at a busy CentOS 6 NFS server I decided to to write a quick script to generate a diff of the NFS v3/v4 statistics every five seconds to see what calls were being made. Easy enough to do, just read the values from the lines starting with proc(2|3|4|4ops) from /proc/net/rpc/nfsd and you are good to go. Several site provide you the mapping, but they are given in they should be in the same order as the output of ‘nfsstat -s’ if you do not want to look around.

I decided to include a diff of the output from the line ‘io … …’ as well as that will give the bytes read and written. Easy enough as well, until you start seeing negative values. This a decent system that when running full speed can do better than 10Gb/s of streaming data, but the bytes counters are down in the GBs.

Searched around a bit and all the sites confirmed that this is a just a counter that does not do any sort of ‘this is the amount changed since the last time you asked’. Decided to check the source code and caught it.


struct nfsd_stats {
...
unsigned int fh_nocache_nondir; /* filehandle not found in dcache */
unsigned int io_read; /* bytes returned to read requests */
unsigned int io_write; /* bytes passed in write requests */
unsigned int th_cnt; /* number of available threads */
...

};

Looks like it probably is a 32 bit unsigned int which would max out at 4GB. Which would explain everything. So not an accurate counter, but as long as you are not doing more than 4GB of io during your interval you can still find the diff…


if new_value > old_value
return new_value - old_value
else
return 4294967296 - old_value + new_value