Child pages
  • Netapp Filer How-to create a custom sysstat output
Skip to end of metadata
Go to start of metadata

Netapp Filer How-to create a custom sysstat output

 

I recently noticed (while playing around with PAM II / Flash-Cache modules) that it was possible to create custom sysstat-like output. Every 'stats' item can be accessed either by shell (e.g. CPU busy, any other counter possible):

filer> stats show -i 1 -n 10 system:system:cpu_busy

 

or by a xml-template which is then called via the 'stats' command. There are some sample reports installed by default on every filer in the folder /filer/etc/stats/preset. Just drop your self created xml-files into this folder and call them on the commandline  without the file suffix: 'stats show -p <template>'

E.g. I created an overview of CPU, protocol and PAM usage/latency which looks like this (updates every 1 second). From left to right: All CPUs, NFS, CIFS, iSCSI and FCP Ops and latencies, PAM latencies and hitrate:

toaster> stats show -p overview
  CPU Stats       NFS Stats                              CIFS Stats          iSCSI Stats           FCP Stats         PAM Stats
  C0  C1  C2  C3  NFS Ops  NFS Read  NFS Write  NFS Avg  CIFS Ops  CIFS Lat  iSCSI Ops  iSCSI Lat  FCP Ops  FCP Lat  PAM/r  PAM/w  PAM/Hit  PAM/Hit  r/repl
   %   %   %   %       /s        ms         ms       us        /s        ms         /s         ms       /s       ms     ms     ms       /s        %      /s
  24  26  26  28     2850      1.52       0.50   374.80       674      0.37        144       0.50        0        0   0.28   2.00      330       22     114
  20  20  26  19     3069      1.39       0.22   134.47       747      0.27         83       0.24        0        0   0.30   2.00       87        4      50
  18  15  22  12     3029      3.76       0.20   213.49       544      0.09         31       0.23        0        0   0.28      0       99        8      57
  24  23  28  19     4638      2.96       0.22   162.60      1118      0.13        110       0.44        0        0   0.22   1.00      215       12     114
  23  22  24  17     2918      1.81       0.20   184.38      1489      0.10        375       1.75        0        0   0.17      0      403       53     334
  22  24  21  17     2115      1.31       0.20   148.60      1824      0.17        232       2.38        0        0   0.23      0      290       28     201
  ... 

Basically one can create any kind of custom report; every system property that can be accessed via the 'stats' command can be used. One can monitor lun or volume i/o or latencies, network throughput, cache statistics in realtime in all its glory. This also makes for a nice semi-transparent desktop background (wink). To create your own reports just take a look at the reference, it's easy to figure out.

Example volume r/w latency

Let's say we want to monitor our virtual environment which is contained in /vol/virtual. First we need to know the object, instance and counter's names.

~# ssh root@toaster stats show | grep virtual
... 
volume:virtual:instance_name:virtual
volume:virtual:node_name:
volume:virtual:avg_latency:155.40us
volume:virtual:total_ops:2211/s
volume:virtual:read_data:2013296b/s
volume:virtual:read_latency:320.60us
volume:virtual:read_ops:273/s
volume:virtual:write_data:17618475b/s
volume:virtual:write_latency:132.13us
volume:virtual:write_ops:1938/s
volume:virtual:other_latency:0us
volume:virtual:other_ops:0/s
... 

 

With this at hand it is easy to construct a xml file.

<?xml VERSION = "1.0" ?>
<!-- Display in column format basic latency performance information
-->
<preset orientation="column" interval="1" 
        print_instance_names="off"
        catenate_instances="true"
        pre_header="vol_virtual r/w latencies">

        <object name="volume">
            <instance name="virtual">
                <counter name="read_latency">
                        <title>read lat.</title>
                        <width>8</width>
                </counter>
                <counter name="write_latency">
                        <title>write lat.</title>
                        <width>8</width>
                </counter>
            </instance>
         </object>
 </preset>

 

Done! Now save it as /filer/etc/stats/preset/virtual.xml and try it out on your toasters cli:

toaster> stats show -p virtual

 

Reference

overview.xml

overview.xml
<?xml VERSION = "1.0" ?>
<!-- Display in column format basic latency performance information
-->
<preset orientation="column" interval="1" 
        print_instance_names="off"
        catenate_instances="true"
        pre_header="  CPU Stats       NFS Stats                              CIFS Stats          iSCSI Stats           FCP Stats         PAM Stats">

        <object name="processor">
            <instance name="processor0">
                <counter name="processor_busy">
                        <title>C0</title>
                        <width>3</width>
                </counter>
            </instance>
            <instance name="processor1">
                <counter name="processor_busy">
                        <title>C1</title>
                        <width>3</width>
                </counter>
            </instance>
            <instance name="processor2">
                <counter name="processor_busy">
                        <title>C2</title>
                        <width>3</width>
                </counter>
            </instance>
            <instance name="processor3">
                <counter name="processor_busy">
                        <title>C3</title>
                        <width>3</width>
                </counter>
            </instance>
        </object>

        <object name="nfsv3">
            <instance name="nfs">
                <counter name="nfsv3_ops">
                        <title>NFS Ops</title>
                        <width>8</width>
                </counter>

                <counter name="nfsv3_read_latency">
                        <title>NFS Read</title>
                        <width>9</width>
                </counter>
                <counter name="nfsv3_write_latency">
                        <title>NFS Write</title>
                        <width>10</width>
                </counter>
                <counter name="nfsv3_avg_op_latency">
                        <title>NFS Avg</title>
                        <width>8</width>
                </counter>
            </instance>
        </object>

        <object name="cifs">
            <instance name="cifs">
                <counter name="cifs_ops">
                        <title>CIFS Ops</title>
                        <width>9</width>
                </counter>
                <counter name="cifs_latency">
                        <title>CIFS Lat</title>
                        <width>9</width>
                </counter>
            </instance>
        </object>
        <object name="iscsi">
            <instance name="iscsi">
                <counter name="iscsi_ops">
                        <title>iSCSI Ops</title>
                        <width>10</width>
                </counter>
                <counter name="iscsi_latency">
                        <title>iSCSI Lat</title>
                        <width>10</width>
                </counter>
            </instance>
        </object>
        <object name="fcp">
            <instance name="fcp">
                <counter name="fcp_ops">
                        <title>FCP Ops</title>
                        <width>8</width>
                </counter>

                <counter name="fcp_latency">
                        <title>FCP Lat</title>
                        <width>8</width>
                </counter>
            </instance>
        </object>
        <object name="ext_cache_obj">
            <instance name="ec0">
                <counter name="readio_avg_latency">
                        <title>PAM/r</title>
                        <width>6</width>
                </counter>
                <counter name="writeio_avg_latency">
                        <title>PAM/w</title>
                        <width>6</width>
                </counter>
                <counter name="hit">
                        <title>PAM/Hit</title>
                        <width>8</width>
                </counter>
                <counter name="hit_percent">
                        <title>PAM/Hit</title>
                        <width>8</width>
                </counter>
                <counter name="disk_reads_replaced">
                        <title>r/repl</title>
                        <width>7</width>
                </counter>
            </instance>
        </object>
</preset>

man (1) na_stats

If I was ever to read man pages I probably would have noticed this earlier, it's all in there right at the end of the file. Oh my, there you go.

 

This page has been viewed

times.

Additions, comments, critisicm? mail to: b2c[at]dest-unreachable.net