Solaris Analytics Collections, partitions, slices and operators

Having poked around with Solaris Analytics, and the WebUI a little, you might have wondered what information or statistics we gather by default. For instance, what are the statistics which we collect to make the Solaris Dashboard sheet useful?


The feature we provide to make this happen is the Collection. Collections give us a handy shorthand for gathering statistics. We ship with several collections:

# sstore list //:class.collection//*

Listing collections is a privileged operation; if I run the command above as myself then I get a very different result:

$ sstore list //:class.collection//*
Warning (//:class.collection//*) - lookup error: no matching collections found

The collection which is enabled by default is //:class.collection//, and you can see what it gathers by running sstore info on it:

# sstore info //:class.collection//
Identifier: //:class.collection//
  ssid: //:class.system//:*
 state: enabled
  uuid: 7a002985-2cf4-4965-adc9-b53116d8ae67
 owner: root
 cname: system
crtime: 1523243338963817

I quite like having the solaris-dashboard and apache-stats collections enabled, and that is really easy to do:

# sstoreadm enable-collection \
    //:class.collection// \

One thing I'm always concerned with, since our family media server is, shall we say, homebrew, is whether my disks are doing ok. Fortunately for me, it is very easy to cons up my own collection and stash it in /usr/lib/sstore/metadata/collections:

        "$schema": "//:collection",
        "description": "disk-related statistics",
        "enabled": true,
        "id": "disk-stats",
        "ssids": [
        "user": "root"

and once you've restarted sstored you can see it like so:

# sstore info -a //:class.collection//
Identifier: //:class.collection//
   ssid: //:class.disk//:res*//:*
  state: enabled
   uuid: bee6c5c5-487e-4376-9d91-f4eb933fd64e
  owner: root
  cname: disk-stats
 crtime: 1525373259871426

[Note that you do need to ensure that your collection validates against the collections schema, so run soljsonvalidate /path/to/my/collection.json, and if you need to reformat it, soljsonfmt /path/to/my/collection.json].

So that's useful - now what? How about looking at the illegal requests counter? When you run iostat -En that information is jumbled up with all the other errors and can be a little difficult to distinguish:

$ iostat -En sd0
c2t0d0           Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
Vendor: ATA      Product: WDC WD30EFRX-68E Revision: 0A82 Serial No: WD-WCC4N7CNYH0S
Size: 3000.59GB <3000592982016 bytes>
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0
Illegal Request: 537 Predictive Failure Analysis: 0 Non-Aligned Writes: 0

With Solaris Analytics, however, we can gather all of those errors together in aggregate and partition them at the same time. This command shows us the most recent data point (the -p -1 argument):

$ sstore export -p -1 "//:class.disk//"
2018-05-08T19:07:44  //:class.disk//
                    device-not-ready: 0.0
                    hard-errors: 0.0
                    illegal-requests: 537.0
                    media-errors: 0.0
                    no-device: 0.0
                    non-aligned-writes: 0.0
                    predictive-failure-analysis: 0.0
                    recoverable: 0.0
                    soft-errors: 0.0
                    transport-errors: 0.0

That's a bit more useful! (Yes, having to use sdN rather than cXtYdZ is a pain, sorry). So... how about just looking for the illegal-requests? That's where we really make use of the partition concept - and let's throw the argument to give a daily total from the start of this month (May 2018):

$ sstore export -t 2018-05-01T00:00:00 -i 86400  "//:class.disk//"
2018-05-01T00:00:00  //:class.disk//
                    illegal-requests: 27.0
2018-05-02T00:00:00  //:class.disk//
                    illegal-requests: 27.0
2018-05-03T00:00:00  //:class.disk//
                    illegal-requests: 27.0
2018-05-04T00:00:00  //:class.disk//
                    illegal-requests: 79.0
2018-05-05T00:00:00  //:class.disk//
                    illegal-requests: 174.0
2018-05-06T00:00:00  //:class.disk//
                    illegal-requests: 270.0
2018-05-07T00:00:00  //:class.disk//
                    illegal-requests: 365.02
2018-05-08T00:00:00  //:class.disk//
                    illegal-requests: 461.0

Much more useful - and observe that because we're using () to extract the partition element, we need to quote the argument so the shell doesn't get snippy with us.

To finish this post, let's take a look at two more really useful features, slices and operators. One operator that I'm particularly happy with is //:op.changed, which shows you when a statistic value changed. While not particularly useful for volatile statistics on a per-second basis (watch //:class.system//:stat.virtual-memory for a few minutes and you'll see what I mean) if you aggregate such stats over a longer time period, such as a day, you can get a better understanding what that stat is doing. So, with disk errors again, but on a daily basis (-i 86400) from the start of this month (-t 2018-05-01T00:00:00):

$ sstore export -t 2018-05-01T00:00:00 -i 86400  "//:class.disk//"
2018-05-01T00:00:00 27.0 //:class.disk//
2018-05-04T00:00:00 79.0 //:class.disk//
2018-05-05T00:00:00 174.0 //:class.disk//
2018-05-06T00:00:00 270.0 //:class.disk//
2018-05-07T00:00:00 365.0 //:class.disk//
2018-05-08T00:00:00 461.0 //:class.disk//

Finally, slices. These are //:s.[....] and you enter the statistic names which you wish to extract inside the brackets - and once again I'm using the //:op.changed to constrain the output:

$ sstore export -i 86400  //:class.disk//[0,28]//:stat.errors//:op.changed //:class.disk//[0,28]//:stat.//:s.[vendor,serial-number]//:op.changed
1970-01-01T10:00:00 27.0 //:class.disk//
2018-05-04T10:00:00 119.0 //:class.disk//
2018-05-05T10:00:00 214.0 //:class.disk//
2018-05-06T10:00:00 309.0 //:class.disk//
2018-05-07T10:00:00 405.0 //:class.disk//
2018-05-08T10:00:00 500.0 //:class.disk//
1970-01-01T10:00:00 169.0 //:class.disk//
2018-05-04T10:00:00 120.0 //:class.disk//
2018-05-05T10:00:00 215.0 //:class.disk//
2018-05-06T10:00:00 310.0 //:class.disk//
2018-05-07T10:00:00 406.0 //:class.disk//
2018-05-08T10:00:00 501.0 //:class.disk//
1970-01-01T10:00:00 ATA      //:class.disk//
1970-01-01T10:00:00 Z1D5K89L //:class.disk//
2018-05-04T10:00:00 WD-WCC4N7CNYH0S //:class.disk//
1970-01-01T10:00:00 ATA      //:class.disk//
1970-01-01T10:00:00  //:class.disk//

For more information about operators, slices and partitions, have a read of ssid-op (aka ssid-op(7)).

Tune in next time when I'll guide you through the process of using a proper certificate for your WebUI instance, rather than the default self-signed certificate.