The following is a brief overview of common performance monitoring tools for Solaris, including built-in tools, freeware add-ons, and commercial applications.

Souce: http://www.sunperf.com/perfmontools.html

Built-in Tools

vmstat

The vm in vmstat stands for Virtual Memory, of course. So let us begin our discussion by going over virtual memory on Solaris. There are several important kernel parameters: lotsfree, freemem, desfree, minfree, fastscan, slowscan, nscan, desscan, pagesio, physmem, handspreadpages, maxpgio, cachefree. The major portion of the behavior of the VM subsystem is described by the following graph. There is a “linear„ relationship between the scan rate and free memory.

Many FAQ’s, documents and article have answered the “Where did my memory go?„ question. Here it is again. Solaris does not continuously run the paging daemon. It only runs when there is a “need„ i.e. when the freemem drops below lotsfree. When memory falls below desfree, Solaris scans for pages, which haven’t been used for a while, 100 times a second. Free memory is wasted memory. Using every bit of memory to cache data makes sense when you read the same data over and over again. This is especially true for binary executables.

Imagine a clock, with two hands. The first hand clears a flag on a page of memory. If that page of memory is accessed, it gets set again. The second hand checks the flag to see if it is set. If it is set, the page is “in use„ or has been used “recently„. If it is not set the page has not been used recently, and will be scheduled to be paged out of memory. The parameter handspreadpages determines how many pages of memory are between the first and second hand. A minor fault occurs when a page is marked for being paged to disk, but gets accessed before the VM subsystem actually pages it. A major fault or reclaim is when the memory page is paged to disk and is accessed again and has to be paged into memory. Minor faults are of no real consequence since the data is still in memory and really amounts to changing a flag for that page of memory. If minor faults are large, it is due to handspreadpages being set too low. Reclaims are a sign of a memory shortage, since memory is being paged out only to be brought back in again.

Here’s the burning question, “what indicates a shortage of what?”

root:sparc > vmstat 5
 procs     memory            page            disk          faults      cpu
 r b w   swap  free  re  mf pi po fr de sr cd cd cd f0   in   sy   cs us sy id
 0 0 0 329064 27332   3  64  2  0  1  0  0  0  0  1  0   16 1517 1259  2  4 94
 0 0 0 325276  2280  45 526  0  6  6  0  0  0  3  4  0   63 23601 21485 20 47 33
 0 0 0 325276  2276  46 517  1  4  4  0  0  0  0  2  0   43 23753 21134 25 48 27
 0 0 0 325276  2276  47 528  0 13 13  0  0  0  0  4  0   59 24152 21759 20 42 38
 0 0 0 325276  2288  47 539  0  8  8  0  0  0  0  3  0   51 24155 21547 19 41 40
 0 0 0 325276  2284  46 527  0  5  5  0  0  0  0  2  0   48 24232 22012 18 45 37
 0 0 0 325276  2280  46 481  0  8  8  0  0  0  0  3  0   53 24264 22292 21 45 34
 0 0 0 325276  2300  47 527  0  8  8  0  0  0  0  3  0   52 23821 21239 22 46 33
 0 0 0 325300  2332  45 526  0  4  4  0  0  0  2  4  0   59 23398 20848 22 42 36

The more important columns are procs: r,b,w, page: re, pi, po, de, sr, cpu: us, sy, id. The r,b,w under proc stands for runnable, blocked and runnable but swapped out. Runnable means the process is in the queue waiting for a CPU to free up and has work to do (is not sleeping). Blocked is for processes waiting for a resource that is busy (see biowait(9f)). Essentially a driver has called biowait and this process is suspended. Having processes in the w column is a bad sign. Having entire processes swapped out even though they are runnable is a sign of severe memory shortage. Unfortunately, if a process does get swapped out and then goes to sleep, it will remain in the w column until it receives a signal to wake it up. Under the page heading the columns re, pi, po, de, and sr stand for reclaims, minor faults, pages paged in, pages paged out, estimated memory shortfall, and scan rate, respectively. Large reclaims and minor faults are not definitive by theirself. They may indicate that the VM subsystem is tuned incorrectly. Pages in and out can also be misleading since Solaris will treat reading an executable from disk as a page in. Also if an application uses mmap to read a file more quickly, then those reads will be counted as page ins. The de column is the kernel’s guess at how much memory needs to be freed. The most important column is sr. If the scan rate is high you have sever memory shortage. Unfortunately most references say 320 is high. I disagree. I have seen boxes go above 10000 on scan rate. This, just like everything else, is proportional to how big your system is. In the past (and present) the VM subsystem has been clamped way too tight. The system would prevent itself form paging and lock up since fastscan was set to a maximum of 8192. You could not scan for pages to page out faster. Hence the box ran out free memory and could do no new work.

mpstat

root:sparc > mpstat 1
CPU minf mjf xcal  intr ithr  csw icsw migr smtx  srw syscl  usr sys  wt idl
  0   28   0    1   112    2  436    1    0  110    0   540    1   3   0  95
  1   24   0    2     3    0  390    1    0  137    0   492    1   2   0  96
CPU minf mjf xcal  intr ithr  csw icsw migr smtx  srw syscl  usr sys  wt idl
  0  248   0    3   144   10 4790   15   10 5813    0 11328   23  65   0  12
  1  290   1   22    10    3 15438    5    7 1505    0 11132   17  25   1  57
CPU minf mjf xcal  intr ithr  csw icsw migr smtx  srw syscl  usr sys  wt idl
  0  303   0    3   142    9 12257   14    8 2641    0 12321   26  60   0  14
  1  231   0   21    23    3 8919   16   10 4151    0 11199   41  18   0  41
CPU minf mjf xcal  intr ithr  csw icsw migr smtx  srw syscl  usr sys  wt idl
  0  392   0    4   139    6 7497   16   12 2503    0 12293   22  67   0  11
  1  369   0   23    14    3 10837    8   19 1810    0  9658   22  25   0  53
CPU minf mjf xcal  intr ithr  csw icsw migr smtx  srw syscl  usr sys  wt idl
  0  365   0    4   133    6 14145   10   12 1706    0 12270   20  63   0  17
  1  169   0   20    19    3 6716   10    9 4029    0 10983   36  30   0  34
CPU minf mjf xcal  intr ithr  csw icsw migr smtx  srw syscl  usr sys  wt idl
  0  187   0    2   132   11 3070   11   11 4338    0 11461   31  57   0  12
  1  346   1    9    19    3 17596   10   11  910    0 11392   11  29   3  57
CPU minf mjf xcal  intr ithr  csw icsw migr smtx  srw syscl  usr sys  wt idl
  0  343   0   35   194   44 1834   44   12 4344    0 10547   31  59   0  10
  1  386   0   24    63    4 16694  125   12  600    0 10397   13  27   0  60
CPU minf mjf xcal  intr ithr  csw icsw migr smtx  srw syscl  usr sys  wt idl
  0  345   0    3   119    6 17230    9   12  713    0 11701   13  66   0  21
  1  210   0    6    22    3 3121   13   12 5833    0 10846   43  37   0  20

mpstat is one of my favorites. It has everything for looking at your processors. mp stands for Multi-Processor. Hence, it gives statistics on each processor. The columns are as follows: CPU id, minor faults, major faults, cross calls, interrupts, thread interrupts, context switches, involuntary context switches, migrations, spin on mutex lock, spin on reader-writer mutex lock, system calls, user cpu time, system cpu time, wait cpu time, and idle cpu time. Some of these are the same as with vmstat. I will look at the new ones which are important in my opinion. A cross happens when an interrupt arrives on one CPU, but is intended for a system call executing on a different CPU. An involuntary context switch happens when a process has work to do, but its time slice has elapsed. A large percentage of context switches being involuntary is an indicator of being CPU bound. A migration occurs when process is switched from one CPU to a different CPU after a context switch. Typically a process will stay on a particular CPU, but will occasionally switch CPU’s. When this happens the cache has to be synced from the original CPU to the new CPU. This can lead to inefficiency with a large number of processors. A spin on a mutex happens when a mutex lock cannot be obtained on the first or subsequent trys. A large number of spins means processes are trying to use the same resource which is becoming a bottleneck. The wt column is the percentage of time slices the CPU spends running biowait. See the discussion for blocked processes for vmstat above.

iostat

root:sparc > iostat -nxtc
      tty         cpu
 tin tout us sy wt id
   0   13  1  0  0 98
                              extended device statistics
  r/s  w/s   kr/s   kw/s wait actv wsvc_t asvc_t  %w  %b device
  0.2  0.1    1.8    1.2  0.0  0.0   22.4    9.6   0   0 c0t0d0
  0.0  0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0 fd0
  0.0  0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0 c0t2d0
  0.0  0.0    0.0    0.0  0.0  0.0    0.0    0.0   0   0 sparc:vold(pid205)

iostat is my favorite way of looking at disk activity. The column that I pay most attention to is the wsvc_t column. This is the average service time in milliseconds. A high number is a sign that the disk is becoming a bottleneck. A rule of thumb is >35 is cause for investigation. Large numbers in the r/s and w/s is an indication of a too small block size. This could also be a poorly tuned application that is making many small reads/writes instead of a few large reads/writes. The kr/s and kr/s give you a good indication of how much bandwidth you are using. For a single Ultra Wide Differential SCSI disk I would expect to get 10MB/s in throughput. For a correctly configured stripe, I would expect to get 10MB/s x number of disks in the stripe. On a read from RAID 5 you should get similar performance. On write the cache will help and you should get close to the same performance while the cache is not being over run.

netstat

root:sparc > netstat -I iprb0 5
    input   iprb0     output           input  (Total)    output
packets errs  packets errs  colls  packets errs  packets errs  colls
41860   0     66310   0     0      42371   0     66821   0     0
8       0     7       0     0      8       0     7       0     0
7       0     7       0     0      7       0     7       0     0
10      0     10      0     0      10      0     10      0     0
8       0     8       0     0      8       0     8       0     0
7       0     7       0     0      7       0     7       0     0
8       0     8       0     0      8       0     8       0     0
8       0     8       0     0      8       0     8       0     0
7       0     8       0     0      7       0     8       0     0
8       0     7       0     0      8       0     7       0     0

netstat is a good for getting a general feel for network traffic. Unlike iostat it does not give you throughput numbers. This is tragic since it is hard to judge whether an interface is pegged by the number of packets. Each packet can vary in size greatly. If you have Sun Trunking software you can use nettr to get this information.

For example

nettr -stats 0 device=qfe interval=1

An undocumented option for netstat is the -k option. It dumps a huge amount of kernel parameters and statistics. This is extremely helpful in diagnosing problems. Unfortunately it is undocumented so you will just have to search the header files for info. maybe Sun will release this info but it seems unlikely.

root:sparc > netstat -k | grep nocanput
retry_error 0 first_collisions 0 nocarrier 0 inits 7 nocanput 0

One key indicator that I have run into is nocanput being non-zero. If this is the case your streams queue is too small. It should be set to 400 per GB of memory. Put a similar line in your /etc/system file. This assumes you have 4GB RAM.

set sq_max_size=1600

I use netstat as a diagnostic tool more than a performance tool. netstat -an | grep ESTA | wc -l will tell you how many connections you have. This would be very important to a web server administrator for example. netstat -an | grep LISTEN is a good way to check for software that should be turned off for security reasons. netstat -nr is a good way to check for routing information.

nfsstat

NFS tuning is going to require a fairly lengthy discussion, so I will hold off for now. For now the best tool is nfsstat which gives a copious amount of information.

root:sparc > nfsstat

Server rpc:
Connection oriented:
calls       badcalls    nullrecv    badlen      xdrcall     dupchecks
36          0           0           0           0           3
dupreqs
0
Connectionless:
calls       badcalls    nullrecv    badlen      xdrcall     dupchecks
0           0           0           0           0           0
dupreqs
0

Server nfs:
calls       badcalls
36          0
Version 2: (0 calls)
null        getattr     setattr     root        lookup      readlink
0 0%        0 0%        0 0%        0 0%        0 0%        0 0%
read        wrcache     write       create      remove      rename
0 0%        0 0%        0 0%        0 0%        0 0%        0 0%
link        symlink     mkdir       rmdir       readdir     statfs
0 0%        0 0%        0 0%        0 0%        0 0%        0 0%
Version 3: (36 calls)
null        getattr     setattr     lookup      access      readlink
1 2%        14 38%      0 0%        0 0%        14 38%      0 0%
read        write       create      mkdir       symlink     mknod
2 5%        0 0%        0 0%        0 0%        0 0%        0 0%
remove      rmdir       rename      link        readdir     readdirplus
0 0%        0 0%        0 0%        0 0%        0 0%        3 8%
fsstat      fsinfo      pathconf    commit
0 0%        1 2%        1 2%        0 0%

Server nfs_acl:
Version 2: (0 calls)
null        getacl      setacl      getattr     access
0 0%        0 0%        0 0%        0 0%        0 0%
Version 3: (0 calls)
null        getacl      setacl
0 0%        0 0%        0 0%

Client rpc:
Connection oriented:
calls       badcalls    badxids     timeouts    newcreds    badverfs
414         0           0           0           0           0
timers      cantconn    nomem       interrupts
0           0           0           0
Connectionless:
calls       badcalls    retrans     badxids     timeouts    newcreds
15          1           0           0           0           0
badverfs    timers      nomem       cantsend
0           12          0           0

Client nfs:
calls       badcalls    clgets      cltoomany
15          1           15          0
Version 2: (14 calls)
null        getattr     setattr     root        lookup      readlink
0 0%        13 92%      0 0%        0 0%        0 0%        0 0%
read        wrcache     write       create      remove      rename
0 0%        0 0%        0 0%        0 0%        0 0%        0 0%
link        symlink     mkdir       rmdir       readdir     statfs
0 0%        0 0%        0 0%        0 0%        0 0%        1 7%
Version 3: (0 calls)
null        getattr     setattr     lookup      access      readlink
0 0%        0 0%        0 0%        0 0%        0 0%        0 0%
read        write       create      mkdir       symlink     mknod
0 0%        0 0%        0 0%        0 0%        0 0%        0 0%
remove      rmdir       rename      link        readdir     readdirplus
0 0%        0 0%        0 0%        0 0%        0 0%        0 0%
fsstat      fsinfo      pathconf    commit
0 0%        0 0%        0 0%        0 0%

Client nfs_acl:
Version 2: (1 calls)
null        getacl      setacl      getattr     access
0 0%        0 0%        0 0%        1 100%      0 0%
Version 3: (0 calls)
null        getacl      setacl
0 0%        0 0%        0 0%

Free Tools

Xkstat

This is a X Windows utility to monitor various subsystems on Solaris servers/workstations. The following can be monitored: virtual memory, network interfaces, disk io, and CPU utilization. It is based on the user accessible interface to the kstat driver. This is unique to Solaris, as far as I know. The kstat driver provides statistics which are kept on every subsystem. This is not by any means an exhaustive tool, there are too many variables to present at once. If one wanted a complete dump of all statistics provided, it would be extremely difficult (probably humanly impossible) to sift through all the data real time. For an example, issue the following command to get a snap shot: netstat -k. This is an absolute gold mine of information on the state of the system.

The syntax for running Xkstat is:

Xkstat -{net | vm | disk | cpu} [sleeptime]
Xkstat -other ks_module ks_instance ks_name name [sleeptime]
Xkstat -otherc ks_module ks_instance ks_name name [sleeptime]
Xkstat -dump file_name

The options are for network, virtual memory, disk IO and CPU utilization, respectively. The optional sleep time can be specified as a decimal number. Hence the intervals can be smaller than one second, except for CPU. The default is 1 second. CPU information is only updated every second, hence it doesn’t make sense to query more often. Beware that the smaller the sleep time the more of a load the tool will put on the system. What load you can withstand will depend on how many CPU’s you have. Most of the load will fall under /usr/openwin/bin/Xsun (Solaris’s Window Manager for CDE) which should be running on a workstation rather than the server.

$ /usr/ucb/ps -aux | head -1
USER       PID %CPU %MEM   SZ  RSS TT       S    START  TIME COMMAND
$ /usr/ucb/ps -aux | grep '[X]kstat'
n         8432  1.0  0.9 2492  996 pts/5    S 11:43:17  0:00 ./Xkstat -vm
n         8434  0.4  0.9 2492 1004 pts/5    S 11:43:17  0:00 ./Xkstat -disk
n         8428  0.3  0.9 2492 1004 pts/5    S 11:43:17  0:00 ./Xkstat -net
n         8430  0.1  0.9 2492  992 pts/5    S 11:43:17  0:00 ./Xkstat -cpu
total           1.8 % CPU Pentium 200 MHz
$

One of the main design goals is for this tool to be “light” on CPU/memory usage. The last thing a performance tuning tool should do is degrade performance. For that reason I have coded it entirely using XLib only. It could be fancier with a lot of Motif (or some other toolkit, GTK, FLTK) buttons and widgets. However, I guarantee it would use more than 0.1-1.0 % CPU and the footprint would be much bigger.

Along the caption bar is the host name and architecture of the box it is running on.

At the top of each column is the name of the interface for which the graphs apply. The arrows at the bottom allow one to scroll left or right if several interfaces exist. The network parameters monitored are opackets, ipackets, oerrors, ierror, collisions, obytes, rbytes, brdcstxmt, brdcstrcv, unknowns, oflo, uflo. These are the names of the variables in the kstat structures. The best way to find information on what these mean is to look for the parameter in the header files under /usr/include or /usr/include/sys.

opackets - output packets.
ipackets - input packets.
oerrors - output errors.
ierror - input errors.
collisions - collisions.
obytes - bytes of data output.
rbytes - bytes of data read.
brdcstxmt - broadcast packets transmitted.
brdcstrcv - broadcast packets received.
unknowns - packets of unknown type.
oflo - over Flows.
uflo - under flows.

The virtual memory info is both static variables and dynamic variables. The static variables are presented on the left and the dynamic variables are presented in the graphs on the right. Again, the names are taken from the variable names in the kstat structure or vm structure. Information can be found for each variable by searching the header files, for example:

$ find /usr/include -name '*.h' -exec grep  physmem {} ;
        struct memlist *physinstalled;  /* amt of physmem installed */
        struct memlist *physavail;      /* amt of physmem avail for use */
extern pgcnt_t  physmem;        /* physical memory (pages) on this CPU */
void add_physmem(page_t *, size_t);
$ for X in kernelbase econtig lotsfree desfree minfree fastscan slowscan pp_kernel
> do
> find /usr/include -name '*.h' -exec grep $X {} ;
> done
 * with installed memory and the eprom variable 'eprom_kernelbase'.
 * which it saves in the variable _kernelbase.
 * _kernelbase with the correct value in i86pc/os/startup.c
 * i86 and i86pc files use kernelbase instead of KERNLEBASE, which is
extern uintptr_t kernelbase;
 * The eprom variable 'eprom-kernelbase' can be used to increase the amount of
extern const uintptr_t _kernelbase;
#define KERNELBASE      _kernelbase
extern pgcnt_t  lotsfree;       /* max free before clock freezes */
extern pgcnt_t swapfs_desfree;
extern pgcnt_t  desfree;        /* minimum free pages before swapping begins */
extern pgcnt_t swapfs_minfree;          /* amount of availrmem (in pages) */
 * tmpfs_minfree is the amount (in pages) of anonymous memory that tmpfs
 * tmpfs could allocate up to 16MB - tmpfs_minfree.  The default value
 * for tmpfs_minfree is btopr(TMPMINFREE) but it can cautiously patched
extern long     tmpfs_minfree;          /* Anonymous memory in pages */
        int     minfree;        /* %age to reserve for root */
        int32_t fs_minfree;             /* minimum percentage of free blocks */
        int     vfs_minfrags;           /* calc. from fs_minfree */
extern pgcnt_t  segspt_minfree; /* low water mark for availrmem in seg_spt */
extern pgcnt_t  minfree;        /* no of pages to try to keep free via daemon */
 *      MAX(ani_max - ani_resv, 0) + (availrmem - swapfs_minfree)
        (k_anoninfo.ani_max + MAX((spgcnt_t)(availrmem - swapfs_minfree), 0))
                        MAX((spgcnt_t)(availrmem - swapfs_minfree), 0))
extern pgcnt_t  fastscan;
extern pgcnt_t  slowscan;
$

The two most important parameters to watch are nscan and desscan. desscan will be a minimum of 25. nscan is the number of pages actually scanned. If these two variables are constantly high, it is time to pony up for some more memory. fastscan and slowscan are poorly tuned for systems with large amounts of memory on Solaris 2.6. If you have more than 256 MB of memory, they are clamped too low. fastscan is important in that this is the maximum number of pages that will be scanned per clock tick (100 clock ticks per second) after the available memory has fallen below minfree. If memory falls below throttlefree, the box will lock until it can free more memory. The scan rate is linear from slowscan to fastscan as memory decreases from lotsfree to minfree.

Unfortunately Solaris represents disks, CDROM’s and other drives internally as some type (usually by the name of the driver) and an instance number. This is not the same as the cXtXdXsX that administrators are used to seeing. All hope is not lost though. With some clever ingenuity one can put the pieces together. Let us take cmdk7 for an example. First we find its entry in the file /etc/path_to_inst. Then we examine the links in the /dev/dsk directory and look for the same device path. The information presented is for the entire disk not just the first slice. This will be different for each type of disk, i.e. ide, scsi, fibre channel, … etc.

$ cat /etc/path_to_inst | grep cmdk
"/pci@0,0/pci-ide@7,1/ata@0/cmdk@0,0" 0 "cmdk"
"/pci@0,0/pci-ide@7,1/ata@0/cmdk@1,0" 1 "cmdk"
"/pci@0,0/pci-ide@7,1/ata@1/cmdk@1,0" 8 "cmdk"
"/pci@0,0/pci-ide@7,1/ata@1/cmdk@0,0" 7 "cmdk"
$ ls -l /dev/dsk/c1d0*
lrwxrwxrwx   1 root     root          50 Aug  5  1999 /dev/dsk/c1d0p0 -> ../../devices/pci@0,0/pci-ide@7,1/ata@1/cmdk@0,0:q
lrwxrwxrwx   1 root     root          50 Aug  5  1999 /dev/dsk/c1d0p1 -> ../../devices/pci@0,0/pci-ide@7,1/ata@1/cmdk@0,0:r
lrwxrwxrwx   1 root     root          50 Aug  5  1999 /dev/dsk/c1d0p2 -> ../../devices/pci@0,0/pci-ide@7,1/ata@1/cmdk@0,0:s
lrwxrwxrwx   1 root     root          50 Aug  5  1999 /dev/dsk/c1d0p3 -> ../../devices/pci@0,0/pci-ide@7,1/ata@1/cmdk@0,0:t
lrwxrwxrwx   1 root     root          50 Aug  5  1999 /dev/dsk/c1d0p4 -> ../../devices/pci@0,0/pci-ide@7,1/ata@1/cmdk@0,0:u
lrwxrwxrwx   1 root     root          50 Aug  5  1999 /dev/dsk/c1d0s0 -> ../../devices/pci@0,0/pci-ide@7,1/ata@1/cmdk@0,0:a
lrwxrwxrwx   1 root     root          50 Aug  5  1999 /dev/dsk/c1d0s1 -> ../../devices/pci@0,0/pci-ide@7,1/ata@1/cmdk@0,0:b
lrwxrwxrwx   1 root     root          50 Aug  5  1999 /dev/dsk/c1d0s10 -> ../../devices/pci@0,0/pci-ide@7,1/ata@1/cmdk@0,0:k
lrwxrwxrwx   1 root     root          50 Aug  5  1999 /dev/dsk/c1d0s11 -> ../../devices/pci@0,0/pci-ide@7,1/ata@1/cmdk@0,0:l
lrwxrwxrwx   1 root     root          50 Aug  5  1999 /dev/dsk/c1d0s12 -> ../../devices/pci@0,0/pci-ide@7,1/ata@1/cmdk@0,0:m
lrwxrwxrwx   1 root     root          50 Aug  5  1999 /dev/dsk/c1d0s13 -> ../../devices/pci@0,0/pci-ide@7,1/ata@1/cmdk@0,0:n
lrwxrwxrwx   1 root     root          50 Aug  5  1999 /dev/dsk/c1d0s14 -> ../../devices/pci@0,0/pci-ide@7,1/ata@1/cmdk@0,0:o
lrwxrwxrwx   1 root     root          50 Aug  5  1999 /dev/dsk/c1d0s15 -> ../../devices/pci@0,0/pci-ide@7,1/ata@1/cmdk@0,0:p
lrwxrwxrwx   1 root     root          50 Aug  5  1999 /dev/dsk/c1d0s2 -> ../../devices/pci@0,0/pci-ide@7,1/ata@1/cmdk@0,0:c
lrwxrwxrwx   1 root     root          50 Aug  5  1999 /dev/dsk/c1d0s3 -> ../../devices/pci@0,0/pci-ide@7,1/ata@1/cmdk@0,0:d
lrwxrwxrwx   1 root     root          50 Aug  5  1999 /dev/dsk/c1d0s4 -> ../../devices/pci@0,0/pci-ide@7,1/ata@1/cmdk@0,0:e
lrwxrwxrwx   1 root     root          50 Aug  5  1999 /dev/dsk/c1d0s5 -> ../../devices/pci@0,0/pci-ide@7,1/ata@1/cmdk@0,0:f
lrwxrwxrwx   1 root     root          50 Aug  5  1999 /dev/dsk/c1d0s6 -> ../../devices/pci@0,0/pci-ide@7,1/ata@1/cmdk@0,0:g
lrwxrwxrwx   1 root     root          50 Aug  5  1999 /dev/dsk/c1d0s7 -> ../../devices/pci@0,0/pci-ide@7,1/ata@1/cmdk@0,0:h
lrwxrwxrwx   1 root     root          50 Aug  5  1999 /dev/dsk/c1d0s8 -> ../../devices/pci@0,0/pci-ide@7,1/ata@1/cmdk@0,0:i
lrwxrwxrwx   1 root     root          50 Aug  5  1999 /dev/dsk/c1d0s9 -> ../../devices/pci@0,0/pci-ide@7,1/ata@1/cmdk@0,0:j
$

On the left is a bar graph of the CPU usage at the moment. Solaris divides the usage up into idle, user, kernel and wait. Hence I have represented it that way here. It is maintained as the number of clock ticks that are spent doing one of the 4 categories of tasks. This is updated once per second. Hence if you specify a sleep time less than 1 second, it will increase it to 1. If you have more than 36 CPU’s, one overall graph will be displayed. This is to try to not present more information than is useful. On the right side is various virtual memory, thread and hardware parameters that are interesting to me. These can be found in /usr/include/sys/sysinfo.h.

Just in case I left out something you need to monitor, I put in two generic options. The option -other is for time dependent counters i.e. counters which are incremented over time. The option -otherc (the c is for “constant”) is for variables which are not counters i.e. the variable is a rate or kernel parameter. This works for KSTAT_TYPE_NAMED variables. To get a list of the available parameters, issue the following command:

Xkstat -dump file

If the parameter you want to monitor is listed under a module that is Data Type NAMED then you can monitor it. See the man page for kstat_lookup(3K). Example:

Xkstat -other eepro 0 eepro0 opackets64
Xkstat -otherc unix -1 system_pages nscan 5

As with all software this needs to be tested in your environment to ensure it operates as expected. No expressed or implied warranty is given. This software is as-is with no statement of merchantability or fitness for any purpose.

Souce: http://www.sunperf.com/perfmontools.html