Tushar Jadhav Blog: 2016-11-20

Install SysStat to enable system monitoring

[1] Install SysStat.

[root@tc ~]#

yum -y install sysstat

[root@tc~]#

/etc/rc.d/init.d/sysstat start

Calling the system activity data collector (sadc):

[root@tc ~]#

chkconfig sysstat on

[2] Logging is executed by cron's setting like follows.

* Logs are stored in /var/log/sa/sa** per 10 minutes with /usr/lib64/sa/sa1 command.

* The Statics of a day is generated to /var/log/sa/sar** with /usr/lib64/sa/sa2 command.

[root@dlp ~]#

cat /etc/cron.d/sysstat

# Run system activity accounting tool every 10 minutes

*/10 * * * * root /usr/lib64/sa/sa1 1 1

# 0 * * * * root /usr/lib64/sa/sa1 600 6 &

# Generate a daily summary of process accounting at 23:53

53 23 * * * root /usr/lib64/sa/sa2 -A

[3] If you'd like to change some settings of SysStat, the configuration file is located like follows.

[root@dlp ~]#

vi /etc/sysconfig/sysstat

# sysstat-9.0.4 configuration file.

# How long to keep log files (in days).

# If value is greater than 28, then log files are kept in

# multiple directories, one for each month.

HISTORY=28

# Compress (using gzip or bzip2) sa and sar files older than (in days):

COMPRESSAFTER=31

# Parameters for the system activity data collector (see sadc manual page)

# which are used for the generation of log files.

# * note

SADC_OPTIONS="-S DISK"

# Compression program to use.

ZIP="bzip2"

# *note : valid options

INT ? System Interrupts

DISK ? Block Devices

SNMP ? SNMP statistics

IPV6 ? IPv6 statistics

POWER ? Power Management statistics

ALL ? All of the above

XDISK ? DISK + Partition statistics

XALL ? All of the above (ALL + XDISK)

################################# END ################################

10 Useful Sar (Sysstat) Examples for UNIX / Linux Performance Monitoring

Using sar you can monitor performance of various Linux subsystems (CPU, Memory, I/O..) in real time.

Using sar, you can also collect all performance data on an on-going basis,

store them, and do historical analysis to identify bottlenecks.

Sar is part of the sysstat package.

This article explains how to install and configure sysstat package (which contains sar utility)

and explains how to monitor the following Linux performance statistics using sar.

Collective CPU usage
Individual CPU statistics
Memory used and available
Swap space used and available
Overall I/O activities of the system
Individual device I/O activities
Context switch statistics
Run queue and load average data
Network statistics
Report sar data from a specific time

Note: This will install sar and other systat utilities under /usr/local/bin

Once installed, verify the sar version using “sar -V”. Version 10 is the current stable version of sysstat.

[root@client sa]# sar -V

sysstat version 9.0.4

Finally, make sure sar works. For example,

the following gives the system CPU statistics 3 times (with 1 second interval).

[root@client /]# sar 1 3

Linux 2.6.32-573.el6.x86_64 (client) 11/26/2016 _x86_64_ (2 CPU)

07:54:06 AM CPU %user %nice %system %iowait %steal %idle

07:54:07 AM all 0.00 0.00 0.00 0.00 0.00 100.00

07:54:08 AM all 0.00 0.00 0.00 0.00 0.00 100.00

07:54:09 AM all 0.00 0.00 0.50 0.00 0.00 99.50

Average: all 0.00 0.00 0.17 0.00 0.00 99.83

Utilities part of Sysstat

Following are the other sysstat utilities.

sar collects and displays ALL system activities statistics.
sadc stands for “system activity data collector”. This is the sar backend tool that does the data collection.
sa1 stores system activities in binary data file. sa1 depends on sadc for this purpose. sa1 runs from cron.
sa2 creates daily summary of the collected statistics. sa2 runs from cron.
sadf can generate sar report in CSV, XML, and various other formats. Use this to integrate sar data with other tools.
iostat generates CPU, I/O statistics
mpstat displays CPU statistics.
pidstat reports statistics based on the process id (PID)
nfsiostat displays NFS I/O statistics.
cifsiostat generates CIFS statistics.

This article focuses on sysstat fundamentals and sar utility.

Collect the sar statistics using cron job – sa1 and sa2

Create sysstat file under /etc/cron.d directory that will collect the historical sar data.

# vi /etc/cron.d/sysstat
*/10 * * * * root /usr/local/lib/sa/sa1 1 1
53 23 * * * root /usr/local/lib/sa/sa2 -A

If you’ve installed sysstat from source, the default location of sa1 and sa2 is /usr/local/lib/sa. If you’ve installed using your distribution update method (for example: yum, up2date, or apt-get), this might be /usr/lib/sa/sa1 and /usr/lib/sa/sa2.

/usr/local/lib/sa/sa1

This runs every 10 minutes and collects sar data for historical reference.
If you want to collect sar statistics every 5 minutes, change */10 to */5 in the above /etc/cron.d/sysstat file.
This writes the data to /var/log/sa/saXX file. XX is the day of the month. saXX file is a binary file. You cannot view its content by opening it in a text editor.
For example, If today is 26th day of the month, sa1 writes the sar data to /var/log/sa/sa26
You can pass two parameters to sa1: interval (in seconds) and count.
In the above crontab example: sa1 1 1 means that sa1 collects sar data 1 time with 1 second interval (for every 10 mins).

/usr/local/lib/sa/sa2

This runs close to midnight (at 23:53) to create the daily summary report of the sar data.
sa2 creates /var/log/sa/sarXX file (Note that this is different than saXX file that is created by sa1). This sarXX file created by sa2 is an ascii file that you can view it in a text editor.
This will also remove saXX files that are older than a week. So, write a quick shell script that runs every week to copy the /var/log/sa/* files to some other directory to do historical sar data analysis.

II. 10 Practical Sar Usage Examples

There are two ways to invoke sar.

1 sar followed by an option (without specifying a saXX data file). This will look for the

current day’s saXX data file and report the performance data that was recorded until that point for the current day.

2 sar followed by an option, and additionally specifying a saXX data file using -f option.

This will report the performance data for that particular day. i.e XX is the day of

the month.

In all the examples below, we are going to explain how to view certain performance data for the current day. To look for a specific day,

add “-f /var/log/sa/saXX” at the end of the sar command.

All the sar command will have the following as the 1st line in its output.

[root@client /]# sar -u

Linux 2.6.32-573.el6.x86_64 (client) 11/26/2016 _x86_64_ (2 CPU)

07:16:31 AM LINUX RESTART

07:20:01 AM CPU %user %nice %system %iowait %steal %idle

07:25:01 AM all 0.03 0.00 0.02 0.08 0.13 99.75

07:30:01 AM all 0.03 0.00 0.04 0.06 0.14 99.73

07:35:01 AM all 0.03 0.00 0.02 0.06 0.11 99.78

07:40:01 AM all 0.03 0.00 0.02 0.06 0.10 99.78

07:45:01 AM all 0.04 0.00 0.04 0.05 0.13 99.75

07:50:01 AM all 0.02 0.00 0.01 0.05 0.11 99.81

07:55:01 AM all 0.03 0.00 0.01 0.05 0.12 99.79

Average: all 0.03 0.00 0.02 0.06 0.12 99.77

Linux 2.6.18-194.el5PAE – Linux kernel version of the system.
(dev-db) – The hostname where the sar data was collected.
03/26/2011 – The date when the sar data was collected.
_i686_ – The system architecture
(8 CPU) – Number of CPUs available on this system. On multi core systems, this indicates the total number of cores.

1. CPU Usage of ALL CPUs (sar -u)

This gives the cumulative real-time CPU usage of all CPUs. “1 3” reports for every 1 seconds a total of 3 times. Most likely you’ll focus on the last field “%idle” to see the cpu load.

[root@client /]# sar -u 1 3

Linux 2.6.32-573.el6.x86_64 (client) 11/26/2016 _x86_64_ (2 CPU)

08:00:09 AM CPU %user %nice %system %iowait %steal %idle

08:00:10 AM all 0.00 0.00 0.00 0.00 0.50 99.50

08:00:11 AM all 0.00 0.00 0.00 0.00 0.00 100.00

08:00:12 AM all 0.00 0.00 0.50 0.00 0.00 99.50

Average: all 0.00 0.00 0.17 0.00 0.17 99.67

Following are few variations:

sar -u Displays CPU usage for the current day that was collected until that point.
sar -u 1 3 Displays real time CPU usage every 1 second for 3 times.
sar -u ALL Same as “sar -u” but displays additional fields.
sar -u ALL 1 3 Same as “sar -u 1 3” but displays additional fields.
sar -u -f /var/log/sa/sa10 Displays CPU usage for the 10day of the month from the sa10 file.

2. CPU Usage of Individual CPU or Core (sar -P)

If you have 4 Cores on the machine and would like to see what the individual cores are doing, do the following.

“-P ALL” indicates that it should displays statistics for ALL the individual Cores.

In the following example under “CPU” column 0, 1, 2, and 3 indicates the corresponding CPU core numbers.

[root@client /]# sar -P ALL 1 1

Linux 2.6.32-573.el6.x86_64 (client) 11/26/2016 _x86_64_ (2 CPU)

08:02:22 AM CPU %user %nice %system %iowait %steal %idle

08:02:23 AM all 0.00 0.00 0.00 0.00 0.50 99.50

08:02:23 AM 0 0.00 0.00 0.00 0.00 0.00 100.00

08:02:23 AM 1 0.00 0.00 0.00 0.00 0.00 100.00

Average: CPU %user %nice %system %iowait %steal %idle

Average: all 0.00 0.00 0.00 0.00 0.50 99.50

Average: 0 0.00 0.00 0.00 0.00 0.00 100.00

Average: 1 0.00 0.00 0.00 0.00 0.00 100.00

“-P 1” indicates that it should displays statistics only for the 2nd Core. (Note that Core number starts from 0).

[root@client /]# sar -P 1 1 1

Linux 2.6.32-573.el6.x86_64 (client) 11/26/2016 _x86_64_ (2 CPU)

08:03:04 AM CPU %user %nice %system %iowait %steal %idle

08:03:05 AM 1 1.00 0.00 0.00 0.00 0.00 99.00

Average: 1 1.00 0.00 0.00 0.00 0.00 99.00

Following are few variations:

sar -P ALL Displays CPU usage broken down by all cores for the current day.
sar -P ALL 1 3 Displays real time CPU usage for ALL cores every 1 second for 3 times (broken down by all cores).
sar -P 1 Displays CPU usage for core number 1 for the current day.
sar -P 1 1 3 Displays real time CPU usage for core number 1, every 1 second for 3 times.
sar -P ALL -f /var/log/sa/sa10 Displays CPU usage broken down by all cores for the 10day day of the month from sa10 file.

3. Memory Free and Used (sar -r)

This reports the memory statistics. “1 3” reports for every 1 seconds a total of 3 times. Most likely you’ll focus on “kbmemfree” and “kbmemused” for free and used memory.

[root@client /]# sar -r 1 3

Linux 2.6.32-573.el6.x86_64 (client) 11/26/2016 _x86_64_ (2 CPU)

08:04:20 AM kbmemfree kbmemused %memused kbbuffers kbcached kbcommit %commit

08:04:21 AM 3145052 829144 20.86 147972 539936 219504 3.62

08:04:22 AM 3145036 829160 20.86 147972 539936 219504 3.62

08:04:23 AM 3145036 829160 20.86 147972 539936 219504 3.62

Average: 3145041 829155 20.86 147972 539936 219504 3.62

Following are few variations:

sar -r
sar -r 1 3
sar -r -f /var/log/sa/sa10

4. Swap Space Used (sar -S)

This reports the swap statistics. “1 3” reports for every 1 seconds a total of 3 times. If the “kbswpused” and “%swpused” are at 0, then your system is not swapping.

[root@client /]# sar -S 1 3

Linux 2.6.32-573.el6.x86_64 (client) 11/26/2016 _x86_64_ (2 CPU)

08:21:17 AM kbswpfree kbswpused %swpused kbswpcad %swpcad

08:21:18 AM 2097148 0 0.00 0 0.00

08:21:19 AM 2097148 0 0.00 0 0.00

08:21:20 AM 2097148 0 0.00 0 0.00

Average: 2097148 0 0.00 0 0.00

Following are few variations:

sar -S
sar -S 1 3
sar -S -f /var/log/sa/sa10

Notes:

Use “sar -R” to identify number of memory pages freed, used, and cached per second by the system.
Use “sar -H” to identify the hugepages (in KB) that are used and available.
Use “sar -B” to generate paging statistics. i.e Number of KB paged in (and out) from disk per second.
Use “sar -W” to generate page swap statistics. i.e Page swap in (and out) per second.

5. Overall I/O Activities (sar -b)

This reports I/O statistics. “1 3” reports for every 1 seconds a total of 3 times.

Following fields are displays in the example below.

tps – Transactions per second (this includes both read and write)
rtps – Read transactions per second
wtps – Write transactions per second
bread/s – Bytes read per second
bwrtn/s – Bytes written per second

[root@client /]# sar -b 1 3

Linux 2.6.32-573.el6.x86_64 (client) 11/26/2016 _x86_64_ (2 CPU)

08:23:34 AM tps rtps wtps bread/s bwrtn/s

08:23:35 AM 0.00 0.00 0.00 0.00 0.00

08:23:36 AM 0.00 0.00 0.00 0.00 0.00

08:23:37 AM 0.00 0.00 0.00 0.00 0.00

Average: 0.00 0.00 0.00 0.00 0.00

Following are few variations:

sar -b
sar -b 1 3
sar -b -f /var/log/sa/sa10

Note: Use “sar -v” to display number of inode handlers, file handlers, and pseudo-terminals used by the system.

6. Individual Block Device I/O Activities (sar -d)

To identify the activities by the individual block devices (i.e a specific mount point, or LUN, or partition), use “sar -d”

[root@client /]# sar -d 1 1

Linux 2.6.32-573.el6.x86_64 (client) 11/26/2016 _x86_64_ (2 CPU)

08:27:35 AM DEV tps rd_sec/s wr_sec/s avgrq-sz avgqu-sz await svctm %util

08:27:36 AM dev202-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

08:27:36 AM dev202-48 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

08:27:36 AM dev253-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

08:27:36 AM dev253-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

Average: DEV tps rd_sec/s wr_sec/s avgrq-sz avgqu-sz await svctm %util

Average: dev202-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

Average: dev202-48 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

Average: dev253-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

Average: dev253-1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

In the above example “DEV” indicates the specific block device.

For example: “dev53-1” means a block device with 53 as major number, and 1 as minor number.

The device name (DEV column) can display the actual device name (for example: sda, sda1, sdb1 etc.,), if you use the -p option (pretty print) as shown below.

[root@client /]# sar -p -d 1 1

Linux 2.6.32-573.el6.x86_64 (client) 11/26/2016 _x86_64_ (2 CPU)

08:31:13 AM DEV tps rd_sec/s wr_sec/s avgrq-sz avgqu-sz await svctm %util

08:31:14 AM xvda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

08:31:14 AM xvdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

08:31:14 AM VolGroup-lv_root 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

08:31:14 AM VolGroup-lv_swap 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

Average: DEV tps rd_sec/s wr_sec/s avgrq-sz avgqu-sz await svctm %util

Average: xvda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

Average: xvdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

Average: VolGroup-lv_root 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

Average: VolGroup-lv_swap 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

Following are few variations:

sar -d
sar -d 1 3
sar -d -f /var/log/sa/sa10
sar -p -d

7. Display context switch per second (sar -w)

This reports the total number of processes created per second, and total number of context switches per second. “1 3” reports for every 1 seconds a total of 3 times.

[root@client /]# sar -w 1 3

Linux 2.6.32-573.el6.x86_64 (client) 11/26/2016 _x86_64_ (2 CPU)

08:32:06 AM proc/s cswch/s

08:32:07 AM 0.00 25.00

08:32:08 AM 0.00 87.25

08:32:09 AM 0.00 86.87

Average: 0.00 66.45

Following are few variations:

sar -w
sar -w 1 3
sar -w -f /var/log/sa/sa10

8. Reports run queue and load average (sar -q)

This reports the run queue size and load average of last 1 minute, 5 minutes, and 15 minutes. “1 3” reports for every 1 seconds a total of 3 times.

[root@client /]# sar -q 1 3

Linux 2.6.32-573.el6.x86_64 (client) 11/26/2016 _x86_64_ (2 CPU)

08:33:24 AM runq-sz plist-sz ldavg-1 ldavg-5 ldavg-15

08:33:25 AM 0 120 0.00 0.00 0.00

08:33:26 AM 0 120 0.00 0.00 0.00

08:33:27 AM 0 120 0.00 0.00 0.00

Average: 0 120 0.00 0.00 0.00

Note: The “blocked” column displays the number of tasks that are currently blocked and waiting for I/O operation to complete.

Following are few variations:

sar -q
sar -q 1 3
sar -q -f /var/log/sa/sa10

9. Report network statistics (sar -n)

This reports various network statistics. For example: number of packets received (transmitted) through the network card, statistics of packet failure etc.,. “1 3” reports for every 1 seconds a total of 3 times.

sar -n KEYWORD

KEYWORD can be one of the following:

DEV – Displays network devices vital statistics for eth0, eth1, etc.,
EDEV – Display network device failure statistics
NFS – Displays NFS client activities
NFSD – Displays NFS server activities
SOCK – Displays sockets in use for IPv4
IP – Displays IPv4 network traffic
EIP – Displays IPv4 network errors
ICMP – Displays ICMPv4 network traffic
EICMP – Displays ICMPv4 network errors
TCP – Displays TCPv4 network traffic
ETCP – Displays TCPv4 network errors
UDP – Displays UDPv4 network traffic
SOCK6, IP6, EIP6, ICMP6, UDP6 are for IPv6
ALL – This displays all of the above information. The output will be very long.

[root@client /]# sar -n DEV 1 1

Linux 2.6.32-573.el6.x86_64 (client) 11/26/2016 _x86_64_ (2 CPU)

08:35:05 AM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s

08:35:06 AM lo 2.02 2.02 0.17 0.17 0.00 0.00 0.00

08:35:06 AM eth0 27.27 1.01 3.02 0.04 0.00 0.00 0.00

08:35:06 AM eth1 0.00 0.00 0.00 0.00 0.00 0.00 0.00

Average: IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s

Average: lo 2.02 2.02 0.17 0.17 0.00 0.00 0.00

Average: eth0 27.27 1.01 3.02 0.04 0.00 0.00 0.00

Average: eth1 0.00 0.00 0.00 0.00 0.00 0.00 0.00

10. Report Sar Data Using Start Time (sar -s)

When you view historic sar data from the /var/log/sa/saXX file using “sar -f” option, it displays all the sar data for that specific day starting from 12:00 a.m for that day.

Using “-s hh:mi:ss” option, you can specify the start time. For example, if you specify “sar -s 10:00:00”, it will display the sar data starting from 10 a.m (instead of starting from midnight) as shown below.

You can combine -s option with other sar option.

For example, to report the load average on 26th of this month starting from 10 a.m in the morning, combine the -q and -s option as shown below.

[root@client /]# sar -q -f /var/log/sa/sa26 -s 08:00:01

Linux 2.6.32-573.el6.x86_64 (client) 11/26/2016 _x86_64_ (2 CPU)

08:00:01 AM runq-sz plist-sz ldavg-1 ldavg-5 ldavg-15

08:05:01 AM 0 121 0.00 0.00 0.00

08:10:01 AM 0 121 0.00 0.00 0.00

08:15:01 AM 0 121 0.00 0.00 0.00

08:20:01 AM 0 121 0.00 0.00 0.00

08:25:01 AM 0 121 0.00 0.00 0.00

08:30:01 AM 0 121 0.00 0.00 0.00

08:35:01 AM 0 121 0.00 0.00 0.00

Average: 0 121 0.00 0.00 0.00

There is no option to limit the end-time. You just have to get creative and use head command as shown below.

For example, starting from 10 a.m, if you want to see 7 entries, you have to pipe the above output to “head -n 10”.

[root@client /]# sar -q -f /var/log/sa/sa26 -s 08:00:01 | head -n 10

Linux 2.6.32-573.el6.x86_64 (client) 11/26/2016 _x86_64_ (2 CPU)

08:00:01 AM runq-sz plist-sz ldavg-1 ldavg-5 ldavg-15

08:05:01 AM 0 121 0.00 0.00 0.00

08:10:01 AM 0 121 0.00 0.00 0.00

08:15:01 AM 0 121 0.00 0.00 0.00

08:20:01 AM 0 121 0.00 0.00 0.00

08:25:01 AM 0 121 0.00 0.00 0.00

08:30:01 AM 0 121 0.00 0.00 0.00

08:35:01 AM 0 121 0.00 0.00 0.00

There is lot more to cover in Linux performance monitoring and tuning. We are only getting started. More articles to come in the performance series.

Tushar Jadhav Blog

Saturday, November 26, 2016

Install SAR System Monitoring

AWS root privileges