Check disk-io¶

Overview¶

Checks disk I/O bandwidth over time and alerts on sustained saturation, not short spikes. The check records per-disk read/write counters and then derives current (R1/W1) and period averages over --count runs (R15/W15, as in load15). It compares the period's total bandwidth against the maximum ever observed for that disk (RWmax). It raises a WARNING when the period average exceeds --warning percent of RWmax, once RWmax has been learned for a week. This bandwidth part only ever warns, never criticals: sustained I/O is a signal to investigate, not an emergency you have to react to at night.

The check also reports per-disk I/O latency (await, not on Windows): the average time a read or write took to complete over the period, in milliseconds. Unlike disk busy percentage, latency is robust against device parallelism, so it is a meaningful "is my storage slow" signal on NVMe, SSD and RAID as well. The optional --await-warning and --await-critical thresholds alert on sustained latency; both are disabled by default. A critical latency threshold is the place to catch a disk that is effectively hung.

Perfdata is emitted for each disk (read/write throughput per second and IOPS, plus I/O latency except on Windows and disk busy percentage on Linux), so you can graph trends. On Linux the check focuses on block devices with a mounted filesystem by default; use --include-unmounted to also include raw, unmounted devices such as multipath SAN volumes. On Windows it uses psutil's disk counters. Optionally, --top lists the processes that generated the most I/O traffic (read/write totals) to help identify offenders.

This check is cross-platform and works on Linux, Windows, and all psutil-supported systems. The check stores its short trend state locally in an SQLite DB to evaluate sustained load across runs.

Important Notes:

--count=5 (the default) while checking every minute means that the check will alert if any of your disks have been above a threshold in the last 5 minutes
The bandwidth warning waits until a disk's maximum bandwidth (RWmax) has been learned for a week. RWmax is the busiest period seen so far, and on a new host that is whatever sustained load comes first, a backup or an installation, which would otherwise sit at 100% of its own maximum. Until then the first line says how long the check is still learning. The await thresholds apply from the start
Plugin execution may take a moment due to process enumeration when --top is enabled
On Windows, the check reports no I/O latency (await) and ignores --await-warning and --await-critical: psutil delivers the underlying disk times with the wrong unit there (psutil#3002)

Data Collection:

Uses psutil to collect per-disk I/O counters (see Perfdata / Metrics for the full list)
On Linux, automatically detects "real" block devices that have mountpoints, filtering out virtual devices
Stores counter snapshots in a local SQLite database and calculates deltas between consecutive runs
IOPS are the completed read and write operations per second. They are reported for graphing only and never alert: they show how much is asked of a disk, not whether it copes. A disk that runs into an IOPS limit, for example a cloud volume with a QoS cap, shows a flat IOPS curve at the limit while its latency (await) rises, which is what --await-warning catches
On the first run, returns "Waiting for more data." until at least two measurements are available
After a system reboot, counter values may be lower than the previous measurement. The check detects this and returns "Waiting for more data." until the next valid measurement pair
Disk I/O bandwidth tracking starts at 10 MiB/sec as a baseline, but stores the highest measured bandwidth, so the RWmax/s value adjusts accordingly over time. The check may throw warnings during the first major disk activities above 10 MiB/sec until the actual maximum bandwidth of the disk has been determined
On Linux, only block devices with a mounted filesystem are monitored by default; --include-unmounted adds raw, unmounted devices (see Troubleshooting). Pseudo devices (loop, ram, zram, floppy, optical) are always excluded
Disks can be filtered by --match, a Python regular expression. It is anchored at the start of the string (Python re.match) and matched against the full device path (for example /dev/sda), the device-mapper path (for example /dev/mapper/vg-lv), and the mountpoint. Prefix the pattern with .* to match anywhere in the path, for example .*sda$ rather than ^sda$

Fact Sheet¶

Fact	Value
Check Plugin Download	https://github.com/Linuxfabrik/monitoring-plugins/tree/main/check-plugins/disk-io
Nagios/Icinga Check Name	`check_disk_io`
Check Interval Recommendation	Every minute
Can be called without parameters	Yes
Runs on	Cross-platform
Compiled for Windows	Yes
3rd Party Python modules	`psutil`
Handles Periods	Yes (values are averaged over `--count` measurements, default 5)
Uses State File	`$TEMP/linuxfabrik-monitoring-plugins-disk-io.db`

Help¶

usage: disk-io [-h] [-V] [--always-ok] [--await-critical AWAIT_CRIT]
               [--await-warning AWAIT_WARN] [--count COUNT]
               [--include-unmounted] [--match MATCH]
               [--no-match-severity {ok,warn,crit,unknown}] [--no-perfdata]
               [--top TOP] [--warning WARN]

Checks disk I/O bandwidth over time and alerts on sustained saturation, not
short spikes. The check records per-disk read/write counters and then derives
current (R1/W1) and period averages over --count runs (R15/W15, as in load15).
It compares the period's total bandwidth against the maximum ever observed for
that disk (RWmax). It raises a WARNING when the period average exceeds
--warning percent of RWmax, once RWmax has been learned for a week. This
bandwidth part only ever warns, never criticals: sustained I/O is a signal to
investigate, not an emergency you have to react to at night. The check also
reports per-disk I/O latency (await, not on Windows): the average time a read
or write took to complete over the period, in milliseconds. Unlike disk busy
percentage, latency is robust against device parallelism, so it is a
meaningful "is my storage slow" signal on NVMe, SSD and RAID as well. Optional
--await-warning and --await-critical thresholds alert on sustained latency;
both are disabled by default. A critical latency threshold is the place to
catch a disk that is effectively hung. Perfdata is emitted for each disk
(read/write throughput per second and IOPS, plus I/O latency except on Windows
and disk busy percentage on Linux), so you can graph trends. On Linux the
check focuses on block devices with a mounted filesystem by default; use
`--include-unmounted` to also include raw, unmounted devices such as multipath
SAN volumes. On Windows it uses psutil's disk counters. Optionally, `--top`
lists the processes that generated the most I/O traffic (read/write totals) to
help identify offenders. This check is cross-platform and works on Linux,
Windows, and all psutil-supported systems. The check stores its short trend
state locally in an SQLite DB to evaluate sustained load across runs.

options:
  -h, --help            show this help message and exit
  -V, --version         show program's version number and exit
  --always-ok           Always returns OK.
  --await-critical AWAIT_CRIT
                        CRIT threshold for per-disk I/O latency (await) in
                        milliseconds, averaged over the last `--count` runs.
                        await is the average time a read or write took to
                        complete. Ignored on Windows. Use this to catch a disk
                        that is effectively hung (sustained latency of
                        seconds), not a merely busy one. Supports Nagios
                        ranges. Disabled by default.
  --await-warning AWAIT_WARN
                        WARN threshold for per-disk I/O latency (await) in
                        milliseconds, averaged over the last `--count` runs.
                        await is the average time a read or write took to
                        complete. Ignored on Windows. A good value depends on
                        the storage (an SSD is well below a millisecond, a
                        busy HDD can sustain tens of milliseconds), so set it
                        to what is abnormal for your disks. Supports Nagios
                        ranges. Disabled by default.
  --count COUNT         Number of consecutive checks the threshold must be
                        exceeded before alerting. Default: 5
  --include-unmounted   Also monitor block devices that have no mounted
                        filesystem (Linux only). By default only block devices
                        with a mounted filesystem are monitored. Enable this
                        to include raw, unmounted devices such as multipath
                        SAN volumes or disks used directly by a database or
                        storage layer. Combine with `--match`, otherwise every
                        unmounted device shows up. Pseudo devices (loop, ram,
                        zram, floppy, optical) are always excluded.
  --match MATCH         Only check disks whose path or mountpoint matches this
                        Python regular expression. Case-sensitive by default;
                        use `(?i)` for case-insensitive matching. The regex is
                        anchored at the start of the string (Python
                        `re.match`) and matched against the full device path
                        (e.g. `/dev/sda`), the device-mapper path (e.g.
                        `/dev/mapper/vg-lv`) and the mountpoint, so prefix
                        with `.*` to match anywhere (`.*sda$` instead of
                        `^sda$`). On Linux only block devices with a mounted
                        filesystem are considered by default; add `--include-
                        unmounted` to also match raw, unmounted devices
                        (multipath SAN volumes, raw LUNs). Default:
  --no-match-severity {ok,warn,crit,unknown}
                        State to report when no item matches the filters and
                        nothing is checked. Default: ok
  --no-perfdata         Suppress the performance data section from the output.
                        The status message and the exit code are unaffected,
                        so alerting keeps working while trending data is
                        dropped.
  --top TOP             Number of top processes to list by I/O traffic. Use
                        `--top=0` to disable. Default: 5
  --warning WARN        WARN threshold for disk bandwidth saturation as a
                        percentage of the observed maximum, measured over the
                        last `--count` runs. Default: >= 80

Documentation:
https://linuxfabrik.github.io/monitoring-plugins/check-plugins/disk-io/

Usage Examples¶

Just check disk dm-0 (if listed as /dev/dm-0):

./disk-io --match='.*dm-0$'

Match all disks except vdc, vdh and vdz:

./disk-io --match='^(?:(?!.*vdc|.*vdh|.*vdz).)*$'

Monitor a raw, unmounted multipath SAN volume (for example an Oracle ASM disk):

./disk-io --include-unmounted --match='.*dm-7$'

Output:

/dev/dm-8: 5.6KiB/s read1, 2.2MiB/s write1, 2.2MiB/s total, 10.0MiB/s max

Name ! MntPnts        ! DvMppr           ! RWmax/s ! R1/s   ! W1/s    ! R15/s  ! W15/s   ! RW15/s
-----+----------------+------------------+---------+--------+---------+--------+---------+---------
dm-0 ! /              ! rl-root          ! 10.0MiB ! 0.0B   ! 426.0B  ! 0.0B   ! 343.0B  ! 343.0B
vda2 ! /boot          !                  ! 10.0MiB ! 0.0B   ! 0.0B    ! 0.0B   ! 0.0B    ! 0.0B
vda1 ! /boot/efi      !                  ! 10.0MiB ! 0.0B   ! 0.0B    ! 0.0B   ! 0.0B    ! 0.0B
dm-5 ! /var           ! rl-var           ! 10.0MiB ! 0.0B   ! 586.0B  ! 0.0B   ! 1.1KiB  ! 1.1KiB
dm-8 ! /data          ! rl-lv_data       ! 10.0MiB ! 5.6KiB ! 2.2MiB  ! 8.3KiB ! 2.3MiB  ! 2.3MiB
dm-6 ! /tmp           ! rl-tmp           ! 10.0MiB ! 0.0B   ! 4.8KiB  ! 0.0B   ! 7.1KiB  ! 7.1KiB
dm-7 ! /home          ! rl-home          ! 10.0MiB ! 0.0B   ! 0.0B    ! 0.0B   ! 0.0B    ! 0.0B
dm-2 ! /var/tmp       ! rl-var_tmp       ! 10.0MiB ! 0.0B   ! 0.0B    ! 0.0B   ! 0.0B    ! 0.0B
dm-4 ! /var/log       ! rl-var_log       ! 10.0MiB ! 0.0B   ! 51.8KiB ! 0.0B   ! 51.2KiB ! 51.2KiB
dm-3 ! /var/log/audit ! rl-var_log_audit ! 10.0MiB ! 0.0B   ! 918.0B  ! 0.0B   ! 876.0B  ! 876.0B

Top 5 processes that generate the most I/O traffic (r/w):
1. nfsd: 149.2GiB/5.7TiB
2. systemd: 695.7GiB/169.9GiB
3. systemd-journald: 33.9MiB/124.4GiB
4. icinga2: 7.9GiB/4.9GiB
5. rsyslogd: 114.8MiB/4.1GiB

States¶

OK if every disk's bandwidth period average is below --warning (default: 80%) of the observed maximum, and I/O latency (await) is below the await thresholds (both off by default).
OK with "Waiting for more data." on the first run or after a reboot.
WARN if the bandwidth period average (computed over the last --count runs, default: 5) reaches --warning (default: 80%) of the observed maximum for a disk, once that maximum has been learned for a week. Bandwidth alerts never escalate beyond WARN: a busy disk is a signal to investigate, not an emergency.
WARN if a disk's average I/O latency (await, over the last --count runs) reaches --await-warning (disabled by default, not on Windows).
CRIT if a disk's average I/O latency reaches --await-critical (disabled by default, not on Windows). This is the place to catch a disk that is effectively hung (sustained multi-second latency); a merely busy disk never criticals.
--no-match-severity sets the state reported when the filters match no disk and nothing is checked (default: ok); set it to warn, crit, or unknown to alert on an empty selection (for example a filter typo or a missing disk) instead of silently returning OK.
--always-ok suppresses all alerts and always returns OK.

Perfdata / Metrics¶

Per matched disk, where <disk> is the block device name. The 1 suffix is the latest interval (like a "load1"), the 15 suffix is the average over the last --count runs (like a "load15"). The byte, throughput and IOPS metrics are cross-platform. The latency metrics (await, read_await, write_await) are not reported on Windows, and busy_percent is Linux only, because the underlying counter (busy_time) is not exposed by psutil on Windows:

Name	Type	Description
`<disk>`_await	Seconds	Average time a read or write took to complete over the last `--count` runs, in milliseconds (iostat's await). This is the latency signal that `--await-warning`/`--await-critical` alert on.
`<disk>`_busy_percent	Percentage	Linux only. Share of wall-clock time the device had at least one I/O in flight over the last interval. This is exactly iostat's %util, derived from the `io_ticks` counter in `/proc/diskstats`. It is a busy/idle indicator, not a saturation measure: on devices that serve requests in parallel (NVMe, SSD, dm/md, ZFS) it can sit near 100% far below the real throughput limit, which is why the check does not alert on it.
`<disk>`_iops1	Number	Completed read plus write operations per second over the latest interval.
`<disk>`_iops15	Number	Completed read plus write operations per second, averaged over the last `--count` runs.
`<disk>`_read_await	Seconds	Average read latency over the last `--count` runs, in milliseconds (iostat's r_await).
`<disk>`_read_bytes_per_second1	Bytes	Bytes read per second over the latest interval.
`<disk>`_read_bytes_per_second15	Bytes	Bytes read per second, averaged over the last `--count` runs.
`<disk>`_read_iops1	Number	Completed read operations per second over the latest interval.
`<disk>`_read_iops15	Number	Completed read operations per second, averaged over the last `--count` runs.
`<disk>`_throughput1	Bytes	Read plus write bytes per second over the latest interval.
`<disk>`_throughput15	Bytes	Read plus write bytes per second, averaged over the last `--count` runs.
`<disk>`_write_bytes_per_second1	Bytes	Bytes written per second over the latest interval.
`<disk>`_write_bytes_per_second15	Bytes	Bytes written per second, averaged over the last `--count` runs.
`<disk>`_write_iops1	Number	Completed write operations per second over the latest interval.
`<disk>`_write_iops15	Number	Completed write operations per second, averaged over the last `--count` runs.

Troubleshooting¶

`Python module "psutil" is not installed.`¶

Install psutil: pip install psutil or dnf install python3-psutil.

Outdated psutil¶

psutil raised error "not sure how to interpret line '...'" or Nothing checked. Running Kernel >= 4.18, this check needs the Python module psutil v5.7.0+

Update the psutil library. On RHEL 8+, use at least python38 and python38-psutil if using dnf.

`Waiting for more data.`¶

This is expected on the first run. The check needs at least two measurements to calculate a delta. Wait for the next check interval.

`No disks found.`¶

On Windows, the disk performance counters may be switched off, so psutil does not see any disk. Enable them with diskperf -y in an elevated command prompt.

A raw or unmounted device is missing¶

By default the check only monitors block devices that have a mounted filesystem, so raw, unmounted devices never appear (for example multipath SAN volumes, raw LUNs, Oracle ASM disks). Add --include-unmounted to also monitor them, and narrow the result with --match, for example --include-unmounted --match '.*dm-7$' for a multipath volume. Pseudo devices (loop, ram, zram, floppy, optical) stay excluded. Note that --match is anchored at the start of the full device path: use .*dm-7$, not ^dm-7$.

Credits, License¶

Authors: Linuxfabrik GmbH, Zurich
License: The Unlicense, see LICENSE file.