Check systemd-units-failed¶
Overview¶
Checks for failed systemd units by running systemctl --state=failed. Reports any unit that is in a failed active state or failed sub state, and says why each unit failed: the exit status or the signal of its main process, a timeout, a start limit or the OOM killer. With read access to the systemd journal, the check also shows the last three log lines of each failed unit, which usually name the actual error.
When no unit is currently failed, the plugin reads the current boot's journal for the most recent system unit-failed event and appends it to the output, so the admin sees how long the host has been clean since the last reboot and which unit last broke.
Data Collection:
- Executes
systemctl --state=failed --no-pager --no-legend - For the failed units, executes
systemctl show --property=Result,ExecMainCode,ExecMainStatusonce, which needs no privileges. Exit statuses from 200 on are set by systemd itself and carry systemd's name (203/EXEC: the executable was not found,217/USER: the user does not exist), see systemd.exec(5) - For the first five failed units, executes
journalctl --boot=0 --output=cat --lines=3 _SYSTEMD_UNIT=<unit>, which shows what the unit's own processes logged, without systemd's messages about the unit. All calls together get 5 seconds. Reading the system journal needs root, sudo or thesystemd-journalgroup; without, the check names thejournalctlcommand to run instead - When everything is OK, additionally runs
journalctl --output=short-unix --boot=0 _PID=1 --grep="Failed with result"and picks the last matching line client-side, to read the most recent system-scope unit-failed event of the current boot. Scoping to--boot=0keeps the call fast regardless of total journal size. Ifjournalctlis unavailable, takes longer than 5 seconds, returns nothing, or its entry cannot be parsed, the suffix is silently omitted - Units can be excluded from the check via
--ignore, which supports glob patterns according to Python'sfnmatchmodule (e.g.--ignore="sshd@*.service"). A pattern matches the unit name as systemd writes it (dev-disk-by\x2duuid-0a1b2c3d.swap) as well as its readable form (dev-disk-by-uuid-0a1b2c3d.swap), in the list of failed units and in the journal alike
Fact Sheet¶
| Fact | Value |
|---|---|
| Check Plugin Download | https://github.com/Linuxfabrik/monitoring-plugins/tree/main/check-plugins/systemd-units-failed |
| Nagios/Icinga Check Name | check_systemd_units_failed |
| Check Interval Recommendation | Every minute |
| Can be called without parameters | Yes |
| Runs on | Linux |
| Compiled for Windows | No |
Help¶
usage: systemd-units-failed [-h] [-V] [--always-ok] [--ignore IGNORE]
[--no-perfdata]
Checks for failed systemd units. Alerts when any unit is in a failed state,
and says why each unit failed, for example the exit status or the signal that
killed it. With read access to the systemd journal, also shows the last log
lines of each failed unit. Specific units can be excluded from the check via
--ignore with glob patterns. When no unit is currently failed, reports the
most recent system unit-failed event from the current boot's journal so
operators see at a glance how long the host has been clean since the last
reboot.
options:
-h, --help show this help message and exit
-V, --version show program's version number and exit
--always-ok Always returns OK.
--ignore IGNORE Unit name to exclude from the check. Supports glob patterns
according to
https://docs.python.org/3/library/fnmatch.html. Case-
sensitive. Can be specified multiple times. Example:
`--ignore=dhcpd.service`.
--no-perfdata Suppress the performance data section from the output. The
status message and the exit code are unaffected, so
alerting keeps working while trending data is dropped.
Documentation:
https://linuxfabrik.github.io/monitoring-plugins/check-plugins/systemd-units-failed/
Usage Examples¶
./systemd-units-failed --ignore=openipmi.service --ignore=dhcpd.service
./systemd-units-failed --ignore=sshd@*.service
Output (something is failing, with read access to the journal):
1 failed unit: httpd.service (exited with status 1)
unit ! load ! active ! sub ! description
--------------+--------+--------+--------+-----------------------
httpd.service ! loaded ! failed ! failed ! The Apache HTTP Server
Last log lines of httpd.service:
httpd.service: Referenced but unset environment variable evaluates to an empty string: OPTIONS
AH00526: Syntax error on line 359 of /etc/httpd/conf/httpd.conf:
Invalid command 'ServerNmae', perhaps misspelled or defined by a module not included in the server configuration
Show the log with `sudo journalctl --boot=0 --unit=httpd.service`
Output (something is failing, without read access to the journal):
1 failed unit: httpd.service (exited with status 1)
unit ! load ! active ! sub ! description
--------------+--------+--------+--------+-----------------------
httpd.service ! loaded ! failed ! failed ! The Apache HTTP Server
Show the log with `sudo journalctl --boot=0 --unit=httpd.service`
Output (currently clean, with last failure surfaced from the journal):
Everything is ok. Last failed: `ipmievd.service` with message "Failed with result 'exit-code'" at 2026-04-30 12:01:53 (1W 2D ago)
States¶
- OK if no units are in a failed state.
- WARN if at least one unit has a failed active state or failed sub state.
--always-oksuppresses all alerts and always returns OK.
Perfdata / Metrics¶
| Name | Type | Description |
|---|---|---|
| systemd-units-failed | Number | Number of failed units |
Troubleshooting¶
No last log lines, only Show the log with sudo journalctl ...¶
The account running the check may not read the system journal. Either run the command by hand, or let the check show the last log lines itself by adding the account to the systemd-journal group, which may read the whole journal:
sudo usermod --append --groups systemd-journal icinga
The group takes effect after a restart of the monitoring agent (sudo systemctl restart icinga2). The last log lines are missing too when the units logged nothing themselves, for example after a timeout; journalctl --unit then still shows systemd's messages about them. The command always follows the output and names every failed unit, since the last log lines are only a first hint and cover the first five units at most.
Resetting a failed unit¶
If you cannot fix the underlying issue and simply want to reset the status of a failed unit:
systemctl reset-failed ipmievd.service
Credits, License¶
- Authors: Linuxfabrik GmbH, Zurich
- License: The Unlicense, see LICENSE file.