Skip to content

Check icinga-topflap-services

Overview

Detects fast-flapping Icinga services by counting state changes per service within a configurable lookback interval. Queries the Icinga DB event history and alerts when any service exceeds the configured number of state changes. State changes that happened during a downtime of the service are not counted.

Important Notes:

  • Requires Icinga DB with the Icinga Web 2 module
  • The Icinga Web 2 user needs at least the "icingadb > General Module Access" permission
  • Downtimes are taken from the same history, so the limit in the URL has to reach back over the lookback period. A downtime whose start and end both lie outside the fetched events is not known, and the state changes during it are counted
  • Instead of specifying URL, username and password on the command line, you can create and specify an INI file:

    [icingaweb2]
    url = http://localhost/icingaweb2/icingadb/history?limit=250
    username = alice
    password = linuxfabrik
    

Data Collection:

  • Fetches data from the Icinga DB event history via the Icinga Web 2 REST API using HTTP Basic authentication
  • Groups events by host and service, then counts state changes per service within the lookback window
  • Skips state changes that fall into a downtime of the service, including flexible downtimes from the moment they were triggered and cancelled downtimes up to their cancellation
  • Uses an in-memory SQLite database to aggregate the events of a check run
  • Credentials can be provided via command-line parameters or a password INI file (command-line takes precedence)

Fact Sheet

Fact Value
Check Plugin Download https://github.com/Linuxfabrik/monitoring-plugins/tree/main/check-plugins/icinga-topflap-services
Nagios/Icinga Check Name check_icinga_topflap_services
Check Interval Recommendation Every 5 minutes
Can be called without parameters Yes (if --pwfile exists at the default path)
Runs on Cross-platform
Compiled for Windows No (runs with Python interpreter)
Requirements Icinga DB, read access to /icingaweb2/icingadb/history

Help

usage: icinga-topflap-services [-h] [-V] [--always-ok] [-c CRIT] [--insecure]
                               [--lookback LOOKBACK] [--no-proxy]
                               [--password PASSWORD] [--proxy PROXY]
                               [--pwfile PWFILE] [--timeout TIMEOUT]
                               [--url URL] [--username USERNAME] [-w WARN]

Detects fast-flapping Icinga services by counting state changes per service
within a configurable lookback interval. Queries the Icinga DB event history
and alerts when any service exceeds the configured number of state changes.
State changes that happened during a downtime of the service are not counted.

options:
  -h, --help           show this help message and exit
  -V, --version        show program's version number and exit
  --always-ok          Always returns OK.
  -c, --critical CRIT  CRIT threshold for the number of state changes per
                       service within the lookback period. Supports Nagios
                       ranges. Default: 19
  --insecure           This option explicitly allows insecure SSL connections.
  --lookback LOOKBACK  State changes are counted within this window. Time
                       window in seconds to look back over, ending at the
                       moment of the run. Only what falls within it is
                       counted, so what is reported is how often something
                       happened lately rather than a total that keeps growing
                       for as long as the source is kept. Example:
                       `--lookback=7200`. Default: 14400 (seconds)
  --no-proxy           Do not use a proxy, not even one the environment names.
                       Overrides `--proxy`.
  --password PASSWORD  Icinga Web 2 password. Takes precedence over the value
                       in `--pwfile`.
  --proxy PROXY        Proxy to reach the target through. The scheme defaults
                       to `http` when omitted. Overrides the proxy the
                       environment names (`http_proxy`, `https_proxy`,
                       `all_proxy`) together with the exceptions it lists in
                       `no_proxy`, and is itself overridden by `--no-proxy`.
                       Without either parameter the environment applies.
                       Credentials belong into the environment variable rather
                       than here, because a command-line argument is visible
                       to every user on the host. Example:
                       `--proxy=http://proxy.example.com:3128`.
  --pwfile PWFILE      Path to a password file containing "url", "user" and
                       "password" for Icinga Web 2. Example:
                       `--pwfile=/var/spool/icinga2/.icingaweb`. Default:
                       /var/spool/icinga2/.icingaweb
  --timeout TIMEOUT    Network timeout in seconds. Default: 8 (seconds)
  --url URL            Icinga DB event history URL including filter
                       parameters. Takes precedence over the value in
                       `--pwfile`. Example: `--url
                       https://icinga/icingaweb2/icingadb/history?limit=250`.
  --username USERNAME  Icinga Web 2 username. Takes precedence over the value
                       in `--pwfile`.
  -w, --warning WARN   WARN threshold for the number of state changes per
                       service within the lookback period. Supports Nagios
                       ranges. Default: 7

Documentation:
https://linuxfabrik.github.io/monitoring-plugins/check-plugins/icinga-topflap-services/

Usage Examples

./icinga-topflap-services \
    --username=alice \
    --password=linuxfabrik \
    --url='https://icinga.example.com/icingaweb2/icingadb/history?limit=250' \
    --lookback=86400

Output:

There are warnings. (lookback=1D warn=7 crit=19)

Host            ! Service                 ! Cnt ! State
----------------+-------------------------+-----+-----------
srv-mon01       ! Swap Usage              ! 12  ! [WARNING]
srv-analytics01 ! Load                    ! 10  ! [WARNING]
srv-analytics01 ! CPU Usage               ! 8   ! [WARNING]
srv-vcs01       ! Swap Usage              ! 6   ! [OK]
srv-cloud02     ! Apache httpd Status     ! 4   ! [OK]
srv-repo01      ! Journald Usage          ! 2   ! [OK]
srv-cloud01     ! Nextcloud Stats         ! 2   ! [OK]

States

  • OK if no service exceeds the warning threshold for state changes within the lookback period.
  • WARN if any service has more than --warning (default: 7) state changes.
  • CRIT if any service has more than --critical (default: 19) state changes.
  • State changes during a downtime of the service are not counted. Like in Icinga, a downtime of the host alone does not cover its services; schedule host downtimes together with their services.
  • UNKNOWN on missing credentials, unreadable password file, or invalid command-line arguments.
  • --always-ok suppresses all alerts and always returns OK.

Perfdata / Metrics

There is no perfdata.

Troubleshooting

Password file does not parse

`/path/to/pwfile` is not a valid INI file, see line 3.

The password file does not parse. The message names the line but not its content, since --pwfile may point to any file. The file needs a section header such as [icingaweb2] and one key = value pair per line, see the example in the Overview.

Credits, License