Skip to content

Check apache-httpd-status

Overview

Monitors Apache httpd performance via the mod_status endpoint (server-status?auto). Alerts when worker usage exceeds the configured thresholds. Reports busy and idle workers, request rates, bytes served, CPU load, connection states, and system load averages. Requires "ExtendedStatus On" in the Apache configuration for full metrics. Uses a local SQLite database to calculate per-second rates from cumulative counters.

Important Notes:

  • Works with any Apache httpd version that provides mod_status
  • Some metrics (connection stats, load averages, processes) are only available in newer versions of mod_status
  • mod_status must be loaded and ExtendedStatus On must be set in the Apache configuration for full metrics. Without ExtendedStatus, only worker counts and scoreboard data are reported.
  • The check alerts on the percentage of busy workers relative to the total number of worker slots (busy + idle + free)
  • Workers in the "Gracefully finishing" (G) state are counted as idle workers, not busy workers

Busy workers (workers serving requests) are:

  • C: Closing connection
  • D: DNS Lookup
  • G: Gracefully finishing
  • I: Idle cleanup of worker
  • K: Keepalive (read)
  • L: Logging
  • R: Reading Request
  • S: Starting up
  • W: Sending Reply

Idle workers are:

  • _: Waiting for Connection

Free workers are:

  • .: Open slot with no current process

Load the Apache module:

LoadModule status_module modules/mod_status.so

If you want to configure /server-status in your main Apache config file:

<IfModule status_module>
    # the alias prevents the processing of .htaccess files, which could contain RewriteRules
    # that interfere with server-status
    Alias /server-status /dev/null
    ExtendedStatus On
    <Location /server-status>
        SetHandler server-status
        Require local
    </Location>
</IfModule>

If you want to configure /server-status in a virtual host:

<IfModule status_module>
    ExtendedStatus On
</IfModule>

<VirtualHost *:80>
    ServerName localhost
    <IfModule status_module>
        # the alias prevents the processing of .htaccess files, which could contain RewriteRules
        # that interfere with server-status
        Alias /server-status /dev/null
        <Location /server-status>
            SetHandler server-status
            Require local
        </Location>
    </IfModule>
</VirtualHost>

Data Collection:

  • Fetches data from the Apache mod_status machine-readable endpoint (server-status?auto)
  • Parses the scoreboard to count workers in each state (reading, replying, keepalive, DNS lookup, closing, logging, starting, finishing, waiting, free slots)
  • Uses a local SQLite database to store previous values and calculate per-interval deltas for accesses, bytes, and duration
  • On the first run (or after a restart), returns "Waiting for more data." until at least two measurements are available

Fact Sheet

Fact Value
Check Plugin Download https://github.com/Linuxfabrik/monitoring-plugins/tree/main/check-plugins/apache-httpd-status
Nagios/Icinga Check Name check_apache_httpd_status
Check Interval Recommendation Every minute
Can be called without parameters Yes
Runs on Cross-platform
Compiled for Windows No
Uses State File $TEMP/linuxfabrik-monitoring-plugins-apache-httpd-status.db

Help

usage: apache-httpd-status [-h] [-V] [--always-ok] [-c CRIT] [--insecure]
                           [--no-proxy] [--test TEST] [--timeout TIMEOUT]
                           [-u URL] [-w WARN]

Monitors Apache httpd performance via the mod_status endpoint (server-
status?auto). Alerts when worker usage exceeds the configured thresholds.
Reports busy and idle workers, request rates, bytes served, CPU load,
connection states, and system load averages. Requires "ExtendedStatus On" in
the Apache configuration for full metrics. Uses a local SQLite database to
calculate per-second rates from cumulative counters.

options:
  -h, --help           show this help message and exit
  -V, --version        show program's version number and exit
  --always-ok          Always returns OK.
  -c, --critical CRIT  CRIT threshold for the percentage of workers processing
                       requests. Default: >= 95
  --insecure           This option explicitly allows insecure SSL connections.
  --no-proxy           Do not use a proxy.
  --test TEST          For unit tests. Needs "path-to-stdout-file,path-to-
                       stderr-file,expected-retc".
  --timeout TIMEOUT    Network timeout in seconds. Default: 8 (seconds)
  -u, --url URL        Apache Server Status URL. Default:
                       http://localhost/server-status
  -w, --warning WARN   WARN threshold for the percentage of workers processing
                       requests. Default: >= 80

Usage Examples

./apache-httpd-status --url http://apache-httpd/server-status --warning 80 --critical 90

Output:

192.168.122.97: 2/400 workers busy (0.5%; 0 "G"), 98 idle, 300 free; 54.0 accesses, 122.0KiB traffic; Up 1W 1D

Key                            ! Value                                               
-------------------------------+-----------------------------------------------------
Current Time                   ! Wednesday, 28-Jul-2021 14:40:48 CEST                
Restart Time                   ! Monday, 19-Jul-2021 20:17:11 CEST                   
Check Interval                 ! 3m 52s                                              
Uptime                         ! 1W 1D                                               
Requests                       ! 54.0                                                
Bytes                          ! 122.0KiB                                            
Request Duration               ! 10s 28ms                                            
Load1                          ! 0.08                                                
Load5                          ! 0.06                                                
Load15                         ! 0.01                                                
Workers Total                  ! 400                                                 
  Busy                         ! 2                                                   
  Idle                         ! 98                                                  
  Usage (%)                    ! 0.5                                                 
Parent Server ConfigGeneration ! 19                                                  
Parent Server MPMGeneration    ! 18                                                  
Server Name                    ! 192.168.122.97                                      
Server MPM                     ! worker                                              
Server Version                 ! Apache/2.4.48 (Fedora) OpenSSL/1.1.1k mod_qos/11.66 
Server Built                   ! Jun  2 2021 00:00:00

States

  • OK if the percentage of busy workers is below the warning threshold.
  • OK with "Waiting for more data." on the first run or after an Apache restart.
  • WARN if the percentage of busy workers is >= --warning (default: 80).
  • CRIT if the percentage of busy workers is >= --critical (default: 95).
  • --always-ok suppresses all alerts and always returns OK.

Perfdata / Metrics

Name Type Description
Accesses Number Total number of accesses during the check interval.
BusyWorkers Number Number of workers currently processing requests.
Bytes Bytes Total bytes served during the check interval.
ConnsAsyncClosing Number Number of async connections in closing state.
ConnsAsyncKeepAlive Number Number of async connections in keep-alive state.
ConnsAsyncWriting Number Number of async connections in writing state.
ConnsTotal Number Total number of connections.
CPULoad Number CPU load of the Apache process.
IdleWorkers Number Number of idle workers (finishing + waiting).
Load1 Number System load average, 1 minute.
Load15 Number System load average, 15 minutes.
Load5 Number System load average, 5 minutes.
ParentServerConfigGeneration Number Apache configuration generation counter.
ParentServerMPMGeneration Number Apache MPM generation counter.
Processes Number Number of Apache processes.
Stopping Number Number of stopping processes.
Total Duration Seconds Total duration of all requests during the check interval.
TotalWorkers Number Total number of worker slots (busy + idle + free).
Uptime Seconds Time the server has been running.
WorkerUsagePercentage Percentage Percentage of workers currently processing requests.
workers_closing Number Workers closing connections ("C" in scoreboard).
workers_dns Number Workers performing DNS lookup ("D" in scoreboard).
workers_finishing Number Workers gracefully finishing ("G" in scoreboard).
workers_free Number Open slots with no current process ("." in scoreboard).
workers_idle Number Workers in idle cleanup ("I" in scoreboard).
workers_keepalive Number Workers in keepalive read ("K" in scoreboard).
workers_logging Number Workers logging ("L" in scoreboard).
workers_reading Number Workers reading request ("R" in scoreboard).
workers_replying Number Workers sending reply ("W" in scoreboard).
workers_starting Number Workers starting up ("S" in scoreboard).
workers_waiting Number Workers waiting for connection ("_" in scoreboard).

Troubleshooting

From https://httpd.apache.org/docs/2.4/mod/mod_status.html#troubleshoot:

The check may be used as a starting place for troubleshooting a situation where your server is consuming all available resources (CPU or memory), and you wish to identify which requests or clients are causing the problem.

First, ensure that you have ExtendedStatus set on, so that you can see the full request and client information for each child or thread.

Now look in your process list (using top, or similar process viewing utility) to identify the specific processes that are the main culprits. Order the output of top by CPU usage, or memory usage, depending on what problem you're trying to address.

Reload the server-status page, and look for those process ids, and you'll be able to see what request is being served by that process, for what client. Requests are transient, so you may need to try several times before you catch it in the act, so to speak.

This process should give you some idea what client, or what type of requests, are primarily responsible for your load problems. Often you will identify a particular web application that is misbehaving, or a particular client that is attacking your site.

Credits, License