Skip to content

Ansible Role linuxfabrik.lfops.system_update

This role configures the server to do (weekly) system updates by deploying two shell scripts: The first script notify-and-schedule checks for available updates (normally during the day), and notifies the system administrators either via email or Rocket.Chat. On update time (usually the next morning at round about 4 AM), the second script update-and-reboot

  • sets a downtime for the host and all its services in Icinga
  • applies all updates
  • and, if necessary, automatically reboots the host after the updates.

On Rocky Linux the role additionally sets up a separate security lane that installs only security hot-fixes daily, independent of the weekly update lane.

Available since LFOps 2.0.0.

How the Role Behaves

  • Two independent lanes. The regular lane (notify-and-schedule / update-and-reboot) applies all available updates on its weekly schedule. On Rocky Linux a second, independent security lane (security-update, twice a day by default) installs only security hot-fixes from the dedicated security repository, isolated from the regular lane via --disablerepo / --enablerepo. Both reboot the host when an update requires it.
  • The security lane is enabled by default, but a no-op without the security repository. That repository is provided by the repo_baseos role. On hosts where it is not present, the security lane installs nothing and never reboots. Turn the lane off entirely with system_update__security_enabled: false.
  • Reboots are steered per host group. A security hot-fix that requires a reboot is scheduled via at; the time comes from system_update__security_reboot_time__*. This can be used so test hosts reboot immediately ('now') while production hosts defer to the evening (for example '19:00').

Dependent Roles

Any LFOps playbook that installs this role runs these for you. Optional ones can be disabled via the playbook's skip variables.

Requirements

Manual steps:

Tags

system_update

  • Sets up automatic system update via systemd timer, and on Rocky Linux hosts the optional security-update timer.
  • Triggers: none.

system_update:state

  • Determines whether notify-and-schedule.timer and security-update.timer are enabled.
  • Triggers: none.

Optional Role Variables

system_update__cache_only

  • Whether to install updates from cache only. This implies to have the cache built beforehand.
  • Type: Bool.
  • Default: false

system_update__icinga2_api_url

  • The URL of the Icinga2 API (usually on the Icinga2 Master). This will be used to set a downtime for the corresponding host and all its services in the reboot alias.
  • Type: String.
  • Default: 'https://{{ icinga2_agent__icinga2_master_host | d("") }}:{{ icinga2_agent__icinga2_master_port | d(5665) }}'

system_update__icinga2_api_user_login

  • The Icinga2 API User to set the downtime for the corresponding host and all its services.
  • Type: Dictionary.
  • Default: unset

system_update__icinga2_hostname

  • The hostname of the Icinga2 host on which the downtime should be set.
  • Type: String.
  • Default: '{{ ansible_facts["nodename"] }}'

system_update__mail_from

  • The email sender account. This will be used as the "from"-address for all notifications.
  • Type: String.
  • Default: '{{ mailto_root__from }}'

system_update__mail_recipients_new_configfiles

  • A list of email recipients to notify if there is a new version of a config file (rpmnew / rpmsave / dpkg-dist / ucf-dist).
  • Type: String.
  • Default: '{{ mailto_root__to }}'

system_update__mail_recipients_updates

  • A list of email recipients to notify about the expected updates and the report of the installed updates.
  • Type: String.
  • Default: '{{ mailto_root__to }}'

system_update__mail_subject_hostname

  • String which will be used as the hostname in the mail subject. You can use $() to call bash code.
  • Type: String.
  • Default: '$(hostname --short)'

system_update__mail_subject_prefix

  • This will set a prefix that will be showed in front of the hostname. Can be used to separate servers by environment or customer.
  • Type: String.
  • Default: ''

system_update__notify_and_schedule_on_calendar

  • When the notification for the expected updates should be sent. Have a look at systemd.time(7) for the format.
  • Type: String.
  • Default: 'mon 10:00'

system_update__post_update_code

  • This codeblock will be executed after the updates have been installed and before a potential reboot.
  • Type: String.
  • Default: unset

system_update__pre_update_code

  • This codeblock will be executed before the update process is started. Can be used to check pre-conditions for updating, for example for checking cluster nodes.
  • Type: String.
  • Default: unset

system_update__rocketchat_msg_suffix

  • A suffix to the Rocket.Chat notifications. This can be used to mention other users.
  • Type: String.
  • Default: ''

system_update__rocketchat_url

  • The URL to a potential Rocket.Chat server to send notifications about the updates to.
  • Type: String.
  • Default: unset

system_update__security_enabled

  • Enables or disables the security lane (the security-update timer), analogous to systemctl enable/disable --now. Rocky Linux only. When enabled but the security repository is not enabled on the host, the lane is a no-op.
  • Type: Bool.
  • Default: true

system_update__security_on_calendar

  • When the security lane checks for and installs security hot-fixes. Have a look at systemd.time(7) for the format.
  • Type: String.
  • Default: '*-*-* 10,16:00'

system_update__security_reboot_time__host_var / system_update__security_reboot_time__group_var

  • When to reboot after a security hot-fix that requires it. Passed verbatim to at. Use this to steer test versus production hosts via inventory group membership: 'now' reboots immediately, a time such as '19:00' defers the reboot.
  • Type: String.
  • Default: 'now'

system_update__security_repos

  • The repositories the security lane installs from. All other repositories are disabled for the security transaction, keeping it separate from the regular update lane.
  • Type: List.
  • Default: ['security']

system_update__update_enabled

  • Enables or disables the system-update timer, analogous to systemctl enable/disable --now.
  • Type: Bool.
  • Default: true

system_update__update_time

  • The time when to actually execute the updates (and automatically reboot if necessary), relative to system_update__notify_and_schedule_on_calendar. Passed verbatim to at. The default schedules the update for the next day between 04:00 and 04:59, with the exact minute derived deterministically from inventory_hostname so multiple hosts spread across the hour instead of all updating at 04:00.
  • Type: String.
  • Default: '04:{{ 59 | random(seed=inventory_hostname) }} + 1 days'

Example:

# optional
system_update__cache_only: true
system_update__icinga2_api_url: 'https://icinga.example.com:5665'
system_update__icinga2_api_user_login:
  username: 'downtime-user'
  password: 'linuxfabrik'
system_update__icinga2_hostname: 'myhost.example.com'
system_update__mail_from: 'noreply@example.com'
system_update__mail_recipients_new_configfiles:
  - 'info@example.com'
  - 'support@example.com'
system_update__mail_recipients_updates:
  - 'info@example.com'
  - 'support@example.com'
system_update__mail_subject_hostname: '$(hostname --long)'
system_update__mail_subject_prefix: '001-'
system_update__notify_and_schedule_on_calendar: 'mon *-*-01..07 10:00' # first monday of the month
system_update__post_update_code: |-
  VAR='hello world'
  echo $VAR
system_update__pre_update_code: |-
  check_dns() {
    local DNS_SERVER=$1
    if ! dig @$DNS_SERVER linuxfabrik.ch +short > /dev/null; then
        SUBJECT="$SUBJECT_PREFIX - System update failed"
        MSGBODY="DNS Server $DNS_SERVER failed to respond. Aborting update."
        send_msg
        exit 1
    fi
  }
  check_dns 192.0.2.10
  check_dns 192.0.2.11
system_update__rocketchat_msg_suffix: '@administrator'
system_update__rocketchat_url: 'https://chat.example.com/hooks/abcd1234'
system_update__security_enabled: true
system_update__security_on_calendar: '*-*-* 10,16:00'
system_update__security_reboot_time__group_var: '19:00'
system_update__security_repos:
  - 'security'
system_update__update_enabled: true
system_update__update_time: '04:{{ 59 | random(seed=inventory_hostname) }} + 1 days'

License

The Unlicense

Author Information

Linuxfabrik GmbH, Zurich