Ansible Role linuxfabrik.lfops.duplicity¶

This role configures daily file-based backups using duplicity. Currently, this role is focused on using OpenStack Object Storage ("Swift") as the storage backend.

Note that this role does not support running with --check, as it first creates a GPG-Key which is required for the rest of the role.

duba (Duplicity Backup)¶

The role comes with the special Python wrapper script duba for duplicity, implemented by Linuxfabrik. The script currently does a massive parallel backup to a Swift storage backend with duplicity, where the number of duplicity processes is min(processor count, 6) + 1. The script's configuration file is located at /etc/duba/duba.json.

To start a backup, simply call duba (or duba --config=/etc/duba/duba.json --command=backup). Have a look at duba --help for details.

Mandatory Requirements¶

On RHEL-compatible systems, enable the EPEL repository. This can be done using the linuxfabrik.lfops.repo_epel role.
Install duplicity, python-swiftclient and python-keystoneclient into a Python 3 virtual environment in /opt/python-venv/duplicity. This can be done using the linuxfabrik.lfops.python_venv role.

Attention

Make sure the virtual environment is not writable by other users to prevent privilege escalation. This is also done by the linuxfabrik.lfops.python_venv role.

Optional Requirements¶

Create a symbolic link from /opt/python-venv/duplicity/bin/duplicity to /usr/local/bin/duplicity for easier usage on the command line.
Either configure journald to persist your logs and do the rotating, or use logrotated.

Tags¶

duplicity

Installs and configures duplicity.
Triggers: none.

duplicity:configure

Deploys the configuration for duplicity.
Triggers: none.

duplicity:script

Just deploys the duba script.
Triggers: none.

duplicity:state

Manages the state of the daily systemd timer.
Triggers: none.

Mandatory Role Variables¶

duplicity__gpg_encrypt_master_key_block

The ASCII-armored public master GPG key. Obtain it using gpg --armor --export $GPG_KEY. This key is imported on the server and is used in addition to the server's own local GPG key to encrypt the backups. This means that the backups can be restored using either the master or the server's local private key (which is pretty cool in case of a desaster recovery). Be aware of the empty line between -----BEGIN PGP PUBLIC KEY BLOCK----- and your public key block.
Type: String.

duplicity__gpg_encrypt_master_key

The long key ID of the master GPG key. Obtain it using gpg --list-keys --keyid-format=long (after importing the key) or gpg /path/to/keyfile.
Type: String.

duplicity__swift_login

The Swift username and password. Usually, this is given by the provider of the Swift Storage.
Type: Dictionary.
Subkeys:
- username:
  - Mandatory. The Swift username.
  - Type: String.
- password:
  - Mandatory. The Swift password.
  - Type: String.

Example:

# mandatory
duplicity__gpg_encrypt_master_key_block: |-
  -----BEGIN PGP PUBLIC KEY BLOCK-----

  6ec3d2aed2a54122817ca02b43a7e340kgKEdlbmVyYXRlZCBieSBBbnNpYmxlLi
  ...
  -----END PGP PUBLIC KEY BLOCK-----
duplicity__gpg_encrypt_master_key: 'LLZGH2BITI2LRLJCLFWEAJQ93N6MWTKBARQDMYX5'
duplicity__swift_login:
  username: 'SBI-MF827483'
  password: 'linuxfabrik'

Optional Role Variables¶

duplicity__backup_backend

The backup backend being used. Possible options: swift, sftp.
Type: String.
Default: 'swift'

duplicity__backup_dest_container

The Swift container. This can be used to separate backups on the destination. By default, this will be used in duplicity__backup_dest.
Type: String.
Default: '{{ ansible_nodename }}'

duplicity__backup_dest

The backup destination. This will be used in combination with the backup source path to create the target URL for duplicity.
Type: String.
Default: 'swift://{{ duplicity__backup_dest_container | regex_replace("/$", "") }}'

duplicity__backup_full_if_older_than

After how long a full backup instead of a incremental one should be done. Time Formats: s, m, h, D, W, M, or Y.
Type: String.
Default: '30D'

duplicity__backup_retention_time

The retention time of the backups. Time Formats: s, m, h, D, W, M, or Y.
Type: String.
Default: '30D'

duplicity__backup_sources__host_var / duplicity__backup_sources__group_var

List of dictionaries with directories to backup.
Type: List of dictionaries.
Default:
- /backup
- /etc
- /home
- /opt
- /root
- /var/spool/cron
Subkeys:
- path:
  - Mandatory. Path to the folder to be backed up.
  - Type: String.
- divide:
  - Optional. Whether to split a large directory at its first level to perform parallel backups. Imagine a computer with 4 processor cores and the folder data containing 100 files and folders. If divide is set to true, duba will start and control 5 duplicate processes at once to speed up the backup process by almost a factor of 5.
  - Type: Bool.
  - Default: false
- excludes:
  - Optional. List of patterns that should not be included in the backup for this path.
  - Type: List of strings.
  - Default: []
- state:
  - Optional. Either present or absent.
  - Type: String.
  - Default: 'present'

duplicity__excludes

List of global exclude shell patterns for duplicity. Have a look at man duplicity for details.
Type: List of strings.
Default: ['**/*.git*', '**/*.svn*', '**/*.temp', '**/*.tmp', '**/.cache', '**/cache', '**/log']

duplicity__loglevel

Set the loglevel. Possible options: error, warning, notice, info, debug.
Type: String.
Default: 'notice'

duplicity__logrotate

Log files are rotated count days before being removed or mailed to the address specified in a logrotate mail directive. If count is 0, old versions are removed rather than rotated. If count is -1, old logs are not removed at all (use with caution, may waste performance and disk space).
Type: Number.
Default: {{ logrotate__rotate | d(14) }}

duplicity__on_calendar_hour

A shorthand to set the hour of duplicity__on_calendar.
Type: String.
Default: '23'

duplicity__on_calendar

The OnCalendar definition for the daily systemd timer. Have a look at man systemd.time(7) for the format.
Type: String.
Default: '*-*-* {{ duplicity__on_calendar_hour }}:{{ 59 | random(seed=inventory_hostname) }}'

duplicity__sftp_password

Password for SSH User that is used by SFTP connection.
Type: String.
Default: unset

duplicity__swift_authurl

The Authentication URL for Swift. Usually, this is given by the provider of the Swift Storage.
Type: String.
Default: 'https://swiss-backup02.infomaniak.com/identity/v3'

duplicity__swift_authversion

The Authentication Version for Swift. Usually, this is given by the provider of the Swift Storage.
Type: String.
Default: '3'

duplicity__swift_tenantname

The Swift Tenantname. Usually, this is given by the provider of the Swift Storage.
Type: String.
Default: 'sb_project_{{ duplicity__swift_login["username"] }}'

duplicity__timer_enabled

The state of the daily systemd timer.
Type: Bool.
Default: true

Example:

# optional
duplicity__backup_dest: 'swift://{{ duplicity__backup_dest_container | regex_replace("/$", "") }}'
duplicity__backup_dest_container: '{{ ansible_nodename }}'
duplicity__backup_full_if_older_than: '30D'
duplicity__backup_retention_time: '30D'
duplicity__excludes:
  - '**/*.git*'
  - '**/*.svn*'
  - '**/*.temp'
  - '**/*.tmp'
  - '**/.cache'
  - '**/cache'
  - '**/log'
duplicity__backup_sources__group_var: []
duplicity__backup_sources__host_var:
  - path: '/var/www/html'
    divide: false
    excludes:
      - '/var/www/html/nextcloud/data'
    state: 'present'
  - path: '/var/www/html/nextcloud/data'
    divide: true
    state: 'present'
  - path: '/backup'
    state: 'absent'
duplicity__loglevel: 'notice'
duplicity__logrotate: 7
duplicity__on_calendar: '*-*-* {{ duplicity__on_calendar_hour }}:{{ 45 | random(seed=inventory_hostname) }}'
duplicity__on_calendar_hour: '23'
duplicity__swift_authurl: 'https://swiss-backup02.infomaniak.com/identity/v3'
duplicity__swift_authversion: '3'
duplicity__swift_tenantname: 'sb_project_SBI-MF827483'
duplicity__timer_enabled: true

Troubleshooting¶

If the gpg --import /tmp/public-master-key task fails with gpg: invalid armor header in stderr, make sure your duplicity__gpg_encrypt_master_key_block is correct and has an empty line after the -----BEGIN PGP PUBLIC KEY BLOCK-----.

If duplicity fails with AttributeError: module 'collections' has no attribute 'Mapping' in /opt/python-venv/duplicity/lib64/python3.11/site-packages/oslo_config/cfg.py, manually install 'oslo.config>=9', e.g. /opt/python-venv/duplicity/bin/pip install 'oslo.config>=9'.

License¶

The Unlicense

Author Information¶

Linuxfabrik GmbH, Zurich