Testing

FirewallFabrik uses expected output regression tests to catch unintended changes in compiler output. Each test compiles a fixture (.fwf or .fwb) with the iptables and/or nftables drivers, normalizes the output, and compares it against a checked-in expected output file.

Quick Start

# Install the package with GUI support and test dependencies
pip install --editable ".[gui]"
pip install pytest

# Run all tests
pytest --verbose

# Run only iptables or nftables tests
pytest --verbose tests/test_compiler_ipt.py
pytest --verbose tests/test_compiler_nft.py

Directory Structure

tests/
├── conftest.py                          # Shared fixtures (compile helpers, test case discovery)
├── normalize.py                         # Output normalization (strip timestamps, chain hashes, etc.)
├── update_expected_output.py            # Script to regenerate expected output files
├── test_addr_contains.py                # Address containment / shadowing detection unit tests
├── test_compiler_ipt.py                 # iptables expected output tests
├── test_compiler_nft.py                 # nftables expected output tests
├── test_interface_autoconfigure.py      # Interface name pattern recognition unit tests
├── test_load_save.py                    # Database load/save round-trip tests
├── test_store_action_processor.py       # Rule processor unit tests
├── fixtures/                            # Test data (one per feature or test suite)
│   ├── basic_accept_deny.fwf            # Hand-crafted YAML fixture
│   ├── cluster-tests.fwb               # C++ cluster regression suite (XML)
│   ├── compiler-tests.fwf              # Hand-crafted YAML fixture
│   ├── addr-table-1.tbl                # Address table data, referenced by .fwb fixtures
│   ├── block-hosts.tbl                 # Address table data, referenced by .fwb fixtures
│   ├── emtpy-table.tbl                 # Address table data, referenced by .fwb fixtures
│   ├── objects-for-regression-tests.fwb # C++ Firewall Builder regression suite (XML)
│   ├── optimizer-test.fwb              # C++ optimizer regression suite (XML)
│   └── reject_actions.fwf             # Hand-crafted YAML fixture
├── expected-output/
│   ├── ipt/                             # Expected iptables output (normalized)
│   │   ├── basic_accept_deny/
│   │   │   └── fw-test.fw
│   │   ├── cluster-tests/              # 32 expected output files, 17 of them the C++ reference
│   │   ├── objects-for-regression-tests/ # 119 C++ reference expected output files
│   │   ├── optimizer-test/             # 2 expected output files, neither a C++ reference
│   │   └── ...
│   └── nft/                             # Expected nftables output (normalized)
│       └── basic_accept_deny/
│           └── fw-test.fw

Unit Tests

In addition to compiler output regression tests, FirewallFabrik includes unit tests for core logic:

• test_addr_contains.py — Tests the _addr_contains() and _addr_range() functions used by the shadowing detector to determine whether one address object is a superset of another. Covers IPv4/IPv6 hosts, networks, and address ranges.
• test_interface_autoconfigure.py — Tests guess_interface_type() which detects interface types (ethernet, VLAN, bridge, bonding) from names and parent context. Covers top-level and sub-interface patterns, VLAN name validation, and bridge/bonding slave detection.
• test_store_action_processor.py — Tests the StoreAction rule processor for iptables and nftables.

How It Works

1. Fixtures (tests/fixtures/*.fwf, tests/fixtures/*.fwb) are firewall databases that exercise specific compiler features. .fwf files are hand-crafted YAML; .fwb files are XML from the C++ Firewall Builder.
2. Tests are auto-discovered from the expected output directory structure. For each tests/expected-output/<platform>/<fixture_name>/<fw_name>.fw, a parametrized test case is generated.
3. The test compiles the fixture, passes the output through a normalize function (which replaces timestamps, version strings, and iptables chain name hashes with stable placeholders), and compares the result against the expected output file.
4. Expected output files are stored in their normalized form, so they are deterministic and diff-friendly.

Normalization

The normalize_ipt() and normalize_nft() functions in tests/normalize.py replace:

Pattern	Replacement	Reason
# Generated <timestamp>	# Generated TIMESTAMP	Varies per run
# Firewall Builder fwb_ipt v<version>	# Firewall Builder fwb_ipt VERSION	Varies per release
log "Activating firewall script generated ..."	...TIMESTAMP...	Varies per run
C<hex>.<N> (iptables chain names)	CHAIN	Hash-based, varies per run
Trailing whitespace	Stripped	Irrelevant noise

Adding Tests for a New Feature

1. Create a fixture

Create a new .fwf file in tests/fixtures/. Name it after the feature being tested (e.g., nat_snat_dnat.fwf).

A fixture is a standard FirewallFabrik YAML database.

2. Generate expected output files

python tests/update_expected_output.py --fixture nat_snat_dnat

This compiles the fixture with both iptables and nftables, normalizes the output, and saves it under tests/expected-output/{ipt,nft}/nat_snat_dnat/fw-test.fw.

3. Review the expected output

Manually inspect the generated expected output files to verify the compiler output is correct. This is the most important step — expected output files encode what "correct" means.

4. Commit

Commit the fixture and expected output files together. From now on, any change to the compiler that alters the output for this feature will cause the test to fail.

5. Verify

pytest --verbose

Updating Expected Output Files

When you intentionally change compiler output (e.g., fixing a bug, adding a feature), recompile and update the expected output files:

# Recompile all expected output files from fixtures
python tests/update_expected_output.py

# Recompile only one fixture
python tests/update_expected_output.py --fixture basic_accept_deny

# Recompile only one platform
python tests/update_expected_output.py --platform nft

# Combine both
python tests/update_expected_output.py --fixture basic_accept_deny --platform ipt

Always review the diff (git diff tests/expected-output/) before committing to confirm the changes are intentional.

Normalizing Existing Expected Output Files

If you have pre-existing .fw files (e.g., from the C++ Firewall Builder compiler or a manual compilation run) that you want to use as expected output files, you can import and normalize them:

1. Copy the files into the appropriate expected output directory:

   bash mkdir --parents tests/expected-output/ipt/my_feature/ cp /path/to/reference/fw-test.fw tests/expected-output/ipt/my_feature/

2. Run --normalize-only to normalize them in-place:

   ```bash

   Normalize all expected output files across both platforms

   python tests/update_expected_output.py --normalize-only

   Normalize only a specific fixture

   python tests/update_expected_output.py --normalize-only --fixture my_feature

   Normalize only a specific platform

   python tests/update_expected_output.py --normalize-only --platform ipt ```

This applies the same normalization (timestamp/version/chain-hash replacement, trailing whitespace stripping) that the test runner applies to compiler output, so the comparison will match.

The script reports which files were modified and skips files that are already normalized, and normalizing a file twice changes nothing - test_normalize_is_idempotent.py asserts both, over every checked-in expected output file. Without that the two writers disagree: a file update_expected_output.py has just produced is changed by the next --normalize-only run, and re-importing the Firewall Builder reference then moves it away from what the C++ compiler produced.

C++ Firewall Builder Regression Suite

Background

The C++ Firewall Builder project (fwbuilder) includes comprehensive iptables regression test suites with expected .fw output. These test suites were developed over many years and cover a wide range of iptables features.

We imported these test suites to serve as a compatibility target for the Python reimplementation. The expected output files represent the C++ compiler's output and define the behavior we aim to match.

Three C++ reference fixtures are currently imported:

Fixture	Expected output files	Of those, the C++ reference
objects-for-regression-tests	119	119
cluster-tests	32	17
optimizer-test	2	0

Not every file under those three directories is the C++ reference. A file is one when fwbuilder5/test/ipt/<name>.fw.orig exists; seventeen do not, and they are FirewallFabrik's own output from whenever they were added. They are xfailed with the rest of the fixture and update_expected_output.py refuses to regenerate the iptables output of a .fwb fixture, so they are frozen and guard nothing - read them as a record, not as a target. The fifteen in cluster-tests are the compiles Firewall Builder has no counterpart for: a cluster compiled as itself, which is fwf's own mode, and a member compiled without its cluster.

A cluster member is named after both objects

A cluster is compiled by compiling each of its members with the cluster named alongside, and both Firewall Builder and compile-corpus.py write the result as <cluster>_<member>.fw. The expected-output file therefore names two objects, and the test harness resolves it that way (_resolve_target in conftest.py): a name that is no firewall is split at a cluster name and the rest is looked up among that cluster's members. Without that the reference output for every cluster member could not be used at all, which is why it went unimported until the cluster port was already finished.

What It Covers

The firewalls in objects-for-regression-tests.fwb exercise:

• Basic policy rules: accept, deny, reject with various combinations of source, destination, service
• NAT: SNAT, DNAT, masquerade, redirect, port translation
• IPv6: dual-stack firewalls with ip6tables rules, neighbor discovery
• Services: TCP, UDP, ICMP, custom protocols, port ranges, multiport
• Addresses: single hosts, networks, address ranges, address tables, DNS names
• Interfaces: multiple interfaces, dynamic addresses, unnumbered interfaces, bridge/bond interfaces
• Rule options: logging, classification/tagging, routing marks, connection marking
• Advanced features: custom chains, rule branching, shadowing detection, multiple rule sets, prolog/epilog script insertion points
• Platform variants: different iptables versions (1.2.5, 1.2.6, 1.3.x, 1.4.x), IPCop, kernel versions

Current Status

All C++ reference tests (across all three fixtures) are marked xfail because the Python compiler does not yet produce identical output.

As the Python compiler is improved, individual tests will start passing. pytest reports these as XPASS (unexpected pass), signaling that the xfail marker can be removed and the test promoted to a proper passing test.

How to Track Progress

# Run all C++ reference regression tests
pytest --verbose -k "objects-for-regression-tests or cluster-tests or optimizer-test"

# See which tests unexpectedly pass (if any)
pytest -rX -k "objects-for-regression-tests or cluster-tests or optimizer-test"

The xfail comes from a marker on the test parameter, so the comparison still runs and a firewall whose output has converged shows up as XPASS. Marking it inside the test body instead would end the test before the comparison and make convergence invisible.

WARNING: Do Not Modify the iptables Expected Output for .fwb Fixtures

The iptables expected output files for objects-for-regression-tests, cluster-tests, and optimizer-test were compiled with the old, known-good C++ fwb_ipt compiler and must not be modified or regenerated.

These files are the ground truth for the Python iptables compiler. They define the correct behavior we are reimplementing. If you regenerate them with update_expected_output.py, you will overwrite the C++ reference with Python compiler output, which defeats the entire purpose of these regression tests.

Do: - Use these files as-is to validate the Python compiler against the C++ reference. - Regenerate expected output only for .fwf fixtures (e.g., compiler-tests, basic_accept_deny, reject_actions), whose expected output is produced by the Python compiler. - Regenerate nftables expected output and carefully review the changes — there is no C++ nftables reference compiler.

Do not: - Run update_expected_output.py --platform ipt on .fwb fixtures. - Manually edit the iptables .fw files under expected-output/ipt/objects-for-regression-tests/, expected-output/ipt/cluster-tests/, or expected-output/ipt/optimizer-test/. - Re-normalize these files (they are already normalized).

update_expected_output.py enforces the first point itself: it skips the iptables output of every .fwb fixture and says so, and exits non-zero if that was all you asked for. This rule was prose only until the reference was overwritten once and nobody noticed, because the tests never read the files back then.

To re-import the reference after a fwbuilder update, copy the matching <firewall>.fw.orig files from the fwbuilder tree (fwbuilder5/test/ipt/) into the expected output directory as <firewall>.fw and run:

python tests/update_expected_output.py --normalize-only --platform ipt --fixture objects-for-regression-tests

Limitation: Compiler Aborts Cannot Be Tested

The expected output regression framework calls pytest.fail() whenever the compiler produces errors (see _compile() in conftest.py). This means compiler aborts cannot be tested with the current framework — there is no way to assert that a specific firewall configuration should cause the compiler to abort.

This affects options whose primary behavior is to abort compilation on invalid input:

Option	Abort behavior	Testable path
firewall_is_part_of_any_and_networks	No abort — changes rule splitting	Yes: produces extra INPUT/OUTPUT chain rules
local_nat	No abort — adds NAT processors	Yes: produces OUTPUT chain NAT rules
ignore_empty_groups	false: aborts on empty groups	Only true: removes empty groups with warnings, produces output
check_shading	true: aborts on shadowed rules	Only false: no shadowing check, passes through

When writing test firewalls for these options, only test the non-abort code paths that produce output. Abort-path testing would require a separate test mechanism (e.g., a compile_expect_error fixture or standalone tests that call the driver directly and assert on driver.all_errors).

Note on nftables Expected Output

Unlike the iptables expected output for objects-for-regression-tests (which comes from the C++ Firewall Builder compiler and serves as a verified reference), the nftables expected output files are generated by our own Python compiler. There is no independent C++ nftables compiler to validate against.

This means the nft expected output files are regression tests only — they capture the current compiler behavior, not necessarily the correct behavior. If the nftables compiler has a bug at the time the expected output is generated, that bug is baked into the expected output.

When reviewing nft expected output files (especially newly created ones), pay extra attention to correctness. The expected output encodes "what the compiler currently produces", not "what is known to be correct".

Running the Output Through the Real Tools

The expected-output tests guard against changes in the compiler output, not against output that does not work. Three tests in the suite do ask the real tools, and each skips itself when it cannot, so the suite still runs without them:

Test	Asks	Needs
test_nft_check.py	Does every checked-in nftables expected output load? Each ruleset goes through nft --check in a private network namespace.	nft and unshare, and a host that grants an unprivileged user namespace
test_nft_identifiers.py	Is the list of nft keywords still true? Every name in it is offered to nft as a table, chain, set and counter name.	the same
test_ipt_version_gates.py	Did the compiler get the release right in which a match first shipped? Re-derived from the netfilter git history.	FWF_IPTABLES_SOURCE pointing at an iptables clone with tags
test_ipt_target_names.py	Is the list of names iptables refuses as a chain name still true? Re-derived from the extension files, and every name is offered to iptables and ip6tables.	FWF_IPTABLES_SOURCE for the first half, unshare and iptables for the second

nft --check cannot initialise its cache without CAP_NET_ADMIN, which is why those two run under unshare -rn: it gives a private network namespace where the parse and evaluation pass work in full.

Having the binaries is not enough. A hardened host, a container and the GitHub runner all refuse to write /proc/self/uid_map, and then every invocation fails for a reason that has nothing to do with what was asked - which reads as "nft refuses this" and turns the suite red. tests/tool_probe.py therefore tries once, per tool, something the tool has to accept: an empty ruleset for nft, an ordinary chain name for iptables. The tests skip unless that worked.

Add the same guard to any new test that reaches for unshare -rn, and make the probe assert a success, not a failure. A test that asks "does the tool refuse this?" cannot guard itself with the same question: a denied namespace looks exactly like an accepted name, so the guard reads green and every case fails.

That still leaves the iptables side and everything around the rules.

tools/compiler-audit/ closes that gap. It compiles a corpus and then asks the real tools whether the result is any good: nft --check on every nftables ruleset and then a real nft -f load of it, a replay of every iptables command in an unprivileged private network namespace, iptables-restore --test on the restore form, and bash -n on the script itself, every ip route command handed to iproute2 in a namespace with a dummy interface per device, and the shell code that fills the named sets an address table, a run-time DNS name or a dynamic interface stands for - run for real, against a loaded ruleset, because a set that stays empty is a set no packet is in. The load is a separate oracle from the check on purpose: nft --check stops after parsing and evaluating, so everything the kernel decides - which statements a hook allows, whether the jumps between the chains form a cycle - is invisible to it, and a ruleset it accepts can still be refused whole. It also compares the iptables output against the Firewall Builder reference and measures which firewalls a change actually affects, which is what a release note needs.

Compiling the corpus with a firewall option forced on every firewall is worth a run of its own, and the result has to go through every oracle rather than the one that motivated it: about a fifth of each print rule is never reached by the corpus as it stands, and two of the twenty-ninth round's four fixes came out of exactly that - one from the iptables replay of a corpus compiled with "Log all rules" on, the other from nft --check on one with thirteen options on.

It also compares three compiles of the same corpus against each other: the IPv4 rules of a firewall compiled with -4 have to be the IPv4 rules the ordinary run produces, and the same for -6. That switch is meant to be a filter and not a different compile, and nothing else here can see it being one - both halves parse, both load, and every set-comparing tool is handed one tree at a time.

It also runs the block every activation runs first and no other oracle touches - configure_interfaces - against real iproute2 in a namespace, and asks the three questions that block can fail: does iproute2 accept every command, is a second activation silent, and do the bridges end up with the ports the script named.

See tools/compiler-audit/README.md. These checks are not part of pytest: they need unshare, nft and iptables, run over a corpus rather than over the fixtures, and a full run takes minutes.

Investigating Failures

When an expected output test fails, the assertion message shows both file paths:

AssertionError: iptables output differs from expected output.
  Actual:   /tmp/pytest-xxx/test_.../fw-test.fw
  Expected: tests/expected-output/ipt/basic_accept_deny/fw-test.fw
Run "python tests/update_expected_output.py --fixture basic_accept_deny --platform ipt" to update.

To investigate:

# Diff the actual vs expected output
diff --unified tests/expected-output/ipt/basic_accept_deny/fw-test.fw /tmp/pytest-xxx/test_.../fw-test.fw

If the change is intentional, update the expected output file. If not, fix the regression.