Tests ===== Efficient and thorough testing is key to ensure bpfilter is stable and reliable. As manual testing is time-consuming and error-prone, bpfilter relies on automated tests to validate its features. Different type of tests are implemented to cover all the components of the project: - Unit tests to test every function part of the public API of ``libbpfilter``. - End-to-end tests to validate ``bfcli``'s language, and the behaviour of the BPF programs generated by ``bpfilter``. - Kernel tests to run the end-to-end suite in virtme-ng with specific kernel releases. - Integration tests to ensure ``libbpfilter`` and the core module can be integrated into other projects without issues. - Check tests to validate code style and quality. bpfilter uses CMake and CTest to manage and run its tests, a test suite is defined for each type of test. You can run the tests using: .. code-block:: shell # Build and run all tests make -C $BUILD_DIR test -j5 # Run a specific test suite (unit, e2e, integration, check) make -C $BUILD_DIR $TEST_SUITE # Run a single test: use the test's path under tests/, replacing / with . # e.g. tests/e2e/matchers/ip4_daddr.cpp becomes e2e.matchers.ip4_daddr ctest --test-dir $BUILD_DIR -R $TEST_NAME --output-on-failure The ``test`` target and the suite targets will run as many tests at once as ``make`` runs jobs: ``-j5`` runs up to five tests concurrently, a bare ``-j`` uses all the CPUs, and ``-j1`` (or no ``-j`` at all) runs them one at a time. The job count can also come from ``MAKEFLAGS``, but ``-j`` on the command line takes precedence: with ``MAKEFLAGS=-j8`` in your environment, ``make test -j1`` still runs one test at a time. This needs the Unix Makefiles generator. Each shell end-to-end test invocation uses unique temporary directories, network namespace names, interface names, and cgroup paths. Privileged shell tests also run in separate host network and mount namespaces, so their routes and addresses do not conflict. The harness removes these resources when the test ends. C++ end-to-end tests use separate directories in bpffs, so flushing one test's ruleset cannot remove another test's chains. The group fixtures flush the ruleset, tear down the context, and remove the directory when the test ends. Test coverage is disabled by default, to enable it, configure CMake with the ``-DENABLE_COVERAGE=1`` flag. Then, you can generate a coverage report using: ``make -C $BUILD_DIR coverage doc``. Unit tests ---------- Unit tests are limited to the libbpfilter library. They are designed to validate the different components of the library in isolation, using mocks and fakes when necessary. We expect every public function of the library to be covered by at least one unit test. **Adding a new unit test** 1. Create a new source file to contain the unit test under ``tests/unit``. You can use existing unit tests as examples. Usually, a source file in ``tests/unit`` will contain tests for a single source file in ``libbpfilter``. 2. Add the new source file to the ``tests/unit/CMakeLists.txt`` file using the ``bf_add_c_test`` function. 3. Implement the unit tests using the ``cmocka`` testing library. You can use existing unit tests as examples. CMocka supports test fixtures to setup and teardown common test data. You can use them to avoid code duplication when multiple tests require the same setup. **Example** The example below is used to test the ``bf_version()`` function from ``libbpfilter``. All the tests source file must include the ``test.h`` header, define a main function, and register the tests to be run. .. code-block:: c /* SPDX-License-Identifier: GPL-2.0-only */ /* * Copyright (c) 2023 Meta Platforms, Inc. and affiliates. */ #include #include "test.h" static void get_version(void **state) { (void)state; assert_non_null(bf_version()); } int main(void) { const struct CMUnitTest tests[] = { cmocka_unit_test(get_version), }; return cmocka_run_group_tests(tests, NULL, NULL); } End-to-end tests ---------------- End-to-end tests are designed to validate bpfilter's behaviour as seen by the user. They cover the entire stack, from ``bfcli`` to the BPF programs generated by ``bpfilter``. End-to-end tests ensure that the filtering rules defined using ``bfcli`` are correctly translated into BPF programs, and that these programs behave as expected when executed. **Adding a new end-to-end test** 1. Create a new source file to contain the end-to-end test under ``tests/e2e``. 2. Call ``bf_add_e2e_shell_test()`` in ``tests/e2e/CMakeLists.txt`` to add the new source file to the end-to-end test suite. 3. In your test, use the functions provided by ``e2e_test_util.sh`` to setup the test environment. Any shell command failure will stop the test immediately and return a failure. **Example** ``e2e_test_util.sh`` provides functions to create a sandboxed environment. Here is an example of a simple end-to-end test that creates a sandbox: .. code-block:: shell #!/usr/bin/env bash set -eux set -o pipefail . "$(dirname "$0")"/../e2e_test_util.sh make_sandbox # Ping the sandbox's IPv4 address from the sandboxed namespace ${FROM_NS} ping -c 1 -W 0.1 ${NS_IP_ADDR} Matcher tests ~~~~~~~~~~~~~ Matcher tests (``e2e.matchers.*``) validate that the BPF programs generated by ``bpfilter`` correctly match packets. They use ``BPF_PROG_TEST_RUN`` to run the generated programs against crafted packet buffers in kernel space, without requiring network namespaces or real traffic. Each matcher test is a C++ binary located under ``tests/e2e/matchers/`` that builds chains with specific matchers, crafts packets using the ``bft::Packet`` builder, and asserts the BPF program's verdict using ``bft_assert_prog_run()``. Tests are automatically parameterized across all hooks that support the matcher under test using the ``MatcherTestsSuite`` class. **Adding a new matcher test** 1. Create a new C++ source file under ``tests/e2e/matchers/`` (e.g. ``tests/e2e/matchers/my_matcher.cpp``). 2. Add a call to ``bf_add_e2e_c_test()`` in ``tests/e2e/CMakeLists.txt`` with your new test. 3. Use ``bf::Chain``, ``bf::Rule``, ``bf::Matcher`` from ``tests/harness/`` to build chains, and ``bft::Ethernet``, ``bft::IPv4``, ``bft::TCP``, etc. from ``tests/harness/Packet.hpp`` to craft packets. 4. Use ``MatcherTestsSuite`` from ``tests/harness/test.hpp`` to automatically run the test across all supported hooks. **Example** .. code-block:: cpp #include "Chain.hpp" #include "Matcher.hpp" #include "Rule.hpp" #include "test.hpp" extern "C" { #include } static void my_matcher_eq(void **state) { auto *test = static_cast(*state); BFT_CHAIN_SET( bf::Chain("test_chain", test->hook(), BF_VERDICT_ACCEPT) << bf::Rule(BF_VERDICT_DROP, bf_counter(), {}, {bf::Matcher(BF_MATCHER_MY_TYPE, BF_MATCHER_EQ, {42})})); // Packet matching the rule -> DROP bft_assert_prog_run( "test_chain", test->hook(), bft::Ethernet() / bft::IPv4 {.saddr = "127.0.0.1", .daddr = "127.0.0.2"} / bft::TCP {.dport = 42}, test->verdictDrop()); // Packet not matching -> ACCEPT (policy) bft_assert_prog_run( "test_chain", test->hook(), bft::Ethernet() / bft::IPv4 {.saddr = "127.0.0.1", .daddr = "127.0.0.2"} / bft::TCP {.dport = 80}, test->verdictAccept()); } int main() { auto suite = MatcherTestsSuite(BF_MATCHER_MY_TYPE); suite << MatcherTest(BF_MATCHER_MY_TYPE, BF_MATCHER_EQ, my_matcher_eq); return suite.run(); } Kernel tests ------------ Kernel tests run the existing end-to-end tests in virtual machines, using virtme-ng. Each test gets its own VM, so we can check bpfilter against different kernel versions and configurations without rebooting the host. Enable kernel tests ~~~~~~~~~~~~~~~~~~~ You will need virtme-ng, QEMU, the dependencies to :doc:`build bpfilter `, and the tools to build a kernel. Use a clone of the Linux stable repository, with ``origin`` pointing to that repository: .. code-block:: shell $ git clone https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git /path/to/linux $ cmake -S . -B build -DTESTS_E2E_KERNEL_PATH=/path/to/linux Already have a stable clone? Use its path instead. CMake queries ``origin`` for the requested releases and fetches missing commits. It leaves your current branch alone and creates a detached worktree for each kernel build. Kernel tests are disabled when ``TESTS_E2E_KERNEL_PATH`` is empty. They also require ``WITH_TESTS_E2E`` to be enabled. Run the tests ~~~~~~~~~~~~~ Let's list the kernel tests before running them: .. code-block:: shell $ ctest --test-dir build -L '^kernel$' -N Each native test has a kernel counterpart. For example, ``e2e.rules.gre`` becomes ``kernel.e2e.rules.gre[v6.6.161]`` when run on Linux 6.6.161. Use the version printed by CTest in the commands below. .. code-block:: shell # Run all kernel tests. $ ctest --test-dir build -L '^kernel$' --parallel "$(nproc)" --output-on-failure # Run all tests on one kernel version, using its label. $ ctest --test-dir build -L '^v6\.6\.161$' --output-on-failure # Run one test on one kernel. Escape brackets and dots in the name. $ ctest --test-dir build -R '^kernel\.e2e\.rules\.gre\[v6\.6\.161\]$' --output-on-failure CTest builds bpfilter, the test binaries, and the selected kernels before starting the VMs. There is no separate build command to remember. You can also run the whole kernel suite with ``make -C build kernel``. The ``e2e`` label still selects tests running on the host. Choose the kernels ~~~~~~~~~~~~~~~~~~ Add a call to ``bf_add_kernel_tests`` in ``tests/kernel/CMakeLists.txt``, then reconfigure CMake: .. code-block:: cmake bf_add_kernel_tests(6.6) bf_add_kernel_tests(6.12) A series such as ``6.6`` selects its latest stable patch release at ``origin``. If it has no stable release yet, we use its latest release candidate. An exact version, such as ``6.6.161`` or ``7.3-rc7``, selects that release. For example, replace ``bf_add_kernel_tests(6.6)`` with ``bf_add_kernel_tests(6.6.161)`` to pin that version. Reconfigure CMake to pick up newer patch releases. Register a release only once unless you give each additional configuration a name. Change kernel options or boot arguments ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ To try a different configuration, add a named variant. For example, to run the 6.6 tests with confidentiality lockdown: .. code-block:: cmake bf_add_kernel_tests(6.6 NAME lockdown KERNEL_CONFIG +CONFIG_SECURITY +CONFIG_SECURITYFS +CONFIG_SECURITY_LOCKDOWN_LSM +CONFIG_SECURITY_LOCKDOWN_LSM_EARLY BOOT_ARGUMENT lockdown=confidentiality loglevel=7 ) ``KERNEL_CONFIG`` overrides ``tests/kernel/kernel.config``. Prefix an option with ``+`` to enable it or ``-`` to disable it. You can omit ``CONFIG_``. ``BOOT_ARGUMENT`` adds arguments to the kernel command line through virtme-ng's ``--append``. The harness passes them unchanged: the test must check any runtime state it relies on. Reconfigure CMake, then select the variant by name: .. code-block:: shell $ cmake -S . -B build $ ctest --test-dir build -L '^v6\.6\.[0-9]+-lockdown$' --output-on-failure The variant becomes part of the version: ``v6.6.161-lockdown``. Its tests are named ``kernel.e2e.rules.gre[v6.6.161-lockdown]`` and carry the ``v6.6.161-lockdown`` label. It has its own worktree and build alongside ``v6.6.161``, so both configurations can be registered together. The kernel release also gets the ``-lockdown`` suffix through ``LOCALVERSION``. What gets built, and where? ~~~~~~~~~~~~~~~~~~~~~~~~~~~ CTest uses two setup fixtures for each VM test: .. list-table:: :header-rows: 1 :widths: 40 60 * - Fixture - Builds * - ``build.e2e`` - bpfilter and the end-to-end binaries, once per CTest run. * - ``kernel.build[vVERSION]`` - The selected kernel, once per CTest run. Named variants use ``kernel.build[vVERSION-NAME]``. The test starts after both fixtures succeed. A failed kernel build prevents its VM tests from running. .. code-block:: text build/kernel/ v6.6.161/ sources/ # Detached Git worktree build/ # Kernel configuration and build outputs v6.6.161-lockdown/ sources/ # Separate worktree at the same kernel commit build/ # Variant configuration and build outputs The builder reuses these directories and runs an incremental build. Changing the configuration fragment or its overrides triggers configuration again. Kconfig must accept the requested settings, otherwise the build fixture fails. A clean, detached worktree can move to another commit; local edits prevent that update. The shared configuration enables the BPF hooks, BTF, namespaces, cgroups, and network devices used by the tests. Enable lockdown explicitly in a named variant. Each VM has one virtual CPU and 1 GiB RAM. The harness mounts bpffs and cgroup2, then runs the selected native test with the prebuilt binaries. The tests still create their own sandboxes inside the VM. Guest output is saved to ``build/vng-logs/TIMESTAMP/guest.log``. The harness returns the VM's exit status, including failures reported by CTest. Run the tools directly ~~~~~~~~~~~~~~~~~~~~~~ Usually, CTest takes care of building and running everything. If you need to build a kernel separately: .. code-block:: shell $ tests/kernel/buildkernel --sources /path/to/linux \ --revision v6.6.161 --output build/kernel/v6.6.161 The revision must exist locally. The builder uses virtme-ng to configure it, then ``make -j"$(nproc)"`` to build it. It prints the build directory to stdout and progress to stderr. Use ``--config +OPTION`` or ``--config -OPTION`` for configuration overrides, and ``--localversion SUFFIX`` for a version suffix (empty by default). Concurrent builds in the same directory are rejected. To run one prebuilt end-to-end test on that kernel: .. code-block:: shell $ tests/kernel/harness --kernel build/kernel/v6.6.161/build \ --build-dir build --test e2e.rules.gre The harness builds nothing. Use ``--memory`` and ``--cpus`` to change the VM's resources, ``--boot-argument`` for each extra kernel argument, and ``--log-dir`` to choose where to save output. ``--dry-run`` prints the command without starting a VM. Integration tests ----------------- Integration tests are designed to ensure that ``libbpfilter`` and the core ``bpfilter`` module can be integrated into other projects without issues. They validate that the public API of ``libbpfilter`` is stable and behaves as expected when used in different contexts. Check tests ----------- Check tests leverage the ``clang-tidy`` and ``clang-format`` tools to ensure code quality and style consistency across the codebase. They help identify potential issues early in the development process. Any warning or error reported by these tools will cause the check tests to fail. Harness ------- The test harness is a set of convenience functions used to ease testing of bpfilter. Test ~~~~ .. doxygenfile:: test.h Mocks ~~~~~ .. doxygenfile:: mock.h Fake ~~~~ .. doxygenfile:: fake.h