crowdsec/test/ansible

Ansible playbooks for functional testing

These playbooks allow you to test crowdsec in a real environment, where the application is running as root, deployed with the OS package manager, and uses the standard init system for the distribution (systemd or other).

This way, you can test not only the application's feature but also the packaging boilerplate, integration scripts, and compatibility with new distribution releases, operating systems, or architectures.

The ansible hosts should be expendable machines with at least 1GB RAM, do not expect them to be stable if you use them for anything else after the tests.

Install (or update) the requirements with ansible-galaxy install -r requirements.yml --force.

There are several Ansible playbooks. You can use run-all.yml to configure the installation and run the tests, or run the playbooks separately to iterate while developing.

• run-all.yml: run the other playbooks in the correct order.

• provision-dependencies.yml: install the bats requirements (bash, cfssl, etc.), compilers, and database.

• provision-test-suite.yml: install the tests scripts and bats environment, and the crowdsec sources if we want to build the crowdsec under test.

• install_binary_package.yml: install the crowdsec under test from a binary package (already released or not).

• prepare-tests.yml: create the test fixture data.

• run-tests.yml: run the functional tests. This is not idempotent and can be run multiple times.

The tasks use the following environment variables. They must be exported or ansible won't be able to see them.

• TEST_SUITE_GIT (default "https://github.com/crowdsecurity/crowdsec"), TEST_SUITE_VERSION (default "master"): repo URL and branch/tag/commit of the crowdsec sources containing the test fixture and scripts.

• TEST_SUITE_ZIP: optional, archive of a crowdsecurity/crowdsec repository containing the test fixture and scripts. Overrides TEST_SUITE_GIT and TEST_SUITE_VERSION. It can be created with zip -r crowdsec.zip . from the root directory of the repository.

• DB_BACKEND: Required. Set to "sqlite", "pgx", "mysql", "postgres". Postgres is automatically provisioned when required. There is no provisioning code for mysql/mariadb yet, but it can be added.

• PACKAGE_TESTING: when set to false or not defined, the crowdsec binaries to be tested are built from the sources that come from TEST_SUITE_GIT or TEST_SUITE_ZIP. Crowdsec is then run as non-root, in a local directory. This is basically a fancy wrapper to run make bats-test in a vm. When PACKAGE_TESTING is set to true, however, crowdsec is installed from a binary package (see variables below), it is run as root from systemd (or equivalent) and uses the system-wide /etc/crowdsec and /var/lib directories to store the test data.

• TEST_PACKAGE_VERSION_DEB, TEST_PACKAGE_VERSION_RPM: Optional, the version of the package under test (ex. "1.4.0-rc5"), can be in the packagecloud "stable" or "testing" repository. Requires PACKAGE_TESTING=true. You must set both variables to reuse the same set of variables for Debian and RedHat-based distributions, because stable releases require a package version suffix in the RPM file names.

• TEST_PACKAGE_FILE: optional, file pointing to the package under test (.deb, .rpm, .pkg...). It can be a glob expression but it must match a single file, and the pattern works only on the filename. If both TEST_PACKAGE_VERSION_* and TEST_PACKAGE_FILE are provided, both are be installed (to test upgrades for example). Requires PACKAGE_TESTING=true

• TEST_PACKAGE_DIR: optional (but conflicts with TEST_PACKAGE_FILE), the path to a directory containing packages with the following layout:

  For DEB: {{ package_dir }}/{{ ansible_distribution_release }}/crowdsec_*_{{ ansible_architecture.replace('x86_64', 'amd64') }}.deb For RPM: {{ package_dir }}/{{ releasever }}/RPMS/{{ ansible_architecture }}/crowdsec-*.{{ releasever }}.{{ ansible_architecture }}.rpm

• TEST_SKIP: optional, comma-separated list of scripts that won't be executed. Example: TEST_SKIP=02_nolapi.bats,03_noagent.bats

Running tests with Vagrant + Ansible

You don't need Vagrant to run the ansible tests, if you can manage your own vm creation and inventory.

However, to avoid relying on (and paying for..) a public cloud, we wrote vagrant configuration files for the most common distributions we support.

To test with Vagrant, you need to:

• have a working libvirt environment (if you can use virt-manager to create VMs, you're golden)

• install the vagrant-libvirt plugin (vagrant plugin install vagrant-libvirt. If it complains about gem versions, blame Ruby and see if you can remove some other conflicting plugin).

• copy one of the ./env/*.sh scripts to environment.sh, edit to your needs, and execute it with "source environment.sh"

• cd vagrant/<distro-of-your-choice>

• vagrant up --no-provision; vagrant provision. The first command creates the VM, the second installs all the dependencies, test suite and package under test, then runs the tests. If you run a plain vagrant up, it does everything with a single command, but also destroys the VM in case of test failure so you are left with nothing to debug.

• vagrant destroy when you want to remove the VM. If you want to free up the space taken by the base VM images, they are in /var/lib/libvirt/images/*VAGRANT*

The above steps are automated in the script ./prepare-run (requires bash

=4.4). It takes an environment file, and optionally a list of directories with vagrant configurations. With a single parameter, it loops over all the directories in alphabetical order, excluding those in the experimental directory. Watch out for running VMs if you break the loop by hand.

After this, you will find up to 30GB of base images in /var/lib/libvirt/images, which you need to remove by hand when you have finished testing or leave them around for the next time.

You can give more memory or CPU juice to the VMs by editing Vagrantfile.common.

Test Matrix

Tests fail with unsupported configurations or when the environment is not prepared correctly due to missing setup/teardown parts in Ansible or functional tests. False positives are also possible due to timing issues or flaky network connections.

If you have a result that deviates from the following matrix, that's probably a genuine bug or regression. The data was created with crowdsec v1.4.1.

	source/sqlite	pkg/sqlite	source/postgres	source/pgx	source/mysql (0)
AmazonLinux 2	✓ (1)	✓ (1)	old-db	old-db	wip
CentOS 7	✓	✓	old-db	old-db	✓
CentOS 8	✓	✓	✓	✓	✓
CentOS 9	✓	✓	✓	✓	✓
Debian 9 (stretch)	✓	✓	old-db	old-db	wip
Debian 10 (buster)	✓	✓	✓	✓	✓
Debian 11 (bullseye)	✓	✓	✓	✓	✓
Debian (testing/bookworm)	✓	✓	✓	✓	wip
Fedora 33	✓	✓	wip	wip	wip
Fedora 34	✓	✓	✓	✓	wip
Fedora 35	✓	✓	✓	✓	wip
Fedora 36	✓	✓	✓	✓	wip
FreeBSD 12	✓	wip	wip	wip	wip
FreeBSD 13	✓	wip	wip	wip	wip
Oracle 7	✓	✓	old-db	old-db	✓
Oracle 8	✓	✓	✓	✓	✓
Ubuntu 16.04 (xenial)	✓	✓	old-db	old-db	✓
Ubuntu 18.04 (bionic)	✓	✓	✓	✓	✓
Ubuntu 20.04 (focal)	✓	✓	✓	✓	✓
Ubuntu 22.04 (jammy)	✓	✓	✓	✓	✓

Note: all tests with local/<database> are expected to pass for pkg/<database> as well.

wip - missing ansible or bats parts, could be fixed in a future release

old-db - the database that ships with the distribution is not supported (Postgres < 10). Won't fix, feel free to install the DB from an unofficial repository.

0 - MySQL or MariaDB, depending on distribution defaults

1 - ansible may hang, passes all tests if run by hand