2022-07-26 11:09:13 +00:00
|
|
|
# Ansible playbooks for functional testing
|
|
|
|
|
|
|
|
These playbooks allow you to test crowdsec in a real environment, where the
|
|
|
|
application is running as root, deployed with the OS package manager, and
|
|
|
|
uses the standard init system for the distribution (systemd or other).
|
|
|
|
|
|
|
|
This way, you can test not only the application's feature but also the packaging
|
|
|
|
boilerplate, integration scripts, and compatibility with new distribution releases,
|
|
|
|
operating systems, or architectures.
|
|
|
|
|
|
|
|
The ansible hosts should be expendable machines with at least 1GB RAM, do not
|
|
|
|
expect them to be stable if you use them for anything else after the tests.
|
|
|
|
|
|
|
|
Install the requirements with `ansible-galaxy install -r requiements.yml`.
|
|
|
|
|
|
|
|
There are several Ansible playbooks. You can use `run-all.yml` to configure the
|
|
|
|
installation and run the tests, or run the playbooks separately to iterate while developing.
|
|
|
|
|
|
|
|
- run-all.yml: run the other playbooks in the correct order.
|
|
|
|
|
|
|
|
- provision-dependencies.yml: install the bats requirements (bash, netcat, cfssl, etc.), compilers, and database.
|
|
|
|
|
|
|
|
- provision-test-suite.yml: install the tests scripts and bats environment.
|
|
|
|
|
|
|
|
- prepare-tests.yml: install the package under test, and create the test fixture data.
|
|
|
|
|
|
|
|
- run-tests.yml: run the functional tests. This is not idempotent and can be run multiple times.
|
|
|
|
|
|
|
|
The tasks use the following environment variables. They must be exported or
|
|
|
|
ansible won't be able to see them.
|
|
|
|
|
|
|
|
- `TEST_SUITE_GIT` (default "https://github.com/crowdsecurity/crowdsec"),
|
|
|
|
`TEST_SUITE_VERSION` (default "master"): repo URL and branch/tag/commit of
|
|
|
|
the crowdsec sources containing the test fixture and scripts.
|
|
|
|
|
|
|
|
- `TEST_SUITE_ZIP`: optional, archive of a `crowdsecurity/crowdsec` repository
|
|
|
|
containing the test fixture and scripts. Overrides `TEST_SUITE_GIT` and
|
|
|
|
`TEST_SUITE_VERSION`. It can be created with `zip -r crowdsec.zip .` from
|
|
|
|
the root directory of the repository.
|
|
|
|
|
|
|
|
- `DB_BACKEND`: Required. Set to "sqlite", "pgx", "mysql", "postgres".
|
|
|
|
Postgres is automatically provisioned when required. There is no
|
|
|
|
provisioning code for mysql/mariadb yet, but it can be added.
|
|
|
|
|
|
|
|
- `PACKAGE_TESTING`: when set to false or not defined, the crowdsec binaries
|
|
|
|
to be tested are built from the sources that come from `TEST_SUITE_GIT` or
|
|
|
|
`TEST_SUITE_ZIP`. Crowdsec is then run as non-root, in a local directory.
|
|
|
|
This is basically a fancy wrapper to run `make bats-test` in a vm.
|
|
|
|
When `PACKAGE_TESTING` is set to true, however, crowdsec is installed from
|
|
|
|
a binary package (see variables below), it is run as root from systemd (or
|
|
|
|
equivalent) and uses the system-wide `/etc/crowdsec` and `/var/lib`
|
|
|
|
directories to store the test data.
|
|
|
|
|
|
|
|
- `TEST_PACKAGE_VERSION_DEB`, `TEST_PACKAGE_VERSION_RPM`: Optional, the
|
|
|
|
version of the package under test (ex. "1.4.0-rc5"), can be in the
|
|
|
|
packagecloud "stable" or "testing" repository. Requires
|
|
|
|
`PACKAGE_TESTING=true`. You must set both variables to reuse the same set of
|
|
|
|
variables for Debian and RedHat-based distributions, because stable releases
|
|
|
|
require a package version suffix in the RPM file names.
|
|
|
|
|
2022-07-29 08:34:03 +00:00
|
|
|
- `TEST_PACKAGE_FILE`: optional, file pointing to the package under test (.deb,
|
|
|
|
.rpm, .pkg...). It can be a glob expression but it must match a single file,
|
|
|
|
and the pattern works only on the filename. If both `TEST_PACKAGE_VERSION_*`
|
|
|
|
and `TEST_PACKAGE_FILE` are provided, both are be installed (to test upgrades
|
2022-07-26 11:09:13 +00:00
|
|
|
for example). Requires `PACKAGE_TESTING=true`
|
|
|
|
|
|
|
|
- `TEST_PACKAGE_DIR`: optional (but conflicts with `TEST_PACKAGE_FILE`), the path
|
|
|
|
to a directory containing packages with the following layout:
|
|
|
|
|
2022-07-29 08:34:03 +00:00
|
|
|
For DEB: `{{ package_dir }}/{{ ansible_distribution_release }}/{{ ansible_architecture.replace('x86_64', 'amd64' }}/crowdsec_*.deb`
|
|
|
|
For RPM: `{{ package_dir }}/{{ releasever }}/RPMS/{{ ansible_architecture }}/crowdsec-*.{{ releasever }}.{{ ansible_architecture }}.rpm`
|
2022-07-26 11:09:13 +00:00
|
|
|
|
|
|
|
- `TEST_SKIP`: optional, comma-separated list of scripts that won't be executed.
|
|
|
|
Example: `TEST_SKIP=02_nolapi.bats,03_noagent.bats`
|
|
|
|
|
|
|
|
## Running tests with Vagrant + Ansible
|
|
|
|
|
|
|
|
You don't need Vagrant to run the ansible tests, if you can manage your own
|
|
|
|
vm creation and inventory.
|
|
|
|
|
|
|
|
However, to avoid relying on (and paying for..) a public cloud, we wrote vagrant
|
|
|
|
configuration files for the most common distributions we support.
|
|
|
|
|
|
|
|
To test with Vagrant, you need to:
|
|
|
|
|
|
|
|
- have a working libvirt environment (if you can use virt-manager to create VMs, you're golden)
|
|
|
|
|
|
|
|
- install the vagrant-libvirt plugin (`vagrant plugin install vagrant-libvirt`.
|
|
|
|
If it complains about gem versions, blame Ruby and see if you can remove some
|
|
|
|
other conflicting plugin).
|
|
|
|
|
|
|
|
- copy one of the `./env/*.sh` scripts to `environment.sh`, edit to your
|
|
|
|
needs, and execute it with "source environment.sh"
|
|
|
|
|
|
|
|
- `cd vagrant/<distro-of-your-choice>`
|
|
|
|
|
|
|
|
- `vagrant up --no-provision; vagrant provision`. The first command creates
|
|
|
|
the VM, the second installs all the dependencies, test suite and package
|
|
|
|
under test, then runs the tests. If you run a plain `vagrant up`, it does
|
|
|
|
everything with a single command, but also destroys the VM in case of test
|
|
|
|
failure so you are left with nothing to debug.
|
|
|
|
|
|
|
|
- `vagrant destroy` when you want to remove the VM. If you want to free up the
|
|
|
|
space taken by the base VM images, they are in
|
|
|
|
`/var/lib/libvirt/images/*VAGRANT*`
|
|
|
|
|
|
|
|
The above steps are automated in the script `./prepare-run` (requires bash >=4.4).
|
|
|
|
It takes an enviroment file, and optionally a list of directories with vagrant
|
|
|
|
configurations. With a single parameter, it loops over all the directories in
|
|
|
|
alphabetical order. Watch out for leftover VMs if you break the loop by hand.
|
|
|
|
|
|
|
|
After this, you will find up to 30GB of base images in `/var/lib/libvirt/images`,
|
|
|
|
which you need to remove by hand when you have finished testing or leave them
|
|
|
|
around for the next time.
|
|
|
|
|
|
|
|
You can give more memory or CPU juice to the VMs by editing [Vagrantfile.common](vagrant/Vagrantfile.common).
|
|
|
|
|
|
|
|
## Test Matrix
|
|
|
|
|
|
|
|
Tests fail with unsupported configurations or when the environment is not prepared correctly
|
|
|
|
due to missing setup/teardown parts in Ansible or functional tests. False positives
|
|
|
|
are also possible due to timing issues or flaky network connections.
|
|
|
|
|
|
|
|
If you have a result that deviates from the following matrix, that's probably a genuine bug or regression.
|
|
|
|
The data was created with crowdsec v1.4.1.
|
|
|
|
|
|
|
|
| | source/sqlite | pkg/sqlite | source/postgres | source/pgx | source/mysql (0) |
|
|
|
|
| ------------------------- | ------------- | ---------- | --------------- | ---------- | ---------------- |
|
|
|
|
| AmazonLinux 2 | ✓ (1) | ✓ (1) | old-db | old-db | wip |
|
|
|
|
| CentOS 7 | ✓ | ✓ | old-db | old-db | ✓ |
|
|
|
|
| CentOS 8 | ✓ | ✓ | ✓ | ✓ | ✓ |
|
|
|
|
| Debian 9 (stretch) | ✓ | ✓ | old-db | old-db | wip |
|
|
|
|
| Debian 10 (buster) | ✓ | ✓ | ✓ | ✓ | ✓ |
|
|
|
|
| Debian 11 (bullseye) | ✓ | ✓ | ✓ | ✓ | ✓ |
|
|
|
|
| Debian (testing/bookworm) | ✓ | ✓ | wip | wip | wip |
|
|
|
|
| Fedora 33 | ✓ | ✓ | wip | wip | wip |
|
|
|
|
| Fedora 34 | ✓ | ✓ | ✓ | ✓ | wip |
|
|
|
|
| Fedora 35 | ✓ | ✓ | ✓ | ✓ | wip |
|
|
|
|
| Fedora 36 | ✓ | ✓ | ✓ | ✓ | wip |
|
|
|
|
| FreeBSD 12 | ✓ | wip | wip | wip | wip |
|
|
|
|
| FreeBSD 13 | ✓ | wip | wip | wip | wip |
|
|
|
|
| Oracle 7 | ✓ | ✓ | wip | wip | ✓ |
|
|
|
|
| Oracle 8 | ✓ | ✓ | ✓ | ✓ | ✓ |
|
|
|
|
| Ubuntu 16.04 (xenial) | ✓ | ✓ | wip | wip | ✓ |
|
|
|
|
| Ubuntu 18.04 (bionic) | ✓ | ✓ | ✓ | ✓ | ✓ |
|
|
|
|
| Ubuntu 20.04 (focal) | ✓ | ✓ | ✓ | ✓ | ✓ |
|
|
|
|
| Ubuntu 22.04 (jammy) | ✓ | ✓ | ✓ | ✓ | ✓ |
|
|
|
|
| | | | | | |
|
|
|
|
|
|
|
|
Note: all tests with `local/<database>` are expected to pass for `pkg/<database>` as well.
|
|
|
|
|
|
|
|
wip - missing ansible or bats parts, working on it
|
|
|
|
|
|
|
|
old-db - the database that ships with the distribution is not supported (Postgres < 10)
|
|
|
|
|
|
|
|
0 - MySQL or MariaDB, depending on distribution defaults
|
|
|
|
|
|
|
|
1 - ansible may hang, passes all tests if run by hand
|