2022-01-20 11:29:19 +00:00
# Quick reference
2020-11-30 09:37:17 +00:00
2022-01-20 11:29:19 +00:00
* Documentation and help: https://docs.crowdsec.net/
* Crowdsec concepts: https://docs.crowdsec.net/docs/concepts
* Where to file issues: https://github.com/crowdsecurity/crowdsec
2020-11-30 09:37:17 +00:00
2022-01-20 11:29:19 +00:00
# What is Crowdsec
2020-11-30 09:37:17 +00:00
2022-12-06 14:55:27 +00:00
Crowdsec - An open-source, lightweight agent to detect and respond to bad behaviors. It also automatically benefits from our global community-wide IP reputation database.
2020-11-30 09:37:17 +00:00
2022-01-20 11:29:19 +00:00
# How to use this image
2020-11-30 09:37:17 +00:00
2022-11-28 09:35:12 +00:00
## Image flavors
2022-11-08 11:28:57 +00:00
2022-11-28 09:35:12 +00:00
All the following images are available on Docker Hub for the architectures
386, amd64, arm/v6, arm/v7, arm64.
2022-11-08 11:28:57 +00:00
### Alpine
2022-11-28 09:35:12 +00:00
- `crowdsecurity/crowdsec:{version}`
2022-11-08 11:28:57 +00:00
Recommended for production usage. Also available on GitHub (ghrc.io).
2022-11-28 09:35:12 +00:00
- `crowdsecurity/crowdsec:latest`
2022-11-08 11:28:57 +00:00
For development and testing.
since v1.4.2:
2022-11-28 09:35:12 +00:00
- `crowdsecurity/crowdsec:slim`
2022-11-08 11:28:57 +00:00
Reduced size by 60%, does not include notifier plugins nor the GeoIP database.
If you need these details on decisions, running `cscli hub upgrade` inside the
container downloads the GeoIP database at runtime.
### Debian (since v1.3.3)
2022-11-28 09:35:12 +00:00
- `crowdsecurity/crowdsec:{version}-debian`
- `crowdsecurity/crowdsec:latest-debian`
2022-11-08 11:28:57 +00:00
The debian version includes support for systemd and journalctl.
### Custom
2022-11-28 09:35:12 +00:00
You can build your images with Dockerfile and Dockerfile-debian.
2022-11-08 11:28:57 +00:00
For example, if you want a Debian version without plugin notifiers:
```console
$ docker build -f Dockerfile.debian --build-arg=BUILD_ENV=slim
```
supported values for BUILD_ENV are: full, with-geoip, with-plugins, slim.
2022-02-15 16:10:15 +00:00
2022-01-20 11:29:19 +00:00
## Required configuration
2020-11-30 09:37:17 +00:00
2022-02-18 16:11:41 +00:00
### Journalctl (only for debian image)
2022-11-28 09:35:12 +00:00
To use journalctl as a log stream, with or without the `DSN` environment variable, it's important to mount the journal log from the host to the container itself.
This can be done by adding the following volume mount to the docker command:
2022-02-15 16:10:15 +00:00
```
-v /var/log/journal:/run/log/journal
```
2022-01-20 11:29:19 +00:00
### Logs ingestion and processing
Collections are a good place to start: https://docs.crowdsec.net/docs/collections/intro
2020-11-30 09:37:17 +00:00
2022-11-28 09:35:12 +00:00
Find collections, scenarios, parsers and postoverflows in the hub: https://hub.crowdsec.net
2020-11-30 09:37:17 +00:00
2022-11-28 09:35:12 +00:00
* Specify collections | scenarios | parsers | postoverflows to install via the environment variables (by default [`crowdsecurity/linux` ](https://hub.crowdsec.net/author/crowdsecurity/collections/linux ) is installed)
2022-01-20 11:29:19 +00:00
* Mount volumes to specify your log files that should be ingested by crowdsec
### Acquisition
2022-11-28 09:35:12 +00:00
`/etc/crowdsec/acquis.yaml` maps logs to the provided parsers. Find out more here: https://docs.crowdsec.net/docs/concepts/#acquisition
2020-11-30 09:37:17 +00:00
2022-01-20 11:29:19 +00:00
acquis.yaml example:
2020-11-30 09:37:17 +00:00
```shell
filenames:
- /logs/auth.log
- /logs/syslog
labels:
type: syslog
---
filename: /logs/apache2/*.log
labels:
type: apache2
```
2022-11-28 09:35:12 +00:00
`labels.type` : use `syslog` if the logs come from syslog, otherwise check the collection's documentation for the relevant type.
2022-01-20 11:29:19 +00:00
## Recommended configuration
### Volumes
2022-11-28 09:35:12 +00:00
We strongly suggest mounting **named volumes** for Crowdsec configuration and database to avoid credentials and decisions loss in case of container destruction and recreation, version update, etc.
2022-01-20 11:29:19 +00:00
* Credentials and configuration: `/etc/crowdsec`
2022-11-28 09:35:12 +00:00
* Database when using SQLite (default): `/var/lib/crowdsec/data`
2020-11-30 09:37:17 +00:00
2022-01-20 11:29:19 +00:00
## Start a Crowdsec instance
2020-11-30 09:37:17 +00:00
```shell
2022-01-20 11:29:19 +00:00
docker run -d \
-v local_path_to_crowdsec_config/acquis.yaml:/etc/crowdsec/acquis.yaml \
-v crowdsec_config:/etc/crowdsec \
-v crowdsec_data:/var/lib/crowdsec/data \
-v /var/log/auth.log:/logs/auth.log:ro \
-v /var/log/syslog.log:/logs/syslog.log:ro \
-v /var/log/apache:/logs/apache:ro \
2020-11-30 09:37:17 +00:00
-e COLLECTIONS="crowdsecurity/apache2 crowdsecurity/sshd" \
-p 8080:8080 -p 6060:6060 \
2021-02-10 08:29:29 +00:00
--name crowdsec crowdsecurity/crowdsec
2020-11-30 09:37:17 +00:00
```
2022-01-20 11:29:19 +00:00
## ... or docker-compose
2021-02-10 08:29:29 +00:00
2022-11-28 09:35:12 +00:00
Check this full-stack example using docker-compose: https://github.com/crowdsecurity/example-docker-compose
2022-01-20 11:29:19 +00:00
# How to extend this image
2022-11-28 09:35:12 +00:00
2022-01-20 11:29:19 +00:00
## Full configuration
2022-11-28 09:35:12 +00:00
The container is built with a specific docker
[configuration ](https://github.com/crowdsecurity/crowdsec/blob/master/docker/config.yaml ).
If you need to change it and the docker variables (see below) are not enough,
2022-12-06 14:55:27 +00:00
you can bind `/etc/crowdsec/config.yaml` to your configuration file.
2022-11-28 09:35:12 +00:00
2022-01-20 11:29:19 +00:00
## Notifications
If you wish to use the [notification system ](https://docs.crowdsec.net/docs/notification_plugins/intro ), you will need to mount at least a custom `profiles.yaml` and a notification configuration to `/etc/crowdsec/notifications`
# Deployment use cases
2022-11-28 09:35:12 +00:00
Crowdsec is composed of an `agent` that parses logs and creates `alerts` , and a
`local API (LAPI)` that transforms these alerts into decisions. Both functions
are provided by the same executables, so the agent and the LAPI can run in the
same or separate containers. In complex configurations, it makes sense to have
agents on each machine that runs the protected applications, and a LAPI that
gathers all signals from agents and communicates with the `central API` .
2022-01-20 11:29:19 +00:00
## Register a new agent with LAPI
2022-12-06 10:58:32 +00:00
2022-12-12 09:57:26 +00:00
Without TLS authentication:
2022-12-06 10:58:32 +00:00
2022-01-20 11:29:19 +00:00
```shell
docker exec -it crowdsec_lapi_container_name cscli machines add agent_user_name --password agent_password
```
2022-12-12 09:57:26 +00:00
With TLS authentication:
2022-12-06 10:58:32 +00:00
Agents are automatically registered and don't need a username or password. The
agents' names are derived from the IP address from which they connect.
2022-01-20 11:29:19 +00:00
## Run an agent connected to LAPI
2022-11-28 09:35:12 +00:00
Add the following environment variables to the docker run command:
2022-01-20 11:29:19 +00:00
* `DISABLE_LOCAL_API=true`
2022-04-27 09:04:12 +00:00
* `AGENT_USERNAME="agent_user_name"` - agent_user_name previously registered with LAPI
2022-01-20 11:29:19 +00:00
* `AGENT_PASSWORD="agent_password"` - agent_password previously registered with LAPI
* `LOCAL_API_URL="http://LAPI_host:LAPI_port"`
# Next steps
2022-11-28 09:35:12 +00:00
2022-01-20 11:29:19 +00:00
## Bouncers
2022-11-28 09:35:12 +00:00
Crowdsec being a detection component, the remediation is implemented using `bouncers` . Each bouncer protects a specific component. Find out more:
2022-01-20 11:29:19 +00:00
https://hub.crowdsec.net/browse/#bouncers
https://docs.crowdsec.net/docs/user_guides/bouncers_configuration/
2022-04-04 08:18:44 +00:00
### Automatic Bouncer Registration
2022-12-12 09:57:26 +00:00
Without TLS authentication:
2022-12-06 10:58:32 +00:00
You can register bouncers with the crowdsec container at startup, using environment variables or Docker secrets. You cannot use this process to update an existing bouncer without first deleting it.
2022-04-04 08:18:44 +00:00
To use environment variables, they should be in the format `BOUNCER_KEY_<name>=<key>` . e.g. `BOUNCER_KEY_nginx=mysecretkey12345` .
2022-11-08 11:28:57 +00:00
To use Docker secrets, the secret should be named `bouncer_key_<name>` with a content of `<key>` . e.g. `bouncer_key_nginx` with content `mysecretkey12345` .
2022-04-04 08:18:44 +00:00
2023-01-04 15:43:35 +00:00
A bouncer key can be any string but we recommend an alphanumeric value for consistency with the keys generated by crowdsec and to avoid problems with escaping special characters.
2022-12-06 10:58:32 +00:00
2022-12-12 09:57:26 +00:00
With TLS authentication:
2022-12-06 10:58:32 +00:00
Bouncers are automatically registered and don't need an API key. The
bouncers' names are derived from the IP address from which they connect.
2022-04-04 08:18:44 +00:00
2022-01-20 11:29:19 +00:00
## Console
2022-11-08 11:28:57 +00:00
We provide a web-based interface to get more from Crowdsec: https://docs.crowdsec.net/docs/console
2022-01-20 11:29:19 +00:00
Subscribe here: https://app.crowdsec.net
# Caveats
2022-11-28 09:35:12 +00:00
Using binds rather than named volumes ([complete explanation here](https://docs.docker.com/storage/volumes/)) results in more complexity as you'll have to bind the relevant files one by one whereas with named volumes you can mount full configuration and data folders. On the other hand, named volumes are less straightforward to navigate.
2022-01-20 11:29:19 +00:00
# Reference
## Environment Variables
2020-11-30 09:37:17 +00:00
2023-01-04 15:43:35 +00:00
Note for persistent configurations (i.e. bind mount or volumes): when a
variable is set, its value may be written to the appropriate file (usually
config.yaml) each time the container is run.
2022-11-28 09:35:12 +00:00
| Variable | Default | Description |
| ----------------------- | ------------------------- | ----------- |
| `CONFIG_FILE` | `/etc/crowdsec/config.yaml` | Configuration file location |
| `DISABLE_AGENT` | false | Disable the agent, run a LAPI-only container |
| `DISABLE_LOCAL_API` | false | Disable LAPI, run an agent-only container |
| `DISABLE_ONLINE_API` | false | Disable online API registration for signal sharing |
2023-01-04 15:43:35 +00:00
| `TEST_MODE` | false | Don't run the service, only test the configuration: `-e TEST_MODE=true` |
| `TZ` | | Set the [timezone ](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones ) to ensure the logs have a local timestamp. |
| `LOCAL_API_URL` | `http://0.0.0.0:8080` | The LAPI URL, you need to change this when `DISABLE_LOCAL_API` is true: `-e LOCAL_API_URL="http://lapi-address:8080"` |
2022-11-28 09:35:12 +00:00
| `PLUGIN_DIR` | `/usr/local/lib/crowdsec/plugins/` | Directory for plugins: `-e PLUGIN_DIR="<path>"` |
| `METRICS_PORT` | 6060 | Port to expose Prometheus metrics |
2023-01-04 15:43:35 +00:00
| | | |
| __LAPI__ | | (useless with DISABLE_LOCAL_API) |
2022-11-30 13:28:33 +00:00
| `USE_WAL` | false | Enable Write-Ahead Logging with SQLite |
2023-01-04 15:43:35 +00:00
| `CUSTOM_HOSTNAME` | localhost | Name for the local agent (running in the container with LAPI) |
| | | |
| __Agent__ | | (these don't work with DISABLE_AGENT) |
| `TYPE` | | [`Labels.type` ](https://docs.crowdsec.net/Crowdsec/v1/references/acquisition/ ) for file in time-machine: `-e TYPE="<type>"` |
| `DSN` | | Process a single source in time-machine: `-e DSN="file:///var/log/toto.log"` or `-e DSN="cloudwatch:///your/group/path:stream_name?profile=dev&backlog=16h"` or `-e DSN="journalctl://filters=_SYSTEMD_UNIT=ssh.service"` |
| | | |
| __Bouncers__ | | |
| `BOUNCER_KEY_<name>` | | Register a bouncer with the name `<name>` and a key equal to the value of the environment variable. |
2022-11-28 09:35:12 +00:00
| | | |
| __Console__ | | |
| `ENROLL_KEY` | | Enroll key retrieved from [the console ](https://app.crowdsec.net/ ) to enroll the instance. |
| `ENROLL_INSTANCE_NAME` | | To set an instance name and see it on [the console ](https://app.crowdsec.net/ ) |
| `ENROLL_TAGS` | | Tags of the enrolled instance, for search and filter |
| | | |
| __Password Auth__ | | |
| `AGENT_USERNAME` | | Agent username (to register if is LAPI or to use if it's an agent): `-e AGENT_USERNAME="machine_id"` |
| `AGENT_PASSWORD` | | Agent password (to register if is LAPI or to use if it's an agent): `-e AGENT_PASSWORD="machine_password"` |
| | | |
2023-01-04 15:43:35 +00:00
| __TLS Encryption__ | | |
| `USE_TLS` | false | Enable TLS encryption (either as a LAPI or agent) |
| `CACERT_FILE` | | CA certificate bundle (for self-signed certificates) |
2022-12-29 21:00:11 +00:00
| `INSECURE_SKIP_VERIFY` | | Skip LAPI certificate validation |
2023-01-04 15:43:35 +00:00
| `LAPI_CERT_FILE` | | LAPI TLS Certificate path |
| `LAPI_KEY_FILE` | | LAPI TLS Key path |
| | | |
| __TLS Authentication__ | | (these require USE_TLS=true) |
| `CLIENT_CERT_FILE` | | Client TLS Certificate path (enable TLS authentication) |
2022-12-29 21:00:11 +00:00
| `CLIENT_KEY_FILE` | | Client TLS Key path |
2022-11-28 09:35:12 +00:00
| `AGENTS_ALLOWED_OU` | agent-ou | OU values allowed for agents, separated by comma |
| `BOUNCERS_ALLOWED_OU` | bouncer-ou | OU values allowed for bouncers, separated by comma |
| | | |
| __Hub management__ | | |
| `COLLECTIONS` | | Collections to install, separated by space: `-e COLLECTIONS="crowdsecurity/linux crowdsecurity/apache2"` |
| `PARSERS` | | Parsers to install, separated by space |
2022-12-16 19:41:39 +00:00
| `SCENARIOS` | | Scenarios to install, separated by space |
2022-11-28 09:35:12 +00:00
| `POSTOVERFLOWS` | | Postoverflows to install, separated by space |
| `DISABLE_COLLECTIONS` | | Collections to remove, separated by space: `-e DISABLE_COLLECTIONS="crowdsecurity/linux crowdsecurity/nginx"` |
| `DISABLE_PARSERS` | | Parsers to remove, separated by space |
| `DISABLE_SCENARIOS` | | Scenarios to remove, separated by space |
| `DISABLE_POSTOVERFLOWS` | | Postoverflows to remove, separated by space |
| | | |
| __Log verbosity__ | | |
| `LEVEL_INFO` | false | Force INFO level for the container log |
| `LEVEL_DEBUG` | false | Force DEBUG level for the container log |
| `LEVEL_TRACE` | false | Force TRACE level (VERY verbose) for the container log |
2023-01-04 15:43:35 +00:00
| | | |
| __Developer options__ | | |
| `CI_TESTING` | false | Used during functional tests |
| `DEBUG` | false | Trace the entrypoint |
2020-11-30 09:37:17 +00:00
2022-01-20 11:29:19 +00:00
## Volumes
2020-11-30 09:37:17 +00:00
* `/var/lib/crowdsec/data/` - Directory where all crowdsec data (Databases) is located
* `/etc/crowdsec/` - Directory where all crowdsec configurations are located
2022-01-20 11:29:19 +00:00
## File Locations
2020-11-30 09:37:17 +00:00
* `/usr/local/bin/crowdsec` - Crowdsec binary
2022-02-15 16:10:15 +00:00
2020-11-30 09:37:17 +00:00
* `/usr/local/bin/cscli` - Crowdsec CLI binary to interact with crowdsec
2022-01-20 11:29:19 +00:00
# Find Us
2020-11-30 09:37:17 +00:00
* [GitHub ](https://github.com/crowdsecurity/crowdsec )
2022-01-20 11:29:19 +00:00
# Contributing
2020-11-30 09:37:17 +00:00
Please read [contributing ](https://docs.crowdsec.net/Crowdsec/v1/contributing/ ) for details on our code of conduct, and the process for submitting pull requests to us.
2022-01-20 11:29:19 +00:00
# License
2020-11-30 09:37:17 +00:00
2021-03-15 08:29:55 +00:00
This project is licensed under the MIT License - see the [LICENSE ](https://github.com/crowdsecurity/crowdsec/blob/master/LICENSE ) file for details.