beenull/crowdsec
1
0
Fork 0
Code Issues Pull requests Packages Projects Releases Wiki Activity

Docker readme: update build instructions, recommend acquis.d and config.yaml.local (#2115)

Browse source
This commit is contained in:
mmetc 2023-03-10 17:30:08 +01:00 committed by GitHub
parent 6aaf3cd50b
commit 9c90144867
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
4 changed files with 81 additions and 28 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

3
Dockerfile
View file

@ -17,12 +17,13 @@ RUN apk add --no-cache git gcc libc-dev make bash gettext binutils-gold coreutil
cscli hub update && \
cscli collections install crowdsecurity/linux && \
cscli parsers install crowdsecurity/whitelists && \
go install github.com/mikefarah/yq/v4@v4.30.6
go install github.com/mikefarah/yq/v4@v4.31.2
FROM alpine:latest as slim
RUN apk add --no-cache --repository=http://dl-cdn.alpinelinux.org/alpine/edge/community tzdata bash && \
mkdir -p /staging/etc/crowdsec && \
mkdir -p /staging/etc/crowdsec/acquis.d && \
mkdir -p /staging/var/lib/crowdsec && \
mkdir -p /var/lib/crowdsec/data

3
Dockerfile.debian
View file

@ -21,7 +21,7 @@ RUN apt-get update && \
cscli hub update && \
cscli collections install crowdsecurity/linux && \
cscli parsers install crowdsecurity/whitelists && \
go install github.com/mikefarah/yq/v4@v4.30.6
go install github.com/mikefarah/yq/v4@v4.31.2
FROM debian:bullseye-slim as slim
@ -37,6 +37,7 @@ RUN apt-get update && \
bash \
tzdata && \
mkdir -p /staging/etc/crowdsec && \
mkdir -p /staging/etc/crowdsec/acquis.d && \
mkdir -p /staging/var/lib/crowdsec && \
mkdir -p /var/lib/crowdsec/data

102
docker/README.md
View file

@ -21,15 +21,19 @@ All the following images are available on Docker Hub for the architectures
Recommended for production usage. Also available on GitHub (ghcr.io).
- `crowdsecurity/crowdsec:latest`
- `crowdsecurity/crowdsec:dev`
For development and testing.
The latest stable release.
- `crowdsecurity/crowdsec:dev`
For development and testing, from the master branch.
since v1.4.2:
- `crowdsecurity/crowdsec:slim`
Reduced size by 60%, does not include notifier plugins nor the GeoIP database.
Reduced size by 60%, it does not include the notifier plugins nor the GeoIP database.
If you need these details on decisions, run `cscli hub upgrade` inside the
container to download the GeoIP database at runtime.
@ -43,27 +47,28 @@ The debian version includes support for systemd and journalctl.
### Custom
You can build your images with Dockerfile and Dockerfile-debian.
You can build your custom images with Dockerfile and Dockerfile-debian.
For example, if you want a Debian version without plugin notifiers:
For example, if you need a Debian version without plugin notifiers:
```console
$ docker build -f Dockerfile.debian --target slim
$ docker build -f Dockerfile.debian --target slim .
```
The supported values for target are: full, with-geoip, with-plugins, slim.
The supported values for target are: full, geoip, plugins, slim.
Note: for crowdsec versions < 1.5.0, the syntax is
```console
$ docker build -f Dockerfile.debian --build-arg=BUILD_ENV=slim
$ docker build -f Dockerfile.debian --build-arg=BUILD_ENV=slim .
```
## Required configuration
### Journalctl (only for debian image)
To use journalctl as a log stream, with or without the `DSN` environment variable, it's important to mount the journal log from the host to the container itself.
To use journalctl as a log stream, with or without the `DSN` environment variable, you need to mount the journal log from the host to the container itself.
This can be done by adding the following volume mount to the docker command:
```
@ -71,19 +76,57 @@ This can be done by adding the following volume mount to the docker command:
```
### Logs ingestion and processing
Collections are a good place to start: https://docs.crowdsec.net/docs/collections/intro
Find collections, scenarios, parsers and postoverflows in the hub: https://hub.crowdsec.net
* Specify collections | scenarios | parsers | postoverflows to install via the environment variables (by default [`crowdsecurity/linux`](https://hub.crowdsec.net/author/crowdsecurity/collections/linux) is installed)
* Mount volumes to specify your log files that should be ingested by crowdsec
### Acquisition
* Mount volumes to specify which log files should be ingested by crowdsec
`/etc/crowdsec/acquis.yaml` maps logs to the provided parsers. Find out more here: https://docs.crowdsec.net/docs/concepts/#acquisition
acquis.yaml example:
```shell
### Acquisition (one file per datasource - recommended)
The files in `/etc/crowdsec/acquis.d/` map the logs to the provided parsers. Find out more here: https://docs.crowdsec.net/docs/concepts/#acquisition
The directory might contain for example
`ssh.yaml`:
```yaml
filenames:
- /logs/auth.log
- /logs/syslog
labels:
type: syslog
```
`apache.yaml`:
``` yaml
filename: /logs/apache2/*.log
labels:
type: apache2
```
`labels.type`: use `syslog` if the logs come from syslog, otherwise check the collection's documentation for the relevant type.
You can bind the directory from the host or have it in a Docker volume, the former is easier to update as you add more applications.
Note: In versions < 1.5, the acquisition directory is not configured by default. You can add it by mounting the `/etc/crowdsec/config.yaml.local` file:
```yaml
crowdsec_service:
acquisition_dir: /etc/crowdsec/acquis.d
```
### Acquisition (single file - deprecated)
Before 1.5.0, it was recommended to put your acquisition configuration in /etc/crowdsec/acquis.yaml. You can still do it
if you prefer but it's more effective to have one file per datasource.
```yaml title="/etc/crowdsec/acquis.yaml"
filenames:
- /logs/auth.log
- /logs/syslog
@ -95,22 +138,26 @@ labels:
type: apache2
```
`labels.type`: use `syslog` if the logs come from syslog, otherwise check the collection's documentation for the relevant type.
## Recommended configuration
### Volumes
We strongly suggest mounting **named volumes** for Crowdsec configuration and database to avoid credentials and decisions loss in case of container destruction and recreation, version update, etc.
We strongly suggest persisting the Crowdsec configuration and database in **named volumes**, or bind-mount them from the host,
to avoid losing credentials and decision data in case of container destruction and recreation, version update, etc.
* Credentials and configuration: `/etc/crowdsec`
* Acquisition: `/etc/crowdsec/acquis.d` and/or `/etc/crowdsec.acquis.yaml` (yes, they can be nested in `/etc/crowdsec`)
* Database when using SQLite (default): `/var/lib/crowdsec/data`
## Start a Crowdsec instance
```shell
docker run -d \
-v local_path_to_crowdsec_config/acquis.yaml:/etc/crowdsec/acquis.yaml \
-v crowdsec_config:/etc/crowdsec \
-v local_path_to_crowdsec_config/acquis.d:/etc/crowdsec/acquis.d \
-v local_path_to_crowdsec_config/acquis.yaml:/etc/crowdsec/acquis.yaml \
-v crowdsec_data:/var/lib/crowdsec/data \
-v /var/log/auth.log:/logs/auth.log:ro \
-v /var/log/syslog.log:/logs/syslog.log:ro \
@ -120,22 +167,31 @@ docker run -d \
--name crowdsec crowdsecurity/crowdsec
```
## ... or docker-compose
Check this full-stack example using docker-compose: https://github.com/crowdsecurity/example-docker-compose
# How to extend this image
## Full configuration
The container is built with a specific docker
[configuration](https://github.com/crowdsecurity/crowdsec/blob/master/docker/config.yaml).
If you need to change it and the docker variables (see below) are not enough,
you can bind `/etc/crowdsec/config.yaml` to your configuration file.
you can mount `/etc/crowdsec/config.yaml.local` from the host.
The file should contain only the options from `config.yaml` that you want to change,
as documented in [`Overriding values`](https://docs.crowdsec.net/docs/configuration/crowdsec_configuration#overriding-values).
It is not recommended anymore to bind-mount the full config.yaml file and you should not need to do it.
## Notifications
If you wish to use the [notification system](https://docs.crowdsec.net/docs/notification_plugins/intro), you will need to mount at least a custom `profiles.yaml` and a notification configuration to `/etc/crowdsec/notifications`
If you want to use the [notification system](https://docs.crowdsec.net/docs/notification_plugins/intro), you have to use the full image (not slim) and mount at least a custom `profiles.yaml` and a notification configuration to `/etc/crowdsec/notifications`
# Deployment use cases
Crowdsec is composed of an `agent` that parses logs and creates `alerts`, and a
`local API (LAPI)` that transforms these alerts into decisions. Both functions
are provided by the same executables, so the agent and the LAPI can run in the
@ -273,12 +329,6 @@ config.yaml) each time the container is run.
| `CI_TESTING` | false | Used during functional tests |
| `DEBUG` | false | Trace the entrypoint |
## Volumes
* `/var/lib/crowdsec/data/` - Directory where all crowdsec data (Databases) is located
* `/etc/crowdsec/` - Directory where all crowdsec configurations are located
## File Locations
* `/usr/local/bin/crowdsec` - Crowdsec binary

1
docker/config.yaml
View file

@ -14,6 +14,7 @@ config_paths:
plugin_dir: /usr/local/lib/crowdsec/plugins/
crowdsec_service:
acquisition_path: /etc/crowdsec/acquis.yaml
acquisition_dir: /etc/crowdsec/acquis.d
parser_routines: 1
plugin_config:
user: nobody