docker: pre-download all hub items and data, opt-in hub update/upgrade (#2933)

* docker: pre-download all hub items and data, opt-in hub update/upgrade

* docker/bars: don't purge anything before pre-downloading hub

* Docker: README update
This commit is contained in:
mmetc 2024-04-08 14:53:12 +02:00 committed by GitHub
parent 990dd5e08e
commit 0e8a1c681b
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
5 changed files with 36 additions and 22 deletions

View file

@ -25,6 +25,7 @@ RUN make clean release DOCKER_BUILD=1 BUILD_STATIC=1 && \
./wizard.sh --docker-mode && \ ./wizard.sh --docker-mode && \
cd - >/dev/null && \ cd - >/dev/null && \
cscli hub update && \ cscli hub update && \
./docker/preload-hub-items && \
cscli collections install crowdsecurity/linux && \ cscli collections install crowdsecurity/linux && \
cscli parsers install crowdsecurity/whitelists cscli parsers install crowdsecurity/whitelists

View file

@ -134,7 +134,6 @@ labels:
type: apache2 type: apache2
``` ```
## Recommended configuration ## Recommended configuration
### Volumes ### Volumes
@ -146,6 +145,14 @@ to avoid losing credentials and decision data in case of container destruction a
* Acquisition: `/etc/crowdsec/acquis.d` and/or `/etc/crowdsec.acquis.yaml` (yes, they can be nested in `/etc/crowdsec`) * Acquisition: `/etc/crowdsec/acquis.d` and/or `/etc/crowdsec.acquis.yaml` (yes, they can be nested in `/etc/crowdsec`)
* Database when using SQLite (default): `/var/lib/crowdsec/data` * Database when using SQLite (default): `/var/lib/crowdsec/data`
### Hub updates
To ensure you have the latest version of the collections, scenarios, parsers, etc., you can set the variable `DO_HUB_UPGRADE` to true.
This will perform an update/upgrade of the hub every time the container is started.
Be aware that if your container is misbehaving and caught in a restart loop, the CrowdSec hub may ban your IP for some time and your containers
will run with the version of the hub that is cached in the container's image. If you enable `DO_HUB_UPGRADE`, do it when your infrastructure is running
correctly and make sure you have some monitoring in place.
## Start a Crowdsec instance ## Start a Crowdsec instance
@ -316,7 +323,7 @@ config.yaml) each time the container is run.
| `BOUNCERS_ALLOWED_OU` | bouncer-ou | OU values allowed for bouncers, separated by comma | | `BOUNCERS_ALLOWED_OU` | bouncer-ou | OU values allowed for bouncers, separated by comma |
| | | | | | | |
| __Hub management__ | | | | __Hub management__ | | |
| `NO_HUB_UPGRADE` | false | Skip hub update / upgrade when the container starts | | `DO_HUB_UPGRADE` | false | Force hub update / upgrade when the container starts. If for some reason the container restarts too often, it may lead to a temporary ban from hub updates. |
| `COLLECTIONS` | | Collections to install, separated by space: `-e COLLECTIONS="crowdsecurity/linux crowdsecurity/apache2"` | | `COLLECTIONS` | | Collections to install, separated by space: `-e COLLECTIONS="crowdsecurity/linux crowdsecurity/apache2"` |
| `PARSERS` | | Parsers to install, separated by space | | `PARSERS` | | Parsers to install, separated by space |
| `SCENARIOS` | | Scenarios to install, separated by space | | `SCENARIOS` | | Scenarios to install, separated by space |

View file

@ -304,9 +304,8 @@ conf_set_if "$PLUGIN_DIR" '.config_paths.plugin_dir = strenv(PLUGIN_DIR)'
## Install hub items ## Install hub items
cscli hub update || true if istrue "$DO_HUB_UPGRADE"; then
cscli hub update || true
if isfalse "$NO_HUB_UPGRADE"; then
cscli hub upgrade || true cscli hub upgrade || true
fi fi

22
docker/preload-hub-items Executable file
View file

@ -0,0 +1,22 @@
#!/usr/bin/env bash
set -eu
# pre-download everything but don't install anything
echo "Pre-downloading Hub content..."
types=$(cscli hub types -o raw)
for itemtype in $types; do
ALL_ITEMS=$(cscli "$itemtype" list -a -o json | itemtype="$itemtype" yq '.[env(itemtype)][] | .name')
if [[ -n "${ALL_ITEMS}" ]]; then
#shellcheck disable=SC2086
cscli "$itemtype" install \
$ALL_ITEMS \
--download-only \
--error
fi
done
echo " done."

View file

@ -9,20 +9,12 @@ THIS_DIR=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)
# pre-download everything but don't install anything # pre-download everything but don't install anything
echo -n "Purging existing hub..." echo "Pre-downloading Hub content..."
types=$("$CSCLI" hub types -o raw) types=$("$CSCLI" hub types -o raw)
for itemtype in $types; do for itemtype in $types; do
"$CSCLI" "${itemtype}" delete --all --error --purge --force ALL_ITEMS=$("$CSCLI" "$itemtype" list -a -o json | itemtype="$itemtype" yq '.[env(itemtype)][] | .name')
done
echo " done."
echo -n "Pre-downloading Hub content..."
for itemtype in $types; do
ALL_ITEMS=$("$CSCLI" "$itemtype" list -a -o json | jq --arg itemtype "$itemtype" -r '.[$itemtype][].name')
if [[ -n "${ALL_ITEMS}" ]]; then if [[ -n "${ALL_ITEMS}" ]]; then
#shellcheck disable=SC2086 #shellcheck disable=SC2086
"$CSCLI" "$itemtype" install \ "$CSCLI" "$itemtype" install \
@ -32,11 +24,4 @@ for itemtype in $types; do
fi fi
done done
# XXX: download-only works only for collections, not for parsers, scenarios, postoverflows.
# so we have to delete the links manually, and leave the downloaded files in place
for itemtype in $types; do
"$CSCLI" "$itemtype" delete --all --error
done
echo " done." echo " done."