diff --git a/docs/faq.md b/docs/faq.md index 59cedd100..db471b67e 100644 --- a/docs/faq.md +++ b/docs/faq.md @@ -53,13 +53,20 @@ SQLite by default as it's suitable for standalone/single-machine setups. ## How to control granularity of actions ? (whitelists, simulation etc.) -{{v1X.crowdsec.name}} support both [whitelists]((/Crowdsec/v1/write_configurations/whitelist/) and [simulation](/Crowdsec/v1/references/simulation/) : +{{v1X.crowdsec.name}} support both [whitelists](/Crowdsec/v1/write_configurations/whitelist/) and [simulation](/Crowdsec/v1/references/simulation/) : - Whitelists allows you to "discard" events or overflows - Simulation allows you to simply cancel the decision that is going to be taken, but keep track of it {{v1X.profiles.htmlname}} allows you to control which decision will be applied to which alert. +## How to know if my setup is working correctly ? Some of my logs are unparsed, is it normal ? + +Yes, crowdsec parsers only parse the logs that are relevant for scenarios :) + +Take a look at `cscli metrics` [and understand what do they mean](/Crowdsec/v1/getting_started/crowdsec-tour/#reading-metrics) to know if your setup is correct. + + ## How to add whitelists ? You can follow this [guide](/Crowdsec/v1/write_configurations/whitelist/) diff --git a/docs/v1.X/docs/getting_started/crowdsec-tour.md b/docs/v1.X/docs/getting_started/crowdsec-tour.md index 855e72c96..3b05d76e3 100644 --- a/docs/v1.X/docs/getting_started/crowdsec-tour.md +++ b/docs/v1.X/docs/getting_started/crowdsec-tour.md @@ -190,6 +190,26 @@ INFO[0000] Local Api Bouncers Metrics: +### Reading metrics + +Those metrics are a great way to know if your configuration is correct : + +The `Acquisition Metrics` is a great way to know if your parsers are setup correctly : + + - If you have 0 **LINES PARSED** for a source : You are probably *missing* a parser, or you have a custom log format that prevents the parser from understanding your logs. + - However, it's perfectly OK to have a lot of **LINES UNPARSED** : Crowdsec is not a SIEM, and only parses the logs that are relevant to its scenarios. For example, [ssh parser](https://hub.crowdsec.net/author/crowdsecurity/configurations/sshd-logs), only cares about failed authentication events (at the time of writting). + - **LINES POURED TO BUCKET** tell you that your scenarios are matching your log sources : it means that some events from this log source made all their way to an actual scenario + + +The `Parser Metrics` will let you troubleshoot eventual parser misconfigurations : + - **HITS** is how many events where fed to this specific parser + - **PARSED** and **UNPARSED** indicate how many events successfully come out of the parser + +For example, if you have a custom log format in nginx that is not supported by the default parser, you will end up seeing a lot of **UNPARSED** for this specific parser, and 0 for **PARSED**. + +For more advanced metrics understanding, [take a look at the dedicated prometheus documentation](/Crowdsec/v1/observability/prometheus/). + + ## Deploy dashboard ```bash @@ -217,3 +237,11 @@ sudo cscli collections install crowdsecurity/nginx Collections are bundles of parsers/scenarios that form a coherent ensemble to analyze/detect attacks for a specific service. It is the most common way to deploy configurations. They can be found and browsed on the {{v1X.hub.htmlname}} + +## Scalability + +Crowdsec uses go-routines for parsing and enriching logs, pouring events to buckets and manage outputs. + +By default, one routine of each exists (should be enough to handle ~1K EP/s), and can be changed in `crowdsec_service` of the main configuration file via the [parser_routines](/Crowdsec/v1/references/crowdsec-config/#parser_routines), [buckets_routines](/Crowdsec/v1/references/crowdsec-config/#buckets_routines) and [output_routines](/Crowdsec/v1/references/crowdsec-config/#output_routines) directives. + +Please keep in mind that thanks to the [http API]({{v1X.lapi.swagger}}), the workload of log parsing can be splitted amongst several agents pushing to a single {{v1X.lapi.Htmlname}}. diff --git a/docs/v1.X/docs/user_guide/forensic_mode.md b/docs/v1.X/docs/user_guide/forensic_mode.md index 5147e9885..555e85389 100644 --- a/docs/v1.X/docs/user_guide/forensic_mode.md +++ b/docs/v1.X/docs/user_guide/forensic_mode.md @@ -35,6 +35,10 @@ INFO[13-11-2020 13:05:24] (14baeedafc1e44c08b806fc0c1cd92c4/crowdsec) crowdsecur And as these alerts are as well pushed to database, it mean you can view them in metabase, or using cscli ! +!!! warning + To work in forensic mode, crowdsec-agent relies on [crowdsecurity/dateparse-enrich](https://hub.crowdsec.net/author/crowdsecurity/configurations/dateparse-enrich) to parse date formats. See dedicated hub page for supported formats. + + ## Injecting alerts into existing database If you already have a running crowdsec/Local API running and want to inject events into existing database, you can run crowdsec directly :