Commit graph

118 commits

Author SHA1 Message Date
mmetc c4473839c4
Refact pkg/parser/node (#2953)
* extract method processFilter()

* extract method processWhitelist()

* lint (whitespace, errors)
2024-04-25 17:53:10 +02:00
Christian Kampka f6bb8412c5
Add patterns_dir configuration option (#2868)
* Add patterns_dir configuration option

* Update config.yaml

---------

Co-authored-by: mmetc <92726601+mmetc@users.noreply.github.com>
2024-03-25 16:20:16 +01:00
mmetc e976614645
cscli metrics: rename buckets -> scenarios (#2848)
* cscli metrics: rename buckets -> scenarios
* update lint configuration
* lint
2024-02-15 14:34:12 +01:00
Thibault "bui" Koechlin 3208a40ef3
Dedicated whitelist metrics (#2813)
* add proper whitelist metrics : both its own table and an extension to acquis metrics to track discarded/whitelisted lines
2024-02-06 18:04:17 +01:00
mmetc f75cdeb239
lint: enalble linter "wastedassign" (#2772) 2024-01-24 17:31:11 +01:00
Laurence Jones 4df4e5b3bf
[parser/scenarios] defer yaml file closure (#2689)
* Defer close the fd's
* Convert fatals into return with errors
2024-01-17 12:09:01 +01:00
Thibault "bui" Koechlin 6ca053ca67
fix #2720 #2719 (#2724)
* fix order of display of parsers

* add a --no-clean opt
2024-01-15 09:16:03 +01:00
mmetc ca784b147b
test and log fixes (#2690)
* cscli inspect: suggest --diff if an item is tainted
* appropriate warning, or error if context configuration file is empty
* fix user/group lookup unit test
* fix: allow hub upgrade --force with local items
* fix pkg/parser lookup for 8.8.8.8
* fix func test
* fix hubtests: machines add --force
2024-01-03 09:33:52 +01:00
mmetc 08694adf1b
lint (errorlint) (#2644) 2023-12-18 09:35:28 +01:00
Thibault "bui" Koechlin 8cca4346a5
Application Security Engine Support (#2273)
Add a new datasource that:
- Receives HTTP requests from remediation components
- Apply rules on them to determine whether they are malicious or not
- Rules can be evaluated in-band (the remediation component will block the request directly) or out-band (the RC will let the request through, but crowdsec can still process the rule matches with scenarios)

The PR also adds support for 2 new hub items:
- appsec-configs: Configure the Application Security Engine (which rules to load, in which phase)
- appsec-rules: a rule that is added in the Application Security Engine (can use either our own format, or seclang)

---------

Co-authored-by: alteredCoder <kevin@crowdsec.net>
Co-authored-by: Sebastien Blot <sebastien@crowdsec.net>
Co-authored-by: mmetc <92726601+mmetc@users.noreply.github.com>
Co-authored-by: Marco Mariani <marco@crowdsec.net>
2023-12-07 12:21:04 +01:00
mmetc ffcab0b2bc
Refactor hub management and cscli commands (#2545) 2023-11-24 15:57:32 +01:00
Thibault "bui" Koechlin 1dcf9d1ae1
Improved expr debugger (#2495)
* new expr debugger

---------

Co-authored-by: mmetc <92726601+mmetc@users.noreply.github.com>
2023-11-24 11:10:54 +01:00
Laurence Jones 19de3a8a77
Runtime whitelist parsing improvement (#2422)
* Improve whitelist parsing

* Split whitelist check into a function tied to whitelist, also since we check node debug we can make a pointer to node containing whitelist

* No point passing clog as an argument since it is just a pointer to node we already know about

* We should break instead of returning false, false as it may have been whitelisted by ips/cidrs

* reimplement early return if expr errors

* Fix lint and dont need to parse ip back to string just loop over sources

* Log error with node logger as it provides context

* Move getsource to a function cleanup some code

* Change func name

* Split out compile to a function so we can use in tests. Add a bunch of tests

* spell correction

* Use node logger so it has context

* alternative solution

* quick fixes

* Use containswls

* Change whitelist test to use parseipsource and only events

* Make it simpler

* Postoverflow tests, some basic ones to make sure it works

* Use official pkg

* Add @mmetc reco

* Add @mmetc reco

* Change if if to a switch to only evaluate once

* simplify assertions

---------

Co-authored-by: bui <thibault@crowdsec.net>
Co-authored-by: Marco Mariani <marco@crowdsec.net>
2023-10-16 10:08:57 +01:00
Laurence Jones ff7acd3347
Reset grokky once all patterns are compiled as we do not need to hold them in memoory (#2420) 2023-10-13 12:53:42 +01:00
Laurence Jones 64deeab1ec
Fix PO expr whitelist (#2471) 2023-09-19 12:51:03 +01:00
mmetc d45bec4047
minor log message improvements (#2455) 2023-09-12 11:04:56 +02:00
Laurence Jones 86d9384954
Whitelist reason (#2439)
* Update node.go

Dont update whitelist reason if event is whitelisted

* oops
2023-08-23 14:51:37 +01:00
Laurence Jones 0334a9afe8
Add method name to child logger so we can see which function is erroring when in enrichers (#2411) 2023-08-08 13:38:11 +01:00
mmetc ffadd42779
update dependency on go-cs-lib; drop the pkg/ part (#2393) 2023-07-28 16:35:08 +02:00
mmetc a01ce18b98
replace imports of path with path/filepath (#2330) 2023-07-26 10:29:58 +02:00
mmetc b9a3acb03f
light pkg/parser cleanup (#2279)
* pkg/parser: clean up imports
* remove duplicate import
* simplify boolean expression
* don't check length before range
* if..else if.. -> switch/case
* errors.Wrap -> fmt.Errorf
* typo, lint
* redundant break
2023-06-13 13:16:13 +02:00
mmetc 6096cb3c9b
Move grok_pattern.go away from pkg/types to trim bouncer dependencies (#2269) 2023-06-08 15:07:30 +02:00
mmetc 396dcf8e6e
dependencies: replaced function calls to pkg/types, errors.Wrap (#2235)
we now use a generic pointer function, and slowly remove the deprecated pkg/errors
2023-06-01 16:31:56 +02:00
blotus 6720d89845
fix lock when dumping the parsing state in explain mode (#2234) 2023-05-26 15:23:50 +01:00
mmetc 534328ca30
decouple bouncer dependencies: use go-cs-lib/pkg/* (#2216)
* decouple bouncer dependencies: use go-cs-lib/pkg/trace
* decouple bouncer dependencies: use go-cs-lib/pkg/version
* decouple bouncer dependencies: use go-cs-lib/pkg/yamlpatch
* decouple bouncer dependencies: use go-cs-lib/pkg/csstring
* unused import
2023-05-23 10:52:47 +02:00
Thibault "bui" Koechlin 77f2968267
fix the behavior of json unmarshal to not return the full map (#2199) 2023-05-16 09:10:38 +02:00
Laurence Jones 424215f228
Add ParseKV helper and rework UnmarshalJSON as a proper helper (#2184) 2023-05-12 09:43:01 +02:00
blotus 772d5b5c32
Add experimental support for re2 (#2138) 2023-03-28 16:26:47 +02:00
blotus 1095f6c875
use expr.Function for custom functions instead of passing them in the env (#2133) 2023-03-28 10:49:01 +02:00
Thibault "bui" Koechlin a3e5f0a3a0
fix dateparse (#2135) 2023-03-22 08:20:21 +01:00
Thibault "bui" Koechlin 618be9ff68
properly update the time structure within event (#2122)
* properly update the time structure within event to ensure it works in time-machine

* move LIVE and TIMEMACHINE to pkg/types : less code needs to import leakybucket package, and we avoid duplicating constants
2023-03-16 16:25:50 +01:00
Thibault "bui" Koechlin 855f9e6f8d
protect map w/ mutex to avoid concurrent map writes with cscli explain when having many concurrent parser routines (#2113) 2023-03-16 11:01:25 +01:00
mmetc e161507d08
Lint (type inference): remove redundant type declarations (#2111) 2023-03-09 11:56:02 +01:00
Thibault "bui" Koechlin 5b0fe4b7f1
support for regexps result cache (#2104)
* support for regexps result cache : gcache + xxhash

Co-authored-by: Marco Mariani <marco@crowdsec.net>
2023-03-08 16:07:49 +01:00
Thibault "bui" Koechlin a0b264047c
allow user to specify stash's cache strategy (#2037) 2023-02-06 15:42:55 +01:00
Thibault "bui" Koechlin 6fb962a941
Allow parsers to capture data for future enrichment (#1969)
* Allow parsers to capture data in a cache, that can be later accessed via expr helpers (fake multi-line support)
2023-01-11 15:01:02 +01:00
AlteredCoder 185f9ad541
Alert context (#1895)
Co-authored-by: bui <thibault@crowdsec.net>
2023-01-04 16:50:02 +01:00
Laurence Jones fd1c38811e
Add parse unix to dateparse enricher (#1958)
Add parse unix is we do have a strTime but wasnt parsed using convential golang time
2022-12-30 12:47:14 +00:00
mmetc cc228f1868
Typos, grammar (#1905) 2022-12-06 15:55:27 +01:00
blotus fdda940ac0
Add Kubernetes audit acquisition (#1767) 2022-12-06 13:47:29 +01:00
mmetc 104f5d1fe6
lint: error handling cleanup (#1877) 2022-11-29 09:16:07 +01:00
mmetc 895691dad1
enabled linters: gocritic, nilerr (#1853) 2022-11-07 10:36:50 +01:00
mmetc 1d9f861f28
unit tests: always capture testcase variable -> allow parallel testing (#1797) 2022-10-10 10:48:26 +02:00
Manuel Sabban 1f06f242cc
fix https://github.com/crowdsecurity/crowdsec/issues/1746 (#1749)
Co-authored-by: sabban <15465465+sabban@users.noreply.github.com>
2022-09-14 15:26:26 +02:00
mmetc 414282a2c9
golangci-lint 1.49 and related fixes (#1736) 2022-09-06 13:55:03 +02:00
Thibault "bui" Koechlin 866c200c31
Generic dateparse approach (#1669)
* Allow any parser to suggest a format string for the date to be parsed.

* allow the enricher functions to get the parser's logger so they can inherit the level
2022-07-28 16:41:41 +02:00
AlteredCoder 39da36361c
Get geoip Country from other objects if not present (#1659) 2022-07-12 15:26:34 +02:00
AlteredCoder 5f62d738fc
Add no-capi flag and review some logs (#1628)
* Add no-capi flag and review some logs
2022-07-01 16:56:13 +02:00
AlteredCoder a4f4eabf0a
support yml file (#1605) 2022-06-22 17:01:27 +02:00
mmetc 628d7be1d8
simplify err.Error() to err when used in printf context (#1603) 2022-06-22 15:53:53 +02:00