Commit graph

481 commits

Author SHA1 Message Date
AlteredCoder 01ea78c10e
Strip version with ~ instead of - (#2076) 2023-02-25 20:05:48 +01:00
Laurence Jones 75d8b821ff
Explain successful parsers only (#2063)
* Add option to filter down explain to successful parsers useful for me who has every collection installed

* Altered naming conventions so it makes more sense when reading
2023-02-24 13:49:17 +00:00
Laurence Jones 8acce4637a
Option to disable remote lapi registration (#2010)
* Allow to disable remote lapi registration

* Extract method and make it extendable as a generic middleware

* Change method name so it make sense to read abort remote if <config>

* golint
2023-02-24 13:44:21 +00:00
mmetc 20a1bc7d44
chore: simplify pkg/database/alerts (#2062) 2023-02-23 10:25:01 +01:00
mmetc be18fea136
Propagate taints to top collections (fix #2064) (#2066) 2023-02-21 22:12:08 +01:00
mmetc 76ea3a063f
fix message "empty scenario" 2023-02-21 09:59:56 +01:00
blotus 90c38db9f2
Stream decisions from db (#1927) 2023-02-20 15:26:30 +01:00
JDEV 12a4a5fb14
CAPI error code handling tests (#2027)
* Registration mocked error cases

* Authentication mock error cases

* mini facto

* check that getMEtric still has bouncers/machines keys in output even with empty collections

* fixed defer body close(), no need to defer and fprint arg

* fix fatal call

---------

Co-authored-by: jdv <julien@crowdsec.net>
2023-02-17 14:57:46 +01:00
blotus 83c3818504
Do not try to refresh JWT token when doing a login request (#2059) 2023-02-16 16:16:26 +01:00
Laurence Jones 5aca11af70
Show s00 stats instead of "first_parser" (#2055)
* show s00 if verbose is provided

* Clean up code

* Fix failing test
2023-02-14 14:36:08 +00:00
Cristian Nitescu ecb32d74c6
optimize blocklist fetch (#2039) 2023-02-13 15:06:14 +01:00
Cristian Nitescu f280505eaa
omtimization - remove useless login call (#2036) 2023-02-13 15:05:58 +01:00
blotus 812b87ab48
Add IsIPV4() and IsIP() helpers (#2050) 2023-02-10 14:44:42 +01:00
Thibault "bui" Koechlin 0f5560b62a
more strings helpers (#2040)
* more strings helpers
2023-02-09 15:23:21 +01:00
Thibault "bui" Koechlin 1d7d377f8b
changes following BL tests (#2038)
Co-authored-by: Sebastien Blot <sebastien@crowdsec.net>
2023-02-08 10:35:21 +01:00
Thibault "bui" Koechlin a0b264047c
allow user to specify stash's cache strategy (#2037) 2023-02-06 15:42:55 +01:00
Cristian Nitescu 987f119c4b
v3 capi and blocklists links support (#2019)
* v3 model generation

* v3 model generation

* comms

* fixes after master merge

* missing reader close

* use constants defined for types

---------

Co-authored-by: bui <thibault@crowdsec.net>
2023-02-06 14:06:14 +01:00
mmetc b6be18ca65
cscli setup (#1923)
Detect running services and generate acquisition configuration
2023-02-06 07:33:04 +01:00
AlteredCoder 7e871d2278
rename PAPI base URL (#2033) 2023-02-03 12:10:02 +01:00
Thibault "bui" Koechlin e927717fa0
Polling API Integration (#1715)
Co-authored-by: alteredCoder <kevin@crowdsec.net>
Co-authored-by: he2ss <hamza.essahely@gmail.com>
Co-authored-by: Sebastien Blot <sebastien@crowdsec.net>
2023-01-31 14:47:44 +01:00
mmetc e37d09e5b4
use helpers for shorter tests, add a couple of error cases (#2016) 2023-01-26 17:13:31 +01:00
mmetc 3fb3decf49
error if tls.key_file or cert_file are missing (#2020) 2023-01-26 17:12:59 +01:00
mmetc 02be5f3618
allow literal $ in plugin configuration (#2015) 2023-01-23 16:28:43 +01:00
mmetc 47cc60bda9
allow use of literal $ in config.yaml (#2012) 2023-01-23 10:29:29 +01:00
mmetc e5833699c0
cscli config feature-flags (#2006) 2023-01-20 09:32:10 +01:00
Thibault "bui" Koechlin 4f29ce2ee7
CTI API Helpers in expr (#1851)
* Add CTI API helpers in expr
* Allow profiles to have an `on_error` option to profiles

Co-authored-by: Sebastien Blot <sebastien@crowdsec.net>
2023-01-19 08:45:50 +01:00
Marco Mariani 0c35d9d43c wip 2023-01-18 15:15:18 +01:00
Marco Mariani 4f25738d6b wip 2023-01-18 15:15:18 +01:00
Marco Mariani 47dbfa770d configure logging earlier 2023-01-18 15:15:18 +01:00
Marco Mariani 91b0f8fee1 load custom configuration paths when agent is disabled 2023-01-18 15:15:18 +01:00
Marco Mariani 2e91a82aa7 load feature.yaml as soon as possible 2023-01-18 15:15:18 +01:00
Thibault "bui" Koechlin f25fdecc3f
normalize scopes for alerts and decisions (#2001)
* normalize scopes for alerts and decisions
2023-01-18 14:50:03 +01:00
mmetc 51800132cd
improve feature flag logging (#1986)
For cscli: it should provide a terse output, not nag users with configuration details. Although it's usually important that cscli and crowdsec have the same enabled features, having it list them every time the command is invoked can be too much.

For crowdsec: when features are set from the environment, it's too early to log where we should. So we can use log.Debug at activation time, and list them again once logging is configured.

 - wrap some functions in csconfig for convenience and DRY
 - for each enabled feature, log.Debug
 - log all enabled features once as Info (crowdsec) or Debug (cscli)
 - file does not exist -> log.Trace
2023-01-13 13:42:42 +01:00
Cristian Nitescu 73663ff9e7
log the request error even in case of retry (#1988) 2023-01-13 12:58:12 +01:00
mmetc ba4396e52c
fix flaky parser unit test (#1985) 2023-01-12 17:03:25 +01:00
Thibault "bui" Koechlin 6fb962a941
Allow parsers to capture data for future enrichment (#1969)
* Allow parsers to capture data in a cache, that can be later accessed via expr helpers (fake multi-line support)
2023-01-11 15:01:02 +01:00
mmetc cd4dabde0e
silence yaml.local explicitly in cscli, keep in crowdsec/bouncer logs (#1981) 2023-01-11 09:50:46 +01:00
Laurence Jones ca12432a2a
Change patch to debug, if user has a local overide they will get informed every cscli call (#1980) 2023-01-10 10:05:18 +00:00
Cristian Nitescu 7284c0a47a
retry with backoff requests to CAPI (#1957)
* backoff on refresh token error

* fix tls communication with lapi and user/pw auth (#1956)

allow self-signed TLS encryption with user/pw auth

docker:
 - remove defaults for certificate file locations
 - new envvar INSECURE_SKIP_VERIFY
 - register agent before TLS settings (cscli machine add removes them
   from the credentials file)

* separate cscli cobra constructors:  lapi, machines, bouncers, postoverflows (#1945)

* use feature toggling to improve testability with http retry backoff

* Add parse unix to dateparse enricher (#1958)

Add parse unix is we do have a strTime but wasnt parsed using convential golang time

* func tests: redirect stderr to filter extra logs (#1961)

* backoff on refresh token error

* use feature toggling to improve testability with http retry backoff

* refactor feature backoff toggle for tests

Co-authored-by: mmetc <92726601+mmetc@users.noreply.github.com>
Co-authored-by: Laurence Jones <laurence.jones@live.co.uk>
2023-01-09 14:49:21 +01:00
blotus a84e4b6b15
Add conditional bucket (#1962) 2023-01-06 09:26:16 +01:00
AlteredCoder 185f9ad541
Alert context (#1895)
Co-authored-by: bui <thibault@crowdsec.net>
2023-01-04 16:50:02 +01:00
mmetc 033082a31e
ParseUnix() test fix: force UTC (#1970) 2023-01-04 16:22:17 +01:00
mmetc 2d81e751a1
fix parser test 2k23 (#1971) 2023-01-04 15:46:16 +01:00
Laurence Jones fd1c38811e
Add parse unix to dateparse enricher (#1958)
Add parse unix is we do have a strTime but wasnt parsed using convential golang time
2022-12-30 12:47:14 +00:00
mmetc 72c1753fb7
fix tls communication with lapi and user/pw auth (#1956)
allow self-signed TLS encryption with user/pw auth

docker:
 - remove defaults for certificate file locations
 - new envvar INSECURE_SKIP_VERIFY
 - register agent before TLS settings (cscli machine add removes them
   from the credentials file)
2022-12-29 22:00:11 +01:00
Laurence Jones 401739b036
Add unix expr helper (#1952)
* Add unix expr helper

* Add original value not parsed error

* return early if cannot parse

* Add tests

* Fix negative value
2022-12-29 14:53:06 +00:00
Thibault "bui" Koechlin e4463c412b
Improve warnings around lack of evt.StrTime field (#1954)
* fix #1951 : improve error messages

* make hubtest warn you if you're missing evt.StrTime in your logs
2022-12-29 15:03:32 +01:00
mmetc 6efc2688b1
simplify feature flags (#1947)
Now checking for a feature flag is a one liner,
with no need to control errors.

if fflag.Crowdsec.CscliSetup.IsEnabled() {
   ...
}
2022-12-26 14:23:41 +01:00
mmetc 5d2c99bb17
runtime feature flag initialization 2022-12-21 17:19:20 +01:00
mmetc ff88faf402
updated localstack dependencies, added build cache 2022-12-21 12:20:01 +01:00