beenull/crowdsec
1
0
Fork 0
Code Issues Pull requests Packages Projects Releases Wiki Activity
1703 commits 83 branches 177 tags 151 MiB
Commit graph

763 commits

Author SHA1 Message Date
mmetc 507da49b5a
send metrics immediately if agents are added or removed (#2296) 2023-06-23 14:06:04 +02:00
mmetc 9beb5388cb
errors.Wrap -> fmt.Errorf; clean up imports (#2301) 2023-06-23 14:04:58 +02:00
mmetc e42841cd00
Change api_key encoding to base64 to comply with bcrypt max size (#2302) 2023-06-23 13:54:36 +02:00
mmetc 62caffb102
update leakybucket readme (#2298) 2023-06-22 15:35:01 +02:00
mmetc fddf597040
errors.Wrap -> fmt.Errorf; clean up imports (#2297) 2023-06-22 15:01:34 +02:00
mmetc 8bfeb7d90d
Update go dependencies (#2293) 
- update fatih/color (fix windows issue)
- update mongo-driver (fix build issue)
- go.mod: merge two "require" blocks
- update semver dependency (same version as indirect dep), fix test checks in cscli setup
- remove gotest.tools dependency (use testify, cstest)
- update x/ exp, mod, sys dependencies
 2023-06-22 11:31:41 +02:00
Emanuel Seemann 40e6b205bc
Add bayesian bucket type (#2290) 2023-06-21 15:08:27 +02:00
mmetc da6106bd23
spellcheck/style leakybucket readme (#2294) 2023-06-21 11:47:07 +02:00
mmetc f7409d47be
fix error message when failing to parse ip address (#2292) 
Co-authored-by: Thibault "bui" Koechlin <thibault@crowdsec.net>
 2023-06-21 09:22:25 +02:00
Laurence Jones 2c8769adf6
Update jsonextract.go (#2287) 
Return nil instead of empty string as ParseKV does the same
 2023-06-16 18:34:55 +01:00
mmetc b9a3acb03f
light pkg/parser cleanup (#2279) 
* pkg/parser: clean up imports
* remove duplicate import
* simplify boolean expression
* don't check length before range
* if..else if.. -> switch/case
* errors.Wrap -> fmt.Errorf
* typo, lint
* redundant break
 2023-06-13 13:16:13 +02:00
mmetc 76429f033a
trim pkg/types: move DataSet/GetData to pkg/cwhub, removed unused Clone function (#2271) 2023-06-08 16:49:51 +02:00
mmetc cf747d65e0
fix missing import (#2275) 2023-06-08 15:49:37 +02:00
mmetc 25bb23d8b7
minor refactor to pkg/types, cscli machines (#2270) 
* cleanup: separate ui and logic
* trim some code from pkg/types
 2023-06-08 15:08:51 +02:00
mmetc 6096cb3c9b
Move grok_pattern.go away from pkg/types to trim bouncer dependencies (#2269) 2023-06-08 15:07:30 +02:00
mmetc 8da9d5eefd
don't log notification error if not running under systemd (#2274) 2023-06-08 15:04:48 +02:00
mmetc 5b3200173e
don't pre-create log files (not required anymore) (#2267) 
The lumberjack package fixed the issue in natefinch/lumberjack#83 (tested with umask 002) and this code is now redundant since we updated the dependency to v2.2.1.
 2023-06-07 12:58:35 +02:00
mmetc edd062522d
build against libre2-dev if found (#2255) 2023-06-06 15:46:25 +02:00
mmetc 3cc6b2c0d0
CI: add tests for metrics configuration (#2251) 2023-06-05 23:17:30 +02:00
mmetc 0191faf3a8
update notif threshold test on windows (#2265) 2023-06-05 22:58:13 +02:00
mmetc e3cb4ab2c4
do not send more than group_threshold alerts at once to a notification plugin (#2264) 
* do not send more than group_threshold alerts at once to a notification plugin
* Use generic Chunks function, updated tests

---------

Co-authored-by: Sebastien Blot <sebastien@crowdsec.net>
 2023-06-05 12:55:03 +02:00
mmetc a4eee41fd7
log.Warning if a notification is configured twice (#2240) 2023-06-02 14:41:50 +02:00
mmetc 396dcf8e6e
dependencies: replaced function calls to pkg/types, errors.Wrap (#2235) 
we now use a generic pointer function, and slowly remove the deprecated pkg/errors
 2023-06-01 16:31:56 +02:00
mmetc 12c32d507c
CI: refactoring pkg/csplugin tests (#2247) 2023-06-01 10:33:08 +02:00
mmetc 92a9d6c321
types.InSlice() -> slices.Contains() (#2246) 2023-05-31 12:39:22 +02:00
Laurence Jones 4fbc3402fb
Update KV ignore whitespace before and after = (#2236) 
* Update KV ignore whitespace before and after `=`

* Update helpers.go

Don't need whitespace infront of KEY

* Add some tests to ensure edge cases

* Ensure quoted and unquoted values act the same
 2023-05-26 15:35:46 +01:00
blotus 6720d89845
fix lock when dumping the parsing state in explain mode (#2234) 2023-05-26 15:23:50 +01:00
blotus f6924f8c57
generate asserts for evt.Unmarshaled in hubtest (#2214) 2023-05-26 11:44:58 +02:00
mmetc 9167bd107d
decouple bouncer dependencies: use go-cs-lib/pkg/ptr (#2228) 2023-05-25 15:43:39 +02:00
mmetc b2d3520519
decouple bouncer dependencies: use go-cs-lib in test code (#2229) 2023-05-25 15:37:44 +02:00
mmetc 364b833d67
test cleanup: remove /tmp/crowdsec_tests* directories (#2232) 2023-05-25 15:32:32 +02:00
Laurence Jones 0416a41d58
Log info capi whitelists (#2220) 
* add infof command if err was nil

* Fix golint

* Make message more readable and log individual stats

* Missed a d

* Remove '

* simplify if/else logic

---------

Co-authored-by: Marco Mariani <marco@crowdsec.net>
 2023-05-25 10:28:08 +01:00
mmetc 025f14f879
merge system cert pool with own certs (#2226) 2023-05-25 10:10:58 +02:00
mmetc e5fe74ce77
decouple bouncer dependencies: use go-cs-lib/pkg/ptr in apiclient (#2227) 2023-05-25 10:08:52 +02:00
mmetc 534328ca30
decouple bouncer dependencies: use go-cs-lib/pkg/* (#2216) 
* decouple bouncer dependencies: use go-cs-lib/pkg/trace
* decouple bouncer dependencies: use go-cs-lib/pkg/version
* decouple bouncer dependencies: use go-cs-lib/pkg/yamlpatch
* decouple bouncer dependencies: use go-cs-lib/pkg/csstring
* unused import
 2023-05-23 10:52:47 +02:00
blotus 6e3ca35941
fallback to master for hub index download if it does not exist (#2210) 2023-05-17 11:20:53 +02:00
blotus 412b4c4b0b
fix incorrect version strip (#2206) 2023-05-17 01:13:55 +02:00
Thibault "bui" Koechlin 77f2968267
fix the behavior of json unmarshal to not return the full map (#2199) 2023-05-16 09:10:38 +02:00
Laurence Jones 424215f228
Add ParseKV helper and rework UnmarshalJSON as a proper helper (#2184) 2023-05-12 09:43:01 +02:00
mmetc e1f5ed41df
Implement "cscli config show-yaml" (#2191) 2023-05-11 21:01:13 +02:00
blotus 4ae41a363d
add Hostname helper in expr and templating (#2193) 2023-05-11 14:25:04 +02:00
blotus 71b7a594bd
add indexes on the FK between alerts and {decisions,metas,events} (#2188) 2023-05-11 13:49:01 +02:00
blotus 2701454f23
defaults to inotify to detect changes in file datasource to avoid too many call to stat() (#2181) 2023-05-09 10:03:55 +02:00
blotus e1f4a71357
readd KeyExists expr helper (#2180) 2023-05-04 16:55:34 +02:00
blotus a753ea6981
Add B64decode expr helper (#2183) 2023-05-04 14:15:20 +02:00
Thibault "bui" Koechlin 8f71edaadd
do not error on this filter (#2182) 2023-05-04 13:06:15 +02:00
Thibault "bui" Koechlin 4ff8f498ce
add a LogInfo expr helper (#2179) 2023-05-03 10:07:11 +02:00
AlteredCoder 6bb20fa951
fix issue #2172 (#2177) 2023-04-28 16:32:46 +02:00
AlteredCoder c0e6c1ac78
Fix chooseHubBranch when latest() doesn't work (#2178) 
* Fix chooseHubBranch when latest() doesn't works
 2023-04-28 11:24:04 +02:00
Thibault "bui" Koechlin 3041023ed8
add an optional flag to disable the fetch (#2169) 2023-04-14 11:39:16 +02:00
Thibault "bui" Koechlin 66dfded0cf
significantly increase the max number of scenarios to be sent (#2170) 2023-04-14 11:39:07 +02:00
mmetc 0c5d233563
Minor cleanup and dead code removal (#2166) 2023-04-12 16:57:38 +02:00
Laurence Jones 9a5a937695
Make it more obvious that parser succeeded but was whitelisted (#2167) 
* Make it more obvious that parser succeeded but was whitelisted

* Add more verbose by placing whitelist reason next to why it is ignored
 2023-04-12 10:48:42 +01:00
blotus 0279e549bd
check if the acquis tomb is dying while processing logs in replay mode for file/s3/docker (#2152) 2023-04-04 13:57:06 +02:00
mmetc 3132aa54b7
Properly load k8s audit configuration (#2158) 2023-04-03 21:55:31 +02:00
mmetc 38ab6be7c2
Allow feature.yml to change available subcommands (#2156) 2023-04-03 10:11:56 +02:00
mmetc 3fa555fb25
Rename k8s_audit to k8s-audit (easier to type, consistent with labels) (#2153) 2023-04-03 09:53:38 +02:00
blotus 61bea26486
Add transform configuration option for acquisition (#2144) 2023-03-29 16:04:17 +02:00
blotus 772d5b5c32
Add experimental support for re2 (#2138) 2023-03-28 16:26:47 +02:00
blotus 1095f6c875
use expr.Function for custom functions instead of passing them in the env (#2133) 2023-03-28 10:49:01 +02:00
Thibault "bui" Koechlin 169b844212
fix awkward stacktrace in conditional filter (#2145) 2023-03-27 16:01:42 +02:00
mmetc d769fff1e8
File acquisition: log "file reopen" events instead of writing to stderr (#2139) 2023-03-24 11:24:36 +01:00
mmetc 3884c5f47d
Unit tests: remove leftover files (#2134) 2023-03-22 13:51:37 +01:00
Thibault "bui" Koechlin a3e5f0a3a0
fix dateparse (#2135) 2023-03-22 08:20:21 +01:00
blotus 91eb39cff6
New PAPI commands: reauth + force_pull (#2129) 2023-03-21 14:06:19 +01:00
blotus dc38e5ac00
S3 acquisition datasource (#2130) 2023-03-21 13:54:52 +01:00
Thibault "bui" Koechlin a74e424d53
support ip and cidr based whitelists for capi and 3rd party blocklists (#2132) 
* support ip and cidr based whitelists for capi and 3rd party blocklist
 2023-03-21 11:50:10 +01:00
Thibault "bui" Koechlin d87f088b8f
match expr helper (#2126) 
* match expr helper
 2023-03-21 10:39:17 +01:00
Thibault "bui" Koechlin 618be9ff68
properly update the time structure within event (#2122) 
* properly update the time structure within event to ensure it works in time-machine

* move LIVE and TIMEMACHINE to pkg/types : less code needs to import leakybucket package, and we avoid duplicating constants
 2023-03-16 16:25:50 +01:00
blotus c77fe16943
actually fix expr-debugger to work with the new version (#2124) 2023-03-16 15:20:48 +01:00
blotus 94c7efdb5b
add ToString() helper (#2100) 2023-03-16 15:20:31 +01:00
blotus b1f2063a9a
Only support pgx driver for postgresql (#2118) 2023-03-16 11:02:31 +01:00
Thibault "bui" Koechlin 855f9e6f8d
protect map w/ mutex to avoid concurrent map writes with cscli explain when having many concurrent parser routines (#2113) 2023-03-16 11:01:25 +01:00
Manuel Sabban b451d190b7
try to make reproducible build work (#2119) 
Co-authored-by: sabban <15465465+sabban@users.noreply.github.com>
 2023-03-13 17:26:33 +01:00
blotus 6aaf3cd50b
Update expr to 1.12.2 (#2110) 2023-03-09 16:56:11 +01:00
mmetc e161507d08
Lint (type inference): remove redundant type declarations (#2111) 2023-03-09 11:56:02 +01:00
Thibault "bui" Koechlin d95b7afe61
Distance support : Impossible travel (#2108) 
* add distance helpers
 2023-03-08 18:29:42 +01:00
Thibault "bui" Koechlin 9d5aaf5ea2
add --origin to cscli decisions delete (#2109) 2023-03-08 18:29:20 +01:00
Thibault "bui" Koechlin 5b0fe4b7f1
support for regexps result cache (#2104) 
* support for regexps result cache : gcache + xxhash

Co-authored-by: Marco Mariani <marco@crowdsec.net>
 2023-03-08 16:07:49 +01:00
blotus 16a3be49e2
do not try to load PAPI is url is not set (#2099) 2023-03-06 15:38:58 +01:00
blotus 85ab9c68a2
Add cscli papi status and cscli papi sync (#2091) 2023-03-03 13:46:28 +01:00
mmetc f6d6c5bb2b
Add tests and typo fixes (#2092) 2023-03-03 11:06:27 +01:00
AlteredCoder 01ea78c10e
Strip version with ~ instead of - (#2076) 2023-02-25 20:05:48 +01:00
Laurence Jones 75d8b821ff
Explain successful parsers only (#2063) 
* Add option to filter down explain to successful parsers useful for me who has every collection installed

* Altered naming conventions so it makes more sense when reading
 2023-02-24 13:49:17 +00:00
Laurence Jones 8acce4637a
Option to disable remote lapi registration (#2010) 
* Allow to disable remote lapi registration

* Extract method and make it extendable as a generic middleware

* Change method name so it make sense to read abort remote if <config>

* golint
 2023-02-24 13:44:21 +00:00
mmetc 20a1bc7d44
chore: simplify pkg/database/alerts (#2062) 2023-02-23 10:25:01 +01:00
mmetc be18fea136
Propagate taints to top collections (fix #2064) (#2066) 2023-02-21 22:12:08 +01:00
mmetc 76ea3a063f
fix message "empty scenario" 2023-02-21 09:59:56 +01:00
blotus 90c38db9f2
Stream decisions from db (#1927) 2023-02-20 15:26:30 +01:00
JDEV 12a4a5fb14
CAPI error code handling tests (#2027) 
* Registration mocked error cases

* Authentication mock error cases

* mini facto

* check that getMEtric still has bouncers/machines keys in output even with empty collections

* fixed defer body close(), no need to defer and fprint arg

* fix fatal call

---------

Co-authored-by: jdv <julien@crowdsec.net>
 2023-02-17 14:57:46 +01:00
blotus 83c3818504
Do not try to refresh JWT token when doing a login request (#2059) 2023-02-16 16:16:26 +01:00
Laurence Jones 5aca11af70
Show s00 stats instead of "first_parser" (#2055) 
* show s00 if verbose is provided

* Clean up code

* Fix failing test
 2023-02-14 14:36:08 +00:00
Cristian Nitescu ecb32d74c6
optimize blocklist fetch (#2039) 2023-02-13 15:06:14 +01:00
Cristian Nitescu f280505eaa
omtimization - remove useless login call (#2036) 2023-02-13 15:05:58 +01:00
blotus 812b87ab48
Add IsIPV4() and IsIP() helpers (#2050) 2023-02-10 14:44:42 +01:00
Thibault "bui" Koechlin 0f5560b62a
more strings helpers (#2040) 
* more strings helpers
 2023-02-09 15:23:21 +01:00
Thibault "bui" Koechlin 1d7d377f8b
changes following BL tests (#2038) 
Co-authored-by: Sebastien Blot <sebastien@crowdsec.net>
 2023-02-08 10:35:21 +01:00
Thibault "bui" Koechlin a0b264047c
allow user to specify stash's cache strategy (#2037) 2023-02-06 15:42:55 +01:00
Cristian Nitescu 987f119c4b
v3 capi and blocklists links support (#2019) 
* v3 model generation

* v3 model generation

* comms

* fixes after master merge

* missing reader close

* use constants defined for types

---------

Co-authored-by: bui <thibault@crowdsec.net>
 2023-02-06 14:06:14 +01:00
mmetc b6be18ca65
cscli setup (#1923) 
Detect running services and generate acquisition configuration
 2023-02-06 07:33:04 +01:00
AlteredCoder 7e871d2278
rename PAPI base URL (#2033) 2023-02-03 12:10:02 +01:00
Thibault "bui" Koechlin e927717fa0
Polling API Integration (#1715) 
Co-authored-by: alteredCoder <kevin@crowdsec.net>
Co-authored-by: he2ss <hamza.essahely@gmail.com>
Co-authored-by: Sebastien Blot <sebastien@crowdsec.net>
 2023-01-31 14:47:44 +01:00
mmetc e37d09e5b4
use helpers for shorter tests, add a couple of error cases (#2016) 2023-01-26 17:13:31 +01:00
mmetc 3fb3decf49
error if tls.key_file or cert_file are missing (#2020) 2023-01-26 17:12:59 +01:00
mmetc 02be5f3618
allow literal $ in plugin configuration (#2015) 2023-01-23 16:28:43 +01:00
mmetc 47cc60bda9
allow use of literal $ in config.yaml (#2012) 2023-01-23 10:29:29 +01:00
mmetc e5833699c0
cscli config feature-flags (#2006) 2023-01-20 09:32:10 +01:00
Thibault "bui" Koechlin 4f29ce2ee7
CTI API Helpers in expr (#1851) 
* Add CTI API helpers in expr
* Allow profiles to have an `on_error` option to profiles

Co-authored-by: Sebastien Blot <sebastien@crowdsec.net>
 2023-01-19 08:45:50 +01:00
Marco Mariani 0c35d9d43c wip 2023-01-18 15:15:18 +01:00
Marco Mariani 4f25738d6b wip 2023-01-18 15:15:18 +01:00
Marco Mariani 47dbfa770d configure logging earlier 2023-01-18 15:15:18 +01:00
Marco Mariani 91b0f8fee1 load custom configuration paths when agent is disabled 2023-01-18 15:15:18 +01:00
Marco Mariani 2e91a82aa7 load feature.yaml as soon as possible 2023-01-18 15:15:18 +01:00
Thibault "bui" Koechlin f25fdecc3f
normalize scopes for alerts and decisions (#2001) 
* normalize scopes for alerts and decisions
 2023-01-18 14:50:03 +01:00
mmetc 51800132cd
improve feature flag logging (#1986) 
For cscli: it should provide a terse output, not nag users with configuration details. Although it's usually important that cscli and crowdsec have the same enabled features, having it list them every time the command is invoked can be too much.

For crowdsec: when features are set from the environment, it's too early to log where we should. So we can use log.Debug at activation time, and list them again once logging is configured.

 - wrap some functions in csconfig for convenience and DRY
 - for each enabled feature, log.Debug
 - log all enabled features once as Info (crowdsec) or Debug (cscli)
 - file does not exist -> log.Trace
 2023-01-13 13:42:42 +01:00
Cristian Nitescu 73663ff9e7
log the request error even in case of retry (#1988) 2023-01-13 12:58:12 +01:00
mmetc ba4396e52c
fix flaky parser unit test (#1985) 2023-01-12 17:03:25 +01:00
Thibault "bui" Koechlin 6fb962a941
Allow parsers to capture data for future enrichment (#1969) 
* Allow parsers to capture data in a cache, that can be later accessed via expr helpers (fake multi-line support)
 2023-01-11 15:01:02 +01:00
mmetc cd4dabde0e
silence yaml.local explicitly in cscli, keep in crowdsec/bouncer logs (#1981) 2023-01-11 09:50:46 +01:00
Laurence Jones ca12432a2a
Change patch to debug, if user has a local overide they will get informed every cscli call (#1980) 2023-01-10 10:05:18 +00:00
Cristian Nitescu 7284c0a47a
retry with backoff requests to CAPI (#1957) 
* backoff on refresh token error

* fix tls communication with lapi and user/pw auth (#1956)

allow self-signed TLS encryption with user/pw auth

docker:
 - remove defaults for certificate file locations
 - new envvar INSECURE_SKIP_VERIFY
 - register agent before TLS settings (cscli machine add removes them
   from the credentials file)

* separate cscli cobra constructors:  lapi, machines, bouncers, postoverflows (#1945)

* use feature toggling to improve testability with http retry backoff

* Add parse unix to dateparse enricher (#1958)

Add parse unix is we do have a strTime but wasnt parsed using convential golang time

* func tests: redirect stderr to filter extra logs (#1961)

* backoff on refresh token error

* use feature toggling to improve testability with http retry backoff

* refactor feature backoff toggle for tests

Co-authored-by: mmetc <92726601+mmetc@users.noreply.github.com>
Co-authored-by: Laurence Jones <laurence.jones@live.co.uk>
 2023-01-09 14:49:21 +01:00
blotus a84e4b6b15
Add conditional bucket (#1962) 2023-01-06 09:26:16 +01:00
AlteredCoder 185f9ad541
Alert context (#1895) 
Co-authored-by: bui <thibault@crowdsec.net>
 2023-01-04 16:50:02 +01:00
mmetc 033082a31e
ParseUnix() test fix: force UTC (#1970) 2023-01-04 16:22:17 +01:00
mmetc 2d81e751a1
fix parser test 2k23 (#1971) 2023-01-04 15:46:16 +01:00
Laurence Jones fd1c38811e
Add parse unix to dateparse enricher (#1958) 
Add parse unix is we do have a strTime but wasnt parsed using convential golang time
 2022-12-30 12:47:14 +00:00
mmetc 72c1753fb7
fix tls communication with lapi and user/pw auth (#1956) 
allow self-signed TLS encryption with user/pw auth

docker:
 - remove defaults for certificate file locations
 - new envvar INSECURE_SKIP_VERIFY
 - register agent before TLS settings (cscli machine add removes them
   from the credentials file)
 2022-12-29 22:00:11 +01:00
Laurence Jones 401739b036
Add unix expr helper (#1952) 
* Add unix expr helper

* Add original value not parsed error

* return early if cannot parse

* Add tests

* Fix negative value
 2022-12-29 14:53:06 +00:00
Thibault "bui" Koechlin e4463c412b
Improve warnings around lack of evt.StrTime field (#1954) 
* fix #1951 : improve error messages

* make hubtest warn you if you're missing evt.StrTime in your logs
 2022-12-29 15:03:32 +01:00
mmetc 6efc2688b1
simplify feature flags (#1947) 
Now checking for a feature flag is a one liner,
with no need to control errors.

if fflag.Crowdsec.CscliSetup.IsEnabled() {
   ...
}
 2022-12-26 14:23:41 +01:00
mmetc 5d2c99bb17
runtime feature flag initialization 2022-12-21 17:19:20 +01:00
mmetc ff88faf402
updated localstack dependencies, added build cache 2022-12-21 12:20:01 +01:00
mmetc a32aa96752
feature flags (#1933) 
Package fflag provides a simple feature flag system.

 Feature names are lowercase and can only contain letters, numbers, undercores
 and dots.

 good: "foo", "foo_bar", "foo.bar"
 bad: "Foo", "foo-bar"

 A feature flag can be enabled by the user with an environment variable
 or by adding it to {ConfigDir}/feature.yaml

 I.e. CROWDSEC_FEATURE_FOO_BAR=true
 or in feature.yaml:
```
 ---
 - foo_bar
```

 If the variable is set to false, the feature can still be enabled
 in feature.yaml. Features cannot be disabled in the file.

 A feature flag can be deprecated or retired. A deprecated feature flag is
 still accepted but a warning is logged. A retired feature flag is ignored
 and an error is logged.

 A specific deprecation message is used to inform the user of the behavior
 that has been decided when the flag is/was finally retired.
 2022-12-20 16:11:51 +01:00
he2ss 579cecde04
apiclient: fix http roundtrip (clone body also) (#1758) 
* apiclient: fix http roundtrip (clone body also)
 2022-12-14 16:42:46 +01:00
Laurence Jones fe23da6e0c
Add postgres socket support, clean some code (#1926) 2022-12-12 16:08:19 +00:00
Laurence Jones 11965f08db
Add socket support to mysql (#1911) 2022-12-08 09:33:08 +00:00
mmetc cc228f1868
Typos, grammar (#1905) 2022-12-06 15:55:27 +01:00
blotus fdda940ac0
Add Kubernetes audit acquisition (#1767) 2022-12-06 13:47:29 +01:00
mmetc fd3e668fe1
add -error flag to crowdsec binary (#1903) 2022-12-03 08:56:11 +01:00
mmetc fa0e590778
removed pid_dir (#1906) 2022-12-02 13:42:43 +01:00
mmetc 4a6a9c4355
acquisition: validate datasources before configuration (static checks) (#1841) 
* acquisition: validate datasources before configuration (allow static configuration checks)

* remove comment

* import reviser, format

* error wrap
 2022-11-30 17:36:56 +01:00
blotus 60f1228030
use a copy of bucket processors in LeakRoutine (#1902) 2022-11-30 10:59:47 +01:00
mmetc 104f5d1fe6
lint: error handling cleanup (#1877) 2022-11-29 09:16:07 +01:00
mmetc 66543493b5
fix nil dereference: check that httpServer is set before shutting down (#1893) 2022-11-28 11:55:08 +01:00
mmetc fde9640364
Docker refactoring, tls setup (#1869) 2022-11-28 10:35:12 +01:00
blotus c5079ac15e
invalidate agent token on 403 as well (#1888) 2022-11-25 14:35:50 +01:00
mmetc 5bdd3bbfcb
require at least go 1.18 to build (#1884) 2022-11-24 11:29:54 +01:00
Laurence Jones 4ac01ed880
Update perms for group read (#1876) 2022-11-21 09:49:56 +00:00
mmetc 3beb84bcfe
print missing "AS" values as empty strings instead of "0 " (#1867) 2022-11-14 09:55:53 +01:00
Thibault "bui" Koechlin 523343b174
notify when community-blocklist starts pull (#1845) 
* minor change to notify blocklist pull update, will make eventual troubleshooting easier
 2022-11-08 10:44:25 +01:00
Thibault "bui" Koechlin 3b4da7e637
fix #1860 : Only repeat the WAL warning once (#1863) 
* fix #1860
 2022-11-07 16:36:39 +01:00
mmetc 895691dad1
enabled linters: gocritic, nilerr (#1853) 2022-11-07 10:36:50 +01:00
Manuel Sabban 8aca00326d
fix ticker (#1858) 
Co-authored-by: sabban <15465465+sabban@users.noreply.github.com>
 2022-11-04 13:56:43 +01:00
Laurence Jones 668627f890
Add error checking to lookup host (#1847) 2022-10-31 18:38:01 +00:00
mmetc 344b1dc559
fixed package tests w/wal, gitignore/typos (#1849) 2022-10-31 10:02:51 +01:00
mmetc df88f4e1e9
randomize pull, push and metric intervals; reload crowdsec only when hub changed (#1846) 2022-10-28 13:55:59 +02:00
mmetc 02d2eab18c
update golangci-lint to 1.50 and fixes (#1828) 2022-10-26 15:11:37 +02:00
ThinkChaos 22479a289d
Add LookupHost expr lib func (#1775) 2022-10-26 10:17:48 +01:00
mmetc 2088bb1f91
fix for #1839 (#1840) 2022-10-26 11:02:12 +02:00
blotus b7c4bfd4e3
Use explicit transaction when inserting community blocklist (#1835) 2022-10-26 10:48:17 +02:00
mmetc e545933923
fix(cscli): correct and test the behavior of "cscli collections delete" (#1824) 2022-10-25 14:10:51 +02:00
blotus bb2f0e938f
Blocklist: Do not duplicate decisions when pulling (#1796) 2022-10-19 15:51:40 +02:00
Thibault "bui" Koechlin ae6bf39495
support decisions deletion via scenario + alerts delete via ID (#1798) 2022-10-19 14:37:27 +02:00
mmetc 6b0097a24b
change warning to debug when directories are missing in hub sync (#1819) 2022-10-18 10:32:54 +02:00
mmetc 2b7e3ff1e7
warn if no acquisition files are found, acquisition_test refactoring, tests (#1816) 2022-10-17 17:32:08 +02:00
mmetc ec0d2a5ed2
refactor broker_test.go, extract cstest/filenotfound*.go (#1815) 2022-10-17 14:17:23 +02:00
mmetc a96b3e077d
rename pkg/cstest -> pkg/hubtest (#1811) 
keep cstest for generic helper functions
this also avoids circular imports in test files
 2022-10-17 09:24:07 +02:00
mmetc 8fecc2c00b
enable staticcheck linter; fixes (#1806) 
- explicitly ignore returned parameters
 - replace Walk with faster WalkDir
 - log path error during hub dir sync
 - colorize static unit tests
 - removed duplicate import in crowdsec/main.go
 - typos
 - func tests: default datasource in tests/var/log instead of /tmp
 - action setup-go v3
 2022-10-14 16:12:21 +02:00
Manuel Sabban 7359586f1c
fix ticker mix up (#1807) 
Co-authored-by: sabban <15465465+sabban@users.noreply.github.com>
 2022-10-13 14:30:27 +02:00
mmetc 4b3c9c2806
print cscli usage in color, fix windows terminal detection (#1801) 2022-10-13 12:28:24 +02:00
mmetc 7674f907c4
replace log.Fatal with t.Fatal (#1805) 
This is required to run deferred teardown functions
 2022-10-13 10:42:46 +02:00
mmetc 1d9f861f28
unit tests: always capture testcase variable -> allow parallel testing (#1797) 2022-10-10 10:48:26 +02:00
Shivam Sandbhor 74659a82ab
Fast bulk alert delete (#1791) 2022-10-07 12:40:30 +02:00
mmetc ddd75eae9a
cscli: new tables, --color yes|no|auto option (#1763) 2022-10-07 11:05:35 +02:00
AlteredCoder b95a67751e
Update ent and grokky package (#1772) 
* Update ent and grokky package
 2022-10-06 14:55:42 +02:00
Manuel Sabban 83841d801c
fork dlog to ease debian packaging on official repos (#1790) 
Co-authored-by: sabban <15465465+sabban@users.noreply.github.com>
 2022-10-06 13:40:31 +02:00
Shivam Sandbhor 65c0b9ebcf
Simplify one shot tests (#1786) 2022-10-06 11:57:26 +02:00
blotus 3ba67bad3d
remove a wrong warning when pulling list content from CAPI (#1789) 2022-10-06 11:48:06 +02:00
mmetc 9b3be5c2e8
Bulk delete alert optimization (#1782) 2022-10-05 17:07:44 +02:00
Shivam Sandbhor b203b3f444
Fix flakey test in file_tests (#1783) 
Signed-off-by: Shivam Sandbhor <shivam.sandbhor@gmail.com>
 2022-10-05 16:40:09 +02:00
mmetc 6120571421
fix & cleanup cloudwatch_test.go (#1780) 2022-10-04 09:48:59 +02:00
mmetc edced6818a
cleanup + fix flaky tests in file_test.go, apic_test.go (#1773) 2022-09-30 16:01:42 +02:00
blotus bfbe180101
Tighten windows sqlite database permissions (#1769) 2022-09-28 16:18:00 +02:00
Sean Kelly 568eb1d4e0
Fix misspelling of instantiate participles (#1759) 2022-09-27 17:13:43 +02:00
Laurence Jones 21e5b0d6d0
Improvement: Docker one shot error message (#1666) 
* In one shot, user would only specify one container?
 2022-09-27 16:20:30 +02:00
Manuel Sabban 1f06f242cc
fix https://github.com/crowdsecurity/crowdsec/issues/1746 (#1749) 
Co-authored-by: sabban <15465465+sabban@users.noreply.github.com>
 2022-09-14 15:26:26 +02:00
blotus 9b3ff82542
add WAL support for sqlite (#1752) 2022-09-14 15:09:54 +02:00
AlteredCoder 7d97729eea
Add config option to enable or not local API and agent (#1730) 
* Add flag to enable or not local API and agent
 2022-09-12 14:38:29 +02:00
AlteredCoder b06167a3fa
Allow plugins to load environment variable (#1727) 
* Allow plugins to load environment variable
 2022-09-08 11:41:28 +02:00
Manuel Sabban b2130b1593
Fix 1737 (#1738) 
* add GetMeta to *types.Event

Co-authored-by: sabban <15465465+sabban@users.noreply.github.com>
 2022-09-07 10:11:39 +02:00
Thibault "bui" Koechlin 9d199fd4a9
fix #1733 : add support for exclusion regexps (#1735) 
* allow to specify a list of regular expressions to skip some specific files
 2022-09-06 14:58:37 +02:00
mmetc 414282a2c9
golangci-lint 1.49 and related fixes (#1736) 2022-09-06 13:55:03 +02:00
Laurence Jones e674537d0b
Update sprig to v3 (#1722) 
* Update sprig to v3
 2022-09-05 09:05:50 +02:00
he2ss ea40ffd655
Datasource/kafka (#1698) 
* add Kafka datasource
 2022-08-30 17:03:45 +02:00
Manuel Sabban 7d0f89df29
Implement reinject command to send notifications of alerts (#1638) 
* implement reinject command to send notifications of alerts using a profile

Co-authored-by: sabban <15465465+sabban@users.noreply.github.com>
 2022-08-30 15:45:52 +02:00
Thibault "bui" Koechlin bacea50485
allow user to disable decision deduplication (#1687) 
* allow user to disable decision deduplication
 2022-08-26 14:17:46 +02:00
blotus 1f5224b74b
switch to go 1.19 (#1709) 2022-08-26 13:31:49 +02:00
mmetc eea07b7a1a
golangci-lint v1.48 and fixes for "usestdlibvars" (#1711) 2022-08-16 09:46:10 +02:00
AlteredCoder 1002affc16
cscli machines delete: return an error if machines doesn't exist (#1689) 
* cscli machines delete: return an error if machines doesn't exist
 2022-07-28 17:32:12 +02:00
Thibault "bui" Koechlin 866c200c31
Generic dateparse approach (#1669) 
* Allow any parser to suggest a format string for the date to be parsed.

* allow the enricher functions to get the parser's logger so they can inherit the level
 2022-07-28 16:41:41 +02:00
Thibault "bui" Koechlin 0eea20fa7c
revert decision dedup behavior to 1.3.4 (#1675) 
* revert decision dedup behavior to 1.3.4
 2022-07-22 11:20:10 +02:00
Thibault "bui" Koechlin bd91ddaf52
logging consistency for .local files (#1655) 2022-07-13 10:56:03 +02:00
blotus 7b8cd63b04
do not set the UDP read buffer size in syslog datasource (#1657) 2022-07-13 10:18:03 +02:00
AlteredCoder 39da36361c
Get geoip Country from other objects if not present (#1659) 2022-07-12 15:26:34 +02:00
Thibault "bui" Koechlin 73f336363a
bump log level when overloading config file with .local (#1646) 
* bump log level

Co-authored-by: Marco Mariani <marco@crowdsec.net>
 2022-07-08 09:29:23 +02:00
blotus 5c1c941851
close response body in heartbeat (#1637) 2022-07-06 14:50:54 +02:00
AlteredCoder 5f62d738fc
Add no-capi flag and review some logs (#1628) 
* Add no-capi flag and review some logs
 2022-07-01 16:56:13 +02:00
Thibault "bui" Koechlin ca4cd6d559
attempt to fix ticker leak (#1620) 2022-06-30 17:36:01 +02:00
AlteredCoder 02e0f3c095
Fix event.timestamp pointer usage (#1621) 
* Fix event.timestamp pointer usage

* avoid returning an error when creating alerts if something goes wrong during the parsing

Co-authored-by: Sebastien Blot <sebastien@crowdsec.net>
 2022-06-30 17:35:42 +02:00
blotus 863facaa33
Properly handle expired decisions with different scopes (#1616) 2022-06-29 16:13:04 +02:00
Thibault "bui" Koechlin 15902dcba6
fix #1615 : cleanup based on heartbeat instead (#1617) 2022-06-29 13:21:58 +02:00
mmetc 8e7e799304
[wip] serve metrics only after agent and/or lapi are ready; fixed some func tests (#1613) 2022-06-24 15:55:21 +02:00
he2ss 7fafb483ad
Pkg/database/fix count decisions since by value (#1606) 
* fix CountDecisionsSinceByValue to have also expired decisions
 2022-06-23 12:04:07 +02:00
AlteredCoder a4f4eabf0a
support yml file (#1605) 2022-06-22 17:01:27 +02:00
mmetc 628d7be1d8
simplify err.Error() to err when used in printf context (#1603) 2022-06-22 15:53:53 +02:00
he2ss 3d6f015211
Add duration expr to add duration formula (#1556) 
* add duration expr to add duration formula
 2022-06-22 11:29:52 +02:00
Thibault "bui" Koechlin a6ed08b239
Add alerts and decisions metrics, LAPI and agent timing prom metrics (#1546) 2022-06-22 11:14:34 +02:00
AlteredCoder 0a39066f9d
Fix #1552 (#1569) 2022-06-22 10:29:02 +02:00
mmetc d71279f023
added flag crowdsec --warning (#1461) 2022-06-22 09:38:23 +02:00
mmetc c78c833400
CI: colored test output, colored crowdsec and crowdsec-api logs, full final db dump for mysql and sqlite (#1596) 
* github-ci: color unit test output and logs
* new config option: force_color_logs (useful in CI)
* bats: show sqlite/mysql dump at the end
* removed "-v" (print package names) from "go build"
* general workflow cleanup
 2022-06-17 16:12:49 +02:00
mmetc 10585bfecc
enabled linters and fixes for: misspell, predeclared, unconvert, ineffassign, gosimple, govet (#1595) 2022-06-16 14:41:54 +02:00
Thibault "bui" Koechlin ff72a3c1c7
avoid create a new name generator at each bucket instanciation, it's not that cheap (#1591) 2022-06-15 10:02:00 +02:00
blotus 9c1b78395a
reduce verbosity of TLS auth and FlushAgentsAndBouncers (#1588) 2022-06-13 16:08:00 +02:00
Thibault "bui" Koechlin 581ddf78fc
Performance improvements (#1583) 
* fix concurrent map write on distinct cache

* cache compiled expressions for groupby and cancel_on filters

* limit objects copy when it's going to lock a shared goroutine
 2022-06-13 14:41:05 +02:00
Thibault "bui" Koechlin 567e0ab7d1
fix concurrent map write on distinct cache (#1582) 2022-06-10 09:39:23 +02:00
Thibault "bui" Koechlin 1c0fe09576
Add support for certificate authentication for agents and bouncers (#1428) 2022-06-08 16:05:52 +02:00
blotus bdda8691ff
New syslog parser for syslog datasource (#1554) 2022-06-08 15:16:58 +02:00
blotus 4b311684ab
Add more JSON expr helpers (#1576) 2022-06-08 12:15:29 +02:00
mmetc 799cc82bb5
functional tests, minor refactoring and lint/cleanup (#1570) 
* cmd/crowdsec: removed log.Fatal()s, added tests and print error for unrecognized argument
* updated golangci-lint to v1.46
* lint/deadcode: fix existing issues
* tests: cscli config backup/restore
* tests: cscli completion powershell/fish
* err check: pflags MarkHidden()
* empty .dockerignore (and explain the reason)
* tests, errors.Wrap
* test for CS_LAPI_SECRET and minor refactoring
* minor style changes
* log cleanup
 2022-06-06 15:24:48 +02:00
mmetc 88a4801d6a
allow run-tests with -f "<test-name>" (#1564) 2022-05-28 22:10:27 +02:00
he2ss e88e9946f9
Crowdsec/decisions_stream bug fix (#1517) 
* Fix bug when stream interval is greater or equal to 60s

Co-authored-by: alteredCoder <kevin@crowdsec.net>
 2022-05-27 15:23:59 +02:00
mmetc 1fc9587919
fix #1283: update and enable error reports from golangci (#1523) 2022-05-25 22:27:50 +02:00
mmetc 1a293a2a27
cwhub: export SetHubBranch (#1559) 2022-05-24 15:46:48 +02:00
mmetc 357899b83e
fixed uid/gid bound check regression (#1555) 2022-05-23 09:46:39 +02:00
Thibault "bui" Koechlin 0483b9c641
do not spew.Sdump() the invalid node on error. It leads to huge memory usage, especially if the parsers refers ie. datafile (#1550) 2022-05-20 13:29:47 +02:00
AlteredCoder 1e1741aa45
Allow to set static to a pointer and add IsIPV6 helper (#1540) 
* Allow to set static to a pointer and add IsIPV6 helper
 2022-05-19 16:28:25 +02:00
Thibault "bui" Koechlin fe09737d80
Add support for machine heartbeat (#1541) 
* add the last_heartbeat field

* add heartbeat controller

* add endpoint of heartbeat

* heartbeat integration

* add last_heartbeat to cscli machines list
 2022-05-19 15:47:27 +02:00
mmetc 131ed1b0a7
error reporting (#1501) 
* unified error reporting, removed redundancy, tests
 2022-05-19 10:48:08 +02:00
Manuel Sabban 18030e6c58
add notifications command (#1537) 
* add notifications command

Co-authored-by: sabban <15465465+sabban@users.noreply.github.com>
 2022-05-18 16:13:33 +02:00
Shivam Sandbhor 220bbe5862
Document LAPI filters (#1535) 
Signed-off-by: Shivam Sandbhor <shivam.sandbhor@gmail.com>
 2022-05-18 11:45:12 +02:00
mmetc 98f2ac5e7c
fix #1385: .yaml.local (#1497) 
Added support for .yaml.local files to override values in .yaml
 2022-05-18 10:08:37 +02:00
blotus 39f7e38444
retry to send alert to plugin channel if it fails (#1530) 2022-05-17 16:57:15 +02:00
Thibault "bui" Koechlin fbcb2ed7fd
Improve distinct/uniq behaviour (#1478) 
* make uniq/distinct use a cache that is independant of the bucket's cache_size

* add testing specifically for cache_size
 2022-05-17 12:45:53 +02:00
blotus 0449ec1868
Windows Support (#1159) 2022-05-17 12:14:59 +02:00
Cristian Nitescu a49b023a28
GetExprEnv usage optimization (#1515) 
* avoid multiples calls to GetExprEnv

* cache ExprEnv in node process

* use global expression env

* remove block profile rate
 2022-05-17 10:50:37 +02:00
blotus 8f111680bf
Allow to override statics in hubtest. (#1495) 2022-04-29 14:24:41 +02:00
blotus 64369b5c2b
add expr XML helpers (#1493) 2022-04-29 13:52:23 +02:00
blotus 392708a804
Fix docker flaky test (#1494) 2022-04-29 12:16:49 +02:00
AlteredCoder f22e4eb24e
Improve MySQL performance (#1477) 
* Improve MySQL performance
 2022-04-28 12:53:14 +02:00
AlteredCoder be977d1cc4
Fix cwhub collections uninstall dependencies (#1486) 
* Fix cwhub collections uninstall dependencies
 2022-04-27 18:28:03 +02:00