* Add parsers length check as it can panic is enrich is empty
* Lets get smarter and loop backwards to find last successful stage
* Shorten code
---------
Co-authored-by: Thibault "bui" Koechlin <thibault@crowdsec.net>
* move function GetLineCountForFile from pkg/types to cscli
* move ParseDuration from pkg/types to pkg/database
* remove unused types.Profile, types.RemediationProfile
now it does not attempt any retry, instead of attempting all retries
immediately
example: cannot reach LAPI
Before:
$ CROWDSEC_FEATURE_DISABLE_HTTP_RETRY_BACKOFF=true cscli decisions list
ERRO[27-07-2023 10:44:44] error while performing request: dial tcp [::1]:8080: connect: connection refused; 4 retries left
INFO[27-07-2023 10:44:44] retrying in 0 seconds (attempt 2 of 5)
[...]
ERRO[27-07-2023 10:44:44] error while performing request: dial tcp [::1]:8080: connect: connection refused; 1 retries left
INFO[27-07-2023 10:44:44] retrying in 0 seconds (attempt 5 of 5)
ERRO[27-07-2023 10:44:44] error while performing request: dial tcp [::1]:8080: connect: connection refused; 0 retries left
FATA[27-07-2023 10:44:44] Unable to list decisions : performing request: Get "http://localhost:8080/v1/alerts?has_active_decision=true&include_capi=false&limit=100": could not get jwt token: Post "http://localhost:8080/v1/watchers/login": dial tcp [::1]:8080: connect: connection refused
After:
$ CROWDSEC_FEATURE_DISABLE_HTTP_RETRY_BACKOFF=true ./test/local/bin/cscli decisions list
FATA[11-08-2023 16:49:58] unable to retrieve decisions: performing request: Get "http://127.0.0.1:8080/v1/alerts?has_active_decision=true&include_capi=false&limit=100": could not get jwt token: Post "http://127.0.0.1:8080/v1/watchers/login": dial tcp 127.0.0.1:8080: connect: connection refused
* apic: minor refactoring
* Add whitelist length check
If user configures the file but fails to define and actual whitelist we should check length to save allocs
* Init with length from file
* extract loop method from ApplyApicWhitelists
* pass pointer
* extract loop method updateBlocklist
---------
Co-authored-by: Laurence Jones <laurence.jones@live.co.uk>
* Add bouncers prune command
* No point overloading functions
* Add prune to list of commands
* change all short desc to be similar, and made it really really clear when pruning it is not recoverable
* Dont use log. and dont return error on user input to abort
* Add machines prune command
* Fix scope variable for naming scheme
* Add some freshness and add new features
* Fix force and fix duration if less than 60
* Allow duration to be more readable
* Fix description
* Improve func wording and make int machines length
* No point overloading functions
* Add prune to list of commands
* Check if GID is already the group if so no need to chown
* Revert "Check if GID is already the group if so no need to chown"
This reverts commit c7cef1773e.
* change all short desc to be similar, and made it really really clear when pruning it is not recoverable
* Better examples
* Match bouncer like for like
* Fix merge error
* Dont use log. and dont return error on user input to abort
* Add version override and update
* Ooppsie
* Quick fix
* fgs copilot
* Allow user to overwrite image, add warning for exposing metabase and general cleanup
* One ix
* Default image if not found in config, and add a warning to remove and update
* Reorder check system memory checks so it inline with @mmetc best pratices
* No need for err
* Clean up some group code
* Change ipv6 as [] seems to wildcard
* Split loopback warn and disclaimer. Add force yes to start to allow user to accept disclaimer by default
* All cmd commands are RunE clean up
* Update flag name and dont allow a shorthand
This on the other hand, gives a new fatal error when there are no valid datasources.
In the previous version, crowdsec kept running with just a warning if no
acquisition yaml or dir were specified.
The lumberjack package fixed the issue in natefinch/lumberjack#83 (tested with umask 002) and this code is now redundant since we updated the dependency to v2.2.1.
* do not send more than group_threshold alerts at once to a notification plugin
* Use generic Chunks function, updated tests
---------
Co-authored-by: Sebastien Blot <sebastien@crowdsec.net>
* Update KV ignore whitespace before and after `=`
* Update helpers.go
Don't need whitespace infront of KEY
* Add some tests to ensure edge cases
* Ensure quoted and unquoted values act the same
* add infof command if err was nil
* Fix golint
* Make message more readable and log individual stats
* Missed a d
* Remove '
* simplify if/else logic
---------
Co-authored-by: Marco Mariani <marco@crowdsec.net>
* properly update the time structure within event to ensure it works in time-machine
* move LIVE and TIMEMACHINE to pkg/types : less code needs to import leakybucket package, and we avoid duplicating constants
* Add option to filter down explain to successful parsers useful for me who has every collection installed
* Altered naming conventions so it makes more sense when reading
* Allow to disable remote lapi registration
* Extract method and make it extendable as a generic middleware
* Change method name so it make sense to read abort remote if <config>
* golint
* Registration mocked error cases
* Authentication mock error cases
* mini facto
* check that getMEtric still has bouncers/machines keys in output even with empty collections
* fixed defer body close(), no need to defer and fprint arg
* fix fatal call
---------
Co-authored-by: jdv <julien@crowdsec.net>
* v3 model generation
* v3 model generation
* comms
* fixes after master merge
* missing reader close
* use constants defined for types
---------
Co-authored-by: bui <thibault@crowdsec.net>
For cscli: it should provide a terse output, not nag users with configuration details. Although it's usually important that cscli and crowdsec have the same enabled features, having it list them every time the command is invoked can be too much.
For crowdsec: when features are set from the environment, it's too early to log where we should. So we can use log.Debug at activation time, and list them again once logging is configured.
- wrap some functions in csconfig for convenience and DRY
- for each enabled feature, log.Debug
- log all enabled features once as Info (crowdsec) or Debug (cscli)
- file does not exist -> log.Trace
* backoff on refresh token error
* fix tls communication with lapi and user/pw auth (#1956)
allow self-signed TLS encryption with user/pw auth
docker:
- remove defaults for certificate file locations
- new envvar INSECURE_SKIP_VERIFY
- register agent before TLS settings (cscli machine add removes them
from the credentials file)
* separate cscli cobra constructors: lapi, machines, bouncers, postoverflows (#1945)
* use feature toggling to improve testability with http retry backoff
* Add parse unix to dateparse enricher (#1958)
Add parse unix is we do have a strTime but wasnt parsed using convential golang time
* func tests: redirect stderr to filter extra logs (#1961)
* backoff on refresh token error
* use feature toggling to improve testability with http retry backoff
* refactor feature backoff toggle for tests
Co-authored-by: mmetc <92726601+mmetc@users.noreply.github.com>
Co-authored-by: Laurence Jones <laurence.jones@live.co.uk>
allow self-signed TLS encryption with user/pw auth
docker:
- remove defaults for certificate file locations
- new envvar INSECURE_SKIP_VERIFY
- register agent before TLS settings (cscli machine add removes them
from the credentials file)
Package fflag provides a simple feature flag system.
Feature names are lowercase and can only contain letters, numbers, undercores
and dots.
good: "foo", "foo_bar", "foo.bar"
bad: "Foo", "foo-bar"
A feature flag can be enabled by the user with an environment variable
or by adding it to {ConfigDir}/feature.yaml
I.e. CROWDSEC_FEATURE_FOO_BAR=true
or in feature.yaml:
```
---
- foo_bar
```
If the variable is set to false, the feature can still be enabled
in feature.yaml. Features cannot be disabled in the file.
A feature flag can be deprecated or retired. A deprecated feature flag is
still accepted but a warning is logged. A retired feature flag is ignored
and an error is logged.
A specific deprecation message is used to inform the user of the behavior
that has been decided when the flag is/was finally retired.
