beenull/crowdsec
1
0
Fork 0
Code Issues Pull requests Packages Projects Releases Wiki Activity
1703 commits 83 branches 177 tags 151 MiB
Commit graph

763 commits

Author SHA1 Message Date
AlteredCoder 7e871d2278
rename PAPI base URL (#2033) 2023-02-03 12:10:02 +01:00
Thibault "bui" Koechlin e927717fa0
Polling API Integration (#1715) 
Co-authored-by: alteredCoder <kevin@crowdsec.net>
Co-authored-by: he2ss <hamza.essahely@gmail.com>
Co-authored-by: Sebastien Blot <sebastien@crowdsec.net>
 2023-01-31 14:47:44 +01:00
mmetc e37d09e5b4
use helpers for shorter tests, add a couple of error cases (#2016) 2023-01-26 17:13:31 +01:00
mmetc 3fb3decf49
error if tls.key_file or cert_file are missing (#2020) 2023-01-26 17:12:59 +01:00
mmetc 02be5f3618
allow literal $ in plugin configuration (#2015) 2023-01-23 16:28:43 +01:00
mmetc 47cc60bda9
allow use of literal $ in config.yaml (#2012) 2023-01-23 10:29:29 +01:00
mmetc e5833699c0
cscli config feature-flags (#2006) 2023-01-20 09:32:10 +01:00
Thibault "bui" Koechlin 4f29ce2ee7
CTI API Helpers in expr (#1851) 
* Add CTI API helpers in expr
* Allow profiles to have an `on_error` option to profiles

Co-authored-by: Sebastien Blot <sebastien@crowdsec.net>
 2023-01-19 08:45:50 +01:00
Marco Mariani 0c35d9d43c wip 2023-01-18 15:15:18 +01:00
Marco Mariani 4f25738d6b wip 2023-01-18 15:15:18 +01:00
Marco Mariani 47dbfa770d configure logging earlier 2023-01-18 15:15:18 +01:00
Marco Mariani 91b0f8fee1 load custom configuration paths when agent is disabled 2023-01-18 15:15:18 +01:00
Marco Mariani 2e91a82aa7 load feature.yaml as soon as possible 2023-01-18 15:15:18 +01:00
Thibault "bui" Koechlin f25fdecc3f
normalize scopes for alerts and decisions (#2001) 
* normalize scopes for alerts and decisions
 2023-01-18 14:50:03 +01:00
mmetc 51800132cd
improve feature flag logging (#1986) 
For cscli: it should provide a terse output, not nag users with configuration details. Although it's usually important that cscli and crowdsec have the same enabled features, having it list them every time the command is invoked can be too much.

For crowdsec: when features are set from the environment, it's too early to log where we should. So we can use log.Debug at activation time, and list them again once logging is configured.

 - wrap some functions in csconfig for convenience and DRY
 - for each enabled feature, log.Debug
 - log all enabled features once as Info (crowdsec) or Debug (cscli)
 - file does not exist -> log.Trace
 2023-01-13 13:42:42 +01:00
Cristian Nitescu 73663ff9e7
log the request error even in case of retry (#1988) 2023-01-13 12:58:12 +01:00
mmetc ba4396e52c
fix flaky parser unit test (#1985) 2023-01-12 17:03:25 +01:00
Thibault "bui" Koechlin 6fb962a941
Allow parsers to capture data for future enrichment (#1969) 
* Allow parsers to capture data in a cache, that can be later accessed via expr helpers (fake multi-line support)
 2023-01-11 15:01:02 +01:00
mmetc cd4dabde0e
silence yaml.local explicitly in cscli, keep in crowdsec/bouncer logs (#1981) 2023-01-11 09:50:46 +01:00
Laurence Jones ca12432a2a
Change patch to debug, if user has a local overide they will get informed every cscli call (#1980) 2023-01-10 10:05:18 +00:00
Cristian Nitescu 7284c0a47a
retry with backoff requests to CAPI (#1957) 
* backoff on refresh token error

* fix tls communication with lapi and user/pw auth (#1956)

allow self-signed TLS encryption with user/pw auth

docker:
 - remove defaults for certificate file locations
 - new envvar INSECURE_SKIP_VERIFY
 - register agent before TLS settings (cscli machine add removes them
   from the credentials file)

* separate cscli cobra constructors:  lapi, machines, bouncers, postoverflows (#1945)

* use feature toggling to improve testability with http retry backoff

* Add parse unix to dateparse enricher (#1958)

Add parse unix is we do have a strTime but wasnt parsed using convential golang time

* func tests: redirect stderr to filter extra logs (#1961)

* backoff on refresh token error

* use feature toggling to improve testability with http retry backoff

* refactor feature backoff toggle for tests

Co-authored-by: mmetc <92726601+mmetc@users.noreply.github.com>
Co-authored-by: Laurence Jones <laurence.jones@live.co.uk>
 2023-01-09 14:49:21 +01:00
blotus a84e4b6b15
Add conditional bucket (#1962) 2023-01-06 09:26:16 +01:00
AlteredCoder 185f9ad541
Alert context (#1895) 
Co-authored-by: bui <thibault@crowdsec.net>
 2023-01-04 16:50:02 +01:00
mmetc 033082a31e
ParseUnix() test fix: force UTC (#1970) 2023-01-04 16:22:17 +01:00
mmetc 2d81e751a1
fix parser test 2k23 (#1971) 2023-01-04 15:46:16 +01:00
Laurence Jones fd1c38811e
Add parse unix to dateparse enricher (#1958) 
Add parse unix is we do have a strTime but wasnt parsed using convential golang time
 2022-12-30 12:47:14 +00:00
mmetc 72c1753fb7
fix tls communication with lapi and user/pw auth (#1956) 
allow self-signed TLS encryption with user/pw auth

docker:
 - remove defaults for certificate file locations
 - new envvar INSECURE_SKIP_VERIFY
 - register agent before TLS settings (cscli machine add removes them
   from the credentials file)
 2022-12-29 22:00:11 +01:00
Laurence Jones 401739b036
Add unix expr helper (#1952) 
* Add unix expr helper

* Add original value not parsed error

* return early if cannot parse

* Add tests

* Fix negative value
 2022-12-29 14:53:06 +00:00
Thibault "bui" Koechlin e4463c412b
Improve warnings around lack of evt.StrTime field (#1954) 
* fix #1951 : improve error messages

* make hubtest warn you if you're missing evt.StrTime in your logs
 2022-12-29 15:03:32 +01:00
mmetc 6efc2688b1
simplify feature flags (#1947) 
Now checking for a feature flag is a one liner,
with no need to control errors.

if fflag.Crowdsec.CscliSetup.IsEnabled() {
   ...
}
 2022-12-26 14:23:41 +01:00
mmetc 5d2c99bb17
runtime feature flag initialization 2022-12-21 17:19:20 +01:00
mmetc ff88faf402
updated localstack dependencies, added build cache 2022-12-21 12:20:01 +01:00
mmetc a32aa96752
feature flags (#1933) 
Package fflag provides a simple feature flag system.

 Feature names are lowercase and can only contain letters, numbers, undercores
 and dots.

 good: "foo", "foo_bar", "foo.bar"
 bad: "Foo", "foo-bar"

 A feature flag can be enabled by the user with an environment variable
 or by adding it to {ConfigDir}/feature.yaml

 I.e. CROWDSEC_FEATURE_FOO_BAR=true
 or in feature.yaml:
```
 ---
 - foo_bar
```

 If the variable is set to false, the feature can still be enabled
 in feature.yaml. Features cannot be disabled in the file.

 A feature flag can be deprecated or retired. A deprecated feature flag is
 still accepted but a warning is logged. A retired feature flag is ignored
 and an error is logged.

 A specific deprecation message is used to inform the user of the behavior
 that has been decided when the flag is/was finally retired.
 2022-12-20 16:11:51 +01:00
he2ss 579cecde04
apiclient: fix http roundtrip (clone body also) (#1758) 
* apiclient: fix http roundtrip (clone body also)
 2022-12-14 16:42:46 +01:00
Laurence Jones fe23da6e0c
Add postgres socket support, clean some code (#1926) 2022-12-12 16:08:19 +00:00
Laurence Jones 11965f08db
Add socket support to mysql (#1911) 2022-12-08 09:33:08 +00:00
mmetc cc228f1868
Typos, grammar (#1905) 2022-12-06 15:55:27 +01:00
blotus fdda940ac0
Add Kubernetes audit acquisition (#1767) 2022-12-06 13:47:29 +01:00
mmetc fd3e668fe1
add -error flag to crowdsec binary (#1903) 2022-12-03 08:56:11 +01:00
mmetc fa0e590778
removed pid_dir (#1906) 2022-12-02 13:42:43 +01:00
mmetc 4a6a9c4355
acquisition: validate datasources before configuration (static checks) (#1841) 
* acquisition: validate datasources before configuration (allow static configuration checks)

* remove comment

* import reviser, format

* error wrap
 2022-11-30 17:36:56 +01:00
blotus 60f1228030
use a copy of bucket processors in LeakRoutine (#1902) 2022-11-30 10:59:47 +01:00
mmetc 104f5d1fe6
lint: error handling cleanup (#1877) 2022-11-29 09:16:07 +01:00
mmetc 66543493b5
fix nil dereference: check that httpServer is set before shutting down (#1893) 2022-11-28 11:55:08 +01:00
mmetc fde9640364
Docker refactoring, tls setup (#1869) 2022-11-28 10:35:12 +01:00
blotus c5079ac15e
invalidate agent token on 403 as well (#1888) 2022-11-25 14:35:50 +01:00
mmetc 5bdd3bbfcb
require at least go 1.18 to build (#1884) 2022-11-24 11:29:54 +01:00
Laurence Jones 4ac01ed880
Update perms for group read (#1876) 2022-11-21 09:49:56 +00:00
mmetc 3beb84bcfe
print missing "AS" values as empty strings instead of "0 " (#1867) 2022-11-14 09:55:53 +01:00
Thibault "bui" Koechlin 523343b174
notify when community-blocklist starts pull (#1845) 
* minor change to notify blocklist pull update, will make eventual troubleshooting easier
 2022-11-08 10:44:25 +01:00
Thibault "bui" Koechlin 3b4da7e637
fix #1860 : Only repeat the WAL warning once (#1863) 
* fix #1860
 2022-11-07 16:36:39 +01:00
mmetc 895691dad1
enabled linters: gocritic, nilerr (#1853) 2022-11-07 10:36:50 +01:00
Manuel Sabban 8aca00326d
fix ticker (#1858) 
Co-authored-by: sabban <15465465+sabban@users.noreply.github.com>
 2022-11-04 13:56:43 +01:00
Laurence Jones 668627f890
Add error checking to lookup host (#1847) 2022-10-31 18:38:01 +00:00
mmetc 344b1dc559
fixed package tests w/wal, gitignore/typos (#1849) 2022-10-31 10:02:51 +01:00
mmetc df88f4e1e9
randomize pull, push and metric intervals; reload crowdsec only when hub changed (#1846) 2022-10-28 13:55:59 +02:00
mmetc 02d2eab18c
update golangci-lint to 1.50 and fixes (#1828) 2022-10-26 15:11:37 +02:00
ThinkChaos 22479a289d
Add LookupHost expr lib func (#1775) 2022-10-26 10:17:48 +01:00
mmetc 2088bb1f91
fix for #1839 (#1840) 2022-10-26 11:02:12 +02:00
blotus b7c4bfd4e3
Use explicit transaction when inserting community blocklist (#1835) 2022-10-26 10:48:17 +02:00
mmetc e545933923
fix(cscli): correct and test the behavior of "cscli collections delete" (#1824) 2022-10-25 14:10:51 +02:00
blotus bb2f0e938f
Blocklist: Do not duplicate decisions when pulling (#1796) 2022-10-19 15:51:40 +02:00
Thibault "bui" Koechlin ae6bf39495
support decisions deletion via scenario + alerts delete via ID (#1798) 2022-10-19 14:37:27 +02:00
mmetc 6b0097a24b
change warning to debug when directories are missing in hub sync (#1819) 2022-10-18 10:32:54 +02:00
mmetc 2b7e3ff1e7
warn if no acquisition files are found, acquisition_test refactoring, tests (#1816) 2022-10-17 17:32:08 +02:00
mmetc ec0d2a5ed2
refactor broker_test.go, extract cstest/filenotfound*.go (#1815) 2022-10-17 14:17:23 +02:00
mmetc a96b3e077d
rename pkg/cstest -> pkg/hubtest (#1811) 
keep cstest for generic helper functions
this also avoids circular imports in test files
 2022-10-17 09:24:07 +02:00
mmetc 8fecc2c00b
enable staticcheck linter; fixes (#1806) 
- explicitly ignore returned parameters
 - replace Walk with faster WalkDir
 - log path error during hub dir sync
 - colorize static unit tests
 - removed duplicate import in crowdsec/main.go
 - typos
 - func tests: default datasource in tests/var/log instead of /tmp
 - action setup-go v3
 2022-10-14 16:12:21 +02:00
Manuel Sabban 7359586f1c
fix ticker mix up (#1807) 
Co-authored-by: sabban <15465465+sabban@users.noreply.github.com>
 2022-10-13 14:30:27 +02:00
mmetc 4b3c9c2806
print cscli usage in color, fix windows terminal detection (#1801) 2022-10-13 12:28:24 +02:00
mmetc 7674f907c4
replace log.Fatal with t.Fatal (#1805) 
This is required to run deferred teardown functions
 2022-10-13 10:42:46 +02:00
mmetc 1d9f861f28
unit tests: always capture testcase variable -> allow parallel testing (#1797) 2022-10-10 10:48:26 +02:00
Shivam Sandbhor 74659a82ab
Fast bulk alert delete (#1791) 2022-10-07 12:40:30 +02:00
mmetc ddd75eae9a
cscli: new tables, --color yes|no|auto option (#1763) 2022-10-07 11:05:35 +02:00
AlteredCoder b95a67751e
Update ent and grokky package (#1772) 
* Update ent and grokky package
 2022-10-06 14:55:42 +02:00
Manuel Sabban 83841d801c
fork dlog to ease debian packaging on official repos (#1790) 
Co-authored-by: sabban <15465465+sabban@users.noreply.github.com>
 2022-10-06 13:40:31 +02:00
Shivam Sandbhor 65c0b9ebcf
Simplify one shot tests (#1786) 2022-10-06 11:57:26 +02:00
blotus 3ba67bad3d
remove a wrong warning when pulling list content from CAPI (#1789) 2022-10-06 11:48:06 +02:00
mmetc 9b3be5c2e8
Bulk delete alert optimization (#1782) 2022-10-05 17:07:44 +02:00
Shivam Sandbhor b203b3f444
Fix flakey test in file_tests (#1783) 
Signed-off-by: Shivam Sandbhor <shivam.sandbhor@gmail.com>
 2022-10-05 16:40:09 +02:00
mmetc 6120571421
fix & cleanup cloudwatch_test.go (#1780) 2022-10-04 09:48:59 +02:00
mmetc edced6818a
cleanup + fix flaky tests in file_test.go, apic_test.go (#1773) 2022-09-30 16:01:42 +02:00
blotus bfbe180101
Tighten windows sqlite database permissions (#1769) 2022-09-28 16:18:00 +02:00
Sean Kelly 568eb1d4e0
Fix misspelling of instantiate participles (#1759) 2022-09-27 17:13:43 +02:00
Laurence Jones 21e5b0d6d0
Improvement: Docker one shot error message (#1666) 
* In one shot, user would only specify one container?
 2022-09-27 16:20:30 +02:00
Manuel Sabban 1f06f242cc
fix https://github.com/crowdsecurity/crowdsec/issues/1746 (#1749) 
Co-authored-by: sabban <15465465+sabban@users.noreply.github.com>
 2022-09-14 15:26:26 +02:00
blotus 9b3ff82542
add WAL support for sqlite (#1752) 2022-09-14 15:09:54 +02:00
AlteredCoder 7d97729eea
Add config option to enable or not local API and agent (#1730) 
* Add flag to enable or not local API and agent
 2022-09-12 14:38:29 +02:00
AlteredCoder b06167a3fa
Allow plugins to load environment variable (#1727) 
* Allow plugins to load environment variable
 2022-09-08 11:41:28 +02:00
Manuel Sabban b2130b1593
Fix 1737 (#1738) 
* add GetMeta to *types.Event

Co-authored-by: sabban <15465465+sabban@users.noreply.github.com>
 2022-09-07 10:11:39 +02:00
Thibault "bui" Koechlin 9d199fd4a9
fix #1733 : add support for exclusion regexps (#1735) 
* allow to specify a list of regular expressions to skip some specific files
 2022-09-06 14:58:37 +02:00
mmetc 414282a2c9
golangci-lint 1.49 and related fixes (#1736) 2022-09-06 13:55:03 +02:00
Laurence Jones e674537d0b
Update sprig to v3 (#1722) 
* Update sprig to v3
 2022-09-05 09:05:50 +02:00
he2ss ea40ffd655
Datasource/kafka (#1698) 
* add Kafka datasource
 2022-08-30 17:03:45 +02:00
Manuel Sabban 7d0f89df29
Implement reinject command to send notifications of alerts (#1638) 
* implement reinject command to send notifications of alerts using a profile

Co-authored-by: sabban <15465465+sabban@users.noreply.github.com>
 2022-08-30 15:45:52 +02:00
Thibault "bui" Koechlin bacea50485
allow user to disable decision deduplication (#1687) 
* allow user to disable decision deduplication
 2022-08-26 14:17:46 +02:00
blotus 1f5224b74b
switch to go 1.19 (#1709) 2022-08-26 13:31:49 +02:00
mmetc eea07b7a1a
golangci-lint v1.48 and fixes for "usestdlibvars" (#1711) 2022-08-16 09:46:10 +02:00
AlteredCoder 1002affc16
cscli machines delete: return an error if machines doesn't exist (#1689) 
* cscli machines delete: return an error if machines doesn't exist
 2022-07-28 17:32:12 +02:00
Thibault "bui" Koechlin 866c200c31
Generic dateparse approach (#1669) 
* Allow any parser to suggest a format string for the date to be parsed.

* allow the enricher functions to get the parser's logger so they can inherit the level
 2022-07-28 16:41:41 +02:00
Thibault "bui" Koechlin 0eea20fa7c
revert decision dedup behavior to 1.3.4 (#1675) 
* revert decision dedup behavior to 1.3.4
 2022-07-22 11:20:10 +02:00
Thibault "bui" Koechlin bd91ddaf52
logging consistency for .local files (#1655) 2022-07-13 10:56:03 +02:00
blotus 7b8cd63b04
do not set the UDP read buffer size in syslog datasource (#1657) 2022-07-13 10:18:03 +02:00
AlteredCoder 39da36361c
Get geoip Country from other objects if not present (#1659) 2022-07-12 15:26:34 +02:00
Thibault "bui" Koechlin 73f336363a
bump log level when overloading config file with .local (#1646) 
* bump log level

Co-authored-by: Marco Mariani <marco@crowdsec.net>
 2022-07-08 09:29:23 +02:00
blotus 5c1c941851
close response body in heartbeat (#1637) 2022-07-06 14:50:54 +02:00
AlteredCoder 5f62d738fc
Add no-capi flag and review some logs (#1628) 
* Add no-capi flag and review some logs
 2022-07-01 16:56:13 +02:00
Thibault "bui" Koechlin ca4cd6d559
attempt to fix ticker leak (#1620) 2022-06-30 17:36:01 +02:00
AlteredCoder 02e0f3c095
Fix event.timestamp pointer usage (#1621) 
* Fix event.timestamp pointer usage

* avoid returning an error when creating alerts if something goes wrong during the parsing

Co-authored-by: Sebastien Blot <sebastien@crowdsec.net>
 2022-06-30 17:35:42 +02:00
blotus 863facaa33
Properly handle expired decisions with different scopes (#1616) 2022-06-29 16:13:04 +02:00
Thibault "bui" Koechlin 15902dcba6
fix #1615 : cleanup based on heartbeat instead (#1617) 2022-06-29 13:21:58 +02:00
mmetc 8e7e799304
[wip] serve metrics only after agent and/or lapi are ready; fixed some func tests (#1613) 2022-06-24 15:55:21 +02:00
he2ss 7fafb483ad
Pkg/database/fix count decisions since by value (#1606) 
* fix CountDecisionsSinceByValue to have also expired decisions
 2022-06-23 12:04:07 +02:00
AlteredCoder a4f4eabf0a
support yml file (#1605) 2022-06-22 17:01:27 +02:00
mmetc 628d7be1d8
simplify err.Error() to err when used in printf context (#1603) 2022-06-22 15:53:53 +02:00
he2ss 3d6f015211
Add duration expr to add duration formula (#1556) 
* add duration expr to add duration formula
 2022-06-22 11:29:52 +02:00
Thibault "bui" Koechlin a6ed08b239
Add alerts and decisions metrics, LAPI and agent timing prom metrics (#1546) 2022-06-22 11:14:34 +02:00
AlteredCoder 0a39066f9d
Fix #1552 (#1569) 2022-06-22 10:29:02 +02:00
mmetc d71279f023
added flag crowdsec --warning (#1461) 2022-06-22 09:38:23 +02:00
mmetc c78c833400
CI: colored test output, colored crowdsec and crowdsec-api logs, full final db dump for mysql and sqlite (#1596) 
* github-ci: color unit test output and logs
* new config option: force_color_logs (useful in CI)
* bats: show sqlite/mysql dump at the end
* removed "-v" (print package names) from "go build"
* general workflow cleanup
 2022-06-17 16:12:49 +02:00
mmetc 10585bfecc
enabled linters and fixes for: misspell, predeclared, unconvert, ineffassign, gosimple, govet (#1595) 2022-06-16 14:41:54 +02:00
Thibault "bui" Koechlin ff72a3c1c7
avoid create a new name generator at each bucket instanciation, it's not that cheap (#1591) 2022-06-15 10:02:00 +02:00
blotus 9c1b78395a
reduce verbosity of TLS auth and FlushAgentsAndBouncers (#1588) 2022-06-13 16:08:00 +02:00
Thibault "bui" Koechlin 581ddf78fc
Performance improvements (#1583) 
* fix concurrent map write on distinct cache

* cache compiled expressions for groupby and cancel_on filters

* limit objects copy when it's going to lock a shared goroutine
 2022-06-13 14:41:05 +02:00
Thibault "bui" Koechlin 567e0ab7d1
fix concurrent map write on distinct cache (#1582) 2022-06-10 09:39:23 +02:00
Thibault "bui" Koechlin 1c0fe09576
Add support for certificate authentication for agents and bouncers (#1428) 2022-06-08 16:05:52 +02:00
blotus bdda8691ff
New syslog parser for syslog datasource (#1554) 2022-06-08 15:16:58 +02:00
blotus 4b311684ab
Add more JSON expr helpers (#1576) 2022-06-08 12:15:29 +02:00
mmetc 799cc82bb5
functional tests, minor refactoring and lint/cleanup (#1570) 
* cmd/crowdsec: removed log.Fatal()s, added tests and print error for unrecognized argument
* updated golangci-lint to v1.46
* lint/deadcode: fix existing issues
* tests: cscli config backup/restore
* tests: cscli completion powershell/fish
* err check: pflags MarkHidden()
* empty .dockerignore (and explain the reason)
* tests, errors.Wrap
* test for CS_LAPI_SECRET and minor refactoring
* minor style changes
* log cleanup
 2022-06-06 15:24:48 +02:00
mmetc 88a4801d6a
allow run-tests with -f "<test-name>" (#1564) 2022-05-28 22:10:27 +02:00
he2ss e88e9946f9
Crowdsec/decisions_stream bug fix (#1517) 
* Fix bug when stream interval is greater or equal to 60s

Co-authored-by: alteredCoder <kevin@crowdsec.net>
 2022-05-27 15:23:59 +02:00
mmetc 1fc9587919
fix #1283: update and enable error reports from golangci (#1523) 2022-05-25 22:27:50 +02:00
mmetc 1a293a2a27
cwhub: export SetHubBranch (#1559) 2022-05-24 15:46:48 +02:00
mmetc 357899b83e
fixed uid/gid bound check regression (#1555) 2022-05-23 09:46:39 +02:00
Thibault "bui" Koechlin 0483b9c641
do not spew.Sdump() the invalid node on error. It leads to huge memory usage, especially if the parsers refers ie. datafile (#1550) 2022-05-20 13:29:47 +02:00
AlteredCoder 1e1741aa45
Allow to set static to a pointer and add IsIPV6 helper (#1540) 
* Allow to set static to a pointer and add IsIPV6 helper
 2022-05-19 16:28:25 +02:00
Thibault "bui" Koechlin fe09737d80
Add support for machine heartbeat (#1541) 
* add the last_heartbeat field

* add heartbeat controller

* add endpoint of heartbeat

* heartbeat integration

* add last_heartbeat to cscli machines list
 2022-05-19 15:47:27 +02:00
mmetc 131ed1b0a7
error reporting (#1501) 
* unified error reporting, removed redundancy, tests
 2022-05-19 10:48:08 +02:00
Manuel Sabban 18030e6c58
add notifications command (#1537) 
* add notifications command

Co-authored-by: sabban <15465465+sabban@users.noreply.github.com>
 2022-05-18 16:13:33 +02:00
Shivam Sandbhor 220bbe5862
Document LAPI filters (#1535) 
Signed-off-by: Shivam Sandbhor <shivam.sandbhor@gmail.com>
 2022-05-18 11:45:12 +02:00
mmetc 98f2ac5e7c
fix #1385: .yaml.local (#1497) 
Added support for .yaml.local files to override values in .yaml
 2022-05-18 10:08:37 +02:00
blotus 39f7e38444
retry to send alert to plugin channel if it fails (#1530) 2022-05-17 16:57:15 +02:00
Thibault "bui" Koechlin fbcb2ed7fd
Improve distinct/uniq behaviour (#1478) 
* make uniq/distinct use a cache that is independant of the bucket's cache_size

* add testing specifically for cache_size
 2022-05-17 12:45:53 +02:00
blotus 0449ec1868
Windows Support (#1159) 2022-05-17 12:14:59 +02:00
Cristian Nitescu a49b023a28
GetExprEnv usage optimization (#1515) 
* avoid multiples calls to GetExprEnv

* cache ExprEnv in node process

* use global expression env

* remove block profile rate
 2022-05-17 10:50:37 +02:00
blotus 8f111680bf
Allow to override statics in hubtest. (#1495) 2022-04-29 14:24:41 +02:00
blotus 64369b5c2b
add expr XML helpers (#1493) 2022-04-29 13:52:23 +02:00
blotus 392708a804
Fix docker flaky test (#1494) 2022-04-29 12:16:49 +02:00
AlteredCoder f22e4eb24e
Improve MySQL performance (#1477) 
* Improve MySQL performance
 2022-04-28 12:53:14 +02:00
AlteredCoder be977d1cc4
Fix cwhub collections uninstall dependencies (#1486) 
* Fix cwhub collections uninstall dependencies
 2022-04-27 18:28:03 +02:00
AlteredCoder a645c928d4
Fix decisions list with --no-simu flag (#1482) 
* Fix decisions list with --no-simu flag
 2022-04-27 11:05:40 +02:00
Greg Myers 0f4ab71f01
Fix typos in docs, comments, code (#1483) 2022-04-27 11:04:12 +02:00
AlteredCoder 44b11c2e5b
Fix hub items installation (#1481) 2022-04-26 17:37:07 +02:00
blotus 1bd8cc79c8
Kill the whole docker acquis in tests (#1475) 2022-04-22 16:56:22 +02:00
blotus 8909fbdb22
cleanup container state if the reader tomb dies by itself (#1470) 2022-04-22 10:52:44 +02:00
Thibault "bui" Koechlin 242706a475
fix journalctl deadlock on shutdown (#1468) 
* avoid being locked sending termination error while the reading routine - on the chan - died
 2022-04-21 14:02:25 +02:00
AlteredCoder 4273a0f243
cscli: add autocompletions for hubitems (#1465) 
* Add autocompletion for hub items in cscli
 2022-04-20 15:44:48 +02:00
he2ss 615895da9d
cscli: add force enroll feature (#1430) 
* cscli: add force enroll feature
 2022-04-20 13:34:17 +02:00
Thibault "bui" Koechlin e6a35e8714
Improve plugins grouping (alternative to #1424) (#1437) 
* Fix races in test (#1446)

Co-authored-by: Shivam Sandbhor <shivam.sandbhor@gmail.com>
Co-authored-by: alteredCoder <kevin@crowdsec.net>
Co-authored-by: AlteredCoder <64792091+AlteredCoder@users.noreply.github.com>
 2022-04-19 19:12:23 +02:00
AlteredCoder 526a4dbd08
Reduce the query unescape helper verbosity (#1447) 2022-04-19 12:31:29 +02:00
Shivam Sandbhor 8060f54f27
Cwhub testing (#1438) 
* Add tests in cwhub and fix collection upgrade(#1431)

Signed-off-by: Shivam Sandbhor <shivam.sandbhor@gmail.com

Co-authored-by: bui <thibault@crowdsec.net>
Co-authored-by: Thibault "bui" Koechlin <orixxx@gmail.com>
 2022-04-19 12:07:35 +02:00
mmetc 4b9a0c4ef7
typos (#1453) 2022-04-19 11:25:27 +02:00
Shivam Sandbhor 4a3ec85686
Update bouncer pull in rupture mode (#1445) 
* Update bouncer pull in rupture mode

Signed-off-by: Shivam Sandbhor <shivam.sandbhor@gmail.com>
 2022-04-15 12:24:01 +02:00
AlteredCoder 71165bcd30
Send all installed scenario to LAPI (#1277) 2022-04-13 17:48:29 +02:00
AlteredCoder 099469c5d2
Fix hub loader to support '.yml' files (#1433) 2022-04-11 16:13:20 +02:00
Shivam Sandbhor a8089c8ddb
Add origins param in decision stream service (#1429) 
Signed-off-by: Shivam Sandbhor <shivam.sandbhor@gmail.com>
 2022-04-07 12:40:27 +02:00
blotus 9cf2d5ab5c
handle containers with TTY in docker acquis (#1422) 2022-04-05 10:31:36 +02:00
Thibault "bui" Koechlin ba7f4fcec0
make this info level (#1409) 
* make this info level
 2022-04-01 15:31:33 +02:00
Thibault "bui" Koechlin d8dc01cd94
Revamp unit tests (#1368) 
* Revamp unit tests
* Increase coverage
* Use go-acc to get cross packages coverage

Signed-off-by: Shivam Sandbhor <shivam.sandbhor@gmail.com>
 2022-03-29 14:20:26 +02:00
Håvard Moen 42ff269bc8
add back dialect to handle pgx correctly (#1376) 2022-03-23 09:29:22 +01:00
AlteredCoder 411baa4dcf
Improve cscli metrics units (#1374) 
* Improve cscli metrics units
 2022-03-21 12:13:36 +01:00
Shivam Sandbhor c5566e92f3
Fix 1262 pgsql conflict resolve (#1363) 
* Fix api for all dbs (#1310)
* DB agnostic lapi sanitize

Signed-off-by: Shivam Sandbhor <shivam.sandbhor@gmail.com>

* Update ent

Signed-off-by: Shivam Sandbhor <shivam.sandbhor@gmail.com>

* Fix go dep mess.

Signed-off-by: Shivam Sandbhor <shivam.sandbhor@gmail.com>
 2022-03-17 14:12:13 +01:00
Shivam Sandbhor 023ac9e138
Add trusted IPs which have admin API access (#1352) 
* Add trusted IPs which have admin API access
 2022-03-16 17:28:34 +01:00
Shivam Sandbhor 42a1bc0260
Add query param to filter decisions by scenarios and origin (#1294) 
* Add query param to filter decisions by scenarios
 2022-03-16 14:37:42 +01:00
Shivam Sandbhor bb30a3f966
Don't omit fields of bouncer in json (#1354) 
Signed-off-by: Shivam Sandbhor <shivam.sandbhor@gmail.com>
 2022-03-16 09:40:34 +01:00
Thibault "bui" Koechlin a74a41dac5
fix #1357 (#1358) 2022-03-16 09:40:00 +01:00
Shivam Sandbhor 76e97303a5
Deprecate pid_file config (#1346) 
* Deprecate pid_file config

Signed-off-by: Shivam Sandbhor <shivam.sandbhor@gmail.com>

* Fix unit test

Signed-off-by: Shivam Sandbhor <shivam.sandbhor@gmail.com>

* Impl review suggestions.

Signed-off-by: Shivam Sandbhor <shivam.sandbhor@gmail.com>
 2022-03-16 09:23:49 +01:00
j-k 2cd3248431
refactor: use runtime rather than ldflags for go details (#1302) 2022-03-14 11:29:34 +01:00
mmetc 4e6b9597f8
fix for https://staticcheck.io/docs/checks#SA2002 (#1334) 2022-03-10 13:53:33 +01:00
mmetc 7c0593c659
noop code removal, typos and lint fixes (#1329) 2022-03-09 16:15:18 +01:00
mmetc 10ce45c054
allow notification plugins to work on freebsd and non-root functional tests (#1253) 
* random uuid for all platforms
* check group writable and setgid; don't check group ownership
* allow user to run plugins without changing desired user/group (set them to "")
 2022-03-09 12:09:50 +01:00
Thibault "bui" Koechlin 5a15f9b39b
bailout on incompatible duration format (#1326) 2022-03-08 18:18:36 +01:00
mmetc e35efc5b2d
fix check uid, gid values (#1309) 2022-03-07 10:16:34 +01:00
Thibault "bui" Koechlin b66366c28c
Revert "Handle decisions with varying expiry for same IP (#1262)" (#1308) 
This reverts commit e4f6cdfc14.
 2022-03-04 10:17:31 +01:00
mmetc c5dda0ffba
fix: deny copy folder to itself or subpath (#1299) 2022-03-02 11:30:04 +01:00
Shivam Sandbhor c3dbe0080c
Exit syslog acquis only after server is dead (#1288) 2022-03-01 11:32:28 +01:00
blotus fb74b2fda7
Improve LAPI performance when under high load (#1273) 2022-02-17 17:52:04 +01:00
Shivam Sandbhor e4f6cdfc14
Handle decisions with varying expiry for same IP (#1262) 
* Upgrade ent and add sql/modifier in codegen

* update db wrappers to sanitize LAPI

Signed-off-by: Shivam Sandbhor <shivam.sandbhor@gmail.com>
 2022-02-16 15:19:14 +01:00
mmetc 9bc7e6ffcf
Refactor unit tests to reduce line count (#1264) 2022-02-15 12:50:33 +01:00
Shivam Sandbhor 43d5690432
Detect missing plugin binary wrt profiles (#1252) 
Signed-off-by: Shivam Sandbhor <shivam.sandbhor@gmail.com>
 2022-02-14 17:45:03 +01:00
AlteredCoder 8b90f4b2b2
Upgrade download datafiles if doesn't exist (#1254) 2022-02-14 16:51:06 +01:00
AlteredCoder 5a0843852a
add IpToRange helpers and allows to have an expression with scope Range (#1260) 
* add IpToRange helpers and allows to have an expression with scope Range
 2022-02-14 16:50:52 +01:00
mmetc 40ab8fa738
Atoi() -> ParseInt() (#1256) 2022-02-14 14:00:42 +01:00
Shivam Sandbhor 76e3612088
Check log level before dumping resp (#1243) 
* Check log level before dumping resp
* Sleep longer in func tests

Signed-off-by: Shivam Sandbhor <shivam.sandbhor@gmail.com>
 2022-02-08 15:40:01 +01:00
mmetc 5c7c12c62d
define cwversion.System (Platform) in "make static" too; show it with --version (#1238) 2022-02-04 13:02:45 +01:00
Thibault "bui" Koechlin dd53d19777
Make whitelist by expr debug level (#1236) 
* fix #616 : simply make it at debug level, so that the user can set his node to debug level if he really wants to see this. Otherwise it can be too spammy
 2022-02-03 17:04:18 +01:00
mmetc ad28a979e9
local control flow cleanup (#1215) 
removed redundant/unreachable returns, else branches, type declarations, unused variables
 2022-02-01 22:08:06 +01:00
mmetc 35eea39db7
allow Makefile to override /etc/crowdsec and /var/lib/crowdsec/data (#1221) 2022-02-01 10:34:53 +01:00
mmetc 8310c10ce3
console_config.yaml -> console.yaml (#1195) 2022-01-21 11:52:23 +01:00
mmetc 240e5ad3ab
remove trailing carriage return (#1194) 2022-01-21 11:35:21 +01:00
blotus 19323ba4aa
fix crash on upgrade with nil last push field (#1191) 2022-01-20 18:10:40 +01:00
AlteredCoder b93b8d9a2e
Support PGX (#1186) 
* Support PGX

* support sslmode
 2022-01-20 11:17:21 +01:00
Shivam Sandbhor 59a537514f
Check for errors before modifying proc attrs (#1181) 
Signed-off-by: Shivam Sandbhor <shivam.sandbhor@gmail.com>
 2022-01-19 15:34:09 +01:00
Thibault "bui" Koechlin cc1ab8c50d
switch to utc time everywhere (#1167) 
* switch to utc time everywhere


Co-authored-by: alteredCoder <kevin@crowdsec.net>
 2022-01-19 14:56:05 +01:00
AlteredCoder b1a7ffb92f
fix postgreSQL count fail (#1184) 2022-01-19 14:50:53 +01:00
Thibault "bui" Koechlin c81fc87d4e
fix #1168 (#1179) 
* fix #1168
 2022-01-19 11:34:40 +01:00
Thibault "bui" Koechlin a88848009a
fix default perms for log file (#1177) 
* fix default perms
 2022-01-18 16:54:02 +01:00
Thibault "bui" Koechlin a17f150e5d
fix #1170 : display full message in debug mode when syslog cannot parse (#1176) 
* fix #1170 : display full message in debug mode when syslog cannot parse
 2022-01-18 09:54:01 +01:00
Thibault "bui" Koechlin 40ed810c0b
Gin upgrade (#1174) 
* upgrade gin / gin-jwt, and add a new 'trusted_proxies' option to provide trusted CIDRs
 2022-01-17 17:18:12 +01:00
Thibault "bui" Koechlin 6e92da76ad
lapi to capi : allow push of tainted/custom/manual decisions (#1154) 
* add console command to control signal sharing
* modify metrics endpoint to add lastpush

Co-authored-by: alteredCoder <kevin@crowdsec.net>
 2022-01-13 16:46:16 +01:00
blotus cc72800f50
Update LAPI swagger (#1155) 2022-01-11 16:45:34 +01:00
Thibault "bui" Koechlin 3bca25fd6d
lists support from central api (#1074) 
* lists support from central api

Co-authored-by: Sebastien Blot <sebastien@crowdsec.net>
 2022-01-11 14:31:51 +01:00
blotus 4a11060930
Kinesis datasource (#1147) 2022-01-11 14:19:43 +01:00
Thibault "bui" Koechlin 6c676c4869
fix #1131 : complain when validating unknown machine (#1146) 2022-01-05 13:50:04 +01:00
Shivam Sandbhor ba71c55492
Fix cscli inpsect json output (#1145) 
* Fix cscli inpsect json output
Signed-off-by: Shivam Sandbhor <shivam.sandbhor@gmail.com>
 2022-01-05 10:42:27 +01:00
Thibault "bui" Koechlin 8e3004ebb3
fix race condition on repetitive trigger buckets creation (#1144) 2022-01-04 14:02:07 +01:00
Shivam Sandbhor 6c4ec64ca9
Fix json output of cscli hub list (#1143) 
* Fix json output of cscli hub list
* Fix functional tests.

Signed-off-by: Shivam Sandbhor <shivam.sandbhor@gmail.com>
 2022-01-04 11:49:23 +01:00
blotus f86ec1c389
Docker api version negotiation (#1135) 2021-12-30 12:21:49 +01:00
blotus 3105897f37
Allow to configure log rotation (#1130) 2021-12-28 11:59:03 +01:00
mmetc 7126f8f0ff
replaced &nbsp; (#1129) 2021-12-28 10:32:46 +01:00
AlteredCoder f86e0c0a5a
don't send decisions with negative duration to bouncers (#1117) 2021-12-21 10:23:30 +01:00
Sykursen 6a3adcff0e
Upgrade metabase to v41.5 (#1109) 2021-12-17 10:29:48 +01:00
Thibault "bui" Koechlin 106254f020
support for cancel_on (#1105) 
* cancel_on filter

* tests
 2021-12-17 09:56:02 +01:00
AlteredCoder d913ac160e
fix create alert bulk for decisions insertion (#1107) 
* fix create alert bulk for decisions insertion
 2021-12-16 18:26:19 +01:00
AlteredCoder 88d06260d7
add cscli decisions import (#1038) 
* add cscli decisions import

Co-authored-by: Sebastien Blot <sebastien@crowdsec.net>
Co-authored-by: bui <thibault@crowdsec.net>
 2021-12-15 11:39:37 +01:00
AlteredCoder 458dcd1979
add more helpers (#1091) 
* add more exprhelpers
 2021-12-14 11:07:40 +01:00
Thibault "bui" Koechlin e5204bc1b1
fix #1083 : do not update/overwrite 'not installed' collections sub-items on 'cscli XX upgrade' (#1089) 
* fix #1083 : do not update/overwrite 'not installed' collections sub-items on 'cscli XX upgrade'
 2021-12-13 19:31:16 +01:00
mmetc c7fb6a1428
enabled -> enabling (#1090) 2021-12-13 13:14:29 +01:00
Manuel Sabban 4e6f6fe3a2
log4j vuln fix for metabase (#1082) 
Co-authored-by: sabban <15465465+sabban@users.noreply.github.com>
 2021-12-13 10:19:20 +01:00
mmetc 7dee103b6e
typos of various nature (#1072) 2021-12-06 17:29:23 +01:00
AlteredCoder 4917aa23c9
Docker datasource (#1064) 
* add docker datasource
 2021-12-02 15:55:50 +01:00
blotus dd03d07355
optimize the flush function by deleting alerts based on their id (#1054) 2021-11-17 10:15:38 +01:00
he2ss 0652e9ed08
feature cscli|crowdsec add additional labels on crowdsec dsn run (#1053) 
* feature cscli|crowdsec add additional labels on crowdsec dsn run
 2021-11-17 10:08:46 +01:00
Thibault "bui" Koechlin 3c768490ba
fix #873 without breaking backward (#1052) 2021-11-15 14:16:18 +01:00
Kerma Gérald 37c2a10e21
Use math.MaxInt32 instead of math.MaxUint32 (#980) 
To fix 32 bits compilation in v1.2.0
https://github.com/crowdsecurity/crowdsec/issues/979

Signed-off-by: Kerma Gérald <gandalf@gk2.net>
 2021-11-15 12:14:04 +01:00
Thibault "bui" Koechlin 7362828a3b
add --failures to explain feature : only display failed lines (#1048) 
* add --failures to explain feature : only display failed lines

* no error no problem
 2021-11-08 18:01:43 +01:00
Thibault "bui" Koechlin 8b0527bf9d
add evt. (#1045) 2021-11-03 15:17:48 +01:00
AlteredCoder fb54388e93
Fix issue 1033 (#1034) 
* Fix issue 1033
 2021-11-02 12:16:33 +01:00
Thibault "bui" Koechlin d1ce543440
Improve explain (#1039) 
* improve explain feature

* nicer display for details, --verbose in favor of --debug for details
 2021-11-02 12:06:01 +01:00
Shivam Sandbhor cbada3d435
Allow using cloudwatch using iam role instead of hardcoded tokens (#1035) 2021-11-02 10:25:35 +01:00
mmetc f10187bd6d
typos (#1036) 2021-11-02 09:19:22 +01:00
Thibault "bui" Koechlin 2b2a11fec7
Extra syslog debug (#1030) 
* extra logging
 2021-11-01 20:55:03 +01:00
AlteredCoder cf57c89177
add name and alias in cscli console enroll (#950) 
* add name and alias in cscli console enroll
 2021-10-26 15:33:17 +02:00
blotus 25a2d528b0
Alerts flush: Optimization of the flush mechanism (batch and limit to one job) + add cscli alerts flush command (#1024) 
- Don't allow running more than one alert flush job at a time to prevent runaway CPU usage in some case. (fix High CPU after Upgrade to 1.2.0 #1022)
 - Add a cscli alerts flush command to manually flush the alerts in the database (fixes Improvement/Manual flush mechanism #1023 ).
 - Enable cascading deletion on alerts as we upgraded ent: Deleting an alert in the database will automatically delete all related decisions, events and meta
 - Add an index on alerts.id to try to improve flush performance with very big sqlite database.
- Flush alert now operates in batch
 2021-10-26 13:33:45 +02:00
Thibault "bui" Koechlin 3f99330b3d
Entgo 0.9 (#1018) 
* update entgo & sqlite to latest version

* schema update
 2021-10-22 16:15:57 +02:00
Shivam Sandbhor a7b1c02bd5
Fix bugs in cloudwatch acq (#991) 
* Fix bugs in cloudwatch acq

- Fix concurrent writes to map streamIndexes
- Fix multiple cases of modifying while iterating on slice.
- Fix order of fetching cloudwatch events.
- Remove `startup` hack.

Signed-off-by: Shivam Sandbhor <shivam.sandbhor@gmail.com>

* Fix cloudwatch tests

Signed-off-by: Shivam Sandbhor <shivam.sandbhor@gmail.com>
 2021-10-22 10:35:05 +02:00
Thibault "bui" Koechlin 3bb2128bf4
fix sort :/ (#1007) 2021-10-12 19:16:24 +02:00
Thibault "bui" Koechlin 1bd6b8f7b9
Multiple fixes (#1006) 
* fix #1005 : timestamp in trigger timemachine buckets

* attempt at consistent bucket order for hubtest
 2021-10-12 14:09:17 +02:00
Thibault "bui" Koechlin 2961a0ed02
ensure machineID is included early enough into the alert (#1004) 2021-10-11 15:02:16 +02:00
blotus 2bc9f33e12
add ParseUri() expr helper (#994) 2021-10-08 16:50:31 +02:00
AlteredCoder 0ccc69696b
Break on success when alert already has decision (#997) (#999) 
* Break on success when alert already has decision (#997)

Signed-off-by: Shivam Sandbhor <shivam.sandbhor@gmail.com>
 2021-10-05 11:30:34 +02:00
Thibault "bui" Koechlin af4bb350c0
hubtests revamp + cscli explain (#988) 
* New hubtest CI for scenarios/parsers from the hub
 * New `cscli explain` command to visualize parsers/scenarios pipeline

Co-authored-by: alteredCoder <kevin@crowdsec.net>
Co-authored-by: Sebastien Blot <sebastien@crowdsec.net>
Co-authored-by: he2ss <hamza.essahely@gmail.com>
Co-authored-by: Cristian Nitescu <cristian@crowdsec.net>
 2021-10-04 17:14:52 +02:00
Thibault "bui" Koechlin c2fd173d1e
fix node success logic (#993) 
* fix node success logic : only fail node on child failure if mother node has no successfull grok
 2021-09-28 17:58:07 +02:00
he2ss fb308d5596
fix plugins logging in right level (#990) 2021-09-28 14:44:21 +02:00
he2ss db5ffb0040
Update test env (#987) 
* update test_env
 2021-09-24 18:06:30 +02:00
blotus f0db3742de
fix usage of regex.Match in cloudwatch module (#986) 2021-09-23 13:52:05 +02:00
Shivam Sandbhor cca76da2d6
Fix crash if plugin config is broken (#964) 
Signed-off-by: Shivam Sandbhor <shivam.sandbhor@gmail.com>
 2021-09-10 14:25:34 +02:00
he2ss e651379964
add jsonExtractUnescape Helper (#962) 
* add jsonExtractUnescape Helper
 2021-09-10 12:43:11 +02:00
AlteredCoder 5ae69aa293
fix stacktrace when mmdb file are not present (#935) 
* fix stacktrace when mmdb file are not present
 2021-09-09 16:27:30 +02:00
blotus 7a1b955ad1
use our fork of grokky (#953) 2021-09-09 14:46:16 +02:00
Shivam Sandbhor b8e24a1e0b
Make plugin runner configurable and run only registered plugins (#944) 
* Make plugin runner configurable and run only registered plugins
 2021-09-08 11:36:42 +02:00
Thibault "bui" Koechlin 0ad6165ed2
fix release drafter + readme + remove dead readme for acquis (#933) 2021-09-03 09:07:24 +02:00
Manuel Sabban d7d591ff84
update to use cdn for hub (#920) 
* update to use cdn for hub
* add cdn for version
* fix unit tests accodingly with new cdn

Co-authored-by: sabban <15465465+sabban@users.noreply.github.com>
 2021-09-02 15:17:37 +02:00
Thibault "bui" Koechlin bed90a832e
fix #919 : display error message (#929) 
* fix #919

* fix tests
 2021-09-02 12:46:32 +02:00
Thibault "bui" Koechlin 589cb72d41
enforce a bit more parsing for resillience (#928) 2021-09-02 12:34:20 +02:00
Shivam Sandbhor b40fd36607
Add plugin interface code in protobufs package (#921) 
Signed-off-by: Shivam Sandbhor <shivam.sandbhor@gmail.com>
 2021-08-31 14:40:17 +02:00
Thibault "bui" Koechlin 68c11dd827
don't try to send/don't notify if plugin chan is nil (#923) 2021-08-31 14:39:32 +02:00
blotus b5d0d56a11
add support for --since in journalctl DSN (#917) 2021-08-31 12:40:22 +02:00
ThinkChaos 448a227079
Minor changes to specific logs (#900) 
- Minor changes to specific logs
- Fix LAPI to not push signals to CAPI when disabled #907
 2021-08-25 18:30:05 +02:00
Thibault "bui" Koechlin c188d401a3
Improve CAPI pull management (#871) 
* prepare for new consensus : thousands of ips

Co-authored-by: Sebastien Blot <sebastien@crowdsec.net>
 2021-08-25 11:45:29 +02:00
Thibault "bui" Koechlin 950759f6d6
Output plugins (#878) 
* Add plugin system for notifications (#857)
 2021-08-25 11:43:29 +02:00
Manuel Sabban 4dbbd4b3c4
Download datafile (#895) 
* add the ability to download datafile on cscli hub upgrade on files are missing
* fix stuff + lint
* fix error management

Co-authored-by: sabban <15465465+sabban@users.noreply.github.com>
 2021-08-19 09:08:20 +02:00
Shivam Sandbhor f64f20fd53
Document scope parameter for stream API (#897) 
Signed-off-by: Shivam Sandbhor <shivam@crowdsec.net>
 2021-08-18 16:05:56 +02:00
Nanik b0746fbc4d
fix: add /health endpoint (#881) 
* fix: add /health endpoint
 2021-08-18 09:06:01 +02:00
Thibault "bui" Koechlin 05ac3ca402
if profile is in debug, log debug even if it matched the profile (#894) 2021-08-17 16:50:16 +02:00
Thibault "bui" Koechlin 25ed1c265d
fix #885 : remove dead dependencies for plugin (#891) 2021-08-17 10:32:15 +02:00
Thibault "bui" Koechlin fc7369c4ea
Fix big serialized entries (#877) 
* bump serialized to 8k

* handle oversized serialized entry : progressively strip its size down
 2021-08-03 15:46:10 +02:00
Thibault "bui" Koechlin 01028d0a09
Goroutine leak hunt (#874) 
* close the writers of gin loggers + kill the tomb of httpServer

* body close defer
 2021-07-30 11:41:17 +02:00
blotus cedfca07c2
don't wait for acquis tomb if we have no sources (#868) 2021-07-28 08:58:44 +02:00
Thibault "bui" Koechlin b6ee006078
ensure decisions from CAPI have proper case (#848) 2021-07-02 11:23:46 +02:00
Thibault "bui" Koechlin 033c8e17e8
fix #842 #837 (#845) 
* fix #842 and move preflight checks tgth

* handle new container name

Co-authored-by: AlteredCoder <AlteredCoder>
 2021-07-01 18:15:22 +02:00
blotus 3994aec7fe
add console enroll command to cscli (#828) 2021-06-28 17:34:19 +02:00
Thibault "bui" Koechlin 7f0cac8ee6
add support for 'expression' (fix #822) in grok patterns (#830) 
* add support for 'expression' (fix #822) in grok patterns

* add tests
 2021-06-21 09:07:33 +02:00
Thibault "bui" Koechlin ce6a61df1c
Refactor Acquisition Interface (#773) 
* Add new acquisition interface + new modules (cloudwatch, syslog)

Co-authored-by: Sebastien Blot <sebastien@crowdsec.net>
 2021-06-11 09:53:53 +02:00
Thibault "bui" Koechlin 71c1d9431f
fix #823 : lower JsonExtract debug (#824) 
* lower key not found log level, fix #823
 2021-06-02 14:27:34 +02:00
Shivam Sandbhor f25d02a7c8
Allow bouncers to filter decisions by scope (#817) 
Signed-off-by: Shivam Sandbhor <shivam@crowdsec.net>
 2021-05-31 15:07:09 +02:00
Thibault "bui" Koechlin bf6b791420
fix #781 - avoid unconsistent body : do not send NbDeleted on error (#812) 2021-05-28 11:17:30 +02:00
blotus c1c76645a7
improve emoji for local configuration when listing (#811) 2021-05-28 11:11:53 +02:00
svesve 6693bff2f5
Add postgres sslmode option (#772) 
Co-authored-by: aleksandr.drozdin <aleksandr.drozdin@karuna.group>
 2021-05-19 17:03:23 +02:00
he2ss eb0bd70046
fix #787 : load simulation config at startup (#793) 
* fix #787 : load simulation config at startup
 2021-05-17 11:54:28 +02:00
Thibault "bui" Koechlin f881510f79
delete orphan nodes (fix #778) (#794) 
* delete orphan nodes (for #778 and partially #781)

* and do it as well for decisions
 2021-05-17 11:45:01 +02:00
AlteredCoder fd830b4293
Fix some bugs (#788) 
* fix config restore

* fix panic on middleware

Co-authored-by: AlteredCoder <AlteredCoder>
 2021-05-07 18:40:01 +02:00
AlteredCoder a19f13ab45
fix cscli alerts delete -all (#769) 
Co-authored-by: AlteredCoder <AlteredCoder>
 2021-04-27 11:59:18 +02:00
Thibault "bui" Koechlin b0d4744b15
add System to cwversion to know platform, add it in UA (#763) 2021-04-23 15:23:46 +02:00
registergoofy 7e9ce901a4
add TimeNow in the exprlib helpers (#756) 
* add TimeNow in the exprlib helpers
* add a default date when none is detected: when no date is recognised by ParseDate, then use time.Now()
 2021-04-16 19:13:48 +02:00
Lars Lehtonen d86ba98cff
pkg/apiserver: fix dropped error (#700) 
* pkg/apiserver: fix dropped error

* pkg/apiserver: remove unused Context from APIServer{}
 2021-04-07 14:51:00 +02:00
Thibault "bui" Koechlin 4bb34d8e77
fix #723 : intercept http2 stream closed errors (#724) 
* fix #723 : intercept http2 stream closed errors

* factorize the 'dump stacktrace' code
 2021-04-07 14:31:03 +02:00
Thibault "bui" Koechlin cd06929e75
honor log levels for api : don't log access logs if level is warn/err (#732) 
* honor log levels for api : don't log access logs if level is warn/err

* add basic test for logging of api server
 2021-04-07 11:39:24 +02:00
Thibault "bui" Koechlin 20ef67a699
cscli hub mgmt improvements (#710) 
* avoid this confusing behaviour where 'cscli parsers/scenarios/... upgrade' won't tell a thing if no arguments are given (and won't do anything neither)

* avoid repeating warnings about available update to the user
 2021-03-29 10:33:23 +02:00
AlteredCoder 1e899c2211
Refactor configuration management (#698) 2021-03-24 18:16:17 +01:00
Thibault "bui" Koechlin 6d28599efa
Ensure LAPI logs respect log_media (#707) 
* if log_media is set to file, don't try to log to stdout

* use the log media no matter what
 2021-03-22 17:46:55 +01:00
AlteredCoder 4166d9ff48
fix pattern registration (#715) 2021-03-22 17:17:24 +01:00
Thibault "bui" Koechlin 1938e1a62d
clarify doc on onsuccess in parsers + add new date formats for dateparse (#703) 2021-03-19 16:33:10 +01:00
Lars Lehtonen 7f8faa7565
pkg/apiclient: pick up dropped errors (#676) 2021-03-17 12:36:47 +01:00
Thibault "bui" Koechlin 28446b6d29
Ent update : 0.7.0 (#692) 
* up regenerate new schema

* new ent

* update documentation for min required versions

* update documentation
 2021-03-15 18:46:52 +01:00
AlteredCoder c1abf69979
fix #677 (#684) 2021-03-12 15:10:56 +01:00
AlteredCoder f2d14c8ca2
update the config.yaml file (#674) 2021-03-11 11:18:09 +01:00
Thibault "bui" Koechlin 0981aa98d8
Pattern syntax consistence (#675) 
* fix #667

* improved error message

* mark the compability, ordered pattern_syntax will be tagged as 'version 2'

* fix tests + add tests to check grok subpattern dependencies
 2021-03-10 18:27:21 +01:00
Lars Lehtonen 7863bad596
pkg/metabase: fix dropped error (#652) 2021-03-10 15:11:56 +01:00
registergoofy a8b16a66b1
truely don't try to send anything with empty online credentials configuration file (#657) 
* truely don't try to send anything with empty online credentials config file

Co-authored-by: AlteredCoder <AlteredCoder>
 2021-03-02 09:25:12 +01:00
Thibault "bui" Koechlin 70055b3fd6
Doc api + minor api fixes (#654) 
* add doc for API

* link users guide on metabase without docker

* rename doc and swagger
 2021-02-26 17:42:45 +01:00
registergoofy 5b7ac4a473
[Rebased] fix races (#633) 
* get rid of dead code
* have LeakRoutined started in a tomb
* fix race and multiple small issues in the way we handle tombs
* yet another race fix
* another race
* get rid of leaky.KillSwitch for proper tomb use
* fix deadlock
* empty overflow before exiting
* fix an obvious typo
* proper use of waitgroup
* have a smart signalisation for allowing LeakRoutine being killed
* ugly workaround
* fix lint error
* fix compilation
* fix panic
* shorten lock
* up lock both copy
* wait for crowdsec to die
* fix coding style and lint issue
* go mod tidy

Co-authored-by: bui <thibault@crowdsec.net>
 2021-02-25 11:26:46 +01:00
AlteredCoder 8b504e9f67
improve logging in cscli and wizard (#643) 2021-02-25 11:20:36 +01:00
Thibault "bui" Koechlin a3d00fe130
skip empty lines to avoid issue of #630 (#631) 
* skip empty lines to avoid issue of #630

* add tests on empty lines and comms
 2021-02-25 09:57:24 +01:00
Thibault "bui" Koechlin 22ada59393
Allow for acquisition files to be specified from a directory as well (#619) 
* allow a acquisition_dir in crowdsec's config + change the behaviour of config loading so that it's working with a list instead. keep backward compat with acquisition_path

* remove the default behaviour of 'guessing' acquis path if param isn't present, and error
 2021-02-17 13:55:36 +01:00
Thibault "bui" Koechlin 7d93302e05
add a prometheus_uri option for cscli's config (#625) 
* add a prometheus_uri option for cscli's config, and update documentation

* specify min version
 2021-02-17 13:53:57 +01:00
Thibault "bui" Koechlin 7f40160f6e
only set logfile dir if media is file (#615) 2021-02-11 18:28:01 +01:00
AlteredCoder dae4458a6f
create crowdsec group for metabase and crowdsec.db (#606) 2021-02-10 09:23:33 +01:00
blotus 260332c726
Add use_forwarded_for_headers configuration option for LAPI (#610) 
* Add use_forwarded_for_headers configuration option for LAPI

* update documentation
 2021-02-09 19:10:14 +01:00
AlteredCoder 22c4962768
don't load lapi creds when running only api (#608) 
Co-authored-by: AlteredCoder <AlteredCoder>
 2021-02-09 17:59:35 +01:00
AlteredCoder 50ee846e87
enable item when they have been added to a collection since previous release (#599) 
Co-authored-by: AlteredCoder <AlteredCoder>
 2021-02-04 17:17:51 +01:00
AlteredCoder 359a9cb8ce
allow environment variable in configuration file (#601) 2021-02-04 17:17:01 +01:00
Thibault "bui" Koechlin e74f221044
Fix default configurations (#597) 
* fix default perms on SQLite file

* seed the prng securely

* fix defaults to enforce certificates verification

* ensure file is within path

* ensure the directory doesn't exist beforehand

* verify certificate by default

* disable http ip forward headers
 2021-02-02 14:15:13 +01:00
Shivam Sandbhor 36844e50b3
Fix typo in apic.go logs (#592) 2021-01-31 11:42:17 +01:00
Thibault "bui" Koechlin 25562e9575
drop the platform argument to avoid being compatible ONLY with API 1.41 (#582) 2021-01-18 15:25:07 +01:00
AlteredCoder 81e7db71ed
Fix bugs in wizard and cscli (#577) 
* fix id generation bug

* fix api client response

Co-authored-by: AlteredCoder <AlteredCoder>
 2021-01-15 18:14:50 +01:00
AlteredCoder 5544000d38
lapi: fix ipv6 operations (#567) 2021-01-14 16:27:45 +01:00
Thibault "bui" Koechlin 9ec0ea08bb
fix jwt token desynchronization between crowdsec and lapi (#572) 2021-01-14 16:04:10 +01:00
AlteredCoder c2517e8eb4
fix docker container creation for metabase (#563) 2021-01-08 14:32:29 +01:00
registergoofy eda9c03c82
jwt token generation improvement (#557) 
* add some warning comment for those who want to choose their secret
* strictly follow the golang doc for using crypto/rand
* fatal if not enough entropy
* add a check when using pre-choosen secret
 2021-01-07 14:24:53 +01:00
Thibault "bui" Koechlin ad4521f2cc
gin: broken pipe (#538) 
* broken pipe

* don't fail if release isn't here
 2020-12-14 17:48:32 +01:00
registergoofy 13881edbaa
export node logger (#537) 2020-12-14 14:12:22 +01:00
Thibault "bui" Koechlin f2b30db684
ensure decisions from local or tainted scenarios aren't push, neither are manual decisions (#536) 2020-12-14 12:46:07 +01:00
Thibault "bui" Koechlin bb679310c7
deal with LAPI down : ensure client will reauthenticate (#527) 
* to avoid keeping apiclient in broken state, reset the token on error
 2020-12-14 11:54:16 +01:00
erenJag b6d73f48cd
Fix some bugs : update doc, codename and fix wizard (#522) 
* change localhost to 127.0.0.1 + fix uninstall in wizard
* remove beta from repo
 2020-12-08 12:45:36 +01:00
erenJag 339cb6cce7
update prometheus doc (#509) 2020-12-04 11:24:12 +01:00
registergoofy f411ab4fcd
Fix a crash (#503) 
* fix a crash
 2020-12-03 17:34:57 +01:00
erenJag fd744408c3
fix cwhub remove func (#501) 2020-12-03 12:05:27 +01:00
erenJag 9d016f262f
fix & improve cscli remove action + improve cscli args vars (#498) 2020-12-02 18:47:17 +01:00
Thibault "bui" Koechlin 2e76097d35
Fix overflows of overflows requesting for different decision scope (#499) 2020-12-02 17:15:48 +01:00
Thibault "bui" Koechlin b7190c9ecc
improve error management of cscli bouncers add (#495) 2020-12-01 16:16:01 +01:00
erenJag 71325d9134
Improve create alerts input (#493) 
* check decisions start_ip & end_ip fields
 2020-12-01 14:42:53 +01:00
erenJag a16fb1475d
add info message when there is no hub index (#492) 2020-12-01 12:33:14 +01:00
AlteredCoder c6eb2afa20
push to CAPI in go routine (#489) 
Co-authored-by: AlteredCoder <AlteredCoder>
 2020-11-30 17:46:02 +01:00
Thibault "bui" Koechlin 71ac0d2fce
Apiclient tests (#484) 
Co-authored-by: AlteredCoder
Co-authored-by: erenJag
 2020-11-30 16:15:07 +01:00
Thibault "bui" Koechlin dbb420f79e
local api (#482) 
Co-authored-by: AlteredCoder
Co-authored-by: erenJag
 2020-11-30 10:37:17 +01:00
registergoofy f0ea8312db
set default hub branch to master in cscli (#279) 2020-10-01 15:02:53 +02:00
registergoofy 9b97633043
fix ban flush (#277) 
* fix ban flush by soft-deleting entries in database
* fix unit tests accordingly
 2020-10-01 08:26:59 +02:00
registergoofy c6aab9893a
add randomness to machine-id when registering. (#261) 
* add randomness to machine-id when registering.

* add some regexp check for machine_id

* typo fix

* fix cwapi unit tests
 2020-09-29 13:17:33 +02:00
AlteredCoder b7286d6a85
make cscli use crowdsec version for hub (#194) 2020-09-01 14:32:45 +02:00