Commit graph

1703 commits

Author SHA1 Message Date
mmetc c10aad79d9
cscli refact / encapsulate methods for capi, hubtest, dashboard, alerts, decisions, simulation (#2650) 2023-12-11 10:32:54 +01:00
mmetc 1b4ec66148
fix package tests for 1.5.6-rc2 (#2652)
* fix go deps for test
* fix package tests for 1.5.6-rc2
* fix context func tests
* docker: pin build image version for alpine

---------

Co-authored-by: sabban <github@sabban.eu>
2023-12-11 10:07:09 +01:00
blotus 04f3dc09f9
remove PAPI feature flag (#2601) 2023-12-08 14:55:45 +01:00
Manuel Sabban c707b72b03
fix lapi credentials creation for debian package (#2646)
* fix lapi credentials creation for package s
---------

Co-authored-by: mmetc <92726601+mmetc@users.noreply.github.com>
2023-12-08 12:10:23 +01:00
mmetc 84cbff16d4
restrict file permissions from "machines add" (#2648) 2023-12-08 10:51:15 +01:00
AlteredCoder b1f85693c2
Appsec improvement and fixes after merge (#2645) 2023-12-08 10:25:00 +01:00
mmetc 518c7f178a
update dependency on aws sdk (#2647) 2023-12-08 10:07:53 +01:00
mmetc 4acb4f8df3
cwhub: context type (#2631)
* add hub type "context"
* cscli lapi: log.Fatal -> fmt.Errorf; lint
* tests for context.yaml
* load console context from hub
* original & compiled context
* deprecate "cscli lapi context delete"
$ cscli lapi context delete
Command "delete" is deprecated, please manually edit the context file.
* cscli completion: add appsec-rules, appsec-configs, explain, hubtest
2023-12-07 16:20:13 +01:00
mmetc 3e86f52250
cscli refact - encapsulation with types (#2643)
* refactor type cliHub, cliBouncers, cliMachines, cliPapi, cliNotifications, cliSupport, type cliExplain
2023-12-07 14:36:35 +01:00
Thibault "bui" Koechlin 8cca4346a5
Application Security Engine Support (#2273)
Add a new datasource that:
- Receives HTTP requests from remediation components
- Apply rules on them to determine whether they are malicious or not
- Rules can be evaluated in-band (the remediation component will block the request directly) or out-band (the RC will let the request through, but crowdsec can still process the rule matches with scenarios)

The PR also adds support for 2 new hub items:
- appsec-configs: Configure the Application Security Engine (which rules to load, in which phase)
- appsec-rules: a rule that is added in the Application Security Engine (can use either our own format, or seclang)

---------

Co-authored-by: alteredCoder <kevin@crowdsec.net>
Co-authored-by: Sebastien Blot <sebastien@crowdsec.net>
Co-authored-by: mmetc <92726601+mmetc@users.noreply.github.com>
Co-authored-by: Marco Mariani <marco@crowdsec.net>
2023-12-07 12:21:04 +01:00
mmetc 90d3a21853
CI: use go 1.21.5 (#2640)
* use go 1.21.5
* Simpler go:build directives
2023-12-06 12:38:36 +01:00
mmetc 1ab4487b65
cscli hub list: show only non-empty tables with -o human
* agent config: remove unused LintOnly bool
* Item.IsLocal() -> Item.State.IsLocal(); split method InstallStatus()
* cscli hub list: show only non-empty tables with -o human
2023-12-05 13:38:52 +01:00
mmetc 486f96e7ac
cscli context detect: fix nil dereference (#2635)
* cscli context detect: fix nil dereference
* Remove log.warning for missing pattern
2023-12-05 12:08:35 +01:00
mmetc 8bb7da3994
docker tests: force local machine creation (#2636)
This is required from 1.5.6 to overwrite the local credentials file
2023-12-05 11:52:04 +01:00
mmetc 0f3ae64062
cscli config show: pretty print with package "litter" (#2633) 2023-12-05 10:38:21 +01:00
mmetc 0c4093dcca
Test for acquisition errors in crowdsec -t (#2629) 2023-12-04 23:09:42 +01:00
mmetc 23968e472d
Refact bouncer auth (#2456)
Co-authored-by: blotus <sebastien@crowdsec.net>
2023-12-04 23:06:01 +01:00
mmetc a5ab73d458
cscli machines add: don't overwrite existing credential file (#2625)
* cscli machines add: don't overwrite existing credential file
* keep old behavior with --force
Now --force is used both to override the replacement of and existing machine,
and an existing credentials file. To retain the old behavior, the
existence of the file is only checked for the default configuration, not
if explicitly specified.
2023-12-04 22:59:52 +01:00
Laurence Jones f8755be9cd
Fix formt on documentation (#2577)
When generating decisions import docusarus v3 now does not allow `{` without escaping this adds escaping
2023-12-04 15:52:14 +00:00
Laurence Jones d1bfaddb69
[Plugin] Pass down ctx and use it (#2626)
* Pass down cancellable context and update http plugin

* Use context where we can
2023-12-04 12:05:26 +00:00
Laurence Jones bfc92ca1c5
[Explain] Ignore blank lines as crowdsec will anyways (#2630)
* Ignore blank lines within file and stdin

* change cleanup to be persistent postrun so if we exit early it always cleans

* When using log flag we should add a newline so we know where EOF is

* Inverse the check for log line since we dont want to modify the line itself

* Wrap run explain with a function that returns the error after cleaning up

* Wrap run explain with a function that returns the error after cleanup

* Use a defer iif instead of global var

* Add invalid len input to err count so it more obvious what is happening

---------

Co-authored-by: Manuel Sabban <github@sabban.eu>
2023-12-04 11:48:12 +00:00
Laurence Jones ed3d501081
[Metabase] QOL Changes and chown wal files (#2627)
* Add detection sqlie wal for dashboard chown

* Lean it down a little

* Change to for loop with extensions

* Keep existing uid on files incase user is running as a unpriviledge user

* I have no idea 🤷

* Exclude dash.go and update windows

* Update

* Renam

* Remove the os check since we no longer get to this stage for those os's

---------

Co-authored-by: Manuel Sabban <github@sabban.eu>
2023-12-04 10:06:41 +00:00
mmetc 7e5ab344a2
command "cscli hub types" (#2632)
* Command "cscli hub types"; de-duplicate test/bin/preload-hub-items
* don't export Hub.Items -> hub.items
2023-12-01 09:36:38 +01:00
Cristian Nitescu 7c5cbef51a
manage force_pull message for one blocklist (#2615)
* manage force_pull message for one blocklist

* fix info message on force pull blocklist
2023-11-29 11:37:46 +01:00
mmetc 6b0bdc5eeb
Refact pkg/cwhub: fix some known issues and reorganize files (#2616)
* bump gopkg.in/yaml.v3
* test: cannot remove local items with cscli
* test dangling links
* test: cannot install local item with cscli
* pkg/cwhub: reorg (move) functions in files
* allow hub upgrade with local items
* data download: honor Last-Modified header
* fatal -> warning when attempting to remove a local item (allows remove --all)
* cscli...inspect -o yaml|human: rename remote_path -> path
* Correct count of removed items
Still no separate counter for the --purge option, but should be clear enough
2023-11-28 23:51:51 +01:00
mmetc 1aa4fc5949
CI: avoid pipe in makefile, correctly report error in CI when tests fail (#2621)
so we don't assume bash+pipefail for the makefile
2023-11-28 17:10:44 +01:00
blotus 380cbf70a9
force rfc 3339 date format in metrics push (#2402) 2023-11-28 16:30:20 +01:00
Laurence Jones 05c1825622
Add to dump after postoverflow so we can test within hubtest (#2511)
Co-authored-by: Thibault "bui" Koechlin <thibault@crowdsec.net>
2023-11-28 13:18:41 +00:00
Laurence Jones 6a61b919e7
[cscli] notifications test command and slight re write (#2391)
* Merge main and apply stash

* Rework some of cscli notif stuff and add a generic test which works with non active profiles

* Update wording

* Fix merge

* Final version

* Cleanup
2023-11-28 13:17:54 +00:00
mmetc 15542b78fb
refact BulkDeleteDecisions (#2308)
Code cleanup and de-duplication.
2023-11-26 22:30:03 +01:00
mmetc b164373997
update dependencies: k8s apiserver, docker and related (#2476) 2023-11-24 16:20:39 +01:00
mmetc ffcab0b2bc
Refactor hub management and cscli commands (#2545) 2023-11-24 15:57:32 +01:00
mmetc 32e9eb4be4
Minor dependency updates (#2505)
* update AlecAivazis/survey
* update Masterminds/semver
* update Masterminds/sprig
* update alexliesenfeld/health
* update golang.org/x/net
2023-11-24 15:30:54 +01:00
mmetc 76d4bc7788
cscli bouncers: increase key size, deprecate and ignore --length option (#2531)
the switch to base64 made the keys shorter (24 characters), this PR increases their size to 32 bytes, 42 chars once encoded

Also deprecate the --length option, users can already provide a key
2023-11-24 15:01:13 +01:00
mmetc ec199162dc
iso8601: use yyyy-mm-dd in log timestamps instead of dd-mm-yyyy (#2564)
Co-authored-by: Thibault "bui" Koechlin <thibault@crowdsec.net>
2023-11-24 14:59:28 +01:00
Thibault "bui" Koechlin 1dcf9d1ae1
Improved expr debugger (#2495)
* new expr debugger

---------

Co-authored-by: mmetc <92726601+mmetc@users.noreply.github.com>
2023-11-24 11:10:54 +01:00
mmetc 7ffa0cc787
docker: replace cp -an with rsync to allow bind-mount of files in /etc/crowdsec (#2611)
fix for https://github.com/crowdsecurity/crowdsec/issues/2480
2023-11-23 11:08:14 +01:00
blotus ec53c672dc
Kafka acquisition: warn if no consumer group id and allow to read from a specific partition (#2612) 2023-11-23 10:02:53 +01:00
lperdereau 92f923cfa8
Loki integration #2 (#2306)
* Add support for loki datasource

---------

Co-authored-by: Mathieu Lecarme <mathieu@garambrogne.net>
Co-authored-by: Sebastien Blot <sebastien@crowdsec.net>
Co-authored-by: Thibault "bui" Koechlin <thibault@crowdsec.net>
2023-11-22 13:31:39 +01:00
he2ss 947b247a40
kafkaAcquisition: add more debug (#2609)
* kafkaAcquisition: add more debug
2023-11-22 09:35:58 +01:00
blotus d7ef51e6ba
properly update the cs_syslogsource_parsed_total metric (#2608) 2023-11-22 09:04:23 +01:00
dependabot[bot] a51bce8f8d
Bump google.golang.org/grpc from 1.56.1 to 1.56.3 (#2566)
Bumps [google.golang.org/grpc](https://github.com/grpc/grpc-go) from 1.56.1 to 1.56.3.
- [Release notes](https://github.com/grpc/grpc-go/releases)
- [Commits](https://github.com/grpc/grpc-go/compare/v1.56.1...v1.56.3)

---
updated-dependencies:
- dependency-name: google.golang.org/grpc
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-11-20 11:55:29 +01:00
mmetc 47eb2e240d
Use go 1.21.4 (#2595) 2023-11-16 11:09:13 +01:00
guangwu ddd6ee8e42
fix: typo (#2582)
Signed-off-by: guoguangwu <guoguangwu@magic-shield.com>
2023-11-08 09:26:34 +01:00
mmetc 5cd4406f5e
typos/grammar (#2561) 2023-11-07 15:07:36 +01:00
Manuel Sabban 4934fce769
update gantsign.golang name (#2558) 2023-11-07 14:53:14 +01:00
mmetc 272cf543b3
Release action: fix asset upload (#2565) 2023-10-25 14:51:36 +02:00
Laurence Jones d2d788c5dc
[hubtest] escpae scenario asssert meta keys (#2551) 2023-10-17 15:29:21 +01:00
Thibault "bui" Koechlin a4dc5053d2
fix null deref in cti calls if key is empty (#2540)
* fix null deref in cti calls if key is empty

* avoid hardcoded error check
2023-10-17 09:34:53 +01:00
Laurence Jones 19de3a8a77
Runtime whitelist parsing improvement (#2422)
* Improve whitelist parsing

* Split whitelist check into a function tied to whitelist, also since we check node debug we can make a pointer to node containing whitelist

* No point passing clog as an argument since it is just a pointer to node we already know about

* We should break instead of returning false, false as it may have been whitelisted by ips/cidrs

* reimplement early return if expr errors

* Fix lint and dont need to parse ip back to string just loop over sources

* Log error with node logger as it provides context

* Move getsource to a function cleanup some code

* Change func name

* Split out compile to a function so we can use in tests. Add a bunch of tests

* spell correction

* Use node logger so it has context

* alternative solution

* quick fixes

* Use containswls

* Change whitelist test to use parseipsource and only events

* Make it simpler

* Postoverflow tests, some basic ones to make sure it works

* Use official pkg

* Add @mmetc reco

* Add @mmetc reco

* Change if if to a switch to only evaluate once

* simplify assertions

---------

Co-authored-by: bui <thibault@crowdsec.net>
Co-authored-by: Marco Mariani <marco@crowdsec.net>
2023-10-16 10:08:57 +01:00