Commit graph

892 commits

Author SHA1 Message Date
blotus d7ef51e6ba
properly update the cs_syslogsource_parsed_total metric (#2608) 2023-11-22 09:04:23 +01:00
Sebastien Blot 5abc8e0e14
merge hub-1.5.6 2023-11-21 17:46:54 +01:00
mmetc 2c652ef92f
pkg/cwhub documentation (#2607)
* pkg/cwhub: package documentation

* Don't repeat local state in "cscli... inspect"

* lint

* use proper name of the hub item instead of the filename for local items

* hub update: avoid reporting local items as tainted
2023-11-21 17:43:10 +01:00
Sebastien Blot 9580f8e14d
merge hub-1.5.6 2023-11-21 17:28:10 +01:00
bui e4b92af78c support dedicated waap rules testing in cscli hubtest 2023-11-21 15:24:51 +01:00
mmetc 1509c2d97c
pkg/cwhub refact (#2606)
* Separate Item and ItemState; fill BelongsToCollections with all ancestors and for uninstalled items too
* fix "installed parents" check when removing an item
* keep BelongsToCollections in order (case insensitive)
2023-11-21 11:06:59 +01:00
mmetc 7b1074f0cb
Refact cwhub (#2603)
* Split RemoteHub.downloadIndex() = Hub.updateIndex() + RemoteHub.fetchIndex()
* Functions safePath(), Item.installPath(), item.downloadPath()
2023-11-20 15:58:42 +01:00
bui 2d01e4680f do not error if no waap rules are present 2023-11-20 14:25:33 +01:00
Sebastien Blot 4a265ca4af
up 2023-11-20 13:27:46 +01:00
mmetc 6b317f0723
Refact cwhub: simplify tree scan and dependency checks (#2600)
* method rename: GetInstalledItemsAsString() -> GetInstalledItemNames()
* use path package
* Comments and method names
* Extract method Item.setVersionState() from Hub.itemVisit()
* refact localSync(), itemVisit() etc.
* fix check for cyclic dependencies, with test
2023-11-20 11:41:31 +01:00
alteredCoder 8173e1ba42 add timeout to auth request 2023-11-20 10:48:21 +01:00
Sebastien Blot 94a378d230
up 2023-11-17 18:07:03 +01:00
bui 017331ca7f nuclei runner 2023-11-17 15:37:32 +01:00
bui 6718d82765 allow testing of waap rules 2023-11-17 15:37:12 +01:00
bui 9af30e2a3d simplify a bit 2023-11-17 15:15:29 +01:00
Sebastien Blot 0e717cb558
up 2023-11-17 13:47:05 +01:00
Sebastien Blot d40e9fb760
do not use filepath.Match 2023-11-17 13:45:43 +01:00
alteredCoder 9864d2c459 Add authentication between bouncers and waf 2023-11-16 18:19:45 +01:00
Sebastien Blot 9db48e2110
fix collections install/inspect with waap-{rules,configs} 2023-11-16 17:17:33 +01:00
Sebastien Blot db40ba7b3b
Merge branch 'hub-1.5.6' into coraza_poc_acquis 2023-11-16 17:12:23 +01:00
mmetc 56ad2bbf98
Refact cwhub: item removal with shared dependencies (#2598)
* Iterate over sub-items in Remove(), not in disable() -- fix shared dependency issue
* Increase hub download timeout to 2 minutes
2023-11-16 17:00:51 +01:00
mmetc 65473d4e05
Refact cwhub: simplify enable/disable/download (#2597)
* Extract methods createInstallLink(), removeInstallLink(), simplify
 - the result of filepath.Join is already Cleaned
 - no need to log the creation of parentDir
 - filepath.Abs() only returns error if the current working directory has been removed
* Extract method Item.fetch()
* Replace Create() + Write() -> WriteFile()
2023-11-16 13:05:55 +01:00
mmetc d9b0d440bf
Refact cwhub (#2596)
* unused param
* (slightly) simpler ListItems() -> listItems()
* listItems(): always showHeader, deduce showType
ref. https://github.com/crowdsecurity/crowdsec/issues/1068
* simplify Item.disable()
also, .tainted and .installed do not need a default since they are always in the json output now
* Drop unused parameters
2023-11-16 11:09:49 +01:00
bui c8af58d1bf ensure we're sending lapi/capi alert if the request matched some inband rules 2023-11-15 17:46:31 +01:00
mmetc 79d019f9a2
Refact cwhub / sort cscli output, case insensitive (#2593)
* dead code: unknown localVersion now defaults to "?"
* skip type declaration; whitespace
* sync: next item if invalid cpath
* func tests for install --force and --ignore
* shorter test names
* sort cscli <itemtype> output, with tests
* cscli: refact hub sort code
2023-11-15 16:59:30 +01:00
bui 056c979455 add support for labels to waap rules 2023-11-15 15:08:57 +01:00
mmetc 4a6fd338e0
replace 'timeout' helper with async python script; allow hub preload in func tests; improve item removal (#2591)
* replace 'timeout' helper with async python script; allow hub preload in func tests; improve item removal
* func tests: cscli hub update/upgrade
* docker test update
* Update docker entrypoint to disable items with --force

The --force flag was not transmitted to cscli, but is required after the hub refact
to disable items inside installed collections
2023-11-14 17:36:07 +01:00
Marco Mariani 120f7cf578 Merge branch 'master' into hub-1.5.6 2023-11-14 15:04:50 +01:00
mmetc 042d316fab
Refact cwhub: remove global hub, func test improvements (#2588)
* csConfig.Cscli is always loaded now, configuration paths too
* Remove global/singleton hub instance
* read {index_path} from config instead of assuming {hub_dir}/.index.json
* fix segfault with cscli explain when no parser is installed
* cscli: help text
* hub download timeout 20 sec
* reduce log verbosity
* allow func tests with empty hub or pre-download
* cscli <itemtype> remove --all --purge
2023-11-14 14:58:36 +01:00
Sebastien Blot 07d463f4f0
up 2023-11-10 17:56:04 +01:00
Sebastien Blot d6f9bbc0c3
merge hub-1.5.6 branch 2023-11-10 17:36:17 +01:00
Sebastien Blot 4bfa0a7b4d
up 2023-11-10 17:33:53 +01:00
mmetc d5c7870826
Refact cwhub: remove global hub instance (#2587)
* csConfig.Cscli is always loaded now, configuration paths too
* Remove global/singleton hub instance
2023-11-10 17:32:12 +01:00
mmetc 9d7ed12950
Refact cwhub (#2586)
* Inspect item: always show tainted, installed, etc. when false
* cleanup, comments, unused stuff
* download collection content after downloading dependencies, avoid duplicate call
* Return instances from Item.SubItems()
* shorter i/o code
* inline / simplify getData()
* Handle timeout connections when downloading from hub or data
2023-11-10 10:25:29 +01:00
mmetc ab8de19506
Refact cwhub: move methods from hub to item (#2585)
* Add back pointer Item.hub
* Hub.enableItem() -> Item.enable()
* Rename variable i -> idx (i is used for item instances)
* Move Hub.purgeItem() -> Item.purge()
* Move Hub.disableItem() -> Item.disable()
* Move Hub.downloadItem() -> Item.download()
* Move Hub.downloadLatest() -> Item.downloadLatest()
* Move Hub.DownloadDataIfNeeded() -> Item.DownloadDataIfNeeded()
* Move Hub.InstallItem() -> Item.Install()
* Move Hub.RemoveItem() -> Item.Remove()
* Move Hub.UpgradeItem() -> Item.Upgrade()
* store hub items as pointers
* No need to re-add items to the hub if we use pointers
* Fix parameter calling order + regression test
2023-11-09 15:19:38 +01:00
mmetc f80d841188
Refact cwhub: make some methods private (#2584)
* make hub.enableItem() private
* make hub.downloadLatest() private
* make getData() private
* make hub.disableItem() private
* make hub.downloadItem() private
* make hub.syncDir() private
* make hub.localSync() private; keep warnings in Hub struct (no need to call LocalSync to get them)
2023-11-09 12:07:09 +01:00
mmetc ec4b5bdc86
Refact cwhub (#2583)
* no need to use NewRequest()
* download error messages
* cscli hub list: fix item stats
* Method item.HasSubItems() - avoid explicit type check
* cscli config restore: drop silent install, just call InstallItem
* no backpointer yet
2023-11-09 11:34:14 +01:00
Sebastien Blot a0b0745f9d
up 2023-11-08 21:14:03 +01:00
Sebastien Blot 927310a439
up 2023-11-08 20:37:05 +01:00
Sebastien Blot 694028f769
merge hub branch 2023-11-08 20:25:42 +01:00
Sebastien Blot 152c940774
wip 2023-11-08 20:24:44 +01:00
mmetc f4b5bcb865
Refact cwhub: version comparison and branch selection (#2581)
* simplify GetItemByPath
* hub: sort version numbers by semver
* replace golang.org/x/mod/semver with github.com/Masterminds/semver/v3 (would not compare correctly)
* fix nil dereference with tainted items
* update tests for collections, postoverflows
* fix nil deref
* don't fallback to master if hub is not found, improve message
* explicit message for unknown version / tainted collections
2023-11-08 13:21:59 +01:00
mmetc 5cd4406f5e
typos/grammar (#2561) 2023-11-07 15:07:36 +01:00
mmetc ad54b99bf9
Refact pkg/hubtest (#2580)
* pkg/hubtest: lint (whitespace, empty lines)
* use existing function to sort keys
* lint
* cscli hubtest: set TZ=UTC
* dedup Coverage struct
* pre-compile regexps
* remove redundant type declarations or global vars
2023-11-07 14:02:02 +01:00
mmetc bfd94ceda7
make ParserIndex(), DownloadIndex() private methods (#2579)
* unnecessary pointer type
* ParseIndex() as hub method, don't collect missing items since they are never used
* don't export hub.parseIndex(), hub.downloadIndex()
2023-11-07 10:27:33 +01:00
mmetc 41d19de092
Refact cwhub (#2578)
* Fix suggest functional tests
* comments
* non-empty SubItems() implies collections type
* use "slices" from stdlib
* No need to repeat author field in the index -- take it from the item key
2023-11-06 17:35:33 +01:00
Sebastien Blot 26c876dc38
merge hub-1.6 branch 2023-11-06 15:02:11 +01:00
mmetc 450c263826
Refact cwhub: minor cleanups and comments (#2574)
* check response status before body; close file
* err check one-liners, lint, comments
* simplify function logic, reduce code
* comments, xxx, whitespace
2023-10-31 16:32:29 +01:00
Marco Mariani fcd6c468c4 fix lint 2023-10-31 13:12:28 +01:00
mmetc 590a19b768
Refact pkg/cwhub: constructor, cscli output
* Single constructor: NewHub() to replace InitHub(), InitHubUpdate()
* sort cscli hub list output
* log.Fatal -> fmt.Errorf
2023-10-31 12:47:39 +01:00
Sebastien Blot 84ffde1844
add body_type in custom rule 2023-10-31 11:53:13 +01:00
mmetc 17662e59a9
Refact pkg/cwhub, cscli: hub upgrades (#2568)
* fix bats test for "upgrade all items"
* refact UpgradeConfig() -> UpgradeItem(): one item only
* refact RemoveMany() -> RemoveItem()
* Computed value: Item.Local -> Item.IsLocal()
* refact url/branch configuration with LocalHubCfg/RemoteHubCfg
2023-10-30 17:23:50 +01:00
bui 2e0b9683f3 logging clean up 2023-10-27 16:10:46 +02:00
bui d136cc4734 logging clean up 2023-10-27 16:10:36 +02:00
bui 81645c96aa logging clean up 2023-10-27 16:07:49 +02:00
bui 83d5211193 logging clean up 2023-10-27 16:07:37 +02:00
bui c96c8f19c9 logging clean up 2023-10-27 16:07:25 +02:00
Sebastien Blot 57b5f5c27c
uip 2023-10-27 11:21:19 +02:00
Sebastien Blot 37c5d54e43
up 2023-10-27 11:17:27 +02:00
Sebastien Blot b0e7da06b9
up 2023-10-27 11:10:40 +02:00
Sebastien Blot e5906e6eea
up 2023-10-27 11:10:40 +02:00
bui 01ddc45a2c use loggeR 2023-10-27 11:09:56 +02:00
bui bb59d9852a make Event viabl 2023-10-27 11:09:38 +02:00
bui 31a3b8a4ef move this to pkg/waf 2023-10-27 11:09:19 +02:00
bui 495c6f9e8a add debug to rule collection 2023-10-27 11:08:54 +02:00
mmetc 6b8ed0c9d0
Refactor hub URL/branch configuration (#2559)
* Refactor hub URL/branch configuration
* docker: using --force to implement $DISABLE (required for items in collections)
* use pointer receiver for consistency
2023-10-27 10:25:29 +02:00
bui cd1cefbc8b fix behavior so we only generate crowdsec events if interrupt was generated in either inband or outofband phases 2023-10-26 15:23:45 +02:00
bui 0cebf833c7 add options via WaapConfig for inband and outofband engines 2023-10-26 14:46:08 +02:00
bui 82bb8a2789 no leak plz 2023-10-26 13:01:11 +02:00
bui f18b554177 warn at start if body reading is disabled 2023-10-26 12:45:59 +02:00
bui 6cbeefead6 up 2023-10-26 12:04:58 +02:00
bui e49f33b4a7 Merge branch 'coraza_poc_acquis' of github.com:crowdsecurity/crowdsec into coraza_poc_acquis 2023-10-26 12:04:12 +02:00
bui 46ae0b3822 properly set default log level 2023-10-26 12:03:57 +02:00
Sebastien Blot 676352b5b1
new custom rule format 2023-10-25 18:45:49 +02:00
bui 4bfca8cab5 fix meta encoding 2023-10-25 13:54:57 +02:00
bui eafffe7c94 up 2023-10-24 18:16:39 +02:00
bui 9edde09608 up 2023-10-24 18:16:30 +02:00
bui 1f3801f390 add the helpers and the type 2023-10-24 17:24:31 +02:00
bui c02c74b5fe shortcut for waap events 2023-10-24 17:24:16 +02:00
bui b2bb15bb49 generate a special event for waap 2023-10-24 17:23:46 +02:00
bui 685006508c make waap rules generate crowdsec events (again) 2023-10-24 13:43:27 +02:00
bui 03650401c5 default level 2023-10-24 10:57:22 +02:00
bui 00e1ffbf58 simplify a bit 2023-10-24 10:49:28 +02:00
bui bd9df8f480 logger 2023-10-23 10:59:02 +02:00
bui 1b9d8c8226 logger 2023-10-23 10:54:26 +02:00
bui c00b1abd72 logger 2023-10-23 10:54:11 +02:00
bui 2ff238d5f8 logger 2023-10-23 10:53:52 +02:00
bui dca6faab08 logger 2023-10-23 10:53:39 +02:00
mmetc ac98256602
Refact pkg/cwhub, cmd/crowdsec-cli (#2557)
- pkg/cwhub: change file layout, rename functions
 - method Item.SubItems
 - cmd/crowdsec-cli: generic code for hub items
 - cscli: removing any type of items in a collection now requires --force
 - tests
2023-10-20 14:32:35 +02:00
bui b110c74487 allow description 2023-10-20 13:49:15 +02:00
bui 5dbc2758fa warn user when setting unexpected default_remediation 2023-10-20 13:32:20 +02:00
Sebastien Blot 0acda36d33
up 2023-10-20 11:58:57 +02:00
Sebastien Blot 1468bb9681
up 2023-10-19 17:25:48 +02:00
Sebastien Blot 68c78249d5
up 2023-10-19 17:20:33 +02:00
Sebastien Blot ef118a49ff
add waap-configs hub item 2023-10-19 16:53:00 +02:00
Sebastien Blot 15120a6d8f
merge hub-1.5.6 2023-10-19 14:19:37 +02:00
Sebastien Blot 350e8979b1
merge hub-1.5.6 branch 2023-10-19 12:18:16 +02:00
Marco Mariani b89c5652ca Merge branch 'master' into hub-1.5.6 2023-10-19 12:05:19 +02:00
mmetc 88e4f7c157
Refact pkg/csconfig, pkg/cwhub (#2555)
* csconfig: drop redundant hub information on *Cfg structs
* rename validItemFileName() -> item.validPath()
* Methods on hub object
* updated tests to reduce need of csconfig.Config or global state
2023-10-19 12:04:29 +02:00
Sebastien Blot ecbdf2f0e1
merge master branch 2023-10-19 10:51:54 +02:00