Commit graph

Select branches

Hide pull requests

add-history

add-term-info-to-index

allow-extra-results

fastapi-users

improve-crawling

local

main

monolith

rishabh-add-launch-json

rishabh-port-configuration

rishabh-unit-testing-with-ci

unique-urls-in-queue

update-index

update-readme-for-new-crawler

#100

#109

#109

#11

#110

#112

#113

#114

#115

#116

#117

#118

#119

#120

#121

#122

#123

#124

#125

#126

#127

#128

#129

#13

#130

#17

#18

#21

#23

#24

#25

#31

#34

#36

#37

#38

#39

#41

#42

#46

#47

#5

#51

#53

#54

#55

#56

#57

#58

#6

#61

#66

#68

#69

#70

#71

#72

#73

#74

#78

#82

#83

#86

#89

#90

#91

#92

#93

#94

#95

#96

#97

#98

Mono Color

• bd0cc3863e Don't try and update an empty list of URLs Daoud Clarke 2023-01-09 21:02:40 +0000
• d347a17d63 Update URL queue separately from the other background process to speed it up Daoud Clarke 2023-01-09 20:50:28 +0000
• 7bd12c1ead Fix some bugs in URL fetching query Daoud Clarke 2023-01-02 20:51:23 +0000
• a50f1d8ae3 Fix postgres install Daoud Clarke 2023-01-02 12:19:10 +0000
• 1ab16b1fb4 Install postgres client Daoud Clarke 2023-01-02 12:18:03 +0000
• dda5a25ad0 Add core domains Daoud Clarke 2023-01-02 12:05:22 +0000
• ab37bbe0a5 Exclude google plus Daoud Clarke 2023-01-01 22:18:47 +0000
• 2336ed7f7d Allow posting extra links with lower score weighting Daoud Clarke 2023-01-01 20:37:41 +0000
• 6edf48693b Check the domain is correct, potential bug in psql Daoud Clarke 2023-01-01 01:30:44 +0000
• b7984684c9 Tidy, improve logging Daoud Clarke 2023-01-01 01:14:05 +0000
• 7c14cd99f8 Update the URL queue earlier Daoud Clarke 2022-12-31 23:37:59 +0000
• 0d33b4f68f
  Merge pull request #86 from mwmbl/improve-crawling Daoud Clarke 2022-12-31 22:56:21 +0000
• a86e172bf3 Reinstate background tasks #86 improve-crawling Daoud Clarke 2022-12-31 22:52:17 +0000
• d9cd3c585b Get results from other domains Daoud Clarke 2022-12-31 22:51:00 +0000
• 77f08d8f0a Update URL status Daoud Clarke 2022-12-31 22:25:05 +0000
• 36af579f7c Sample domains Daoud Clarke 2022-12-31 17:04:38 +0000
• ea16e7b5cd WIP: improve method of getting URLs for crawling Daoud Clarke 2022-12-31 13:37:40 +0000
• 7dae39b780 WIP: improve method of getting URLs for crawling Daoud Clarke 2022-12-31 13:32:15 +0000
• c69108cfcc Don't delete an index if the sizes don't match Daoud Clarke 2022-12-27 10:52:46 +0000
• bb8a36a612 Number of pages is an int Daoud Clarke 2022-12-27 10:40:53 +0000
• c01129cdb9 Merge branch 'master' of github.com:mwmbl/mwmbl Daoud Clarke 2022-12-27 10:25:41 +0000
• 26351a1072 Use the correct storage location in prod Daoud Clarke 2022-12-27 10:24:48 +0000
• f3f3831a97
  Merge pull request #83 from omasanori/spacy-deps-rework Daoud Clarke 2022-12-27 10:20:52 +0000
• 71187a3938 Rework installation of spaCy models for clarity #83 Masanori Ogino 2022-12-27 11:30:35 +0900
• b8f7775b49 Add configuration for Devbox #82 Masanori Ogino 2022-12-27 03:33:47 +0900
• d85067ec09 Remove apt command Daoud Clarke 2022-12-24 20:20:53 +0000
• 1ef60e8d5d Put install in correct place Daoud Clarke 2022-12-24 20:18:02 +0000
• 8e613dd368 Install psql client Daoud Clarke 2022-12-24 20:13:53 +0000
• 80282cfc7a Exclude a domain Daoud Clarke 2022-12-24 19:59:56 +0000
• 8676abbc63 Format fetched url Daoud Clarke 2022-12-24 19:59:15 +0000
• 57295846cb
  Update README.md Daoud Clarke 2022-12-21 21:49:56 +0000
• 0a4e1e4aee Add endpoint to fetch a URL and return title and extract Daoud Clarke 2022-12-21 21:15:34 +0000
• c7571120cc Implement validation Daoud Clarke 2022-12-21 15:32:30 +0000
• 061462460b Separate out the curation to make it easier to store in a comment Daoud Clarke 2022-12-20 19:11:01 +0000
• 6cf27fa47f Fix serialisation issue Daoud Clarke 2022-12-19 23:19:32 +0000
• b559a50506 Require the whole result Daoud Clarke 2022-12-19 22:18:28 +0000
• 5eab543f3b Merge branch 'master' into user-registration Daoud Clarke 2022-12-19 21:53:11 +0000
• a88a1a3e95 Rename some parameters; return curation ID Daoud Clarke 2022-12-19 21:51:26 +0000
• efc8e8e383
  Merge pull request #78 from mwmbl/make-dev-easier Daoud Clarke 2022-12-19 21:50:54 +0000
• 31c27daca4 Add curations Daoud Clarke 2022-12-11 18:48:25 +0000
• f89e1d6043 Create a post when beginning curation Daoud Clarke 2022-12-10 23:45:10 +0000
• eadb7f3e28 Follow a begin curate/update curation workflow Daoud Clarke 2022-12-10 22:49:06 +0000
• f8ab6092b0 Suggest using dokku instead of docker directly #78 Daoud Clarke 2022-12-08 22:33:58 +0000
• 8aa51e548b Allow login Daoud Clarke 2022-12-08 22:23:48 +0000
• cf6ceedfd5 Actually allow registration Daoud Clarke 2022-12-07 22:56:20 +0000
• a50bc28436 Make it easier to rum mwmbl locally Daoud Clarke 2022-12-07 20:01:31 +0000
• d8d7149f4a Start to implement user registration using Lemmy as a back end Daoud Clarke 2022-12-06 22:36:38 +0000
• c0f89ba6c3
  Update matrix badge Daoud Clarke 2022-12-05 18:47:26 +0000
• dd4dd8a752 Exclude an annoying web site Daoud Clarke 2022-12-02 21:29:06 +0000
• 40f9eade9a Update index name Daoud Clarke 2022-08-27 09:38:39 +0100
• b6183e00ea
  Merge pull request #74 from mwmbl/evaluate-indexing Daoud Clarke 2022-08-27 09:37:22 +0100
• cf253ae524 Split out URL updating from indexing #74 Daoud Clarke 2022-08-26 22:20:35 +0100
• f4fb9f831a Use terms and bigrams from the beginning of the string only Daoud Clarke 2022-08-26 17:20:11 +0100
• 619b6c3a93 Don't remove stopwords Daoud Clarke 2022-08-24 21:08:33 +0100
• 578b705609 Don't replace full stops and commas Daoud Clarke 2022-08-23 22:06:43 +0100
• 4779371cf3 Use a custom tokenizer Daoud Clarke 2022-08-23 21:57:38 +0100
• b1eea2457f Script to index local batch for evaluation Daoud Clarke 2022-08-22 22:47:42 +0100
• 480be85cfd Fix bug in completions with duplicated terms Daoud Clarke 2022-08-14 22:03:50 +0100
• f7660bcd27
  Merge pull request #73 from mwmbl/completion Daoud Clarke 2022-08-13 23:55:22 +0100
• 627f82d19f Suggest searching Google if there are no search results #73 Daoud Clarke 2022-08-13 23:54:57 +0100
• f1c77d1389 Search google if there are no results Daoud Clarke 2022-08-13 23:47:48 +0100
• fe5eff7b64 Exclude web.archive.org as we're only crawling that right now Daoud Clarke 2022-08-13 10:52:31 +0100
• 9920fc5ddd Disguise URLs so Firefox doesn't recognise them and filter them out local Daoud Clarke 2022-08-13 10:49:55 +0100
• a8bbb9f303 Missing import Daoud Clarke 2022-08-13 10:14:28 +0100
• 6022d867a3 Merge branch 'completion' into local Daoud Clarke 2022-08-13 10:08:37 +0100
• 00705703f3 Require matching at least half the terms Daoud Clarke 2022-08-11 23:27:30 +0100
• eda7870788 Restrict to https and strip the prefix and / on the end Daoud Clarke 2022-08-11 22:23:14 +0100
• 23e47e963b Simplify completions Daoud Clarke 2022-08-11 17:34:52 +0100
• c6773b46c4
  Merge pull request #72 from mwmbl/improve-ranking-with-multi-term-search Daoud Clarke 2022-08-10 21:43:51 +0100
• 74107667b4 Improve printing of search results in script #72 Daoud Clarke 2022-08-10 21:43:13 +0100
• 3bcb7f42c1 Use heuristic ranker Daoud Clarke 2022-08-09 22:56:12 +0100
• 9b22c32322 Merge branch 'improve-ranking-with-multi-term-search' into local Daoud Clarke 2022-08-09 22:50:56 +0100
• c1b9e70743 Add new LTR model Daoud Clarke 2022-08-09 22:47:59 +0100
• 57476ed2c8 Tweak features Daoud Clarke 2022-08-09 22:23:36 +0100
• c99e813398 Get best-performing configuration Daoud Clarke 2022-08-09 20:56:15 +0100
• 8b50643303 Add in match score feature (although it hurts the results) Daoud Clarke 2022-08-09 00:08:55 +0100
• c60b73a403 Create a get_features function and make it work like the heuristic approach Daoud Clarke 2022-08-08 23:42:34 +0100
• c1d361c0a0 New LTR model trained on more data Daoud Clarke 2022-08-08 22:52:37 +0100
• b99d9d1c6a Search for the term itself as well as its completion Daoud Clarke 2022-08-01 23:38:14 +0100
• a40259af30 Search for the term itself as well as its completion Daoud Clarke 2022-08-01 23:38:14 +0100
• 87d2e9474c Merge branch 'improve-ranking-with-multi-term-search' into local Daoud Clarke 2022-08-01 23:33:52 +0100
• f40d82c449 Allow running with no background script Daoud Clarke 2022-08-01 23:33:02 +0100
• 046f86f7e3
  Merge pull request #71 from mwmbl/fix-missing-scores Daoud Clarke 2022-08-01 23:32:24 +0100
• ae658906dd Store the best items, not the worst ones #71 Daoud Clarke 2022-07-31 22:55:15 +0100
• aa5878fd2f
  Merge pull request #70 from mwmbl/reduce-new-batch-contention Daoud Clarke 2022-07-31 21:02:05 +0100
• fc1742e24f Reinstate correct num_pages #70 Daoud Clarke 2022-07-31 00:45:00 +0100
• bb5186196f Use an in-memory queue Daoud Clarke 2022-07-31 00:43:58 +0100
• 62ba9ddc7e Use a randomised timeout for getting a new batch Daoud Clarke 2022-07-30 23:10:37 +0100
• a54e093cf1
  Merge pull request #69 from mwmbl/reduce-contention-for-client-queries Daoud Clarke 2022-07-30 17:11:34 +0100
• 2942d83673 Get URL scores in batches #69 Daoud Clarke 2022-07-30 14:35:21 +0100
• 3709cb236f Use correct index path; retrieve historical batches Daoud Clarke 2022-07-30 11:08:15 +0100
• 063ebb4504 args.index no longer exists Daoud Clarke 2022-07-30 10:57:15 +0100
• ea32c0ba00 Double index size Daoud Clarke 2022-07-30 10:37:07 +0100
• 2d5235f6f6 More threads for retrieving batches Daoud Clarke 2022-07-30 10:09:27 +0100
• 218d873654 Delete unused SQL Daoud Clarke 2022-07-30 09:27:44 +0100
• 3137068c77 More threads for retrieving batches Daoud Clarke 2022-07-30 10:09:27 +0100
• e79f1ce10b Delete unused SQL Daoud Clarke 2022-07-30 09:27:44 +0100
• c52faeaddc Merge branch 'reduce-contention-for-client-queries' into local Daoud Clarke 2022-07-24 17:02:37 +0100
• 6209382d76 Index batches in memory Daoud Clarke 2022-07-24 15:44:01 +0100
• 1bceeae3df Implement new indexing approach Daoud Clarke 2022-07-23 23:19:36 +0100