Commit graph

163 commits

Author SHA1 Message Date
vishnukvmd 52943f38ab Remove normalizations 2023-12-20 15:18:58 +05:30
vishnukvmd 79956502ae Init ObjectBox only in the foreground 2023-12-19 13:03:45 +05:30
vishnukvmd 6c242aa1d2 Implement the API to release resources held by ORT 2023-12-19 00:35:45 +05:30
vishnukvmd 836fb97598 Use the file directly to create an OrtSession 2023-12-19 00:35:16 +05:30
vishnukvmd b4019f3c50 Refactor commented code 2023-12-18 22:21:01 +05:30
vishnukvmd a5e5c60111 Initialize SSS only if it has been opted-in for 2023-12-18 22:17:19 +05:30
vishnukvmd 4357795d50 Remove unnecessary check 2023-12-18 22:05:03 +05:30
vishnukvmd 248f45c2c3 Reduce size of LRU cache 2023-12-18 12:55:15 +05:30
vishnukvmd d4ebb5c493 Add an LRU cache for queries 2023-12-18 12:54:54 +05:30
vishnukvmd 073e2a06e6 Update model path 2023-12-18 12:14:17 +05:30
vishnukvmd 22d24cfd73 Fix warnings 2023-12-18 11:28:32 +05:30
vishnukvmd 8c0b569702 Remove all release calls 2023-12-17 17:46:06 +05:30
vishnukvmd fb11e730ce Skip image model loading 2023-12-16 17:10:39 +05:30
vishnukvmd 621599496e Use ONNX for text embeddings 2023-12-16 16:50:07 +05:30
vishnukvmd d0a85aa146 Fix warnings 2023-12-16 16:22:01 +05:30
0nullpointer 04b4a1fbba Add ggml image preprocessing 2023-12-15 02:45:00 +05:30
vishnukvmd 61a6481de4 Extract variable 2023-12-14 15:38:42 +05:30
0nullpointer cfc0d05a14 image normalization with tensors 2023-12-14 12:02:14 +05:30
vishnukvmd 03f471623b Temporarily parse model from assets 2023-12-14 11:31:01 +05:30
vishnukvmd c35ee381d3 Add ability to clear local embeddings 2023-12-14 11:30:09 +05:30
vishnukvmd 1c488e86a5 Increase batch size for caching thumbs 2023-12-14 11:25:16 +05:30
vishnukvmd 7bbcdb0010 Increase batch size for caching thumbnails 2023-12-13 19:08:05 +05:30
vishnukvmd f5b7897838 Fix thumbnail caching logic 2023-12-13 17:24:27 +05:30
vishnukvmd ad8f7cbccd Cache all thumbnails first 2023-12-13 15:16:34 +05:30
vishnukvmd e1ea9530f1 Remove score threshold 2023-12-13 15:02:35 +05:30
vishnukvmd bbee1c0e41 Fetch only relevant embeddings from disk 2023-12-13 14:56:08 +05:30
vishnukvmd 3ea8fca2f2 Retrieve OrtSession from cached address 2023-12-13 14:55:51 +05:30
vishnukvmd 14acf95a15 Disable embedding push 2023-12-13 14:55:13 +05:30
vishnukvmd 01c82eaf82 Store framework name along with the model name 2023-12-13 14:42:02 +05:30
vishnukvmd 1c0ff2f10c Use the cached address to retrieve the encoder 2023-12-13 14:39:18 +05:30
vishnukvmd b2f9dd2c8b Initialize tokenizer outside the isolate 2023-12-13 14:38:24 +05:30
vishnukvmd f0f4f7f429 Fix logger name 2023-12-13 14:37:36 +05:30
vishnukvmd de3db565f5 Fix asset path 2023-12-13 14:37:26 +05:30
vishnukvmd b59e928a1a Pass final file path to model loader 2023-12-13 12:54:03 +05:30
vishnukvmd dd846d7920 Split ONNX framework into multiple files 2023-12-13 12:46:07 +05:30
vishnukvmd 4ed3720906 Fix compilation 2023-12-13 12:39:08 +05:30
vishnukvmd 4c44fc521c Fix warnings 2023-12-13 12:36:14 +05:30
0nullpointer d7cacce1d8 Add bpePath to CLIPTokenizer 2023-12-13 12:33:28 +05:30
0nullpointer 02114efc2b [WIP] onnx local test 2023-12-13 12:28:04 +05:30
0nullpointer f7859cdb7f [WIP] semantic search with onnx 2023-12-12 21:53:59 +05:30
0nullpointer e2c7cdf9f8 Add initial onnx framework 2023-12-12 21:39:35 +05:30
vishnukvmd 75704f0687 Update logger name 2023-12-05 22:37:56 +05:30
vishnukvmd 708b1c8d23 Refactor ML framework 2023-12-05 22:36:02 +05:30
vishnukvmd 05ba553ebc Define a dummy init method 2023-12-05 21:33:03 +05:30
vishnukvmd ac82f5b282 Create a generic EmbeddingService 2023-12-05 21:31:25 +05:30
vishnukvmd dc16c52ff6 Move score computation to app layer 2023-12-05 21:17:28 +05:30
vishnukvmd a0a5640e61 Add logs 2023-11-18 16:57:25 +05:30
vishnukvmd 1956a589d7 ping-pong 2023-11-17 17:26:02 +05:30
vishnukvmd 6faa408204 Re-enable computation of image embedding 2023-11-17 15:44:16 +05:30
vishnukvmd f54d82be34 Perform no image embedding 2023-11-15 17:35:04 +05:30
vishnukvmd cf3634f707 Ignore spurious requests to backfill 2023-11-15 10:50:48 +05:30
vishnukvmd 179dc0ce89 Minor refactor 2023-11-15 10:47:25 +05:30
vishnukvmd 4c2ec1da50 Wait until model is loaded 2023-11-15 10:42:03 +05:30
vishnukvmd 2e15d953f7 Ignore search if models haven't been loaded 2023-11-15 10:19:05 +05:30
vishnukvmd d64d3c4179 Reflect the correct index status 2023-11-15 02:17:53 +05:30
vishnukvmd df5c776dc8 Simplify state 2023-11-15 02:08:26 +05:30
vishnukvmd 76f3c43ae3 Use a different event to pull from server 2023-11-15 02:03:32 +05:30
vishnukvmd a28219807b Remove extra sync event 2023-11-15 01:36:14 +05:30
vishnukvmd 688b43c75a Listen to changes in the DB and update the cache 2023-11-15 01:16:17 +05:30
vishnukvmd 4740d723af Simplify scoring function 2023-11-15 00:16:49 +05:30
vishnukvmd ce2e034085 Play with an optimized computeScore method 2023-11-14 23:33:10 +05:30
vishnukvmd 6ad7911f1f Update indexing status in real time 2023-11-14 23:32:03 +05:30
vishnukvmd 98b1c1689d Update key 2023-11-14 22:48:03 +05:30
vishnukvmd ef5a5b7c40 Ignore files that have already been queued 2023-11-14 21:01:16 +05:30
vishnukvmd 7c2bf8bd97 Block back fill on remote sync 2023-11-14 20:50:46 +05:30
vishnukvmd 1f17e892c0 Disable search if magic search is disabled 2023-11-14 19:30:06 +05:30
vishnukvmd cd296373d2 Reorder methods 2023-11-14 14:34:42 +05:30
vishnukvmd 34028dc5d7 Reduce visibility of method 2023-11-14 14:33:16 +05:30
vishnukvmd 419467cfac Minor refactor 2023-11-14 14:31:48 +05:30
vishnukvmd e101462eee Decode fetched embeddings on a separate isolate 2023-11-14 14:18:22 +05:30
vishnukvmd 739ef4c21e Refactor code 2023-11-14 13:41:32 +05:30
vishnukvmd ea76fd2ce8 Warm up the text encoder 2023-11-14 13:34:28 +05:30
vishnukvmd b54f082cb6 Move from SQLite to ObjectBox for storing embeddings 2023-11-14 13:26:54 +05:30
vishnukvmd b9cd9cf9cd Fix bug when 0 embeddings are returned 2023-11-14 12:12:58 +05:30
vishnukvmd 289fbc8a11 Move entire score computation to an isolate 2023-11-14 10:14:54 +05:30
vishnukvmd 4d095208f6 Load embeddings in a batch 2023-11-06 19:00:08 +05:30
vishnukvmd 71234dfa7e Cache local embeddings 2023-10-28 17:47:17 +05:30
vishnukvmd e971bf6bc5 Ensure that each entry within an embedding is cast to double 2023-10-28 17:37:24 +05:30
vishnukvmd fb07b954a5 Simplify how low quality results are filtered out 2023-10-28 16:25:06 +05:30
vishnukvmd e6c5bf0bbe Update method name 2023-10-28 16:23:48 +05:30
vishnukvmd 1a01fbd5d9 Index the latest item first 2023-10-28 16:15:42 +05:30
vishnukvmd 4c593e097a Clear the queue when indexing is disabled 2023-10-28 16:15:10 +05:30
vishnukvmd 89e7606d7a Update score threshold 2023-10-27 13:04:31 +05:30
vishnukvmd c0b1667962 Return empty results if magic search hasn't been enabled 2023-10-27 13:04:31 +05:30
vishnukvmd b7e434c5ec Disable semantic search on iOS 2023-10-27 13:04:31 +05:30
vishnukvmd 2827bf238a Ignore if embedding lengths don't match 2023-10-27 13:04:31 +05:30
vishnukvmd e36ec7cc94 Use ChaCha to encrypt embeddings 2023-10-27 13:04:31 +05:30
vishnukvmd 54a78555d8 Deal with JSONified strings instead of FloatArray buffers 2023-10-27 13:04:31 +05:30
vishnukvmd 6acb2951b9 Remove batching 2023-10-27 13:04:31 +05:30
vishnukvmd 5b86080910 Minor refactor 2023-10-27 13:04:30 +05:30
vishnukvmd d25938aeac Pull model from remote 2023-10-27 13:04:30 +05:30
vishnukvmd 3dda5af52a Fix issue with the ordering of fetched embeddings 2023-10-27 13:04:30 +05:30
vishnukvmd 6f251a80a2 Load models in a separate isolate 2023-10-27 13:04:30 +05:30
vishnukvmd 638ff0547e Update dependency on clip_ggml 2023-10-27 13:04:30 +05:30
vishnukvmd 720ac428db Fix attribute name for remote embeddings 2023-10-27 13:04:30 +05:30
vishnukvmd ce1d6c0805 Ignore items not owned by the user 2023-10-27 13:04:30 +05:30
vishnukvmd 53fa9c6830 Load models separately 2023-10-27 13:04:29 +05:30
vishnukvmd a2f66f1e4e Switch to using uploaded IDs 2023-10-27 13:04:29 +05:30
vishnukvmd 4ea80d391b Display index status within the UI 2023-10-27 13:04:29 +05:30
vishnukvmd 804178e03d Fire an event once a files are indexed 2023-10-27 13:04:29 +05:30