This removes the dependency on Redis for core app functionality. Rather
than using the key/value store provided by Redis, Farside now uses a
key/val store provided by [cubdb](https://github.com/lucaong/cubdb) for
identical functionality but without reliance on a non-Elixir service.
This solution was chosen instead of ets, because storing instance data
in memory leads to a period of broken functionality whenever the app
restarts and hasn't re-populated instance data yet. It was also chosen
instead of dets, because the documentation for dets was pretty hard to
understand at first glance.
Tests and the CI build were updated to reflect the removed dependency on
Redis.
New environment variable `FARSIDE_DATA_DIR` can be used to point to a
directory where the instance data can be stored by cubdb.
Co-authored-by: Jason Clark <mithereal@gmail.com>
Connection values (such as redis server port and the port to run farside
on) as well as the services json file to use can now be set via
environment variables:
FARSIDE_PORT sets the port for Farside to run on
FARSIDE_REDIS_PORT sets the redis server port for Farside to use
FARSIDE_SERVICES_JSON sets the services json file for Farside to use
This partially addresses the move towards de-listing Cloudflare
instances by default by allowing different services json files to be
used with different redis servers.
See #43
The dependency took a long time to compile, and was causing problems for
a user who was attempting to build the project.
Since it wasn't a strictly necessary dependency, and `jason` was already
included in the project, all instances of `poison` have been replaced
with `jason`.
The only additional code that this introduced was converting from
generic maps returned by `Jason.decode` into Service structs.
Rather than enforcing a 200 status code, the instance query is deemed a
success if the status code is <400. Various services return 200-399
status codes that don't necessarily indicate an error, but may have to
do with how the instance was configured.
The FARSIDE_NO_ROUTER variable wasn't terribly useful after refactoring
the app to include the update routine internally (rather than available
externally as an elixir script).
Now the only supported environment variable is FARSIDE_TEST, which is
still useful for tests and quick validation of functionality.
Although the current list of queries is only 2 elements ("time" and
"weather"), this allows searches for Whoogle and Searx to be slightly
more random, and potentially avoid any issues with their parent engine
rate limiting them.
* Create nightly update workflow for instances
A nightly GitHub Actions CI workflow has been added to fetch new
instances of supported services within Farside.
Currently only Searx is supported, but obviously others could be added
if there are similarly easy ways to fetch and filter instances
programmatically.
services.json has also been updated with the initial results of the
workflow script.
* Set headers for every HTTPoison request
This serves as a workaround for bot blocking via filtron.
* Expand filtering of searx instances
New filter enforces:
- No Cloudflare
- Good TLS config
- Good HTTP header config
- Vanilla instances or forks
- Instances with 100% search success
Rather than requiring a traditional crontab install, the app now
leverages quantum-core (link below) to schedule the instance update/sync
task every 5 minutes. Some updates as a result:
- The new job is scheduled at runtime in server.ex.
- The update.exs script was refactored to be compiled along with the
rest of the app as instances.ex.
- Scheduler and Server modules were added for creating and executing
the new update task
- All shell scripts were removed, as they are no longer needed
https://github.com/quantum-elixir/quantum-core
