Request types don't really matter in Farside, and should be ignored.
This updates the router to handle all request types (GET, HEAD, POST,
etc) the same as GET requests were handled previously.
The previous approach to filtering cloudflare instances was checking if
cloudflare nameservers were assigned to that domain. I believe this was
too harsh of a filter, as quite a few instances were removed even though
they were not using cloudflare's proxying feature.
To solve this, the filtering script has been updated to manually fetch
each IP associated with a domain and see if it returns the 1003 error
that cloudflare proxied sites return when queried directly.
The majority of searx instances returned by
https://searx.space/data/instances.json seem to be running non-release
versions of searx (i.e. versions like "2022.11.06-ae54c7d5" and not
"1.0.0"). Since the version itself doesn't indicate reliability alone
imo, I don't think it's necessary to exclude instances based on this
criteria in the auto-update nightly build.
This removes the dependency on Redis for core app functionality. Rather
than using the key/value store provided by Redis, Farside now uses a
key/val store provided by [cubdb](https://github.com/lucaong/cubdb) for
identical functionality but without reliance on a non-Elixir service.
This solution was chosen instead of ets, because storing instance data
in memory leads to a period of broken functionality whenever the app
restarts and hasn't re-populated instance data yet. It was also chosen
instead of dets, because the documentation for dets was pretty hard to
understand at first glance.
Tests and the CI build were updated to reflect the removed dependency on
Redis.
New environment variable `FARSIDE_DATA_DIR` can be used to point to a
directory where the instance data can be stored by cubdb.
Co-authored-by: Jason Clark <mithereal@gmail.com>
