This removes the dependency on Redis for core app functionality. Rather
than using the key/value store provided by Redis, Farside now uses a
key/val store provided by [cubdb](https://github.com/lucaong/cubdb) for
identical functionality but without reliance on a non-Elixir service.
This solution was chosen instead of ets, because storing instance data
in memory leads to a period of broken functionality whenever the app
restarts and hasn't re-populated instance data yet. It was also chosen
instead of dets, because the documentation for dets was pretty hard to
understand at first glance.
Tests and the CI build were updated to reflect the removed dependency on
Redis.
New environment variable `FARSIDE_DATA_DIR` can be used to point to a
directory where the instance data can be stored by cubdb.
Co-authored-by: Jason Clark <mithereal@gmail.com>
Connection values (such as redis server port and the port to run farside
on) as well as the services json file to use can now be set via
environment variables:
FARSIDE_PORT sets the port for Farside to run on
FARSIDE_REDIS_PORT sets the redis server port for Farside to use
FARSIDE_SERVICES_JSON sets the services json file for Farside to use
This partially addresses the move towards de-listing Cloudflare
instances by default by allowing different services json files to be
used with different redis servers.
See #43
This adds a straightforward way of preserving Farside's redirecting
behavior in the user's browser history. That way if an instance becomes
unavailable between the 5 min scans, the user can opt to navigate back
one page and be taken to a new instance.
This is accomplished using a single line of JS, and could potentially
work as the default behavior of Farside (with the current default
behavior requiring a path prefix instead). This should be revisited down
the road when more people are using this service.
Although the current list of queries is only 2 elements ("time" and
"weather"), this allows searches for Whoogle and Searx to be slightly
more random, and potentially avoid any issues with their parent engine
rate limiting them.
* Create nightly update workflow for instances
A nightly GitHub Actions CI workflow has been added to fetch new
instances of supported services within Farside.
Currently only Searx is supported, but obviously others could be added
if there are similarly easy ways to fetch and filter instances
programmatically.
services.json has also been updated with the initial results of the
workflow script.
* Set headers for every HTTPoison request
This serves as a workaround for bot blocking via filtron.
* Expand filtering of searx instances
New filter enforces:
- No Cloudflare
- Good TLS config
- Good HTTP header config
- Vanilla instances or forks
- Instances with 100% search success
Setting the aforementioned env var skips creation of the app router,
which is useful for running update.exs when the main app is already
running (otherwise there's a port conflict).
Introduces a new db key "<service>-previous" to track which instance was
last selected for a particular service. This allows for filtering the
list of available instances to exclude the instance that was last
picked, to ensure a (slightly) more even distribution of traffic.
There's still the possiblity of the following scenario, however:
:service instances > 2
/:service request #1 -> instance #1
/:service request #2 -> instance #2
/:service request #3 -> instance #1
/:service request #4 -> instance #2
where there are many ignored instances for a particular service. One
possible solution would be to implement the "<service>-previous" value
to be a list, rather than a single value, and push to that list until
only one element is left in the original "instance" array after
filtering, and then delete the "<service>-previous" key.
This introduces a number of new changes:
- Services are now inserted into redis with a prefix prepended to the
key name. This allows for easier filtering to get only live instances.
- The home page now uses an eex template for displaying all live
instances for every service, determined by the last update
- A "last_updated" field was added
- farside.ex was added to contain all functionality related to querying
for instances (WIP)
- Other improvements
The name of the project is being refactored from Privacy Revolver to
Farside. The reasoning behind this is:
1. A shorter name is easier to remember
2. It can stand for "FOSS alternative redirecting service" (which I know
doesn't encapsulate all letters from "farside", but it's close enough).
This commit also includes improvements to the update script for
determining how far along the script is.
Not sure if this is the Elixir-y way to do this, but seems more logical
than hardcoding values such as redis connection.
Also went through and improved how string formatting was performed
throughout the app. Rather than "combining" <> "strings" this way, I'm
now just doing "#${variable}#{formatting}", which looks a lot cleaner.
Ben Busby
Ben Busby