From the recent changes to twitter/X, it sounds like guest accounts are
now required for nitter, which are more easily rate limited. To avoid
any impact from Farside, the instances are now health checked in the
nightly build using https://status.d420.de (this doesn't seem to be
directly associated with the nitter maintainers, so might not be
entirely future-proof).
FARSIDE_CRON allows turning on/off the scheduled instance availability
check that occurs every 5 minutes by setting the variable to 1 (on) or 0
(off). The default behavior is "on".
Instance URLs and API URLs are not always the same, and require
different testing strategies. This allows the two types of URLs for an
instance, as well as the testing path, to be separated by a "|" character.
When Farside runs the instance checker, it will split instance strings
by that character and perform separate tests. Instances without a
separate API URL will only be tested once against their primary instance
URL.
Currently the only service using this paradigm is Piped, but allows for
others using the same approach to be added/tested in Farside in the
future.
Closes#115
Too many instances redirect to either non-functional or spam sites. The
new behavior is to only allow 200-300 responses to mark an instance as
accessible.
searx.space includes metrics for instance uptime, which is now
implemented as part of farside's nightly build. Accordingly, the
instance availability task built into farside now excludes searxng
instances.
Closes#95
The redirect regex for google search (~r/google.com) would also match
for google translate requests. This updates the search regex to also
require the "/search" path to be present in the original link.
Request types don't really matter in Farside, and should be ignored.
This updates the router to handle all request types (GET, HEAD, POST,
etc) the same as GET requests were handled previously.
This removes the dependency on Redis for core app functionality. Rather
than using the key/value store provided by Redis, Farside now uses a
key/val store provided by [cubdb](https://github.com/lucaong/cubdb) for
identical functionality but without reliance on a non-Elixir service.
This solution was chosen instead of ets, because storing instance data
in memory leads to a period of broken functionality whenever the app
restarts and hasn't re-populated instance data yet. It was also chosen
instead of dets, because the documentation for dets was pretty hard to
understand at first glance.
Tests and the CI build were updated to reflect the removed dependency on
Redis.
New environment variable `FARSIDE_DATA_DIR` can be used to point to a
directory where the instance data can be stored by cubdb.
Co-authored-by: Jason Clark <mithereal@gmail.com>
Connection values (such as redis server port and the port to run farside
on) as well as the services json file to use can now be set via
environment variables:
FARSIDE_PORT sets the port for Farside to run on
FARSIDE_REDIS_PORT sets the redis server port for Farside to use
FARSIDE_SERVICES_JSON sets the services json file for Farside to use
This partially addresses the move towards de-listing Cloudflare
instances by default by allowing different services json files to be
used with different redis servers.
See #43
Bibliogram links require a "/u/" prefix when viewing a user's profile,
which does not match Instagram's URL paradigm. This adds a bit of logic
for Bibliogram links to prepend this prefix if the user is not visiting
a post (indicated by a "/p/" prefix, which Bibliogram and Instagram BOTH
use) or the home page of an instance (indicated by an empty path value).
Farside was previously only looking for links containing "youtube.com"
when performing the redirect for full URLs to youtube, which obviously
doesn't work for youtu.be shortened links. This updates the matching
logic to use a regex element for every key in the services map in order
to match against multiple possible domains for each parent service.
Fixes#40
Query params were not included when using the "/_/" prefix for farside
links. This extracts the logic for parsing query params into a function
that is then used for both /_/ and non-/_/ links.
Farside now supports redirecting based on a provided link to a "parent"
service, if such a parent service is supported.
For example, a link such as:
farside.link/https://www.youtube.com/watch?v=dQw4w9WgXcQ
will now redirect to any of the available YouTube related frontends.
This works by matching against a mapping of "parent" service domains
("youtube.com", "reddit.com", etc) to a list of their respective frontend
alternatives (["invidious", "piped"], ["libreddit", "teddit"], etc). A
random element is chosen from this list, and the remainder of Farside's
routing logic proceeds as if the user had chosen the service directly to
begin with.
Closes#37
The dependency took a long time to compile, and was causing problems for
a user who was attempting to build the project.
Since it wasn't a strictly necessary dependency, and `jason` was already
included in the project, all instances of `poison` have been replaced
with `jason`.
The only additional code that this introduced was converting from
generic maps returned by `Jason.decode` into Service structs.
This adds a straightforward way of preserving Farside's redirecting
behavior in the user's browser history. That way if an instance becomes
unavailable between the 5 min scans, the user can opt to navigate back
one page and be taken to a new instance.
This is accomplished using a single line of JS, and could potentially
work as the default behavior of Farside (with the current default
behavior requiring a path prefix instead). This should be revisited down
the road when more people are using this service.
Rather than enforcing a 200 status code, the instance query is deemed a
success if the status code is <400. Various services return 200-399
status codes that don't necessarily indicate an error, but may have to
do with how the instance was configured.
The FARSIDE_NO_ROUTER variable wasn't terribly useful after refactoring
the app to include the update routine internally (rather than available
externally as an elixir script).
Now the only supported environment variable is FARSIDE_TEST, which is
still useful for tests and quick validation of functionality.
Although the current list of queries is only 2 elements ("time" and
"weather"), this allows searches for Whoogle and Searx to be slightly
more random, and potentially avoid any issues with their parent engine
rate limiting them.
* Create nightly update workflow for instances
A nightly GitHub Actions CI workflow has been added to fetch new
instances of supported services within Farside.
Currently only Searx is supported, but obviously others could be added
if there are similarly easy ways to fetch and filter instances
programmatically.
services.json has also been updated with the initial results of the
workflow script.
* Set headers for every HTTPoison request
This serves as a workaround for bot blocking via filtron.
* Expand filtering of searx instances
New filter enforces:
- No Cloudflare
- Good TLS config
- Good HTTP header config
- Vanilla instances or forks
- Instances with 100% search success
Rather than requiring a traditional crontab install, the app now
leverages quantum-core (link below) to schedule the instance update/sync
task every 5 minutes. Some updates as a result:
- The new job is scheduled at runtime in server.ex.
- The update.exs script was refactored to be compiled along with the
rest of the app as instances.ex.
- Scheduler and Server modules were added for creating and executing
the new update task
- All shell scripts were removed, as they are no longer needed
https://github.com/quantum-elixir/quantum-core
Setting the aforementioned env var skips creation of the app router,
which is useful for running update.exs when the main app is already
running (otherwise there's a port conflict).
Query params ("/watch?v=dQw4w9WgXcQ" for instance) would previously be
lost in Farside redirects. This now includes them if they were included
in the original request.
This introduces a way of throttling requests in a way that makes sense
for the purpose of the app. The app only supports redirecting to one
particular service when browsing, which would seldom be required more
than once per second for normal "human" browsing.
Without this, the service could easily be used to DOS multiple instances
at once. That being said, anyone concerned about someone DOS-ing
multiple instances at once should be aware that this would be trivial to
do with a simple bash script. This is simply a preventative measure to
hopefully deter people from trying to attack all public instances of
private frontends using farside.link.
Note that this throttling applies to all routes in the app, including
the homepage. This could be updated to exclude the homepage I guess,
but I'm not really sure what the use case would be for that.
Introduces a new db key "<service>-previous" to track which instance was
last selected for a particular service. This allows for filtering the
list of available instances to exclude the instance that was last
picked, to ensure a (slightly) more even distribution of traffic.
There's still the possiblity of the following scenario, however:
:service instances > 2
/:service request #1 -> instance #1
/:service request #2 -> instance #2
/:service request #3 -> instance #1
/:service request #4 -> instance #2
where there are many ignored instances for a particular service. One
possible solution would be to implement the "<service>-previous" value
to be a list, rather than a single value, and push to that list until
only one element is left in the original "instance" array after
filtering, and then delete the "<service>-previous" key.
The process of selecting a random (working) instance for a specified
service has been moved out of the router and into lib/farside.ex. Moving
forward, the router itself should have very simple and easy to follow
logic for all paths.
Service struct now defined in lib/service.ex
This makes a bit more sense now that its a shared resource, rather than
just defining it only in the update.exs script.
This introduces a number of new changes:
- Services are now inserted into redis with a prefix prepended to the
key name. This allows for easier filtering to get only live instances.
- The home page now uses an eex template for displaying all live
instances for every service, determined by the last update
- A "last_updated" field was added
- farside.ex was added to contain all functionality related to querying
for instances (WIP)
- Other improvements
The name of the project is being refactored from Privacy Revolver to
Farside. The reasoning behind this is:
1. A shorter name is easier to remember
2. It can stand for "FOSS alternative redirecting service" (which I know
doesn't encapsulate all letters from "farside", but it's close enough).
This commit also includes improvements to the update script for
determining how far along the script is.
Not sure if this is the Elixir-y way to do this, but seems more logical
than hardcoding values such as redis connection.
Also went through and improved how string formatting was performed
throughout the app. Rather than "combining" <> "strings" this way, I'm
now just doing "#${variable}#{formatting}", which looks a lot cleaner.
Router has been updated with a basic /:service/*glob endpoint, which
retrieves a list of instances for the requested service and picks a
random one to forward the user to (or falls back to a default instance
if none are found).
Should probably add a check to make sure the requested service exists
first.
Rather than use a full blown framework*, adding basic routing with
Plug.Router seems to make more sense, since I'm not planning on hosting
any content through this app. The app itself will just be endpoints for
all available services that redirect the user to an available instance
for the requested service.
Note that I might change my mind about this, but that's unlikely. At
most there would just be a home page with info about available
instances, but even then that seems kinda pointless. Trying to keep this
as absolutely simple as possible.
*like Phoenix
Ben Busby
NoPlagiarism
FriendlyPigeon
Kian-Meng Ang
alvv-z
Ben Busby