This is related to the previous commit, where instances can now specify
a separate API url. Since users aren't visiting the API urls directly,
however, they shouldn't be included in the cf proxy test.
The previous approach to filtering cloudflare instances was checking if
cloudflare nameservers were assigned to that domain. I believe this was
too harsh of a filter, as quite a few instances were removed even though
they were not using cloudflare's proxying feature.
To solve this, the filtering script has been updated to manually fetch
each IP associated with a domain and see if it returns the 1003 error
that cloudflare proxied sites return when queried directly.
The cloudflare filter, when performed as part of the github action
workflow, doesn't seem to work nearly as reliably when run on an actual
machine.
The farside server will instead run the un-cloudflare script whenever it
pulls in new changes to services-full.json, which should be a much more
reliable approach to filtering out cloudflare instances.
If dig returns exit code 9 (no reply from server) when checking an
instance for cloudflare records, it shouldn't fail the CI build but
rather just skip adding the instance to the non-cloudflare services
list.
This should be re-evaluated soon to see if the CI build routinely has
issues with getting a server reply. If so, a different approach needs to
be taken to check if an instance is using cloudflare.
This new dir will probably stay fairly empty, but this script could come
in handy for anyone seeking to add a new instance (for services that
aren't supported by the auto update workflow).
Ben Busby