I used "git grep -FIn http://" to find all occurrences, and looked at
each one. If an occurrence was really just a link, and if a https
version exists, and if our Browser can access it at least as well as the
http version, then I changed the occurrence to https.
I'm happy to report that I didn't run into a single site where Browser
can't deal with the https version.
This contains all the bits and pieces necessary to build a Clang binary
that will correctly compile SerenityOS.
I had some trouble with getting LLVM building with a single command, so
for now, I decided to build each LLVM component in a separate command
invocation. In the future, we can also make the main llvm build step
architecture-independent, but that would come with extra work to make
library and include paths work.
The binutils build invocation and related boilerplate is duplicated
because we only use `objdump` from GNU binutils in the Clang toolchain,
so most features can be disabled.
Rather than having the toolchain build fail half-way through we should
check whether the user has installed all the required tools and
libraries early on.
Previously the buildstep function would obscure error codes because
the return value of the function was the exit code for the sed command
which caused us to continue execution even though one of the build
steps had failed.
With set -o pipefail the return value of the buildstep function is
the real command's exit code.
This ensures inter-machine compatibility by not emitting any processor
specific instructions. This fixes the issue raised by the non AVX-512
supporting GitHub actions runners.
-march=native specializes the binaries for the CPU features available on
the CPU the binary is being compiled on. This matches the needs of the
Toolchain, as it's always built and used on that machine only.
This should be safe for the github actions VMs as well, as they all run
on a standard VM SKU in "the cloud".
I saw small but notable improvements in end-2-end build times in my
local testing. Each compilation unit is on average around a second
faster on my Intel(R) Core(TM) i7-8705G CPU @ 3.10GHz.
BuildIt.sh had a bunch of SC2086 errors, where we were not quoting
variables in variable expansions. The logic being:
Quoting variables prevents word splitting and glob expansion,
and prevents the script from breaking when input contains spaces,
line feeds, glob characters and such.
Reference: https://github.com/koalaman/shellcheck/wiki/SC2086
As bcoles noticed in #6772, shellcheck actually found a real bug here,
where the user's build directory included spaces.
Close: #6772
Make this stuff a bit easier to maintain by using the
root level variables to build up the Toolchain paths.
Also leave a note for future editors of BuildIt.sh to
give them warning about the other changes they'll need
to make.
This enables building usermode programs with exception handling. It also
builds a libstdc++ without exception support for the kernel.
This is necessary because the libstdc++ that gets built is different
when exceptions are enabled. Using the same library binary would
require extensive stubs for exception-related functionality in the
kernel.
GCC determines whether the system's <limits.h> header is usable
and installs a different version of its own <limits.h> header
depending on whether the system header file exists.
If the system header is missing GCC's <limits.h> header does not
include the system header via #include_next.
For this to work we need to install LibC's headers before
attempting to build GCC.
Also, re-running BuildIt.sh "hides" this problem because at that
point the sysroot directory also already has a <limits.h> header
file from the previous build.
realpath(1) is specific to coreutils and its behavior can be had
with readlink -f
Create the Toolchain Build directory if it doesn't exist before
calling readlink, since realpath(3) on at least OpenBSD will error
on a non-existent path
* Add SERENITY_ARCH option to CMake for selecting the target toolchain
* Port all build scripts but continue to use i686
* Update GitHub Actions cache to include BuildIt.sh
A good number of contributors use macOS. However, we have a bit of
a tendency of breaking the macOS build without realising it.
Luckily, GitHub Actions does actually supply macOS environments,
so let's use it.
This is necessary because cache reusability will be determined by Github Actions.
Note that we only cache if explicitly asked to do so,
which only happens on Github Actions.
When libstdc++ was added in 4977fd22b8, just calling
'make install' was the easiest way to install the headers. And the headers are all
that is needed for libstdc++ to determine the ABI. Since then, BuildIt.sh was
rewritten again and again, and somehow everyone just silently assumed that
libstdc++ also depends on libc.a and libm.a, because surely it does?
Turns out, it doesn't! This massively reduces the dependencies of libstdc++,
hopefully meaning that the Toolchain doesn't need to be rebuilt so often on Travis.
Furthermore, the old method of trying to determine the dependency tree with
bash/grep/etc. has finally broken anyways:
https://travis-ci.com/github/SerenityOS/serenity/builds/179805569#L567
In summary, this should eliminate most of the Toolchain rebuilds on Travis,
and therefore make Travis build blazingly fast! :^)
./configure generates about 3500 lines in a few seconds. Noone will ever read
those lines and they make loading the Travis webpage slower. And if there is
ever a problem, it will be because the Travis base image changed (which happens
only rarely) in a way that interferes with compiling gcc (which is incredibly
unlikely), or we update gcc (which happens very rarely) and gcc doesn't like
the Travis iamge (which again is incredibly unlikely). In all of these cases,
finding the culprit will be self-evident.
Empirically, every single push or PR has to download *and then upload*
about 3.6 GiB of "cache stuff", which takes up about 400 seconds:
https://travis-ci.com/github/SerenityOS/serenity/builds/177500795
On every single push/PR! No matter what!
Those 3.6 GB consist of:
- 3.2 GB Toolchain cache (around 260 MB per compressed item)
- 0.4 GB ccache, but is capped at 0.5 GB: https://travis-ci.com/github/BenWiederhake/serenity/builds/177528549
- (And 200 KB for some weird debian package? Dunno.)
Investigating in the size, the Toolchain consists mostly of *DEBUG SYMBOLS IN
THE COMPILER BINARIES* which comically misses the point. If we ever run into
compiler crashes, any stacktrace would be lost anyway as soon as the Travis VM
shuts down. Furthermore, Travis will only ever compile Serenity itself, and
Serenity forbids C in it's Contribution Guidelines. That's another 20 MB we
don't need to cache.
Stripping the binaries and deleting the C compiler reduces the uncompressed size
from 1200 MB down to 220 MB. The compressed size gets reduced from 260 MB to 70MB.
That's a reduction of 73%.
It'll take a while until the 'old' toolchains get deleted.
I guess it'll take less than a week.
From that point onward, the Travis cache will be 1.2 GB, consisting of:
- 0.7 GB Toolchain cache
- 0.5 GB ccache
- (And that weird 200 KB deb file)
If network speeds are linear, then this should reduce the "cache network
overhead time" from about 400 seconds to about 120 seconds.
tl;dr: Strip unnecessary debug infos, delete an unused files, and speed
everything up by two minutes. (Both Toolchain cache hits and Toolchain rebuilds!)
