Rather than invoking AK::Time::from_timestamp at runtime, we can do so
at compile time. This reduces invoking TimeZone::get_time_zone_offset
100,000 times in a loop from about 7 seconds to 30 milliseconds.
We compute the effective overload sets for each argument count at build
time, to save us having to do it every time a function with overloads
is called.
As part of this, I've moved a couple of methods for checking for
null/undefined from UnionType to Type, and filled in more of their
steps.
This now detects more, and so causes us to hit a `TODO()` which is too
big for me to go after right now, so I've replaced that assertion with
a log message.
Track the kind of Type it is, and use that to provide some convenient
`is_foo()` / `as_foo()` methods. While I was at it, made these all
classes instead of structs and made their data private.
IDL function overload resolution requires knowing each IDL function's
parameters and their types at runtime. The simplest way to do that is
just to make the types the generator uses available to the runtime.
Parsing has moved to LibIDL, but code generation has not, since that is
very specific to WrapperGenerator.
This script is useful when wanting to install lagom libraries for
projects using Lagom via FetchContent, but trips over itself if the
project links other non-Lagom imported targets to itself. So, let's just
skip them.
This mainly changes two aspects:
- The category can now be a single letter, such as 'w' to indicate the
file Utilities/w.cpp
- Spaces in the category (or list) are no longer allowed. This follows
the lived practice of writing category lists as "Foo+Bar: Quux"
Closes#15243.
This was apparently never used by anyone except me, and currently
fails silently.
The script originally allowed easy inspection of potentially missing
resources, but that seems no longer useful. Even after restoring the
script to a working state, I found nothing with it.
A somewhat usable version might be available at
https://github.com/BenWiederhake/serenity/tree/historic/lint-missing-resources.sh
However, there seems to be no interest in the script, so it is better to
remove it.
This was apparently never used by anyone except me, and currently
fails silently.
The script originally allowed easy inspection of the difference between:
1. The list of declared syscalls according to Kernel/API/Syscall.h
2. The list of syscalls implemented by the UserspaceEmulator according
to Userland/DevTools/UserspaceEmulator/Emulator_syscalls.cpp
3. The list of syscalls documented in Base/usr/share/man/man2/.
Here's how the script could have been updated:
SYSCALLS_KERNEL="$(echo 'Kernel syscalls'; echo; \
grep -Eo '^ +S\(.*, NeedsBigProcessLock::' Kernel/API/Syscall.h | \
sed -Ee 's-^ +S\((.+), .*-\1-' | sort)"
SYSCALLS_UE="$(echo 'Implemented in UserspaceEmulator'; echo; \
grep -Eo '^ +case SC_.*:$' \
Userland/DevTools/UserspaceEmulator/Emulator_syscalls.cpp | \
sed -Ee 's,^ +case SC_(.*):$,\1,' | sort)"
SYSCALLS_MAN2="$(echo 'Documented syscalls'; echo; \
find Base/usr/share/man/man2/ ! -type d -printf '%f\n' | \
sed -Ee 's,\.md,,' | sort)"
diff --color=always \
<(echo "${SYSCALLS_KERNEL}") <(echo "${SYSCALLS_UE}")
diff --color=always \
<(echo "${SYSCALLS_KERNEL}") <(echo "${SYSCALLS_MAN2}")
A more readable version might be available at
https://github.com/BenWiederhake/serenity/tree/historic/syscall-linting
However, there seems to be no interest in the script, so it is better to
remove it.
basename from GNU coreutils 8.32 (the default on Debian Bullseye, as
bash 5.2.0 does not include basename as a built-in command) does not
accept multiple arguments. This will cause the command to fail with no
output, and the error code not being propagated for some reason. This
means that the loop never gets executed, and thus the check never
actually does anything. This commit fixes that behavior by calling
'basename' multiple times.
Parse emoji from emoji-serenity.txt to allow displaying their names and
grouping them together in the EmojiInputDialog.
This also adds an "Unknown" value to the EmojiGroup enum. This will be
useful for emoji that aren't found in the UCD, or for when UCD downloads
are disabled.
This allows us to find emoji data for files such as /res/emoji/U+A9.png.
U+00A9 is not fully-qualified (its full form is U+00A9 U+FE0F). But the
UCD has unqualified data for this code point; generating it allows us to
categorize these emoji appropriately in the EmojiInputDialog.
For now this is a lagom only application as it is not compatible with
serenity in its current state.
The only change is that it is released under a different license with
permission from all the authors.
Newer cmake's have internal functions to un-compress files. These
functions will work on pure windows - as well as linux. This
eliminates the need to search for external tools (TAR,GZIP,ZIP) - and
helps fixing #9866.
In order to finally fix#9866 we need to decide to bump the cmake
version requirements and remove the checks. If we demand a newer cmake
version, we will loose Ubuntu 20.04 as a build target - as it ships
with CMake 3.16.
For now - we keep compatibility with CMake 3.16 - and only if CMake
3.18 as been found - we use its new functionality.
Remove the Corrosion dependency, and use the now-builtin
add_jakt_executable function from the Jakt install rules to build our
example application.
By using find_package(Jakt), we now have to set ENABLE_JAKT manually on
both serenity and Lagom at the same time, so the preferred method to do
this for now is:
cmake -B Build/superbuild<arch><toolchain> \
-S Meta/CMake/Superbuild \
-DENABLE_JAKT=ON \
-DJAKT_SOURCE_DIR=/path/to/jakt
Where omitting JAKT_SOURCE_DIR will still pull from the main branch of
SerenityOS/jakt. This can be done after runing Meta/serenity.sh run.
According to TR #51, the "best definition of the full set [of emojis] is
in the emoji-test.txt file". This defines not only the emoji themselves,
but the order in which they should be displayed, and what "group" of
emojis they belong to.
There are still some remaining cases where generated code depends on the
existence of FooWrapper => Web::NS::Foo mappings. Fixing those will
require figuring out the appropriate namespace for all IDL types, not
just the currently parsed interface.
Unlike ensure_web_prototype<T>(), the cached version doesn't require the
prototype type to be fully formed, so we can use it without including
the FooPrototype.h header. It's also a bit less verbose. :^)
This is a monster patch that turns all EventTargets into GC-allocated
PlatformObjects. Their C++ wrapper classes are removed, and the LibJS
garbage collector is now responsible for their lifetimes.
There's a fair amount of hacks and band-aids in this patch, and we'll
have a lot of cleanup to do after this.
This patch moves the following things to being GC-allocated:
- Bindings::CallbackType
- HTML::EventHandler
- DOM::IDLEventListener
- DOM::DOMEventListener
- DOM::NodeFilter
Note that we only use PlatformObject for things that might be exposed
to web content. Anything that is only used internally inherits directly
from JS::Cell instead, making them a bit more lightweight.
This tells the wrapper generator that there is no separate wrapper class
for this interface, and it should refer directly to the C++ "Foo" object
instead of "FooWrapper".
To enable incremental movement towards the removal of DOM object
instance wrappers, this patch adds a NO_INSTANCE argument that can be
passed to libweb_js_wrapper().
The UCD only cares about a few locales for special casing rules (az, lt,
and tr). Unfortunately, LibUnicode cannot use LibLocale once the
libraries are separate because LibLocale will need to use LibUnicode for
many more things; thus there would be a circular dependency. Instead,
just generate the small enum needed for this one use case.
When LibLocale is placed in the Locale namespace, this will conflict
with the Locale structure in each CLDR generator. Rename this to
"LocaleData", and rename its parent UnicodeLocaleData to just "CLDR"
to avoid confusion between LocaleData and UnicodeLocaleData.
Currently, LibUnicodeData contains the generated UCD and CLDR data. Move
the UCD data to the main LibUnicode library, and rename LibUnicodeData
to LibLocaleData. This is another prepatory change to migrate to
LibLocale.
To prepare for placing all CLDR generated data in a new library,
LibLocale, this moves the code generators for the CLDR data to the
LibLocale subfolder.
The FLAC "spec tests", or rather the test suite by xiph that exercises
weird FLAC features and edge cases, can be found at
https://github.com/ietf-wg-cellar/flac-test-files and is a good
challenge for our FLAC decoder to become more spec compliant. Running
these tests is similar to LibWasm spec tests, you need to pass
INCLUDE_FLAC_SPEC_TESTS to CMake.
As of integrating these tests, 23 out of 63 fail. :yakplus:
We don't need this AHCI controller to be present as we already have one
in the Q35 machine. This will help using the correct boot device in GRUB
setups later on.
The current lookup code and emoji.txt generator expects codepoints to
not include leading zeros. This may change in the future, but it's worth
enforcing the current convention until then.
Since LibUnicode depends on this data it used to include
Intl/AbstractOperations which in turn includes a number of other LibJS
headers. By moving this to its own header with minimal includes we can
save on rebuilding LibUnicode for unrelated LibJS header changes.
Intrinsics, i.e. mostly constructor and prototype objects, but also
things like empty and new object shape now live on a new heap-allocated
JS::Intrinsics object, thus completing the long journey of taking all
the magic away from the global object.
This represents the Realm's [[Intrinsics]] slot in the spec and matches
its existing [[GlobalObject]] / [[GlobalEnv]] slots in terms of
architecture.
In the majority of cases it should now be possibly to fully allocate a
regular object without the global object existing, and in fact that's
what we do now - the realm is allocated before the global object, and
the intrinsics between both :^)
The primary motivation for this is to make `generate-emoji-txt.sh` more
useful for generating a compact list of new emoji being added (e.g.
for use in commit messages / PRs) if it's run with an emoji image
directory that contains only the new emojis.
It currently takes upwards of 40 minutes to download the ccache on macOS
and often errors-out near the end. Change the cache version to bust it
so we can start anew. Reduce its max size to 2 GB (a clean build is ~0.9
GB, so this allows just over 2 clean builds to be cached).
- Prefer VM::current_realm() over GlobalObject::associated_realm()
- Prefer VM::heap() over GlobalObject::heap()
- Prefer Cell::vm() over Cell::global_object()
- Prefer Wrapper::vm() over Wrapper::global_object()
- Inline Realm::global_object() calls used to access intrinsics as they
will later perform a direct lookup without going through the global
object
This is needed so that the allocated NativeFunction receives the correct
realm, usually forwarded from the Object's initialize() function, rather
than using the current realm.
Global object initialization is tightly coupled to realm creation, so
simply pass it to the function instead of relying on the non-standard
'associated realm' concept, which I'd like to remove later.
This works essentially the same way as regular Object::initialize() now.
Additionally this allows us to forward the realm to GlobalObject's
add_constructor() / initialize_constructor() helpers, so they set the
correct realm on the allocated constructor function object.
Similar to create() in LibJS, wrap() et al. are on a low enough level to
warrant passing a Realm directly instead of relying on the current realm
from the VM, as a wrapper may need to be allocated while no JS is being
executed.
This is a continuation of the previous six commits.
The global object is only needed to return it if the execution context
stack is empty, but that doesn't seem like a useful thing to allow in
the first place - if you're not currently executing JS, and the
execution context stack is empty, there is no this value to retrieve.
This is a continuation of the previous five commits.
A first big step into the direction of no longer having to pass a realm
(or currently, a global object) trough layers upon layers of AOs!
Unlike the create() APIs we can safely assume that this is only ever
called when a running execution context and therefore current realm
exists. If not, you can always manually allocate the Error and put it in
a Completion :^)
In the spec, throw exceptions implicitly use the current realm's
intrinsics as well: https://tc39.es/ecma262/#sec-throw-an-exception
This is a continuation of the previous three commits.
Now that create() receives the allocating realm, we can simply forward
that to allocate(), which accounts for the majority of these changes.
Additionally, we can get rid of the realm_from_global_object() in one
place, with one more remaining in VM::throw_completion().
This is a continuation of the previous two commits.
As allocating a JS cell already primarily involves a realm instead of a
global object, and we'll need to pass one to the allocate() function
itself eventually (it's bridged via the global object right now), the
create() functions need to receive a realm as well.
The plan is for this to be the highest-level function that actually
receives a realm and passes it around, AOs on an even higher level will
use the "current realm" concept via VM::current_realm() as that's what
the spec assumes; passing around realms (or global objects, for that
matter) on higher AO levels is pointless and unlike for allocating
individual objects, which may happen outside of regular JS execution, we
don't need control over the specific realm that is being used there.
This is a continuation of the previous commit.
Calling initialize() is the first thing that's done after allocating a
cell on the JS heap - and in the common case of allocating an object,
that's where properties are assigned and intrinsics occasionally
accessed.
Since those are supposed to live on the realm eventually, this is
another step into that direction.
No functional changes - we can still very easily get to the global
object via `Realm::global_object()`. This is in preparation of moving
the intrinsics to the realm and no longer having to pass a global
object when allocating any object.
In a few (now, and many more in subsequent commits) places we get a
realm using `GlobalObject::associated_realm()`, this is intended to be
temporary. For example, create() functions will later receive the same
treatment and are passed a realm instead of a global object.
The recently added generate-emoji-txt.sh script uses a Bash 4
substitution feature, causing CI to fail as macOS ships an ancient Bash
3.x. We'll want to use a more recent version anyway, so let's do that
instead of updading the script to older syntax.
The current emoji_txt.cmake does not handle download errors (which were
a common source of issues in the build problems channel) or Unicode
versioning. These are both handled by unicode_data.cmake. Move the
download to unicode_data.cmake so that we can more easily handle next
month's Unicode 15 release.
Instead of manually updating emoji.txt whenever new emoji are added,
we use Unicode's emoji-test.txt to generate emoji.txt on each build,
including only the emojis that Serenity supports at that time.
By using emoji-test.txt, we can also include all forms of each emoji
(fully-qualified, minimally-qualified, and unqualified) which can be
helpful when double-checking how certain forms are handled.
Verifies that emoji filenames:
- Contain only uppercase letters, numbers, +, and _
- Use _ and a separator between codepoints, not +
- Do not include the U+FE0F emoji presentation specifier
Similar to commit becec35, our code point display name data was a large
list of StringViews. RLE can be used here as well to remove about 32 MB
from the initialized data section to the read-only section.
Some of the refactoring to store strings as indices into an RLE array
also lets us clean up some of the code point name generators.
Currently, the unique string lists are stored in the initialized data
sections of their shared libraries. In order to move the data to the
read-only section, generate the strings using RLE arrays.
We generate two arrays: the first is the RLE data itself, the second is
a list of indices into the RLE array for each string. We then generate a
decoding method to convert an RLE string to a StringView.
We are downloading these directly into the build directory now, and
generating the source code from there, so we no longer need the
manually created directory.
While we are at it, remove two variables that seem to be no longer in
use, and at least one of which is confusing regarding a missing prefix.
This cache was disabled in 3127454 because it wasn't needed and there
was a race between the builders for this cache. Then commit 0c95d99
started fuzzing the generated Unicode / TZDB data. Since then, we've
been pulling this data from the live servers instead of Azure's cache.
We do a similar trick for the compiler cache. This allows each builder
to separately push their local data cache (if it changed) while pulling
a shared cache, without the race outlined in commit 3127454. This is
needed for a subsequent commit which will enable this cache for Fuzzer
builds.
This isn't called out in TR-35, but before ICU even looks at CLDR data,
it adds a hard-coded set of default patterns to each locale's calendar.
It has done this since 2006 when its DateTimeFormat feature was first
created. Several test262 tests depend on this, which under ECMA-402,
falls into "implementation defined" behavior. For compatibility, we
can do the same in LibUnicode.
In the generated unique string list, index 0 is the empty string, and is
used to indicate a value doesn't exist in the CLDR. Check for this
before returning an empty calendar symbol.
For example, an upcoming commit will add the fixed day period "noon",
which not all locales support.
Commit ec7d535 only partially handled the case of flexible day periods
rolling over midnight, in that it only worked for hours after midnight.
For example, the en locale defines a day period range of [21:00, 06:00).
The previous method of adding 24 hours to the given hour would change
e.g. 23:00 to 47:00, which isn't valid.
This abstraction layer is mainly for ATA ports (AHCI ports, IDE ports).
The goal is to create a convenient and flexible framework so it's
possible to expand to support other types of controller (e.g. Intel PIIX
and ICH IDE controllers) and to abstract operations that are possible on
each component.
Currently only the ATA IDE code is affected by this, making it much
cleaner and readable - the ATA bus mastering code is moved to the
ATAPort code so more implementations in the near future can take
advantage of such functionality easily.
In addition to that, the hierarchy of the ATA IDE code resembles more of
the SATA AHCI code now, which means the IDEChannel class is solely
responsible for getting interrupts, passing them for further processing
in the ATAPort code to take care of the rest of the handling logic.
This is a start to properly letting us cross-compile Lagom where both
the Tools and the BUILD_LAGOM=ON build are using Lagom CMakeLists.
The initial cut allows an Android build to succeed, more or less.
But there are issues with namespace clashes when using FetchContent with
this approach.
When patterns, grouping digits, symbols, etc. for a requested numbering
system are not found, use the locale's default numbering system. This
will allow using the correct digits e.g. for the locale "en-u-nu-arab"
even though the "en" locale only contains patterns for the "latn"
numbering system.
Replacement conditions for `requires_argument` have been chosen based
on what would be most convenient for implementing an eventual optional
argument mode.
While null StringViews are just as bad, these prevent the removal of
StringView(char const*) as that constructor accepts a nullptr.
No functional changes.
This prevents us from needing a sv suffix, and potentially reduces the
need to run generic code for a single character (as contains,
starts_with, ends_with etc. for a char will be just a length and
equality check).
No functional changes.
Each of these strings would previously rely on StringView's char const*
constructor overload, which would call __builtin_strlen on the string.
Since we now have operator ""sv, we can replace these with much simpler
versions. This opens the door to being able to remove
StringView(char const*).
No functional changes.
Error::from_string_literal now takes direct char const*s, while
Error::from_string_view does what Error::from_string_literal used to do:
taking StringViews. This change will remove the need to insert `sv`
after error strings when returning string literal errors once
StringView(char const*) is removed.
No functional changes.
This commit moves the length calculations out to be directly on the
StringView users. This is an important step towards the goal of removing
StringView(char const*), as it moves the responsibility of calculating
the size of the string to the user of the StringView (which will prevent
naive uses causing OOB access).
These are mostly minor mistakes I've encountered while working on the
removal of StringView(char const*). The usage of builder.put_string over
Format<FormatString>::format is preferrable as it will avoid the
indirection altogether when there's no formatting to be done. Similarly,
there is no need to do format(builder, "{}", number) when
builder.put_u64(number) works equally well.
Additionally a few Strings where only constant strings were used are
replaced with StringViews.
The generate-manpages script needs to be updated again to handle the new
PNGs in section 1. (I'm intentionally not making this a multi-directory
glob.)
The reason empty string was treated as JSON null was to paper over an
issue where UTMP would start out as the empty string and presumably
cause errors when trying to parse it as JSON. This was added in
commit a409b832.
This changes that by making UTMP start out as an empty JSON object
instead of the empty string.
Homebrew does not add upstream LLVM's install location to $PATH so as
not to conflict with XCode tools, so we should look for it by its
absolute path. LLVM is installed to /opt/homebrew/opt/llvm on ARM Macs,
and is a symlink that points to the latest stable LLVM version.
To prepare for using plural rules within number & duration format, this
removes the NumberFormat::Plurality enumeration.
This also adds PluralCategory::ExactlyZero & PluralCategory::ExactlyOne.
These are used in locales like French, where PluralCategory::One really
means any value from 0.00 to 1.99. PluralCategory::ExactlyOne means only
the value 1, as the name implies. These exact rules are not known by the
general plural rules, they are explicitly for number / currency format.
The PluralCategory enum is currently generated for plural rules. Instead
of generating it, this moves the enum to the public LibUnicode header.
While it was nice to auto-discover these values, they are well defined
by TR-35, and we will need their values from within the number format
code generator (which can't rely on the plural rules generator having
run yet). Further, number format will require additional values in the
enum that plural rules doesn't know about.
This patch adds support for URLSearchParams to XHR::send() and
introduces the union type XMLHttpRequestBodyInit.
XHR::send() now has support for String and URLSearchParams.
Parts of our build system and scripts rely on the fact that we are
cross-compiling. For now, remove the "try to build natively" part to get
the build running and leave a TODO for later.
Plural rules in the CLDR are of the form:
"cs": {
"pluralRule-count-one": "i = 1 and v = 0 @integer 1",
"pluralRule-count-few": "i = 2..4 and v = 0 @integer 2~4",
"pluralRule-count-many": "v != 0 @decimal 0.0~1.5, 10.0, 100.0 ...",
"pluralRule-count-other": "@integer 0, 5~19, 100, 1000, 10000 ..."
}
The syntax is described here:
https://unicode.org/reports/tr35/tr35-numbers.html#Plural_rules_syntax
There are up to 2 sets of rules for each locale, a cardinal set and an
ordinal set. The approach here is to generate a C++ function for each
set of rules. Each condition in the rules (e.g. "i = 1 and v = 0") is
transpiled to a C++ if-statement within its function. Then lookup tables
are generated to match locales to their generated functions.
NOTE: -Wno-parentheses-equality is added to the LibUnicodeData compile
flags because the generated plural rules have lots of extra parentheses
(because e.g. we need to selectively negate and combine rules). The code
to generate only exactly the right number of parentheses is quite hairy,
so this just tells the compiler to ignore the extras.
This includes:
* The minimum number of days in a week for that week to count as the
first week of a new year.
* The day to be shown as the first day of the week in a calendar.
* The start/end days of the weekend.
Like the existing hour cycle data, week data is presented per-region in
the CLDR, rather than per-locale. The method to add likely subtags to a
locale to perform region lookups is the same.
The list of regions in the CLDR for hour cycle, minimum days, first day,
and weekend days are quite different. So rather than changing the
existing HourCycleRegion enum to a generic Region enum, we generate
separate enums for each of the week data fields. This allows each lookup
into these fields to remain simple array-based index access, without any
"jumps" for regions that don't have CLDR data for a field.
Currently contains just each locale's character order, but is set up to
easily add other text layout fields from the CLDR if ECMA-402 eventually
requires them.
The zone1970.tab file in the TZDB contains regional time zone data, some
of which we already parse for the system time zone settings map.
This parses the region names from that file and generates a list of time
zones which are used in each of those regions.
Add overrides for serenity_bin and serenity_lib to allow the actual
CMakeLists.txt from Userland to be used to build as many services as
possible without adding more clutter to Meta/Lagom/CMakeLists.txt
This matches the target names for the main serenity build, and will make
simplifying the Lagom build much easier going forward.
The LagomFoo name came from a time when we had both library builds in
the same CMake generated project and needed to deconflict the names.
In preparation for future refactoring of Lagom, let's use the variables
from GNUInstallDirs as much as possible for the helper macros and other
scripts used by the main build already.
This commit has no behavior changes.
In particular, this does not fix any of the wrong uses of the previous
default parameter (which used to be 'false', meaning "only replace the
first occurence in the string"). It simply replaces the default uses by
String::replace(..., ReplaceMode::FirstOnly), leaving them incorrect.
The main thing that is missing is validating certain pieces of data
against XML productions in well-formed mode, but nothing uses
well-formed mode right now.
Required by Closure Library for sanitising HTML.
e687b3d8ab/closure/goog/html/sanitizer/safedomtreeprocessor.js (L117)
The Mach-O file format does not have ELF's interposition rules, so this
flag does not make sense for macOS builds. While GCC silently accepts
the unsupported option, Clang issues a warning for it.
This commit makes it possible to build Lagom with LLVM from Homebrew.
I saw one site relying on this, where they are trying to set
XHR.responseType to "text/plain", which is not a valid responseType.
However, they also don't expect it to throw. The IDL spec special cases
enumerations to make it return instead of throwing in this case.
This uses optipng to check how much size can be reduced on PNG files. If
that's more than 2 KiB for at least one file, the check fails. As with
other checks, it doesn't run if optipng is not installed.
Previously we only threw an error if the enum was used as a function
argument. However, we are supposed to throw an error no matter the
context it is used in.
