Commit graph

998 commits

Author SHA1 Message Date
Timothy Flynn ff66218631 LibWeb: Allow creating "any" types in IDL with integral default values
This enables defining "any" types in IDL files such as:

    any threshold = 0;

This isn't able to parse decimal values yet.
2021-10-14 10:32:51 +02:00
Timothy Flynn f91d63af83 LibUnicode: Generate enum/alias from-string methods without a HashMap
The *_from_string() and resolve_*_alias() generated methods are the last
remaining users of HashMap in the LibUnicode generated files (read: the
last methods not using compile-time structures). This converts these
methods to use an array containing pairs of hash values to the desired
lookup value.

Because this code generation is the same between GenerateUnicodeData.cpp
and GenerateUnicodeLocale.cpp, this adds a GeneratorUtil.h header to the
LibUnicode generators to contain the method that generates the methods.
2021-10-13 16:38:51 +02:00
Linus Groh 52976bfac6 LibJS: Convert to_object() to ThrowCompletionOr 2021-10-13 09:55:10 +01:00
Linus Groh 4d8912a92b LibJS: Convert to_string() to ThrowCompletionOr
Also update get_function_name() to use ThrowCompletionOr, but this is
not a standard AO and should be refactored out of existence eventually.
2021-10-13 09:55:10 +01:00
Linus Groh 44e70d1bc0 LibJS+LibWeb: Let WrapperGenerator deal with legacy_null_to_empty_string
This concept is not present in ECMAScript, and it bothers me every time
I see it.
It's only used by WrapperGenerator, and even there only relevant in two
places, so let's fully remove it from LibJS and use a simple ternary
expression instead:

    cpp_name = js_name.is_null() && legacy_null_to_empty_string
        ? String::empty()
        : js_name.to_string(global_object);
2021-10-11 23:36:03 +01:00
Linus Groh 661dd32432 LibWeb: Add support for the Promise<T> IDL type to WrapperGenerator
This includes parsing parameterized types (foo<T>) as well as generating
the appropriate code in generate_wrap_statement() and generate_to_cpp().
2021-10-11 13:30:17 +01:00
Linus Groh 7afd215e95 LibWeb: Initialize IDL any values without default value to undefined
Previously this would generate the following code:

    JS::Value foo_value;
    if (!foo.is_undefined())
        foo_value = foo;

Which is dangerous as we're passing an empty value around, which could
be exposed to user code again. This is fine with "= null", for which it
also generates:

    else
        foo_value = JS::js_null();

So, in summary: a value of type `any`, not `required`, with no default
value and no initializer from user code will now default to undefined
instead of an empty value.
2021-10-11 13:30:17 +01:00
Linus Groh a9a7d65099 LibWeb: Replace heycam.github.io/webidl URLs with webidl.spec.whatwg.org
Web IDL is now a WHATWG standard and the specification was moved
accordingly: https://twitter.com/annevk/status/1445311275026821120

The old URLs now redirect, but let's use canonical ones.
2021-10-11 13:15:16 +01:00
Andreas Kling fdc1c15064 LibWeb: Stub out a basic ResizeObserver interface
This patch establishes scaffolding for the ResizeObserver API.
2021-10-11 00:54:01 +02:00
Andreas Kling 5c9ca5c2dc LibWeb: Stub out a basic Selection interface
This patch establishes scaffolding for the Selection API.
2021-10-11 00:32:19 +02:00
Ben Wiederhake e900f94a03 Meta: Run check-markdown as part of 'Azure Linux NoFuzz' 2021-10-10 15:18:55 -07:00
Ben Wiederhake a296f6bb8d Meta: Invoke markdown-checker if available while linting 2021-10-10 15:18:55 -07:00
Ben Wiederhake c06a0bae04 Meta: Fix broken external links
Meta/Lagom/ReadMe.md never had any other name; not sure how that typo
happened.

The link to the non-existent directory is especially vexing because the
text goes on to explain that we don't want such a directory to exist.

Found by running markdown-checker, and 'wget'ing all external links.
2021-10-10 15:18:55 -07:00
Ben Wiederhake 3f88d65b78 markdown-checker: New tool that checks document links 2021-10-10 15:18:55 -07:00
Timothy Flynn 597379e864 LibUnicode: Generate and use unique locale-related alias strings
Almost all of these are already in the unique string list.
2021-10-10 22:21:48 +02:00
Timothy Flynn acb7bd917f LibUnicode: Generate and use unique subtag and complex alias strings 2021-10-10 22:21:48 +02:00
Timothy Flynn 3d67f6bd29 LibUnicode: Generate and use unique list-format strings
The list-format strings used for Intl.ListFormat are small, but quite
heavily duplicated. For example, the string "{0}, {1}" appears 6,519
times. Generate unique strings for this data to avoid duplication.
2021-10-10 22:21:48 +02:00
Timothy Flynn f9e605397c LibUnicode: Generate and use a set of unique locale-related strings
In the generated UnicodeLocale.cpp file, there are 296,408 strings for
localizations of languages, territories, scripts, currencies & keywords.
Of these, only 43,848 (14.8%) are actually unique, so there are quite a
large number of duplicated strings.

This generates a single compile-time array to store these strings. The
arrays for the localizations now store an index into this single array
rather than duplicating any strings.
2021-10-10 22:21:48 +02:00
Timothy Flynn 3f0095b57a LibUnicode: Skip unknown languages and territories
Some CLDR languages.json / territories.json files contain localizations
for some lanuages/territories that are otherwise not present in the CLDR
database. We already don't generate anything in UnicodeLocale.cpp for
these anomalies, but this will stop us from even storing that data in
the generator's memory.

This doesn't affect the output of the generator, but will have an effect
after an upcoming commit to unique-ify all of the strings in the CLDR.
2021-10-10 22:21:48 +02:00
Ben Wiederhake 6d99b7b72e Meta: Re-enable warnings for deprecated copies also for Lagom 2021-10-10 21:21:35 +01:00
Timothy Flynn 79707d83d3 LibUnicode: Stop generating large UnicodeData hash map
The data in this hash map is now available by way of much smaller arrays
and is now unused.
2021-10-10 13:49:37 +02:00
Timothy Flynn d83b262e64 LibUnicode: Generate standalone compile-time array for combining class 2021-10-10 13:49:37 +02:00
Timothy Flynn 9f83774913 LibUnicode: Generate standalone compile-time array for special casing
There are only 112 code points with special casing rules, so this array
is quite small (compared to the size 34,626 UnicodeData hash map that is
also storing this data). Removing all casing rules from UnicodeData will
happen in a subsequent commit.
2021-10-10 13:49:37 +02:00
Timothy Flynn da4b8897a7 LibUnicode: Generate standalone compile-time arrays for simple casing
Currently, all casing information (simple and special) are stored in a
compile-time array of size 34,626, then statically copied to a hash map
at runtime. In an effort to reduce the resulting memory usage, store the
simple casing rules in standalone compile-time arrays. The uppercase map
is size 1,450 and the lowercase map is size 1,433. Any code point not in
a map will implicitly have an identity mapping.
2021-10-10 13:49:37 +02:00
Idan Horowitz 9958277317 Meta: Disable -Wmaybe-uninitialized for Lagom 2021-10-07 21:56:03 +03:00
Nico Weber a7b558bf7d Meta: Warn on rebuild-toolchain i686 clang
It's "Clang" (capitalized). Silently building gcc if the toolchain
isn't know seems a bit unfriendly. So print a warning for this
(and for other unknown toolchains).
2021-10-07 18:34:13 +03:00
Ben Wiederhake 52e9f25403 Everywhere: Change from http to https where feasible
I used "git grep -FIn http://" to find all occurrences, and looked at
each one. If an occurrence was really just a link, and if a https
version exists, and if our Browser can access it at least as well as the
http version, then I changed the occurrence to https.

I'm happy to report that I didn't run into a single site where Browser
can't deal with the https version.
2021-10-05 02:08:08 +02:00
Andreas Kling bf43b0f884 LibWeb: Make IDL-constructed objects aware of their JS wrapper
Having IDL constructors call FooWrapper::create(impl) directly was
creating a wrapper directly without telling the impl object about the
wrapper. This meant that we had wrapped C++ objects with a null
wrapper() pointer.
2021-10-04 12:13:25 +02:00
Linus Groh 2f42675ebd LibJS: Convert ordinary_set_with_own_descriptor() to ThrowCompletionOr 2021-10-04 09:52:15 +01:00
Linus Groh 3be26f56db LibJS: Convert has_own_property() to ThrowCompletionOr 2021-10-03 20:14:03 +01:00
Linus Groh fb443b3fb4 LibJS: Convert create_data_property() to ThrowCompletionOr 2021-10-03 20:14:03 +01:00
Linus Groh b7e5f08e56 LibJS: Convert Object::get() to ThrowCompletionOr
To no one's surprise, this patch is pretty big - this is possibly the
most used AO of all of them. Definitely worth it though.
2021-10-03 20:14:03 +01:00
davidot ac2c3a73b1 LibJS: Add a specific test for invalid unicode characters in the lexer
Also fixes that it tried to make substrings past the end of the source
if we overran the source length.
2021-10-03 17:42:05 +02:00
Luke Wilde 8d6db36cbb LibWeb: Add support for NodeList
This introduces 3 classes: NodeList, StaticNodeList and LiveNodeList.
NodeList is the base of the static and live versions. Static is a
snapshot whereas live acts on the underlying data and thus inhibits
the same issues we have currently with HTMLCollection.

They were split into separate classes to not have them weirdly
mis-mashed together.

The create functions for static and live both return a NNRP to the base
class. This is to prevent having to do awkward casting at creation
and/or return, as the bindings expect to see the base NodeList only.
2021-10-03 00:18:52 +02:00
Luke Wilde bfd089fbd1 LibWeb: Add support for IDL value iterators
This also renames Interface::iterator_types to pair_iterator_types to
reduce confusion between value and pair iterators.
2021-10-03 00:18:52 +02:00
Linus Groh ba6e4c7ae1 LibJS+LibWeb: Use Object::set_prototype() in more places 2021-10-02 14:53:06 +01:00
Idan Horowitz a4bc3fa255 LibWeb: Initialize IDL-generated prototypes' prototype directly
Instead of setting it to the default object prototype and then
immediately setting it again via internal_set_prototype_of, we can just
set it directly in the parent constructor call.
2021-10-02 13:23:59 +01:00
Andreas Kling 77f0e57b27 RequestServer: Don't hide the SIGINFO state dump behind a debug macro
Until we're confident that RequestServer doesn't need this runtime debug
dump helper, it's much nicer if everyone has it built in, so they can
simply send a SIGINFO if they see it acting up.
2021-10-01 20:17:15 +02:00
Idan Horowitz 43482dfde3 LibWeb: Support dictionary-only IDL files 2021-10-01 20:14:45 +02:00
Idan Horowitz ac25c28c43 LibWeb: Add the missing UIEvent IDL constructor 2021-10-01 20:14:45 +02:00
Idan Horowitz 7b2c63fd87 LibWeb: Add support for unwrapping the WindowObject to WrapperGenerator 2021-10-01 20:14:45 +02:00
Idan Horowitz 4d71f22673 LibWeb: Add the missing ProgressEvent IDL constructor 2021-10-01 20:14:45 +02:00
Idan Horowitz 7f551d7f6a LibWeb: Use the LibWeb source directory as the IDL #import base path
This allows us to include IDL files from other base LibWeb directories
wihout using relative `../foo.idl` references.
2021-10-01 20:14:45 +02:00
Nico Weber 9ec9886b04 Meta: Fix typos 2021-10-01 01:06:40 +01:00
Idan Horowitz 2c6c9b73c8 LibWeb: Add the Web::Crypto namespace, built-in, and getRandomValues
Since we don't support IDL typedefs or unions yet, the responsibility
of verifying the type of the argument is temporarily moved from the
generated Wrapper to the implementation.
2021-09-30 20:02:09 +02:00
Timothy Flynn becbb0ea97 LibUnicode: Upgrade to Unicode version 14.0.0 2021-09-30 17:37:57 +01:00
Timothy Flynn c8dbcdb0bc LibUnicode: Do not compare generated file contents before writing
This is now covered by unicode_data.cmake after the superbuild changes.
2021-09-30 17:37:57 +01:00
Timothy Flynn a466b78bf3 LibUnicode: Remove cached UCD and CLDR files when their version changes 2021-09-30 17:37:57 +01:00
Timothy Flynn 0ec6b4f132 LibUnicode: Use consistent variable naming in unicode_data.cmake
I kept mixing these up in my head.
2021-09-30 17:37:57 +01:00
Timothy Flynn 583d703f61 LibUnicode: Functionalize extraction of CLDR data files 2021-09-30 17:37:57 +01:00
Timothy Flynn ead30c26e5 LibUnicode: Functionalize downloading of UCD data files 2021-09-30 17:37:57 +01:00
Timothy Flynn 87bca78ef2 LibUnicode: Extract UCD and CLDR versions to a CMake variable
This also surrounds expansion of affected URL and path variables with
quotes.
2021-09-30 17:37:57 +01:00
Linus Groh ee8380edea LibJS: Convert internal_own_property_keys() to ThrowCompletionOr 2021-09-29 23:49:53 +01:00
Linus Groh fbfb0bb908 LibJS: Convert internal_delete() to ThrowCompletionOr 2021-09-29 23:49:53 +01:00
Linus Groh e5409c6ead LibJS: Convert internal_set() to ThrowCompletionOr 2021-09-29 23:49:53 +01:00
Linus Groh 6c2b974db2 LibJS: Convert internal_get() to ThrowCompletionOr 2021-09-29 23:49:53 +01:00
Linus Groh d9895ec12d LibJS: Convert internal_has_property() to ThrowCompletionOr 2021-09-29 23:49:53 +01:00
Linus Groh 5da210125e LibJS: Convert internal_define_own_property() to ThrowCompletionOr 2021-09-29 23:49:53 +01:00
Linus Groh 0e69a6e487 LibJS: Convert internal_get_own_property() to ThrowCompletionOr 2021-09-29 23:49:53 +01:00
Linus Groh 73bae7d779 LibJS: Convert internal_prevent_extensions() to ThrowCompletionOr 2021-09-29 23:49:53 +01:00
Linus Groh 8c81c84c18 LibJS: Convert internal_set_prototype_of() to ThrowCompletionOr 2021-09-29 23:49:53 +01:00
Linus Groh 5148150e1c LibJS: Convert internal_get_prototype_of() to ThrowCompletionOr 2021-09-29 23:49:53 +01:00
Andreas Kling 6cda24097b LibWeb: Add the CSSStyleRule interface with some limited functionality 2021-09-30 00:00:55 +02:00
Andreas Kling dadb92a155 LibWeb: Generate CSS::property_id_from_camel_case_string()
This allows us to resolve a "camelCase" CSS property name to our own
CSS::PropertyID enum. This will be used by CSSOM bindings.
2021-09-29 21:21:57 +02:00
Andreas Kling 3a4565beec LibWeb: Make CSSRule and CSSRuleList available to JavaScript :^)
This patch makes both of these classes inherit from RefCounted and
Bindings::Wrappable, plus some minimal rejigging to allow us to keep
using them internally while also exposing them to web content.
2021-09-29 21:21:57 +02:00
Idan Horowitz e04f3c713c LibWeb: Add support for custom #import IDL statements
This will currently be used to support dictionary inheritance between
dictionaries defined across different IDL definition files.
2021-09-29 19:38:41 +02:00
Luke Wilde 2202428ca4 LibWeb: Add initial support for the IDL [Unscopable] extended attribute
This adds support for the [Unscopable] extended attribute to attributes
and functions.

I believe it should be applicable to all interface members, but I
haven't done that here.
2021-09-29 17:56:13 +02:00
Luke Wilde 881e9d1341 LibWeb: Make StyleSheetList.item an IDL getter 2021-09-29 14:57:59 +01:00
Ali Mohammad Pur 398435277b RequestServer: Use an OwnPtr for cached connections
Otherwise we'd end up trying to delete the wrong connection if a
connection made before us is deleted.
Fixes _some_ RequestServer spins (though not all...).
This commit also adds a small debug mechanism to RequestServer (which
can be enabled by turning REQUEST_SERVER_DEBUG on), that can dump all
the current active connections in the cache, what they're doing, and how
long they've been doing that by sending it a SIGINFO.
2021-09-29 11:47:18 +02:00
Idan Horowitz 071a358ffb LibWeb: Add support for converting IDL dictionaries to native structs 2021-09-29 10:09:33 +03:00
Idan Horowitz c3810b827a LibWeb: Add support for parsing IDL dictionaries 2021-09-29 10:09:33 +03:00
Idan Horowitz 87f655f2af LibWeb: Change IDL::parse_interface's return type to NonnullOwnPtr
We always returned an interface, so there's no need for the null state.
2021-09-29 10:09:33 +03:00
Idan Horowitz cdde3ba5c5 LibWeb: Add partial support for IDL Iterable declarations
This currently only supports pair iterables (i.e. iterable<key, value>)
support for value iterables (i.e. iterable<value>) is left as TODO().

Since currently our cmake setup calls the WrapperGenerator separately
and unconditionally for each (hard-coded) output file iterable wrappers
have to be explicitly marked so in the CMakeLists.txt declaration, we
could likely improve this in the future by querying WrapperGenerator
for the outputs based on the IDL.
2021-09-28 16:51:27 +02:00
Idan Horowitz 2fab43ba5d LibWeb: Add support for wrapping arbitrary values to WrapperGenerator
This patch essentially just splits the non return-specific logic from
generate_return_statement (i.e. the wrapping of the cpp value into
a javascript one) into a separate function generate_wrap_statement that
can be used to wrap any cpp value during wrapper generation.
2021-09-28 16:51:27 +02:00
Idan Horowitz a11f7868a4 LibWeb: Only consume [a-zA-Z0-9_] characters for IDL types 2021-09-28 16:51:27 +02:00
Luke Wilde 067d839615 LibWeb: Add support for the any type in returning and parameters
Required for CustomEvent.
2021-09-27 18:45:45 +02:00
Luke Wilde d30ec4d790 LibWeb: Add [CustomVisit] IDL interface extended attribute
This custom attribute will be used for objects that hold onto arbitrary
JS::Value's. This is needed as JS::Handle can only be constructed for
objects that implement JS::Cell, which JS::Value doesn't.

This works by overriding the `visit_edges` function in the wrapper.
This overridden function calls the base `visit_edges` and then forwards
it to the underlying implementation.

This will be used for CustomEvent, which must hold onto an arbitrary
JS::Value for it's entire lifespan.
2021-09-27 18:45:45 +02:00
Nico Weber 5565db5aa1 Meta: Pass -serial stdio to qemu on aarch64
With this, `Meta/serenity.sh run aarch64` produces some output on
the terminal :^)
2021-09-26 23:14:01 +00:00
Andreas Kling 43d378940f LibWeb: Add DOMRect and Element.getBoundingClientRect()
This marks our entry into the Web::Geometry namespace, based on the
"Geometry" spec at https://drafts.fxtf.org/geometry/
2021-09-27 01:01:29 +02:00
Luke Wilde 41ae0c0216 LibWeb: Add support for IDL legacy platform objects
A legacy platform object is a non-global platform object that
implements a special operation. A special operation is a getter, setter
and/or deleter. This is particularly used for old collection types,
such as HTMLCollection, NodeList, etc.

This will be used to make these spec-compliant and remove their custom
wrappers. Additionally, it will be used to implement collections that
we don't have yet, such as DOMStringMap.
2021-09-26 18:59:56 +02:00
Idan Horowitz 8dcf4e0346 LibWeb: Return undefined from generated EventHandler setters
Returning an empty value without throwing an exception is no longer
valid.
2021-09-25 23:35:54 +02:00
Sam Atkins 4b554ba92a LibWeb: Clarify StyleValue API with new naming scheme
This does a few things, that are hard to separate. For a while now, it's
been confuzing what `StyleValue::is_foo()` actually means. It sometimes
was used to check the type, and sometimes to see if it could return a
certain value type. The new naming scheme is:

- `is_length()` - is it a LengthStyleValue?
- `as_length()` - casts it to LengthStyleValue
- `has_length()` - can it return a Length?
- `to_length()` - gets the internal value out (eg, Length)

This also means, no more `static_cast<LengthStyleValue const&>(*this)`
stuff when dealing with StyleValues. :^)

Hopefully this will be a bit clearer going forward. There are lots of
places using the original methods, so I'll be going through them to
hopefully catch any issues.
2021-09-24 15:01:43 +02:00
Linus Groh a1a164e6b8 LibWeb: Return undefined from generated setters, not an empty value
These now crash as VM::call() uses ThrowExceptionOr<T>, which refuses to
hold an empty JS::Value as its non-exception result.
We only need to return an empty value when should_return_empty() says
so for the return value of throw_dom_exception_if_needed().

Co-authored-by: Luke Wilde <lukew@serenityos.org>
2021-09-24 12:57:04 +02:00
Sam Atkins b927972da7 LibWeb: Add range-checking to property_accepts_value()
For `number` and `integer` types, you can add a range afterwards to add
a range check, using similar syntax to that used in the CSS specs. For
example:

```json
"font-weight": {
  ...
  "valid-types": [
    "number [1,1000]"
  ],
  ...
}
```

This limits any numbers to the range `1 <= n <= 1000`.
2021-09-23 17:47:40 +02:00
Sam Atkins 11d3098f40 LibWeb: Generate property_accepts_value() function :^)
Previously, we have not been validating the values for CSS declarations
inside the Parser. This causes issues, since we should be discarding
invalid style declarations, so that previous ones are used instead. For
example, in this code:

```css
.foo {
  width: 2em;
  width: orange;
}
```

... the `width: orange` declaration overwrites the `width: 2em` one,
even though it is invalid. According to the spec, `width: orange` should
be rejected at parse time, and discarded, leaving `width: 2em` as the
resulting value.

Many properties (mostly shorthands) are parsed specially, and so they
are already rejected if they are invalid. But for simple properties, we
currently accept any value. With `property_accepts_value()`, we can
check if the value is valid in `parse_css_value()`, and reject it if it
is not.
2021-09-23 17:47:40 +02:00
Sam Atkins a1bc89b814 LibWeb: Generate property_maximum_value_count()
This will allow the CSS Parser to check if a property has been give too
many arguments, and if so, reject it as invalid.
2021-09-23 17:47:40 +02:00
Andreas Kling ca45d34055 LibWeb: Don't add shorthand CSS properties to cascaded values
We already expand shorthands in the cascade, so there's no need to
preserve them in the output.

This patch reorganizes the CSS::PropertyID enum values so that we can
easily iterate over all shorthand or longhand properties.
2021-09-21 16:54:28 +02:00
Timothy Flynn 8084957e88 CI: Create a secondary ccache for the Clang toolchain build
We bust the prebuilt cache when any header in e.g. LibC changes. Doing a
full toolchain rebuild probably isn't necessary, so this adds a separate
ccache to speed up toolchain builds.
2021-09-21 15:39:17 +03:00
Timothy Flynn 4a4e614387 CI: Set ccache path based on template parameter
Currently, the templated steps in Caches.yml rely on the environment
variable CCACHE_DIR being set to configure the ccache location. To
prepare for multiple ccache paths, do not rely on this environment
variable because only one ccache can use it at a time. Instead, pass
the path into the template as a parameter.
2021-09-21 15:39:17 +03:00
Andreas Kling 7ed49e954f LibWeb: Don't print debug spam when looking up missing initial values
I'm about to look up a lot of missing values, and the spam was getting
out of hand.
2021-09-21 12:33:15 +02:00
Andreas Kling 95559c4277 LibWeb: Implement basic support for MessageChannel and MessagePort
This patch adds a basic initial implementation of these API's.

Since LibWeb currently doesn't support workers, this implementation of
messaging doesn't bother with serializing and deserializing messages.
2021-09-19 22:34:44 +02:00
Andreas Kling dd29abe4d6 Meta: Bump default VM memory size to 1 GiB 2021-09-19 22:34:43 +02:00
Brian Gianforcaro aa118642aa Lagom/Fuzzers: Add fuzzer for the LibCrypto PEM parser 2021-09-18 17:15:08 +00:00
Brian Gianforcaro 4f5d71b4e8 Lagom/Fuzzers: Add fuzzer for the LibTLS ASN1 parser 2021-09-18 17:15:08 +00:00
Andrew Kaster aed56b46d4 Documentation: Update Lagom ReadMe with new fuzzer build instructions 2021-09-18 16:38:59 +03:00
Sam Atkins 6ea9edf84b LibWeb: Generate shorthand initial values after their longhands
When parsing shorthand values, we'd like to use
`property_initial_value()` to get their longhand property values,
instead of hard-coding them as we currently do. That involves
recursively calling that function while the `initial_values` map is
being initialized, which causes problems because the shorthands appear
alphabetically before their longhand components, so the longhands aren't
initialized yet!

The solution here is to perform 2 passes when generating the code,
outputting properties without "longhands" first, and the rest after.
This could potentially cause issues when shorthands have multiple
levels, in particular `border` -> `border-color` -> `border-left-color`.
But, we do not currently define a default value for `border`, and
`border-color` takes only a single value, so it's fine for now. :^)
2021-09-17 23:06:45 +02:00
Andreas Kling 11bb6e045f LibWeb: Add the IdleDeadline interface from the RequestIdleCallback spec 2021-09-17 13:20:18 +02:00
Brian Gianforcaro 5c244a7893 Meta: Set SERENITY_ARCH if it is not set in debug-kernel.sh
This appears to have regressed recently in commit 783a58dbc.
2021-09-17 09:16:40 +00:00
Andrew Kaster d0506730b8 Meta: Ensure BUILD_LAGOM is set when running serenity.sh test lagom
If the superbuild created the lagom binary directory, it won't set
BUILD_LAGOM=ON in the CMake cache for that binary directory. Insert a
check into build_target() to make sure that if the user explicitly asks
for the lagom target, we have all our ducks in a row.
2021-09-16 15:47:13 +00:00
Andrew Kaster 454a839f49 CI+Meta: Update Sonar Cloud CI job for new SuperBuild configuration
This requires exposing the `configure` step on the `serenity`
ExternalProject in the SuperBuild CMakeLists so that we can continue to
only build the generated sources and not the entire OS.
2021-09-16 15:47:13 +00:00
Andrew Kaster 04d91e90c2 Meta: Fix Lagom RPATH for non-Ubuntu Linux and macOS hosts
Multi-lib distros like Gentoo and Fedora install lagom-core.so into
lagom-install/lib64 rather than lib. Set the install RPATH based on
CMAKE_INSTALL_LIBDIR to avoid the wrong path being set in the binaries.

Also apply macOS specific RPATH rules to fix the build on that platform.
2021-09-16 15:47:13 +00:00
Ben Wiederhake 51b42e0463 Meta: Make QtCreator aware of all CMake files 2021-09-15 20:21:19 +00:00
Andrew Kaster 368d2ace55 Meta: Add FIXME for not setting BUILD_SHARED_LIBS in Lagom build
This is really the business of the consuming project. We will need to
make changes to libjs-test262 and to oss-fuzz to address this properly.
2021-09-15 19:04:52 +04:30
Andrew Kaster bd7b158c71 Meta: Update serenity.sh for the SuperBuild
Direct build commands to the SuperBuild's binary directory, and
image/run commands to the Serenity binary directory.

As a side benefit, make the lagom target only build Lagom instead of the
entire OS alongside Lagom.
2021-09-15 19:04:52 +04:30
Andrew Kaster b5c98ede08 Meta: Switch to a SuperBuild that splits host and target builds
Replace the old logic where we would start with a host build, and swap
all the CMake compiler and target variables underneath it to trick
CMake into building for Serenity after we configured and built the Lagom
code generators.

The SuperBuild creates two ExternalProjects, one for Lagom and one for
Serenity. The Serenity project depends on the install stage for the
Lagom build. The SuperBuild also generates a CMakeToolchain file for the
Serenity build to use that replaces the old toolchain file that was only
used for Ports.

To ensure that code generators are rebuilt when core libraries such as
AK and LibCore are modified, developers will need to direct their manual
`ninja` invocations to the SuperBuild's binary directory instead of the
Serenity binary directory.

This commit includes warning coalescing and option style cleanup for the
affected CMakeLists in the Kernel, top level, and runtime support
libraries. A large part of the cleanup is replacing USE_CLANG_TOOLCHAIN
with the proper CMAKE_CXX_COMPILER_ID variable, which will no longer be
confused by a host clang compiler.
2021-09-15 19:04:52 +04:30
Andrew Kaster 904a268872 Meta: Move all options to targetname_options.cmake files
This common strategy of having a serenity_option() macro defined in
either the Lagom or top level CMakeLists.txt allows us to do two things:

First, we can more clearly see which options are Serenity-specific,
Lagom-specific, or common between the target and host builds.

Second, it enables the upcoming SuperBuild changes to set() the options
in the SuperBuild's CMake cache and forward each target's options to the
corresponding ExternalProject.
2021-09-15 19:04:52 +04:30
Andrew Kaster 6e7cc40b18 Meta: Add Meta/CMake to the CMAKE_MODULE_PATH for Serenity and Lagom
This makes it so we don't need to specify the full path to all the
helper scripts we include() from different places in the codebase and
feels a lot cleaner.
2021-09-15 19:04:52 +04:30
Andrew Kaster a269a32a17 Meta: Use Lagom:: namespaced names for code generators
This will be required when we switch to a SuperBuild that has Lagom as
a configure time dependency, but is a distinct enough change to be
separate.
2021-09-15 19:04:52 +04:30
Andrew Kaster fc8d1bf3ce Meta: Allow specifying alternative paths for downloaded Unicode data
This lets us possibly share downloaded artifacts between different
builds without re-downloading them every time you change toolchains.
2021-09-15 19:04:52 +04:30
Andrew Kaster a6d83e02d2 Meta: Define and use lagom_tool() CMake helper function for all Tools
We'll use this to prevent repeating common tool dependencies. They all
depend on LibCore and AK only. We also want to encapsulate common
install rules for them.
2021-09-15 19:04:52 +04:30
Brian Gianforcaro fbb31b4519 Kernel: Disable lock rank enforcement by default for now
There are a few violations with signal handling that I won't be able to
fix it until later this week. So lets put lock rank enforcement under a
debug option for now so other folks don't hit these crashes until rank
enforcement is more fleshed out.
2021-09-14 18:31:16 +00:00
Ali Mohammad Pur 7e75a16e6f Meta: Tweak the error in check-ak-test-files.sh a bit
This can also be tripped when the change was simply not being committed,
avoid confusing the author by simply mentioning that.
2021-09-14 21:33:15 +04:30
Idan Horowitz fe32c9c3bd LibWeb: Add the URL::searchParams attribute 2021-09-14 00:14:45 +02:00
Idan Horowitz 23997005cf LibWeb: Allow IDL attribute setters to throw DOMExceptions 2021-09-14 00:14:45 +02:00
Idan Horowitz f9a169380c LibWeb: Consume the extra whitespace in stringifier attributes 2021-09-14 00:14:45 +02:00
Andreas Kling 2e4b34b8c3 LibWeb: Generate CSS::{first,last}_property_id constants
These will allow us to iterate through all known CSS properties.
2021-09-13 22:21:57 +02:00
Idan Horowitz 4629f2e4ad LibWeb: Add the Web::URL namespace and move URLEncoder to it
This namespace will be used for all interfaces defined in the URL
specification, like URL and URLSearchParams.

This has the unfortunate side-effect of requiring us to use the fully
qualified AK::URL name whenever we want to refer to the AK class, so
this commit also fixes all such references.
2021-09-13 01:43:10 +02:00
Idan Horowitz 2b78e227f2 LibWeb: Add support for generating a stringifier method/attribute 2021-09-13 01:43:10 +02:00
Idan Horowitz 47e261c691 LibWeb: Escape reserved names of generated wrapper functions
This also adds 'delete' to the list of reserved cpp names.
2021-09-13 01:43:10 +02:00
Sam Atkins af58bddfc8 LibWeb: Generate CSS::property_has_quirk() function
This lets you query if a given Quirk applies to a given PropertyID.
Currently this applies only to the "Hashless hex color" and "Unitless
length" quirks.
2021-09-12 16:30:38 +02:00
Luke Wilde 678dd2d180 LibWeb: Expose the location object via Document.location
Both Window.location and Document.location use the same instance of the
Location object. Some sites use it via Window, some via Document.
2021-09-12 16:07:24 +02:00
Luke Wilde 3faed65e2b LibWeb: Add full support for optional and nullable to IDL string types
Used by History.
2021-09-12 01:41:44 +02:00
Luke Wilde f8eb616fe3 LibWeb: Add support for the IDL any type
The any type is essentially a raw JS::Value.
2021-09-12 01:41:44 +02:00
Timothy Flynn c59b97043e LibWeb: Use ErrorType::NotAnObjectOfType instead of NotA 2021-09-12 00:16:39 +02:00
Idan Horowitz 6704961c82 AK: Replace the mutable String::replace API with an immutable version
This removes the awkward String::replace API which was the only String
API which mutated the String and replaces it with a new immutable
version that returns a new String with the replacements applied. This
also fixes a couple of UAFs that were caused by the use of this API.

As an optimization an equivalent StringView::replace API was also added
to remove an unnecessary String allocations in the format of:
`String { view }.replace(...);`
2021-09-11 20:36:43 +03:00
Timothy Flynn b1d4bcf364 LibUnicode: Generate numeric keyword values for each locale
This is needed for Intl.NumberFormat's usage of the ResolveLocale AO,
where the [[RelevantExtensionKeys]] internal slot will be "nu".
2021-09-11 11:05:50 +01:00
Timothy Flynn 32a2a02489 LibUnicode: Fix typo in listPatterns.json parsing method 2021-09-08 21:08:48 +01:00
Idan Horowitz 24ed8511dd CI+Meta: Add Signed-off-by tag filter to the commit message linter 2021-09-07 20:46:34 +01:00
Idan Horowitz f56bdd2bb7 Meta: Don't immediately continue in serenity.sh's gdb option
This allows us to step through the initial boot instructions.
2021-09-07 12:58:46 +02:00
Idan Horowitz 783a58dbc7 Toolchain: Build aarch64-gdb for cross-debugging on x86 2021-09-07 12:58:46 +02:00
Timothy Flynn 4ad2159812 LibUnicode: Remove Unicode locale variants from CLDR path names
There's only a couple of cases like this, but there are some locale
paths in the CLDR that contain variants. For example, there isn't a
en-US path, but there is a en-US-POSIX path. This interferes with the
operation to search for locales by name. The algorithm is such that
searching for en-US will not result in en-US-POSIX being found. To
resolve this, we should remove variants from the locale name.
2021-09-06 23:49:56 +01:00
Timothy Flynn 3f64a14e06 LibUnicode: Parse and generate the Unicode locale list patterns dataset
This data informs consumers how to join lists of values. For example,
in en-US, the list ["a", "b", "c"] formatted to a string should become
"a, b, and c".
2021-09-06 23:49:56 +01:00
Timothy Flynn 9cd986d8c0 LibUnicode: Extract cldr-misc dataset from CLDR database 2021-09-06 23:49:56 +01:00
Nico Weber f1d1418256 Meta: Unbreak serenity.sh run aarch64 after 918f7cb4a8 2021-09-06 21:57:59 +01:00
Timothy Flynn 077a693de6 LibUnicode: Sort special casing array by locale specificity
This is to simply the Default Case Conversion implementation. Otherwise,
the implementation would need to determine which special casing rule to
apply, instead of just picking the first match.
2021-09-06 15:24:27 +01:00
Timothy Flynn 91db61ae8d LibUnicode: Generate canonical combining class in Unicode data
Will be used by special casing rules.
2021-09-06 15:24:27 +01:00
Ali Mohammad Pur d20fc922c5 Meta: Correct misuse of ByteBuffer::resize() as grow() in FuzziliJS 2021-09-06 01:53:26 +02:00
Ali Mohammad Pur 97e97bccab Everywhere: Make ByteBuffer::{create_*,copy}() OOM-safe 2021-09-06 01:53:26 +02:00
Tom 918f7cb4a8 Meta: Force legacy xAPIC when using 8 cores or less
This is merely so that we exercise the legacy xAPIC code when we don't
really need x2APIC.
2021-09-04 22:22:58 +02:00
Timothy Flynn e6a2ab1202 LibUnicode: Generate an implementation of the Add Likely Subtags method 2021-09-04 13:51:40 +01:00
Timothy Flynn 28ae63177e LibUnicode: Generate the entire locale likely-subtags dataset
The amount of aliases in the likely-subtags dataset is quite large, so
this also needed to change the way the data is generated. Otherwise, the
compiler would complain about the size of the generated code.

Previously, a static method was generated that would effectively parse
the dataset into a HashMap of Unicode::LanguageID at runtime. We now
perform that parsing at generation-time, and instead generate an Array
of a structure similar to Unicode::LanguageID (we cannot use the same
structure because it contains String and Optional, which cannot be used
at compile-time).
2021-09-04 13:51:40 +01:00
Liav A 8cb4fcdbd8 Meta: Add q35grub option in run.sh
This option is similar to the qgrub option, but instead of starting a
QEMU PIIX4 machine, it starts a QEMU Q35 machine, booting a grub image
disk within it.
2021-09-04 12:54:00 +02:00
Tom 77953a937d Meta: Add the ability to specify clang with serenity.sh
This enables maintaining gcc and clang builds side-by-side.
2021-09-03 23:12:17 +02:00
Luke Wilde ed97ee902b LibWeb: Add support for IDL static functions 2021-09-03 23:11:58 +02:00
Luke Wilde 1d8f8ea5b1 LibWeb: Add initial support for AbortController and AbortSignal
The DOM specification says that the primary use case for these is to
give Promises abort semantics. It is also a prerequisite for Fetch,
as it is used to make Fetch abortable.
a
2021-09-02 09:12:17 +02:00
Timothy Flynn 1fbc5dba08 LibUnicode: Generate Unicode locale likely subtag data
CLDR contains a set of likely subtag data where, given a locale, you can
resolve what is the most likely language, script, or territory of that
locale. This data is needed for resolving territory aliases. These
aliases might contain multiple territories, and we need to resolve which
of those territories is most likely correct for a locale.

Note that the likely subtag data is quite huge (a few thousand entries).
As an optimization encouraged by the spec, we only generate the smallest
subset of this data that we actually need (about 150 entries).
2021-09-01 14:14:47 +01:00
Timothy Flynn 9ae7ac4c87 LibUnicode: Generate complex Unicode locale alias matching
Most alias substitutions are "simple", meaning that alias matching is
done by examining a single locale subtag. However, there are a handful
of "complex" aliases where matching is done by examining multiple
subtags. For example, the variant subtag "lojban" causes the locale
"art-lojban" to be canonicalized to "jbo", but only when the language
subtag is "art" (i.e. this should not occur for the locale "en-lojban").

This generates a method to perform complex alias matching.
2021-09-01 14:14:47 +01:00
Timothy Flynn 9b118f1f06 LibUnicode: Generate Unicode locale alias data
CLDR contains a set of aliases for languages, territories, etc. that no
longer are meant to be used (e.g. due to deprecation). For example, the
language "aam" is deprecated and should be canonicalized as "aas".
2021-09-01 14:14:47 +01:00
Timothy Flynn caf5b6fa6f LibUnicode: Extract cldr-core dataset from CLDR database 2021-09-01 14:14:47 +01:00
Peter Elliott 8d2c04821f Tests: Test LibMarkdown against commonmark test suite
TestCommonmark runs the CommonMark test suite
(https://spec.commonmark.org/0.30/spec.json) against LibMarkdown.
Currently 44/652 tests pass.
2021-08-31 16:53:51 +02:00
Ali Mohammad Pur 1465b11b58 Meta: Implement support for the "unlinkable" Wasm spectest assertion 2021-08-30 22:47:02 +02:00
Ali Mohammad Pur e93c740df5 Meta: Generate bigints for i64 values in LibWasm test suite files 2021-08-30 22:47:02 +02:00
Andreas Kling 57ac8ff1fd Meta: Update lint-ports.py for the SerenityPorts move 2021-08-30 20:49:37 +02:00
Brian Gianforcaro 619200774b CMake: Add custom target to build only the generated sources
This is needed so all headers and files exist on disk, so that
the sonar cloud analyzer can find them when executing the compilation
commands contained in compile_commands.json, without actually building.

Co-authored-by: Andrew Kaster <akaster@serenityos.org>
2021-08-30 16:44:16 +02:00
Linus Groh 747af8a031 Meta: Remove duplicate valid target check from serenity.sh 2021-08-29 13:50:07 +01:00
Linus Groh 2b70f32876 Meta: Mention aarch64 as a supported TARGET in serenity.sh's help 2021-08-29 13:48:35 +01:00
Nico Weber ce98aa567c Meta: Don't pass net -device flags in aarch64 buils in run.sh 2021-08-28 21:51:30 +01:00
Nico Weber ad480505af Meta: Introduce SERENITY_MACHINE variable in run.sh
Set it to all the `-m`, `-display`, `-device` flags for Intel,
and to just "-M raspi3" for aarch64 builds.
2021-08-28 21:51:30 +01:00
Nico Weber 4f3c0fa659 Meta: Disable spice in aarch64 buils in run.sh 2021-08-28 21:51:30 +01:00
Nico Weber 230ea3754d Meta: Run qemu-system-aarch64 in aarch64 builds in run.sh 2021-08-28 21:51:30 +01:00
Nico Weber 00c5e68b11 Meta: Disable kvm for aarch64 in run.sh 2021-08-28 21:51:30 +01:00
Nico Weber 23bc752c5d Meta: Make build-root-filesystem.sh omit libgcc_so.so in aarch64 builds 2021-08-28 21:51:30 +01:00
Andrew Kaster 72de228695 Kernel: Verify interrupts are disabled when interacting with Mutexes
This should help prevent deadlocks where a thread blocks on a Mutex
while interrupts are disabled, and makes it impossible for the holder of
the Mutex to make forward progress because it cannot be scheduled in.

Hide it behind a new debug macro LOCK_IN_CRITICAL_DEBUG for now, because
Ext2FS takes a series of Mutexes from the page fault handler, which
executes with interrupts disabled.
2021-08-28 20:53:38 +02:00
Nico Weber 7052f403c8 Toolchain: Enough to make rebuild-toolchain aarch64 work
The gcc patch might not be completely correct, but at least the
toolchain completes building.
2021-08-28 14:43:07 +01:00
Linus Groh 822566ff1a Meta: Let serenity.sh's TARGET default to SERENITY_ARCH, not i686
This environment variable is already widely used across our build
scripts and tooling, so serenity.sh should respect it as well.
It still uses i686 as the fallback.
2021-08-28 11:20:52 +01:00
Andrew Kaster 834be1a895 Meta: Remove write-only-on-difference script
It's now been replaced with "${CMAKE_PROGRAM}" -E copy_if_different in
all previous usage locations.
2021-08-28 08:44:17 +01:00
Andrew Kaster 63956b36d0 Everywhere: Move all host tools into the Lagom/Tools subdirectory
This allows us to remove all the add_subdirectory calls from the top
level CMakeLists.txt that referred to targets linking LagomCore.

Segregating the host tools and Serenity targets helps us get to a place
where the main Serenity build can simply use a CMake toolchain file
rather than swapping all the compiler/sysroot variables after building
host libraries and tools.
2021-08-28 08:44:17 +01:00
Andrew Kaster e88761b2b9 Meta+LibUnicode: Move unicode_data helper to Meta/CMake
Moving this helper CMake file to the centralized Meta/CMake folder helps
to get a better grasp on what extra files are required for the build,
and what files are generated.

While we're at it, don't use add_compile_definitions for
ENABLE_UNICODE_DATA, which only needs to be seen by LibUnicode sources.
2021-08-28 08:44:17 +01:00
Andrew Kaster 47471c0ec2 Meta: Prefer SerenityOS_SOURCE_DIR to CMAKE_SOURCE_DIR
By using SerenityOS_SOURCE_DIR we can make custom targets and commands
agnostic to the actual location of the root CMakeLists directory.
All we care about is the root of the SerenityOS project.
2021-08-28 08:44:17 +01:00
Andrew Kaster 20e904d87c Meta: Move code generator helpers to their own CMake helper file
compile_gml, compile_ipc, and generate_state_machine all use host
tools to generate sources for the target build. As part of trying to
organize host tools into a common area, let's move these helper rules to
a common file that we can add other host tools to later. And, keep the
host tool helpers separate from the CMake target helpers for apps and
libraries.
2021-08-28 08:44:17 +01:00
Andrew Kaster a83847f8bf Meta: Move components CMake helpers to their own file
It's hard to follow how all the functions in the utils.cmake helper file
flow together, so let's move the pieces that are related to each other
into specialized helpers. First up: all the ConfigureComponents related
properties and functions.
2021-08-28 08:44:17 +01:00
kleines Filmröllchen 2dc614127e Meta: Add System32 to the PATH so that reg.exe is always found
On my machine (c), /mnt/c/Windows/System32 is not on the PATH by
default. This causes reg.exe to fail, which is responsible for detecting
the presence of QEMU. By putting this path on the PATH on WSL, it will
always work regardless of the specific PATH configuration, and QEMU is
always detected.
2021-08-27 21:24:25 +01:00
Timothy Flynn 9663525542 CI: Ensure unzip is installed on the builders
The CLDR database comes in a .zip file.
2021-08-26 23:40:23 +02:00
Timothy Flynn 3127454642 CI: Skip Unicode caches for Fuzzer build
There is a bit of a race here between the Fuzzer and non-Fuzzer Lagom
builds. If the Unicode caches are empty, and the Fuzzer build completes
first, then the UCD and CLDR directories will be empty or won't exist.
Skip handling the Unicode caches for this build.
2021-08-26 22:04:09 +01:00
Timothy Flynn e883792fd4 CI: Cache downloaded Unicode CLDR database 2021-08-26 22:04:09 +01:00
Andreas Kling c915174563 Userland: Remove IRC Client
The IRC Client application made some sense while our main communication
hub was an IRC channel. Now that we've moved on, IRC is just a random
protocol with no particular relevance to this project.

This also has the benefit of removing one major client of the single-
process Web::InProcessWebView class.
2021-08-24 16:37:28 +02:00
Itamar 6f4324e64e Meta: Add 'copy-src' command to serenity.sh
This command copies the project's source tree to
/home/anon/Source/serenity in the built disk image.

This will be useful for working on serenity inside serenity :^)
2021-08-22 10:28:13 +02:00
Timothy Flynn ddfd4cced7 CI: Set on-target test timeout to 60 minutes
The on-target pipelines have a timeout of 6 hours to allow time for a
clean toolchain + Serenity build. Tests should time out much sooner than
that though.
2021-08-20 10:34:52 +02:00
Timothy Flynn 25ae1f7f72 CI: Include a timestamp in the Azure ccache key
Caches on Azure are immutable - so if a cache changes, but its key does
not, then the cache is not updated. Include a timestamp in the ccache
key so that we always push an updated cache from the master branch. Then
use a subkey without the timestamp to pull the cache.

We use a similar trick on GitHub Actions.
2021-08-20 10:34:52 +02:00
Brian Gianforcaro ca9619c750 Meta: Explicitly configure source directory in debug-kernel.sh
Previously debug-kernel.sh wouldn't detect source if you launched the
script from the wrong path. By explicitly setting the name, source will
be loaded by gdb in all situations.
2021-08-19 11:42:20 +02:00
networkException 446bd1e0de README+Meta: Update the screenshot :^) 2021-08-18 14:27:37 +02:00
Martin Åberg 0ed2a03450 Meta: Assume and find GNU du
We want to use use the 'du' option '--apparent-size' which is a
GNU coreutils extension. GNU coreutils is a build dependency so
we know it is available. With this commit we first try to pick up
du as 'gdu', and if that fails, try 'du' instead.
2021-08-18 10:59:50 +02:00
Martin Åberg 0bf867bb9a Meta: Create memory disk before mounting it on FreeBSD
There was previously a case where the build-image-qemu.sh script
decided to mount an existing disk image, but without creating the
memory disk device and recording its /dev file name.

After this commit, We create the memory disk device just before
it is used to mount the disk image.
2021-08-18 10:25:00 +02:00
Andrew Kaster 9bdb44c5d2 CI: Ensure relevant patch files are part of Toolchain cache hash
We were over-hashing for the GNU build on GitHub Actions by including
the LLVM patch as well. The GNU Toolchain doesn't care about our LLVM
patches.

For Azure, fix the inversion of the condition for which jobs check which
Build*.sh script, and add the Toolchain patch files to the cache
hash calculation.
2021-08-18 08:26:38 +02:00
Timothy Flynn a8f9f160f9 CI: Print on-target debug logs when the tests fail 2021-08-18 08:47:23 +04:30
Timothy Flynn e3d4778a63 CI: Build Lagom with fuzzing on Azure 2021-08-17 21:06:15 +01:00
Timothy Flynn 3ce6f26924 CI: Build and test SerenityOS with the Clang toolchain on Azure 2021-08-17 13:31:43 +01:00
Timothy Flynn 5eba8c948d CI: Use Azure caching for Lagom build
Adds a compiler cache and a cache for the UCD files.
2021-08-17 13:31:43 +01:00
Timothy Flynn 831f90c9d4 Meta: Semi-automatically detect ccache for Lagom builds
The top-level CMakeLists.txt already automatically detects ccache, but
CI will invoke CMake with Lagom's CMakeLists.txt. Add an option to Lagom
to do the same detection.
2021-08-17 13:31:43 +01:00
Timothy Flynn 27804a6e67 CI: Build and test Lagom (non-fuzzer) on Azure 2021-08-16 18:04:33 +01:00
Brian Gianforcaro 72f348a6cf Meta: Disable printing frame args when debugging the kernel with gdb
The function arguments almost always optimized away, so you never get
much value out of showing these in the default back trace view, it just
adds a bunch of extra stuff that you need to visual wade through.
So lets disable showing them.
2021-08-14 21:18:05 +02:00
Itamar e57fdb63f8 Tests: Add regression tests for the LibCpp preprocessor
Similarly to the LibCpp parser regression tests, these tests run the
preprocessor on the .cpp test files under
Userland/LibCpp/Tests/preprocessor, and compare the output with existing
.txt ground truth files.
2021-08-14 12:40:55 +02:00
Itamar a38c330c68 LibCpp: Move parser tests to Userland/Libraries/LibCpp/Tests/parser 2021-08-14 12:40:55 +02:00
sin-ack 1d2bde2403 Meta: Add new-project.sh :^)
This script will instantiate a HackStudio template into a project on the
host. It currently supports all templates used by HackStudio.

To avoid having to maintain compatibility between other shells and the
Serenity shell in the postcreate scripts, we build the Serenity shell
with Lagom and use that to run the script.
2021-08-13 21:12:17 +04:30
Jean-Baptiste Boric f641cc6470 Meta: Tune default QEMU disk size
Having lots of small files in Base/ may require more inodes in the
ext2 filesystem than the format utility sets aside by default. Let's
make a more educated guess since we have a rough idea of how many
inodes we need by counting files and directories.
2021-08-13 08:04:59 +02:00
Ali Mohammad Pur f0e7e5bbe8 Meta: Un-escape escaped strings when generating Wasm tests 2021-08-12 21:03:53 +02:00
Ali Mohammad Pur 799471d16f Meta: Don't roundtrip floats for i64/i32 hex literals in wasm tests 2021-08-12 21:03:53 +02:00
sin-ack a871a2040b Meta: Properly quote some CMake variables in serenity_component
This probably isn't all of them, because I'm no CMake expert. :^)
It does however allow "/bin/false" to build now.
2021-08-12 07:48:50 +02:00
Tom e3e52a4e61 Meta: Add SERENITY_CPUS environment variable to control number of CPUs 2021-08-09 21:02:23 +02:00
Daniel Bertalan 13e3df41de Meta: Add Clang support to the CMake build scripts 2021-08-08 10:55:36 +02:00
Daniel Bertalan 15e217ea68 Toolchain: Add LLVM patch and script for building it
This contains all the bits and pieces necessary to build a Clang binary
that will correctly compile SerenityOS.

I had some trouble with getting LLVM building with a single command, so
for now, I decided to build each LLVM component in a separate command
invocation. In the future, we can also make the main llvm build step
architecture-independent, but that would come with extra work to make
library and include paths work.

The binutils build invocation and related boilerplate is duplicated
because we only use `objdump` from GNU binutils in the Clang toolchain,
so most features can be disabled.
2021-08-08 10:55:36 +02:00
Gunnar Beutner 2fec891d8e Meta: Allow attaching GDB to the QEMU instance on the WSL2 host 2021-08-08 01:46:41 +02:00
Gunnar Beutner 45f2e9f3cf Meta: Don't use x86_64 QEMU for the i686 kernel
This seemed like a good idea at the time to avoid an unnecessary
dependency on qemu-system-i386. However this makes debugging the
kernel with GDB more difficult because GDB assumes that the QEMU
architectures matches the kernel architecture.
2021-08-08 01:46:41 +02:00
Andrew Kaster 891bc81f30 Meta: Remove syslog.h from clang-format exclusion list
Before Libraries was moved to Userland/Libraries syslog.h had a bunch
of manually aligned defines and array initializations.

Andreas seems to have formatted the file with clang-format as part of
that file move. Since syslog.h is now properly formatted, we don't
need to exclude it from the linter list.
2021-08-07 15:36:51 +02:00
sin-ack 0d468f2282 Kernel: Implement a ISO 9660 filesystem reader :^)
This commit implements the ISO 9660 filesystem as specified in ECMA 119.
Currently, it only supports the base specification and Joliet or Rock
Ridge support is not present. The filesystem will normalize all
filenames to be lowercase (same as Linux).

The filesystem can be mounted directly from a file. Loop devices are
currently not supported by SerenityOS.

Special thanks to Lubrsi for testing on real hardware and providing
profiling help.

Co-Authored-By: Luke <luke.wilde@live.co.uk>
2021-08-07 15:21:58 +02:00
Liav A 02c72de090 Meta: Remove legacy hardware components from Q35 machine
As this is a test machine I use personally to test "modern" hardware
setups, it feels quite comfortable to not care too much about VGA with
this type of machine.
Also, we don't actively use the IDE controller on this machine type, so
let's just remove it :^)
2021-08-07 11:49:07 +02:00
sin-ack bb609cee7f Meta: Add run-local.sh
This allows one to set their desired parameters for run.sh without the
need to set them in every terminal session or add it to the user account
shell files. If a run-local.sh file exists at the repository root and is
executable, it will be sourced. The file can contain any variables that
are expected to be set in run.sh.
2021-08-07 11:47:07 +02:00
Mandar Kulkarni 566702ffa2 Meta: Fix wrong path for AK Tests in check-ak-test-files.sh 2021-08-05 19:58:55 +02:00
Gunnar Beutner d9d2287a11 Meta: Detect nested KVM support for WSL2 and use that when available
This allows running QEMU inside WSL2 for hosts which have nested KVM
and WSLg support (e.g. Windows 11).

Running QEMU inside the WSL2 VM is slightly slower than running QEMU
on Windows, probably because of how WSLg handles screen updates.
2021-08-04 21:15:41 +02:00
Gunnar Beutner 723c89af67 Meta: Automatically enable WHPX when possible 2021-08-04 21:15:41 +02:00
Gunnar Beutner 5445155dba Meta: Auto-detect where QEMU is installed on Windows 2021-08-04 21:15:41 +02:00
Liav A cca1498e09 Meta: Make the Q35 machine more realistic
Although it is nice to test the system without too many devices, in
reality bare metal hardware is far more complex than the default skeleon
that QEMU provides. As a preparation of supporting more devices, we
need to ensure we are capable of at least booting on complex hardware
setups without easily-observable problems. Later on, this can be the
foundations of testing new drivers :^)
2021-08-03 21:08:01 +02:00
Gunnar Beutner 31655c9486 Meta: Prefer to use the 64-bit QEMU binary if that's available
Some users might not have qemu-system-i386 installed.
2021-08-03 13:59:57 +01:00
Gunnar Beutner b082488e94 Meta: Prefer to use -machine pcspk-audiodev for QEMU >=5.1
This gets rid of the following warning message from QEMU on startup:

qemu-system-i386: warning: '-soundhw pcspk' is deprecated, please set a
backend using '-machine pcspk-audiodev=<name>' instead

Fixes #4093.
2021-08-03 13:59:57 +01:00
Gunnar Beutner 74af43ed97 Meta: Improve WSL detection for the run.sh script
For users who use a custom kernel with WSL our previous method of
detecting WSL doesn't work. This new check instead detects WSL by
checking if the wslpath utility is available.
2021-08-02 11:47:29 +01:00
Andrew Kaster 40b0767d88 Meta: Add BUILD_SHARED_LIBS option for Lagom builds
This standard CMake option controls whether add_library() calls will
use STATIC or SHARED by default. The flag is set to on by default
since that's what we want for normal CI jobs and local builds and the
test262 runner, but disabled for oss-fuzz builds.

This should finally fix the oss-fuzz build after it was broken in #9017

oss-fuzz un-breakage was verified by running the following commands in
the oss-fuzz repo:

python infra/helper.py build_image serenity
python infra/helper.py build_fuzzers --sanitizer address --engine afl \
    --architecture x86_64 serenity /path/to/local/checkout/Meta/Lagom
python infra/helper.py check_build --sanitizer address --engine afl \
    --architecture x86_64 serenity
2021-08-02 09:05:28 +02:00
x-yl 8e8b1c8333 Meta: Explicitly specify the disk format in the QEMU options
Otherwise we're getting this warning:

WARNING: Image format was not specified for '_disk_image' and probing
         guessed raw. Automatically detecting the format is dangerous
         for raw images, write operations on block 0 will be restricted.
         Specify the 'raw' format explicitly to remove the restrictions.
2021-08-02 00:35:22 +02:00
Gunnar Beutner c73c40a21d Meta: Fix compatibility with QEMU 5.x
QEMU 5 doesn't support -machine pcspk-audiodev so we need to fall back
to using -soundhw for that.
2021-08-02 00:35:22 +02:00
Sam Atkins e54531244f LibWeb: Define proper debug symbols for CSS Parser and Tokenizer
You can now turn debug logging for them on using `CSS_PARSER_DEBUG` and
`CSS_TOKENIZER_DEBUG`.
2021-07-31 00:18:11 +02:00
Gunnar Beutner 8b2e76b838 Meta: Make audiodev detection more robust
This fixes audio on Windows as well as Linux systems which don't have
SDL.

Fixes #8903.
2021-07-30 23:51:26 +02:00
Gunnar Beutner dff8b3d2d9 Meta: Make rsync a hard dependency and remove the fallback code
Previously we'd fall back to using cp if rsync wasn't available. Not
only is this considerably slower it also breaks when some of the files
in the target directory are symlinks because cp tries to dereference
them.

Fixes #8672.
2021-07-30 23:50:29 +02:00
Timothy Flynn d485cf29d7 LibRegex+LibUnicode: Begin implementing Unicode property escapes
This supports some binary property matching. It does not support any
properties not yet parsed by LibUnicode, nor does it support value
matching (such as Script_Extensions=Latin).
2021-07-30 21:26:31 +01:00
Brian Gianforcaro a4dc1eb330 Lagom: Fix oss-fuzz build error due to CMake typo
The OSS-Fuzz build was failing with the following error:

 /usr/bin/ld: cannot find -l{}
2021-07-30 20:28:59 +01:00
Andrew Kaster 32d076ef54 Lagom: Add proper install rules
Create the proper export files to allow Lagom to be a well-behaved
ExternalProject, based on the example project from the cmake-init
project generator here:

https://github.com/friendlyanon/cmake-init-shared-static
2021-07-29 21:46:25 +01:00
Linus Groh 7392a73066 Lagom: Exclude GMLAutocompleteProvider.cpp from GML library sources 2021-07-29 21:46:25 +01:00
Andrew Kaster b6e5117f65 Lagom: Change to shared library build for all Lagom code
Split the Lagom build into shared libraries to match the Serenity build.

This reduces the cognitive load when trying to edit the Lagom CMakeLists
significantly. It also reduces the amount of source files that must be
compiled to run each test or host program significantly.

Also re-organize all the build rules into sections. And reorganize the
CMakeLists file in general.
2021-07-29 21:46:25 +01:00
Andrew Kaster de2bf3f333 Meta/LibTest: Build object libraries for test main files
By using the power of object libraries and $<TARGET_OBJECTS> we can make
sure to only build TestMain.cpp and JavaScriptTestRunnerMain.cpp once.

Previously we built these cpp files into object files once for every
single test executable. This change reduces the number of total compile
jobs in a Serenity target build by around 100.
2021-07-29 21:46:25 +01:00
Andrew Kaster af9be6e093 LibTTF/LibGfx: Remove circular dependency by merging LibTTF into LibGfx
LibTTF has a concrete dependency on LibGfx for things like Gfx::Bitmap,
and LibGfx has a concrete dependency in the TTF::Font class in
Gfx::FontDatabase. This circular dependency works fine for Serenity and
Lagom Linux builds of the two libraries. It also works fine for static
library builds on Lagom macOS builds.

However, future changes will make Lagom use shared libraries, and
circular library dependencies are not tolerated in macOS.
2021-07-29 21:46:25 +01:00
Andreas Kling b3d27c2340 LibGUI: Add GUI_HOVER_DEBUG runtime debugging flag (environment)
You can now see the outline of GUI widgets when hovering them.
For example:

    $ export GUI_HOVER_DEBUG=1
    $ FileManager

Then move the mouse around in the file manager. :^)
2021-07-28 01:01:02 +02:00
Karol Kosek 220dd28b02 Meta: Ignore everything after the git cut line in a lint-commit.sh hook
I have set up a commit.verbose variable in my git config,
which shows the patch diff on bottom of the commit message.

Unfortunately the character limit was also applied to the diff,
which meant that I got a false-positive lint error almost every time.
2021-07-27 22:35:49 +01:00
Idan Horowitz 350fb0e3db Meta: Bring lint-commit.sh up to date with the CI commit linter
This was missing 2 of the recently added checks. Also added a reminder
in the CI linter to update the Meta (commit hook) version.
2021-07-27 20:41:49 +01:00
x-yl 4bdd62939a Meta: Set -audiodev explictly
Unfortunately seems like QEMU tries to use SPICE for audio so we need to
explicitly set -audiodev to use either SDL or coreaudio (depending on
platorm)
2021-07-27 19:15:37 +02:00
Gunnar Beutner 57417a3d6e Kernel: Support loading the kernel at almost arbitrary virtual addresses
This enables further work on implementing KASLR by adding relocation
support to the pre-kernel and updating the kernel to be less dependent
on specific virtual memory layouts.
2021-07-27 13:15:16 +02:00
Timothy Flynn 98d8274040 Meta: Add LibUnicode (and its tests) to Lagom
This is primarily to allow using LibUnicode within LibJS and its REPL.

Note: this seems to be the first time that a Lagom dependency requires
generated source files. For this to work, some of Lagom's CMakeLists.txt
commands needed to be re-organized to include the CMake files that fetch
and parse UnicodeData.txt. The paths required to invoke the generator
also differ depending on what is currently building (SerenityOS vs.
Lagom as part of the Serenity build vs. a standalone Lagom build).
2021-07-26 17:03:55 +01:00
Ali Mohammad Pur 466e2a2fb7 Meta: Don't reboot on triple-fault in CI mode
Instead, just make QEMU quit immediately.
2021-07-26 02:29:25 +04:30
Thomas Wagenveld 2002b7e2e3 Meta/run.sh: Allow for overriding of QEMU ethernet device type
You can set the SERENITY_ETHERNET_DEVICE_TYPE environment variable to
pick another device type (i.e. ne2k_pci). Defaults to e1000 as before.
2021-07-24 21:28:22 +02:00
Luke 7e7c65abb6 Lagom/Fuzzers: Add fuzzer for the quoted printable decoder 2021-07-24 20:11:28 +04:30
Gunnar Beutner cc0914ae58 Meta: Add Homebrew paths for macOS on M1 to the PATH env variable 2021-07-24 14:04:13 +02:00
Andreas Kling 3f9e018d9a CrashDaemon: Remove BACKTRACE_DEBUG debugging code
This thing seems to work fine, no need to hang on to old debug code.
2021-07-22 23:34:33 +02:00
Gunnar Beutner 9031ed0b84 Meta: Add a new qextlinux target for the run.sh script
This allows testing the extlinux image with QEMU.
2021-07-20 15:12:19 +02:00
Gunnar Beutner 696e15fcd8 Meta: Update extlinux config for the recent prekernel changes 2021-07-20 15:12:19 +02:00
Gunnar Beutner 6a45ebe282 Meta: Fix syslinux detection on Arch Linux 2021-07-20 15:12:19 +02:00
Gunnar Beutner e3b7ae0c77 Meta: Remove unused bootloader_test target
Due to other changes this is now just a copy of the q35 target.
2021-07-18 22:08:03 +02:00
Gunnar Beutner 61a8f3e81a Kernel: Fix GRUB config by specifying the correct file name 2021-07-18 22:08:03 +02:00
Gunnar Beutner 7e94b090fe Kernel: Introduce basic pre-kernel environment
This implements a simple bootloader that is capable of loading ELF64
kernel images. It does this by using QEMU/GRUB to load the kernel image
from disk and pass it to our bootloader as a Multiboot module.

The bootloader then parses the ELF image and sets it up appropriately.
The kernel's entry point is a C++ function with architecture-native
code.

Co-authored-by: Liav A <liavalb@gmail.com>
2021-07-18 17:31:13 +02:00
Gunnar Beutner 98f8ecd9d2 Kernel: Split debug symbols into a separate file
This speeds up the boot process considerably when specifying the kernel
image via -initrd.
2021-07-18 17:31:13 +02:00
Riyyi 5d1676b05a Meta: Do not warn user about too modern clang-format
To prevent warnings for users of varying distributions, do not warn the
user if the clang-version is higher than the expected version.
2021-07-15 09:27:44 +02:00
Sam Atkins fa7a6fbedd Meta: Fix SPICE detection in run.sh
The previous fix did not work for me, but this does. :^)

Credit goes to @X-yl for actually figuring it out.
2021-07-14 21:17:56 +02:00
x-yl 410eb9e9ff Meta: Don't use SPICE if QEMU doesn't support it
I do seem to have a tendency for breaking everyone's builds :/
2021-07-14 17:17:06 +02:00
x-yl 6b68f16f2c Meta: Change QEMU options in run.sh to use SPICE
If QEMU has the qemu_vdagent chardev (should be present in QEMU 6.1)
then we use that as the SPICE client.

If qemu_vdagent is not present, no SPICE client will be launched by
default because it makes the display a bit choppy.

Set SERENITY_SPICE to override the default behavior and use your default
SPICE client.
2021-07-14 12:33:07 +02:00