This patch adds generic slab allocators to kmalloc. In this initial
version, the slab sizes are 16, 32, 64, 128, 256 and 512 bytes.
Slabheaps are backed by 64 KiB block-aligned blocks with freelists,
similar to what we do in LibC malloc and LibJS Heap.
There are no more users of the C-style kfree() API in the kernel,
so let's get rid of it and enjoy the new world where we always know
how much memory we are freeing. :^)
This patch does two things:
- Combines kmalloc_aligned() and kmalloc_aligned_cxx(). Templatizing
the alignment parameter doesn't seem like a valuable enough
optimization to justify having two almost-identical implementations.
- Stores the real allocation size of an aligned allocation along with
the other alignment metadata, and uses it to call kfree_sized()
instead of kfree().
Since we allocate the subheap in the first page of the given storage
let's assert that the subheap can actually fit in a single page, to
prevent the possible future headache of trying to debug the cause of
random kernel memory corruption :^)
This avoids getting caught with our pants down when heap expansion fails
due to missing page tables. It also avoids a circular dependency on
kmalloc() by way of HashMap::set() in MemoryManager::ensure_pte().
Previously, the heap expansion logic could end up calling kmalloc
recursively, which was quite messy and hard to reason about.
This patch redesigns heap expansion so that it's kmalloc-free:
- We make a single large virtual range allocation at startup
- When expanding, we bump allocate VM from that region
- When expanding, we populate page tables directly ourselves,
instead of going via MemoryManager.
This makes heap expansion a great deal simpler. However, do note that it
introduces two new flaws that we'll need to deal with eventually:
- The single virtual range allocation is limited to 64 MiB and once
exhausted, kmalloc() will fail. (Actually, it will PANIC for now..)
- The kmalloc heap can no longer shrink once expanded. Subheaps stay
in place once constructed.
cert-dcl50-cpp: No variadic functions, suppressed in RefCounted and
ThreadSafeRefCounted for implementing the magic one_ref_left and
will_be_destroyed functions.
cert-dcl58-cpp: No opening ::std, suppressed in the places we put names
in ::std to aid tools (move, forward, nullptr_t, align_val_t, etc).
SonarCloud flagged this "Code Smell", where we are accessing these
static methods as if they are instance methods. While it is technically
possible, it is very confusing to read when you realize they are static
functions.
PVS-Studio flagged these as uninitialized. While there is no bug here,
it is our policy to always initialize members to avoid potential bugs
in the future.
This expands the reach of error propagation greatly throughout the
kernel. Sadly, it also exposes the fact that we're allocating (and
doing other fallible things) in constructors all over the place.
This patch doesn't attempt to address that of course. That's work for
our future selves.
This is the idiomatic way to declare type aliases in modern C++.
Flagged by Sonar Cloud as a "Code Smell", but I happen to agree
with this particular one. :^)
The C++ standard specifies that `free` and `operator delete` should
be callable with nullptr. The non-aligned `kfree` already handles this,
but because of the pointer arithmetic to obtain the allocation start
pointer, the aligned version would produce undefined behavior.
In e7fb70b05, regular kmalloc was changed to return nullptr on
allocation failure instead of crashing. The `kmalloc_aligned_cxx`
wrapper used by the aligned operator new should do the same.
Now that we have a significant amount of code paths handling OOM, lets
enable kmalloc and friends to actually return nullptr. This way we can
start stressing these paths and validating all of they work as expected.
By making these functions static we close a window where we could get
preempted after calling Processor::current() and move to another
processor.
Co-authored-by: Tom <tomut@yahoo.com>
Kernels built with Clang seem to be quite allocation-heavy compared to
their GCC counterparts. We would sometimes end up crashing during boot
because the eternal ranges had no free capacity.
This is a much more ergonomic option than getting a
`VERIFY_NOT_REACHED()` failure at run-time. I encountered this issue
with Clang, where sized deallocation is not the default due to ABI
breakage concerns.
Note that we can't simply just not declare these functions, because the
C++ standard states:
> If this function with size parameter is defined, the program shall
> also define the version without the size parameter.
The compiler will use these to allocate objects that have alignment
requirements greater than that of our normal `operator new` (4/8 byte
aligned).
This means we can now use smart pointers for over-aligned types.
Fixes a FIXME.
The C++ standard says that it's legal to call the `delete` operator with
a null pointer argument, in which case it should be a no-op. I
encountered this issue when running a kernel that's compiled with Clang.
I assume this fact was used for some kind of optimization.
This was only used by a single class (AK::ByteBuffer) in the kernel
and not in an OOM-safe way.
Now that ByteBuffer no longer uses it, there's no need for the kernel
heap to burden itself with supporting this.
C++14 gave us sized operator delete, but we haven't been taking
advantage of it. Let's get to a point where it can help us by
adding kfree_sized(void*, size_t).
When creating uninitialized storage for variables, we need to make sure
that the alignment is correct. Fixes a KUBSAN failure when running
kernels compiled with Clang.
In `Syscalls/socket.cpp`, we can simply use local variables, as
`sockaddr_un` is a POD type.
Along with moving the `alignas` specifier to the correct member,
`AK::Optional`'s internal buffer has been made non-zeroed by default.
GCC emitted bogus uninitialized memory access warnings, so we now use
`__builtin_launder` to tell the compiler that we know what we are doing.
This might disable some optimizations, but judging by how GCC failed to
notice that the memory's initialization is dependent on `m_has_value`,
I'm not sure that's a bad thing.
In standard C++, operators `new` and `new[]` are guaranteed to return a
valid (non-null) pointer and throw an exception if the allocation
couldn't be performed. Based on this, compilers did not check the
returned pointer before attempting to use them for object construction.
To avoid this, the allocator operators were changed to be `noexcept` in
PR #7026, which made GCC emit the desired null checks. Unfortunately,
this is a non-standard feature which meant that Clang would not accept
these function definitions, as it did not match its expected
declaration.
To make compiling using Clang possible, the special "nothrow" versions
of `new` are implemented in this commit. These take a tag type of
`std::nothrow_t` (used for disambiguating from placement new/etc.), and
are allowed by the standard to return null. There is a global variable,
`std::nothrow`, declared with this type, which is also exported into the
global namespace.
To perform fallible allocations, the following syntax should be used:
```cpp
auto ptr = new (nothrow) T;
```
As we don't support exceptions in the kernel, the only way of uphold the
"throwing" new's guarantee is to abort if the allocation couldn't be
performed. Once we have proper OOM handling in the kernel, this should
only be used for critical allocations, where we wouldn't be able to
recover from allocation failures anyway.
