1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
|
# Virtual Memory Overcommit
_2026 August_
There's a distinction between reserving virtual memory and having it
"committed" by the kernel.
This isn't the same thing as it being backed by physical memory. One
could think of the state of vmem as having three levels:
1. Reserved
2. Committed
3. Physically backed
Also, the physical backing could be RAM or SWAP, but that's not
important for our case. What matters is that reserving is always
"free" because it's just a reservation within the virtual address
space of the process itself, whereas committing *may* be checked
against the available RAM/SWAP of the whole system, and possibly
denied even before it needs physical backing. There may also be
explicit commit limits unrelated to RAM/SWAP availability.
Committing memory despite the system potentially not having enough
RAM/SWAP is called overcommit.
## When is overcommit allowed
According to some quick research (don't cite this) the breakdown of
which kernels allow overcommit is as follows:
* MacOS: Always allowed.
* Linux, FreeBSD, NetBSD: Allowed by default; configurable.
* OpenBSD, MS Windows: Not allowed.
So, for Zisp to cover even just all POSIX systems, it will need some
way to reserve without committing.
## How to reserve without commit
Under POSIX, you reserve without committing by calling `mmap()` with
`PROT_NONE` for the protection argument. Of course, the behavior is
not actually documented as part of POSIX, but I suppose it should be
safe to assume that any sane implementation would not count memory
that's inaccessible towards any kind of commit limit. Surprisingly,
even the Linux documentation doesn't *explicitly* guarantee that a
`PROT_NONE` mapping is "free" but I think it's safe to assume given
there's projects in the wild like jemalloc and the JVM that already
use this trick, so Linux wouldn't dare change the behavior.
There's also `MAP_NORESERVE` under Linux, which can achieve the same
effect, but I guess it's less portable. Funnily, there's even some
old discussions about the kernel not honoring `MAP_NORESERVE` under
strict overcommit policies, while `PROT_NONE` works. Oh and also,
"NORESERVE" flies in the face of the terminology I use here, and
there's also comments on LWN about how it's a misnomer:
[https://lwn.net/Articles/627557/](https://lwn.net/Articles/627557/)
Anyhow, TL;DR: Use `PROT_NONE` on POSIX; it should hopefully work as
desired on any common POSIX system besides OpenBSD.
On MS Windows, you reserve by calling `VirtualAlloc()` with the
`MEM_RESERVE` flag, and commit with the `MEM_COMMIT` flag.
## Committing as you go
When you've reserved without commit, you then actually need to tell
the kernel when you need the memory committed.
On POSIX, you can't use it under `PROT_NONE` anyway (the whole point)
and asking the system to commit it is a matter of calling `mprotect()`
to set the protection mode to `PROT_READ | PROT_WRITE` instead.
Likewise, on MS Windows, trying to access in any way a region of vmem
not yet committed is an error; you need to call `VirtualAlloc` again,
on a sub-range of the full range, with `MEM_COMMIT` this time.
## Disadvantages
Syscalls are costly! Would be great if we only ever needed a single
one at startup, but alas.
Thankfully, after my latest tweaks to Meta Alloc, it already bumps
watermarks in chunks of at least 64 KiB. That's a perfect spot for
inserting the `mprotect()` or `VirtualAlloc()` calls.
I may just do that unconditionally, at least for now, because it's
easy to make it configurable if ever needed.
|
