diff options
Diffstat (limited to 'notes')
| -rw-r--r-- | notes/250219-reader.md | 5 | ||||
| -rw-r--r-- | notes/250920-libgccjit.md | 2 | ||||
| -rw-r--r-- | notes/260107-decoder.md | 2 | ||||
| -rw-r--r-- | notes/260109-uninterned.md | 2 | ||||
| -rw-r--r-- | notes/260516-joke-rant.md | 69 | ||||
| -rw-r--r-- | notes/260522-interpreter.md | 261 | ||||
| -rw-r--r-- | notes/index.md | 2 |
7 files changed, 339 insertions, 4 deletions
diff --git a/notes/250219-reader.md b/notes/250219-reader.md index 4454783..e930e21 100644 --- a/notes/250219-reader.md +++ b/notes/250219-reader.md @@ -526,6 +526,5 @@ Just to summarize what I actually ended up implementing in the end: See here for full documentation of Zisp expressions as implemented: -- [Informal docs](https://git.tkammer.de/zisp/tree/docs/parser.md) -- [Formal spec](https://git.tkammer.de/zisp/tree/spec/syntax.md) -- [ABNF](https://git.tkammer.de/zisp/tree/spec/syntax.abnf) +- [Informal docs](https://tkammer.de/zisp/docs/c1/1-parse.html) +- [Formal spec](https://tkammer.de/zisp/docs/c1/grammar/index.html) diff --git a/notes/250920-libgccjit.md b/notes/250920-libgccjit.md index 8ecf4f2..882b69a 100644 --- a/notes/250920-libgccjit.md +++ b/notes/250920-libgccjit.md @@ -1,5 +1,7 @@ # Using libgccjit? +_2025 September_ + The installed size of libgccjit varies from 18 MB (armhf) to 53 MB (riscv64), with 37 MB for amd64. diff --git a/notes/260107-decoder.md b/notes/260107-decoder.md index a1118b7..5fa3677 100644 --- a/notes/260107-decoder.md +++ b/notes/260107-decoder.md @@ -191,7 +191,7 @@ However, this causes such custom object literals to not stand out: ``` For this reason, it would be a convention that decoder rules are used -to implement new object literal syntax, and macros used for then you +to implement new object literal syntax, and macros used for when you want to output code, with hygienic bindings. ```scheme diff --git a/notes/260109-uninterned.md b/notes/260109-uninterned.md index 5899cba..5d914b1 100644 --- a/notes/260109-uninterned.md +++ b/notes/260109-uninterned.md @@ -1,5 +1,7 @@ # What strings NOT to intern? +_2026 January_ + Although it's become fairly normal these days to define grammars in terms of Unicode, there's a certain simplicity to just defining them in terms of bytes instead. diff --git a/notes/260516-joke-rant.md b/notes/260516-joke-rant.md new file mode 100644 index 0000000..a1c6f64 --- /dev/null +++ b/notes/260516-joke-rant.md @@ -0,0 +1,69 @@ +# Compilers are FAKE + +_2026 May NOT_ + +Languages typically either + +- follow the "interpreted language" mentality and at best add things + like bytecode cache files or JIT as behind-the-scenes optimizations, + or + +- follow the "compiled language" mentality where code written in the + language is a representation of the intended behaviour of an + executable that will eventually be produced. + +In the latter model, compile-time logic is often limited to primitive +pre-processors, declarative DSLs or otherwise limited subsystems (C++ +templates, Zig comptime, etc.) and often even offloaded to an entirely +separate "build system" language. In the former model, compilation is +swept under the rug like it's shameful dirt. + +Why? + +How about designing a language that embraces the reality of what it +means to "compile code": Run an executable with source code as input. +Sound familiar? IT'S WHAT AN INTERPRETER DOES! So, in my preferred +language: + +- The "interpreter" includes APIs for explicitly compiling any of the + currently defined functions, modules, etc. straight from their in + memory representation (be it AST or bytecode) into native code or + assembly snippets, with the ability to then "serialize" or assemble + and link these to create object files or executables. + +- The "compiler" is just a thin wrapper around these APIs, loading + some source files that define functions and globals etc. (all of + which is first interpreted) then compiling them and producing + executables. + +- The "top level" of any source file is naturally executed at compile + time. You get meta programming and a build system language "for + free" since these kinds of features can simply be libraries. + +Some lisp/scheme implementations already contain an interpreter and +compiler in one, exposing parts of the compiler through APIs, and +allow full compile time code execution via macros. However, in my +experience, they still tend to embrace the traditional interpreted +vs. compiled mentality, which sometimes just makes things awkward. +For example, GNU Guile insists on automatically compiling everything +into a bytecode cache on the filesystem, unless you pass a CLI flag +disabling this behavior. And if you write a Guile Scheme codebase, +you still typically use Autotools and Make to call `guild` to compile +the Scheme files. Just why? You have the full Guile runtime at your +disposal, at compile time... I propose a change in mentality. + +## Dial the schizophrenia to 11 + +Compilers are all FAKE. They just pretend to be compilers. All a +compiler is, is an interpreter for a language that pretends to be a +"compiled language" when in reality it's just an interpreted DSL for +declaring the desired behavior of an executable, which is produced +when the codebase written in this DSL is interpreted ("compiled") by +the interpreter ("compiler"). + +They have played us for absolute fools. + +C is a declarative, interpreted DSL to describe executables. + +GCC is an interpreter that runs C, a declarative DSL describing the +executable that GCC should shit out. diff --git a/notes/260522-interpreter.md b/notes/260522-interpreter.md new file mode 100644 index 0000000..cf87180 --- /dev/null +++ b/notes/260522-interpreter.md @@ -0,0 +1,261 @@ +# The interpreter and the compiler + +_2026 May_ + +Last December, I wrote the following in the context of how one might +[bootstrap](250329-boot.html) Zisp even if it uses a self-hosting +compiler: + +* There will be a Zisp interpreter written in Zig, which is fairly + simple and naive in its implementation and, for example, ignores + static type declarations. It should support the full Zisp language + including hygienic macros, but be as easy as possible to maintain. + +* The Zisp compiler will be written in Zisp. The interpreter can run + the compiler (since it can run any Zisp program) and will be used to + compile the compiler. + +After some pondering on a variety of topics, I've decided to stick +with this, just with one significant added insight: + +The interpreter will not be some bootstrapping hack and then put in +the dustbin until someone needs to bootstrap from scratch again. +Rather, the interpreter will be a first-class citizen of the Zisp +implementation. + +This is because a simple interpreter without any compilation overhead +is useful for an entire class of applications: Small to medium size +scripts that you simply plop into `~/bin` with a shebang line at the +top, or other similarly small programs that are simply distributed as +monolithic source files, or at most a small collection of files. + +The interpreter may be slow, but these would be the kinds of programs +one might otherwise write in GNU Bash or the like (which is also quite +slow) except GNU Bash doesn't even have proper data structures, so it +becomes a terrible choice very quickly. The next consideration after +Bash would typically be a language like Python, and although even the +CPython interpreter might beat the naive Zisp interpreter (because the +former at least uses bytecode and had a ton of engineering poured into +it) this shouldn't really matter, since the kind of tiny application +we're talking about typically wouldn't involve heavy computation. + +(Besides, a Zisp script could choose to compile parts of itself; more +on this later.) + +Another example are build scripts. One of the first ideas I had when +pondering on Zisp's design is how [compilation](250210-compile.html) +should automatically evaluate the top-level of a program, simply +because this feels most natural to me. Furthermore, I've pondered +about how it should be possible to [serialize](250210-serialize.html) +everything in the language, so compiling a program would be a matter +of calling something like `(write main)` after the main function is +defined. Both of these fit naturally with the idea that a build +script for a Zisp program would essentially just be a Zisp script +which imports all the files in the codebase, compiles everything, and +writes out the result. Such a build script would be interpreted, with +the compiler being a shared library it loads. + +The compiler itself would typically still be shipped in compiled form, +as well as the rest of the standard library, though it's conceivable +that there might be benefits to having stdlib sources available; the +compiler may be able to do better whole-program analysis, achieving +better results than what you might get from LTO. + +## The programmer is in control of compilation + +Shipping an interpreter, with a compiler as a library, being able to +compile things on-the-fly as instructed by the interpreted source +itself, enables some novel strategies in development and deployment. + +### Manual JIT + +First, imagine you started developing a program as a fairly small +script but at some point begin to realize that it does, after all, +involve some heavy computations that could benefit from improved +performance. + +Maybe it takes 10-20 minutes to run, with the majority of that time +spent on one or two functions sifting through massive amounts of data +and doing some heavy computation, involving some tight loops. Well, +your interpreter includes a compiler, so what about you simply just +call the compiler on those functions right after defining them? + +Note that we're not talking about compiling *files* but simply some +functions that are sitting in memory as AST and would otherwise be +interpreted naively and slowly. + +It's said that the difference between a naive AST interpreter, and +compiled native code, can be as high as a 5-20x difference, so your +script running in 20 minutes could be reduced down to 1-2 minutes; a +little extra computation is added up-front to compile a function or +two, then they run blazing fast. + +### Native targeting, and user data/code specialization + +The fact that you have a compiler in your runtime, and that it has a +well-designed easy to use API, opens the door to a somewhat unusual +software deployment strategy: + +Despite the fact that your application is rather sophisticated and +needs to run at peak performance, you distribute it as source code, +with a "boot" process that compiles all the sources every time when +it's started up on the end user's machine. (Well, the compilation +result could be cached into files on disk too, but that's a detail.) + +This has two advantages. For one, the code is always compiled for the +exact native architecture, not just an ISA family. This can improve +performance a little, sometimes. + +Secondly, and more interestingly, data *and even code* read from a +configuration file can be compiled straight into the native code +that's being generated. + +If you know Nginx's configuration format, you may know that it has +some limitations that appear a bit strange, typically because the +directives need to be "compiled" into something efficient if they +declare some logic that has to be executed on every single request. +Since Nginx doesn't want to implement a sophisticated compiled DSL +like Varnish, it ends up being somewhat limited. Varnish does make +that jump and implements a whole DSL for per-request decisions, which +is transpiled to C, compiled into a dynamic lib and loaded. + +Imagine Nginx was written in Zisp, and distributed in source format. +You could have arbitrary code in your configuration, for per-request +decisions, which would be compiled into native code and potentially +inlined straight into Nginx's request handler. Imagine Varnish was +written in Zisp. It wouldn't need to invent a whole new language! + +(I just realized Varnish has been renamed to Vinyl Cache, but I +suspect most people still know it as Varnish, like me just now.) + +Just as an aside, I think this "compile at startup and cache it" +strategy is used by Elixir. Or maybe I just got that impression +because I've installed Pleroma (an Elixir application) from Git. +Either way, I doubt my idea is entirely new; this is definitely a +strategy that can already be used by any application written in a +language with a compiler built into the runtime, like many Lisp or +Scheme implementations. + +## Why not automatic JIT? + +Although a more "proper" JIT has some advantages, like being able to +specialize on arbitrary run-time data (not just config files or other +such "boot-time" data), they typically produce significantly worse +code than a "full AOT compiler in a JIT-shaped trench coat" because +the AOT compiler simply spends a *lot* more time on analysis upfront. +Don't cite me on this, but it appears to be the current consensus. + +Traditional JIT, as opposed to what LLVM and GCC offer (i.e., AOT in a +JIT shaped trench coat), needs to be low latency, since it's done on +the fly, transparently, and concurrently. Imagine your browser ran +GCC or LLVM for every JS file it received. That would be ridiculous. +Note that JS is special in that it's basically the only programming +language where arbitrary new code is loaded *all the time* during the +normal course of operations. Other languages just don't need this. +It's just JS where high upfront latency is unacceptable. + +Why do Java, Lua, and a bunch of other dynamic languages use JIT? +Partly, it may be cultural: Native AOT compilation feels yucky, +invoking associations such as long compile times multiplied by the +number of target architectures, needing to ship binary blobs, and the +primitive C ABI. Java can have its own rich ABI, and languages like +Lua don't have an ABI at all because everything is source code. If +programmers can simply ship source files, or at worst cross-platform +byte code like for the JVM, and then the JIT magically makes things +faster, there's less headache I guess. (There is AOT for Java, but +it's a niche.) + +Another reason, probably, is that many high-level languages are very +dynamic and lack a serious static type system that would be needed to +generate peak performance AOT compiled code. + +Zisp is all about breaking norms, and giving the programmer maximum +freedom. The interpreter might one day incorporate some lightweight +JIT, but my aim is to ensure that a Zisp programmer always has the +ability to generate peak-performance native compiled binaries, through +a combination of features such as: An optional but serious static type +system, the ability to completely take control over memory management +rather than relying on GC, and integrating with a high-end AOT native +compiler like GCC. + +Tall claims, I know. Stop looking at me like that. Yes I know, all I +have so far is a fucking s-expression parser, a NaN packing strategy +for dynamic typing, and dreams. But if I keep dreaming and planning, +I'm sure the implementation will spontaneously pop into existence any +day now. + +## Summary of planned implementation architecture + +Just to recap, here's the plan so far: + +1. A code base in a low level language (probably Zig but not married + to it) implements the Zisp core, meaning interpreter, basic data + types, and a slim standard library. Comparable to R7RS-small in + complexity, give or take. The interpreter accepts but ignores + advanced code constructs intended to help the compiler, such as + declarations and directives related to static typing and explicit + object lifetime management. (Simple bindings to libgccjit are + exposed; libgccjit.so is an optional run-time dependency.) This + yields libzisp.so and the zisp executable, which are like liblua + and the lua executable. You *can* use just this if you need a + minimal Zisp interpreter with a barebones stdlib; OS package + repositories could deploy these in a "zisp-core" package. + +2. Richer standard library routines are written in Zisp, but the + sources are meant to stay in the source code repo; wait for it. + +3. An advanced compiler, which actually understands the constructs + mentioned in point 1, is written in Zisp. The compiler infers + static types where possible, and applies strategies to decrease GC + pressure, such as escape analysis, even if compiled code offers no + helpful declarations at all. But with full static typing and + manual memory management, Zisp can practically be used as if it's + yet another low-level language front-end for GCC; it's up to the + programmer how much effort they want to put into improving the + performance of their code. The compiler implementation may use + parts of the richer standard library mentioned above, which is not + yet compiled, mind you. + +4. The interpreter runs the compiler to compile the compiler; this + yields libzispcomp.so which Zisp can load dynamically so when + deploying Zisp you don't need to compile the compiler on every + end-user machine. (Zisp can load any .so dynamically really.) + Standard library routines written in Zisp are imported directly + from within the source code repo at this point, and are merely + interpreted, since the compiler itself wasn't ready yet. + (Actually, you could run the compiler with the interpreter to + compile the stdlib first, then use the compiled stdlib while + compiling the compiler. But this would probably be slower.) + +5. The richer standard library routines are finally compiled, giving + us libzisputil.so, which contains goodies that interpreted Zisp + code can also load and use, so Zisp scripts aren't limited to the + barebones stdlib anymore. + +In OS package repositories, you'd have zisp-core which only contains +libzisp.so and the zisp executable, and then you'd have the standard +zisp package which also pulls in libzispcomp and libzisputil as two +additional packages. + +Actually, libzispcomp itself would probably depend on libzisputil +anyway, but if you're an absolute nerd you *could* manually install +only zisp-core and libzisputil, giving you an interpreter and rich +standard library, without a compiler. This would allow you to omit +libgccjit as well, which could be useful if you want to use the Zisp +interpreter for simple scripts on some minimal systems. + +## Closing up + +Funny, I had totally forgotten about this note: + +- [Using libgccjit?](250920-libgccjit.html) + +Yes, I will most definitely be using libgccjit. If Zisp is to be a +true [full-stack language](260102-full-stack.html) then it must be +able to produce code rivaling C in efficiency, and that requires +either GCC or LLVM. + +Some of the other considerations in the above linked note, like the +"ZispScript" idea, are obsolete. Unless I've totally goofed up and +planned some illogical nonsense above, I'll be going with what I've +written here, not in the previous note. diff --git a/notes/index.md b/notes/index.md index 197261f..afc627d 100644 --- a/notes/index.md +++ b/notes/index.md @@ -27,3 +27,5 @@ * [Simplifying S-Expression Grammar](260106-simpler-grammar.html) * [Decoder](260107-decoder.html) * [What not to intern?](260109-uninterned.html) +* [JOKE RANT](260516-joke-rant.html) +* [Interpreter](260522-interpreter.html) |