# Reader? Decoder? I barely know 'er!

*This started from an expansion to the following, then became its own
article:*

[Symbols are strings are symbols](symbols.html)

OK but hear me out... What if there were different reader modes, for
code and (pure) data?

I want Zisp to have various neat [syntactic extensions](sugar.html)
for programming purposes anyway, like the lambda shorthand, and these
shouldn't apply to configuration files, either.  (Although they seem
unlikely to occur by accident.)

So what if the transform from string literal to quoted string literal
only occurred in code reading mode?

At least one problem remains, which is that `'(foo "bar")` would turn
into `(quote (foo (quote bar)))` because the reader would be in code
mode while reading it...

This reminds me of the long-standing annoyance in Scheme that "quote
is unhygienic" and maybe we can tackle this problem as well now.

Also, expressions like `'(foo '(bar))` always seemed weird to me, and
probably have no place in Scheme, because we don't generate code via
quote; we generate it with macros that operate on explicit syntax
objects rather than pure data.

I want to experiment with an idea like this:

    ;; "code reader mode" transformations

    '(foo bar)     -> (#quote foo bar)
    
    '(foo 'bar)    -> ERROR

    "foo"          -> (#quote . foo)

    "foo bar"      -> (#quote . "foo bar")

    '(foo "bar")   -> (#quote foo bar)

    '(foo "x y")   -> (#quote foo "x y")

The right-hand side shows what you would get if you read the form on
the left in code reader mode, then wrote it back out in data mode.

The writer could also have a code writer mode, which applies the
reverse transformations.  There should be a one to one mapping,
unambiguous, so this always works.  A "hygienic" way to quote is
imperative here, since the writer could otherwise not know whether
some `quote` keyword in a list is *the* quote special form, or just
part of some data.

We've made quote into a special token, `#quote`, to solve that.
Instead of adding a separate symbol data type that's a subtype of
strings, I think I'll add something called a "rune" or such that's
represented like `#foo` and allows for custom reader extensions, or
rather, writer extensions.

Essentially, these runes would be bound to a pair of the following:

1. A procedure that accepts a datum and returns some type of value.

2. A procedure that takes values of that type, and turns them back
   into written format.

For `#quote`, the reader procedure would be the identity function.
The writer procedure would need to be a little more sophisticated.

Note that the first procedure would not actually be called during
reading of data.  Somewhat confusingly, it would only be called in
evaluation of code.

Let's recap.  Starting with pure data reading and writing:

1. There is no special reader syntax.  This s-expression format is a
bare minimum of what's needed to represent sequential data i.e. lists
and lists are the only compound data type recognized by the reader.
Anything that isn't a list is either an atomic value, or a string
which may or may not be considered atomic depending on how pedantic
you want to be.  Oh and runes are allowed.

   A. This is basically "classic" s-expressions with runes added.
   Only lists, numbers, and strings/symbols are recognized.

   B. Heck, numbers may not be recognized.  Or maybe they will be
   limited to integers and floats, but no rationals or such, and
   reading a float will guarantee no loss of precision?

2. Writing data returned by the data reader back out, in data form,
will produce exactly what was read, with the sole exception being
whitespace differences.  The data is not allowed to contain any
non-atomic values other than proper lists.

   A. It's important not to allow floats that IEEE 754 doubles can't
   represent, since then differences between input and output would
   occur.  But what about numbers like "10.00"?  That would also
   become something else when written back out.

   B. OK, maybe numbers represented in a non-canonical way are a
   second source of difference between reading and writing back out,
   but let's at least guarantee there's no loss of precision.

(I've not considered comments.  Maybe they will be preserved?  Maybe
they should be implemented as code reader syntax sugar as well??)

And now code reading and writing:

1. Various syntax sugar is internally transformed into runes, with
non-list compound data literals (vectors, hash tables, etc.) needing
this type of representation to appear in code.

   A. Writing that data back out in data mode will reveal the inner
   workings of the language, producing output containing runes.

   B. Direct use of runes may be forbidden; not sure about this.

   C. Evaluating this data containing runes will produce, in-memory,
   the actual values being represented.  The "reader procedure" tied
   to the rune is responsible for this, though the fact that it's
   evaluation and not reading that calls that procedure makes it
   confusing so a better name is needed.  Maybe just "decoder."

2. For every data type that falls outside the pure data syntax, there
is a procedure that turns it into a canonical data representation
based on lists and atomics, always using the format `(#rune ...)`.

   A. Another procedure is capable of turning that back into reader
   sugar, but this is not terribly important.  Although it would be
   neat to be able to write out code that looks like hand-written
   program code, this really is just a bonus feature.

   B. For some types, turning them back into code without any runes
   may be highly complicated; procedures, in particular, would need
   decompilation to make this work.


## Recap (or not?)

Wow, that was a long "recap."  I actually came up with new ideas in
writing that.  Let's recap the recap.  I'll represent the mechanisms
as different pipelines that can happen using the various features.

Typical pipeline when reading and evaluating code:

    code-file --[code-reader]--> code-data --[eval]--> values
                 ^^^^^^^^^^^                  ^^^^
              turns sugar into        calls rune decoders
                 rune calls            to produce values
            i.e. desugars code          & compiles code

Reading in a [serialized program](compile.html):

    data-file --[data-reader]--> data --[eval]--> values
                                         ^^^^
                                    fairly trivial
                                (no lambdas, only runes)

Reading pure and simple data like a config file:

    data-file --[data-reader]--> data (no runes to eval)

Note that "data" is a subset of "values" basically.  And the term
"code-data" which was used above just means data that is meant to be
evaluated as code, but is totally valid as pure data.  This is not to
be confused with the "data" that existed in the intermediate step
while we were reading a serialized program; that was absent of any
forms like lambdas that need compilation.

OK, that last bit was a bit confusing, and I realize it stems from
conflating rune decoding with code compilation, so let's split that
further up.  Above, "eval" is "decode + compile" basically, but it's
possible to separate them, for example if we want to read a file of
serialized values that should not contain any code:

    values-file --[data-reader]--> values-data --[decode]--> values

This is a secure way to read complex data even if it comes from an
untrusted source.  It may contain runes that represent code, such as
in the form of `(#program "binary")` (compiled procedure) or even
`(#lambda (x) (do-things))` but so long as you don't actually call
those things after having decoded them, they can't do anything.
Decoding runes can't define macros or register new rune decoders,
meaning there's no way to achieve arbitrary code execution.

Heck, although `#lambda` exists to represent the desugaring of the
`{...}` convenience syntax, it wouldn't actually work here because
decoding runes would happen in a null-environment without any bound
identifiers, meaning that e.g. `(#lambda (x) (+ x x))` would just
raise an error during decoding, because the compiler would consider
`+` unbound.

Alternatively, instead of calling the compiler, the `#lambda` decoder
could just be a no-op that returns the same form back, but without the
rune, like `(lambda (x) (+ x x))`, because the compiler will take care
of that later.  Yeah, I think this makes more sense.  Why doesn't the
code reader directly give `(lambda ...)` for the `{...}` sugar?  Well,
actually, the `#lambda` decoder may yield a syntax object where the
first element specifically refers to the binding of `lambda` in the
default environment, so you could use `{...}` in an environment where
`lambda` is bound to something else, and you would still hygienically
get the default lambda behavior from `{...}`.  Yay!

(Wow, it's rabbit hole after rabbit hole today.  This is good though.
I'm coming up with some crazy stuff.)

It would be possible to decode "code-data" and get an internal memory
representation of an uncompiled program which however already has
various data structure literals turned into values.  This is super
obscure but for sake of completeness:

    code-file --[code-reader]--> code-data --[decode]--> code-values

(These so-called "code-values" would only ever be useful for piping
them into the compiler.  By the way, I initially used "eval" in the
example of reading a serialized program, but "decode" would have been
sufficient there.)


## Here's a well-deserved break

(There wasn't a new header in a while.  This seemed a good spot.)

Now writing pipelines.  Let's reverse the above pipelines, from the
bottom back towards eventually the first...

The reverse of the super obscure thing above:

    code-values --[encode]--> code-data --[code-writer]--> code-file

That would only ever be useful for debugging things.  Now writing a
data structure into a serialized file, without unnecessarily invoking
the decompiler:

    values --[encode]--> values-data --[data-writer]--> data-file

That gives you a file containing only data, but the data is the
encoded format of various data structures Zisp recognizes...
Actually, that may include compiled procedures as well.

Now the simple config file case, being serialized:

    data -[data writer]-> data-file

Now serializing a compiled program to a file, without decompilation:

    values --[encode]--> values-data --[data-writer]--> data-file
              ^^^^^^                    ^^^^^^^^^^^
         data structures             no decompilation
        become rune calls            or "re-sugaring"

Oh, look at that.  It's the same as writing out data structures, as
we've already seen previously...  This recap of a recap will need
another recap for sure.

And now, the full decompiler:

    values --[uneval]--> code-data --[code-writer]--> code-file
              ^^^^^^
          decompilation

Actually, just like "eval" is "decode + compile", the "uneval" here
really is "decompile + encode".


## The Revised Recap of the Recap

The following exist:

1. Readers:

   1. Data reader: Reads lists, strings/symbols, runes, integers, and
   IEEE 754 double-precision floats without loss of precision.

   2. Code reader: Reads code that can contain various syntax sugar,
   all of which has an equivalent representation with runes.

2. In-memory transformers:

   1. Decoder: Calls decoders for runes in data, to yield values.

   2. Evaluator: [Executes aka compiles](compile.html) decoded values
   into other values.[*]

3. Reverse in-memory transformers:

   1. Encoder: Reverse of the decoder. (Lossless?)

   2. Unevaluator: Reverse of the evaluator. (Lossy.)

4. Writers:

   1. Data writer: Reverse of data reader. (Lossless.)

   2. Code writer: Reverse of code reader. (Lossy?)

(*) This needs decoding to run first, because otherwise it wouldn't
    realize that you're e.g. calling `+` on a pair of rational number
    constants represented through runes, so constant folding wouldn't
    work.  Same with `vector-ref` on a vector literal represented as a
    rune, and so on.


## How in the seven hells did I arrive at this point?

Jesus Christ!

This was about symbols and strings being the same thing.

But I love these rabbit holes.  They're mind expanding and you find
out so many new things you never thought about.

Did you notice, by the way, that the code reader/writer above is
essentially a parser (and unparser) you would have in a regular
programming language, where syntax becomes an AST?  The pure data
format is basically our AST!

But this doesn't mean we lost homoiconicity.  No, we merely expanded
upon it by providing a more detailed explanation of the relationship
between textual representation of code and in-memory data that exists
at various stages before ultimate compilation.

Oh, and did we achieve our strategy of strings = symbols now, or does
it need to be dropped?  I think we achieved it.  The code reader, as
described all the way up where the section "Reconsidering AGAIN"
begins --in the original article; see top-- will desugar string
literals into:

    "foo" -> (#quote foo)

(As already described originally...)

And the `#quote` rune?  Well, it will not actually just return its
operand verbatim, no!  It will return a syntax object that's a list
with the first element specifically refers to the binding of `quote`
from the standard library.  In other words, it's the evaluator that
actually implements quote, not the decoder.

Oh yes, this is very satisfying.  Everything is coming together.

Syntax objects, by the way, will also have a rune-based external
representation, so you can inspect the result of macro expansion.

And yes, I think using runes directly in code mode should be illegal,
because it allows referring to bindings in the standard library, or
even bindings in arbitrary libraries by crafting syntax objects
represented via runes, to bypass environment limits.

That bug actually existed in Guile at some point, where one could
craft syntax objects, represented as vector literals, to refer to
bindings in other modules, making it impossible to run code in a
sandboxed environment.  (It was fixed long ago, I believe.)

Oh, but what about `#true` and `#false`?  OK, maybe there will be a
whitelist of runes that are allowed in code.  That makes sense.

We will see.  Still more details to be fleshed out.

In any case, some runes must be able to declare that they don't take
arguments, in which case `(#rune ...)` isn't decoded by passing the
entire form to the decoder of `#rune`, but rather treated as a normal
list whose first element is decoded as a nullary rune.  That's how
boolean literals in code will be implemented.


## Looking at more of the initial problems

What happened to `'(quote "foo")` in code mode being weird?  Well,
encountering an apostrophe tells the code reader that the next
expression is a datum, so it switches to data mode for that.

Wow, that was easy.

This also means you can't use syntax sugar inside it, which is good
because as we said previously, we don't want to use quoting to create
code; we want to use syntax objects for that.

This is really orthogonal to the whole runes issue, and could have
been solved without that mechanism, but I'm happy I came up with it
because it resolves hygiene issues.

The syntax `#'(quote "foo")` would be sugar for a different rune, and
the reader would remain in code mode, further desugaring any sugar
found within, so this works: `#'{x (+ x x)}`

Oh and I mentioned reader extensions (for code mode) but then didn't
expand on that.  Well, whenever the code reader encounters this:

    #foo(blah blah blah)

It will turn that into:

    (#foo blah blah blah)

After which the decoder for `#foo` will be invoked, which could have
been registered by the programmer.

Can that registration be done in the same file though?  Normally, the
execution step comes after decoding, and we decided that we don't want
to allow arbitrary code execution to happen just when reading a data
file and decoding it.  So something exceptional would need to happen
for this to work.  Or maybe not.

Remember that [compilation is execution](compile.html) in Zisp,
meaning that compiling a file looks like this in pseudo-Scheme:

    (define env (null-environment))    ;start out empty
    
    (while (not (eof? input))
      (let* ((datum (read-code input)) ;desugar
             (value (decode datum)))   ;decode
        (eval! value env)))            ;eval in mutable env

    (write (env-lookup env 'main))     ;serialize

I've called eval `eval!` to indicate that it can mutate the env it
receives, which is what import statements and defines would do.

Let's modify that a little further to indicate the fact that reader
macros, or in our terms, custom rune decoders, can be defined in the
middle of the code file by affecting the environment:

    (define env (null-environment))      ;start out empty
    
    (while (not (eof? input))
      (let* ((datum (read-code input))   ;desugar
             (value (decode datum env))) ;decode in env
        (eval! value env)))              ;eval in mutable env

    (write (env-lookup env 'main))       ;serialize

Since the `decode` procedure is given an environment, it will look up
decoders from therein.  So, after the evaluation of each top-level
expression, the expressions coming after it could be using a custom
decoder.

What our reader macros cannot do is completely affect the lexical
syntax of the language, as in, add more sugar.  You must rely on the
global desugaring feature of `#x(...) -> (#x ...)` which, now that I
think about it, is completely useless because a regular macro could
have achieved exactly the same thing.

OK, let's try that again.  The global desugaring wouldn't work on
lists only, it would work on a number of things:

    #x"foo"    -> (#x #string . foo)

    #x[foo]    -> (#x #square . foo)

    #x{foo}    -> (#x #braces . foo)

You get the idea!

(I've changed my mind that `"foo"` should desugar into a call to the
regular `#quote` rune; it should be `#string` instead to disambiguate
from the apostrophe if needed.)

Also, all those would work without a rune as well, to allow a file to
change the meaning of some of the default syntax sugar if desired:

    "foo"    -> (#string . foo)

    [foo bar]    -> (#square foo bar)

    {foo bar}    -> (#braces foo bar)

Or something like that.  I'm making this all up as I go.