+++
title = "§18 What's Next: Extensions and Further Reading"
priority = 5
status = "done"
ticket_type = "task"
dependencies = []
+++

## §18 What's Next: Extensions and Further Reading — Stub to fill

File: `edu/src/lisp-compiler.md`, section `### 18. What's Next: Extensions and Further Reading`

Replace the stub line with full content. Target 600–800 words. Survey the directions the compiler can be taken and provide a curated reading list. Reading-only, no code.

## Learning objectives

- Understand what limitations the current compiler has and why
- Know the conceptual approaches for each major extension
- Have a reading list for going deeper into compiler theory and Lisp implementation

## Content to write

### Congratulations — and what you skipped

Open by acknowledging what the reader has built: a complete compiler with a lexer, parser, semantic analyser, code generator, and test suite. Then honestly catalog what was left out:

### Extension 1: Closures and Lambda Lifting

The current compiler does not support closures — lambdas cannot capture variables from enclosing functions. Adding closures requires **lambda lifting**: transforming each lambda that captures free variables into a top-level function that takes those variables as extra parameters. This is a classical technique. Real Lisp runtimes use **closure records** (a struct containing the function pointer and captured values) allocated on the heap.

### Extension 2: Tail-Call Optimisation (TCO)

`(define (loop n) (if (= n 0) n (loop (- n 1))))` will stack overflow for large `n` in the current compiler because each recursive call pushes a new C stack frame. TCO transforms tail calls into jumps. In C, this can be approximated with the `__attribute__((optimize("O2")))` pragma or by using a trampoline pattern. A proper solution requires detecting tail-call position during code generation and emitting a `goto` loop.

### Extension 3: A Type System

Add type inference (Hindley-Milner or a simpler Hindley-style bidirectional checker) so that type errors are caught before C is emitted. This would allow the code generator to choose the correct `ml_display_*` variant and generate proper C function signatures for string-returning functions.

### Extension 4: Pairs, Lists, and a Runtime

`(cons a b)`, `(car p)`, `(cdr p)` require heap-allocated pair objects — a proper C struct. This opens the door to proper Lisp list processing. Once you have heap allocation, you need a garbage collector. The simplest GC is reference counting; a more robust approach is mark-and-sweep.

### Extension 5: Macros

Lisp macros transform code before compilation. A simple approach is **syntax transformers**: functions that run at compile time and return transformed AST nodes. This requires a small interpreter for the macro language. Hygienic macros (as in Scheme) are significantly more complex.

### Extension 6: A REPL

A read-eval-print loop compiles and runs one expression at a time. This requires either an interpreter (easier) or incremental native code emission (harder). An interpreter over the AST is a natural extension once the parser is complete — it's essentially the code generator replaced with a recursive evaluator.

### Extension 7: Self-Hosting

The ultimate milestone: rewrite the MiniLisp compiler in MiniLisp itself. This requires the language to be expressive enough (strings, I/O, some form of list processing) and the compiler to be complete enough to compile itself. Self-hosting is the proof that you've really built something.

### Further Reading

**Compiler theory:**
- *Crafting Interpreters* by Robert Nystrom — free online; builds a language in two complete implementations (tree-walking and bytecode)
- *Modern Compiler Implementation in ML/Java/C* by Andrew Appel — classic academic compiler textbook
- *Engineering a Compiler* by Cooper & Torczon — comprehensive modern treatment

**Lisp implementation:**
- *Structure and Interpretation of Computer Programs* (SICP) — chapters 4 and 5 cover interpreters and compilers for Scheme
- *Lisp in Small Pieces* by Christian Queinnec — 11 different Lisp implementations, from interpreter to compiler
- *Build Your Own Lisp* by Daniel Holden — free online, C implementation

**Parsing:**
- *Parsing Techniques* by Grune & Jacobs — comprehensive reference (free PDF)
- nom documentation and recipes: https://github.com/rust-bakery/nom/tree/main/doc

**Rust and compilers:**
- The `cranelift` crate — a code generator backend (Rust, used in Wasmtime)
- `inkwell` crate — safe Rust bindings to LLVM for native code generation

## Style notes

- Open warmly — the reader has accomplished something real
- Each extension should be a paragraph: what it is, why it is non-trivial, and the key technique
- The reading list is the most durable part of this section; keep it current and annotated
- Close the course with encouragement: the concepts learned here (parsing, AST manipulation, code generation) apply to every compiler, transpiler, and language tool the reader will ever build