Home Compiler Pipeline

This document describes the complete compilation pipeline for the Home programming language, including all integrated features.

Overview

The Home compiler implements a sophisticated multi-pass architecture that includes lexical analysis, parsing, macro expansion, compile-time evaluation, type checking, borrow checking, and native code generation.

Compilation Stages

1. Lexical Analysis (Lexer)

Location: packages/lexer/src/lexer.zig

The lexer tokenizes source code into a stream of tokens, identifying:

• Keywords (fn, let, struct, if, match, etc.)
• Identifiers and literals
• Operators and punctuation
• Comments (automatically stripped)

Output: Array of tokens with location information

2. Parsing

Location: packages/parser/src/parser.zig

The parser constructs an Abstract Syntax Tree (AST) from the token stream.

Key Features:

• Macro Expansion: Built-in macros (todo!, assert!, unreachable!, debug_assert!, unimplemented!) are expanded directly during parsing into their equivalent AST representations
• Error Recovery: Continues parsing after errors to report multiple issues
• Module Resolution: Handles import statements and module dependencies

Macro Expansion Examples:

// Input:
todo!("implement feature");

// Expanded to:
panic("not yet implemented: implement feature");

// Input:
assert!(x > 0, "x must be positive");

// Expanded to:
if (!(x > 0)) {
    panic("assertion failed: x must be positive");
}

Output: Complete AST (ast.Program)

3. Compile-Time Evaluation

Location: packages/comptime/

The comptime executor initializes before type checking to prepare for compile-time code execution.

Features:

• Type Reflection: Query struct fields, type properties, sizes, and alignments
• String Operations: concat, toUpper, toLower, trim, split, join (16 operations)
• Array Operations: map, filter, reduce, sum, min, max (15 operations)
• Compile-Time Constants: Values computed at compile time and embedded in the binary

Integration: The ComptimeExecutor is initialized and used by the type checker to evaluate comptime blocks and expressions.

Example:

comptime {
    const numbers = [1, 2, 3, 4, 5];
    const sum = array.sum(numbers);  // Computed at compile time: 15
    pub const TOTAL = sum;  // Exported as a constant
}

4. Type Checking

Location: packages/types/src/type_system.zig

The type checker verifies type correctness and performs type inference.

Features:

• Type Inference: Deduces types from context when not explicitly specified
• Generic Type Resolution: Handles parametric polymorphism
• Trait Bounds: Validates trait implementations
• Comptime Integration: Evaluates compile-time expressions during type checking

Output: Typed AST with resolved types

5. Borrow Checking

Location: packages/compiler/src/borrow_checker.zig

The borrow checker enforces Rust-style ownership and borrowing rules.

Features:

• Ownership Tracking: Ensures each value has a single owner
• Lifetime Analysis: Validates that references don't outlive their referents
• Move Semantics: Prevents use-after-move errors
• Aliasing Rules: Enforces XOR between mutable and immutable borrows
• Enhanced Diagnostics: Provides Rust-quality error messages with suggestions

Example Error:

error[E0502]: cannot borrow `x` as mutable because it is also borrowed as immutable
  --> main.home:25:5
   |
23 |     let r = &x;
   |             -- immutable borrow occurs here
25 |     let m = &mut x;
   |             ^^ mutable borrow occurs here
26 |     println(r);
   |             - immutable borrow later used here

Output: Borrow-checked AST (compilation fails if borrow check errors occur)

6. Code Generation

Location: packages/codegen/src/

The code generator produces native machine code or IR.

Backends:

• LLVM Backend: Generates optimized native code via LLVM
• Kernel Mode: Special codegen for bare-metal kernel development
• Interpreter Backend: Direct AST interpretation for REPL and testing

Optimizations:

• Constant folding
• Dead code elimination
• Inline expansion
• SIMD vectorization (via LLVM)

Output: Native object files (.o) or executables

7. Incremental Compilation Cache

Location: packages/cache/src/incremental_compiler.zig

The cache system dramatically speeds up incremental builds.

Features:

• Content-Based Fingerprinting: SHA-256 hashes detect real changes (not just timestamps)
• Dependency Tracking: Automatically invalidates dependent modules
• Artifact Management: Caches IR, object files, and metadata
• LRU Eviction: Manages cache size (default 1GB limit)

Workflow:

1. Compute SHA-256 fingerprint of source file
2. Check if fingerprint matches cached version
3. If match and dependencies unchanged: use cached artifacts (< 1ms)
4. If mismatch: recompile and update cache

Performance: 10x+ speedup on incremental builds for large projects

8. Diagnostics & Error Reporting

Location: packages/diagnostics/src/enhanced_reporter.zig

The diagnostics system provides compiler-quality error messages.

Features:

• Rich Visual Formatting: Color-coded errors, warnings, and notes
• Source Context: Shows relevant code lines with line numbers
• Multi-Character Spans: Highlights full identifiers (^~~~)
• Contextual Labels: Primary, secondary, and note annotations
• Actionable Suggestions: Provides fix recommendations with code replacements

Example:

error[E0308]: mismatched types
  --> main.home:10:5
   |
 8 |     let x: i32 = 42;
 9 |     let y: string = "hello";
10 |     let z = x + y;
   |                 ^ expected `i32`, found `string`
   |
  help: use parseInt() to convert strings to integers
  help: try converting with parseInt
   |
   | parseInt(value)

Complete Pipeline Visualization

Source Code (.home)
    │
    ↓
┌───────────────────────────┐
│  1. Lexer                 │  Tokenization
│  - Keywords, identifiers  │
│  - Operators, literals    │
└───────────┬───────────────┘
            │ Tokens
            ↓
┌───────────────────────────┐
│  2. Parser                │  AST Construction
│  - Syntax analysis        │
│  - Macro expansion        │  ← todo!, assert!, etc.
│  - Module resolution      │
└───────────┬───────────────┘
            │ AST
            ↓
┌───────────────────────────┐
│  3. Comptime Executor     │  Compile-Time Evaluation
│  - Type reflection        │
│  - String/array ops       │
│  - Constant folding       │
└───────────┬───────────────┘
            │ AST + Comptime Values
            ↓
┌───────────────────────────┐
│  4. Type Checker          │  Type Analysis
│  - Type inference         │
│  - Generic resolution     │
│  - Trait validation       │
│  - Comptime evaluation    │
└───────────┬───────────────┘
            │ Typed AST
            ↓
┌───────────────────────────┐
│  5. Borrow Checker        │  Ownership & Borrowing
│  - Ownership tracking     │
│  - Lifetime analysis      │
│  - Move semantics         │
│  - Aliasing rules         │
└───────────┬───────────────┘
            │ Verified AST
            ↓
┌───────────────────────────┐
│  6. Code Generator        │  Native Code
│  - LLVM backend           │
│  - Optimization passes    │
│  - Object file emission   │
└───────────┬───────────────┘
            │ .o / executable
            ↓
┌───────────────────────────┐
│  7. Cache System          │  Artifact Storage
│  - SHA-256 fingerprinting │
│  - Dependency tracking    │
│  - LRU eviction           │
└───────────────────────────┘

Cross-Cutting Concerns

Diagnostics (All Stages)

The enhanced diagnostics reporter is integrated throughout the pipeline:

• Parser: Syntax errors with source context
• Type Checker: Type mismatches with suggestions
• Borrow Checker: Ownership violations with detailed explanations
• All Stages: Color-coded severity levels (Error, Warning, Note, Help)

Incremental Compilation (Build System)

The cache system optimizes the entire pipeline:

• Before Parsing: Check if source fingerprint matches cache
• Cache Hit: Skip compilation, use cached artifacts (< 1ms)
• Cache Miss: Run full pipeline, update cache
• Dependency Changes: Invalidate and recompile affected modules

Feature Integration Summary

Example: Full Compilation

Input (main.home)

comptime {
    const numbers = [1, 2, 3];
    pub const SUM = array.sum(numbers);  // Computed: 6
}

fn divide(a: i32, b: i32): i32 {
    assert!(b != 0, "division by zero");
    return a / b;
}

fn main(): i32 {
    let result = divide(10, SUM);
    return result;
}

Compilation Steps

1. Lexer: Tokenizes source → 45 tokens
2. Parser:
   • Parses AST
   • Expands assert!(b != 0, "division by zero") → if (!(b != 0)) { panic(...) }
3. Comptime:
   • Evaluates array.sum([1, 2, 3]) → 6
   • SUM = 6 (compile-time constant)
4. Type Checker:
   • Infers result: i32
   • Validates divide(10, 6) call
5. Borrow Checker:
   • No borrows to check (only value types)
6. Code Generator:
   • Generates optimized native code
   • Inlines divide(10, 6) → direct computation
7. Cache:
   • Stores artifacts with SHA-256 fingerprint
   • Next build: instant if unchanged

Output

• Executable binary
• Cached artifacts in .home-cache/
• Zero runtime overhead for comptime evaluation

Performance Characteristics

First Build (Cold Cache)

• Small project (< 1000 LOC): ~200ms
• Medium project (1K-10K LOC): ~2s
• Large project (10K-100K LOC): ~20s

Incremental Build (Warm Cache, 1 file changed)

• Cache hit (unchanged files): < 1ms per file
• Recompile (changed file): ~200ms for typical file
• Speedup: 10x-100x depending on project size

Memory Usage

• Compiler: _{50MB base +} 100KB per 1000 LOC
• Cache: ~1GB default limit (configurable)
• Runtime: Zero overhead for comptime features

Testing

Each component has comprehensive tests:

# Test macro expansion
./pantry/.bin/zig test packages/macros/tests/macro_test.zig

# Test comptime evaluation
./pantry/.bin/zig test packages/comptime/tests/comptime_test.zig

# Test borrow checker
./pantry/.bin/zig test packages/compiler/tests/borrow_check_test.zig

# Test diagnostics
./pantry/.bin/zig test packages/diagnostics/tests/enhanced_reporter_test.zig

# Test incremental compilation
./pantry/.bin/zig test packages/cache/tests/incremental_test.zig

# Full integration test
home build examples/test_borrow_checking.home
home build examples/test_macros.home
home build examples/test_comptime.home

Future Enhancements

Potential improvements to the pipeline:

1. Parallel Compilation: Compile independent modules concurrently
2. Link-Time Optimization (LTO): Cross-module optimizations
3. Profile-Guided Optimization (PGO): Use runtime profiles to guide optimization
4. Distributed Compilation: Share cache across machines
5. Custom Macro Expanders: User-defined procedural macros
6. Advanced Comptime: Compile-time code generation and reflection

Conclusion

The Home compiler is a multi-pass pipeline that integrates macro expansion, compile-time evaluation, ownership checking, and incremental compilation. Many pieces are still maturing — see the Capability Matrix for what is stable today.

Key achievements:

• ✅ Rust-quality borrow checking
• ✅ Compile-time evaluation with type reflection
• ✅ Built-in macro system
• ✅ Enhanced diagnostics with suggestions
• ✅ Incremental compilation (10x+ speedup)
• ✅ Zero-cost abstractions

Home is under active development; not all stages are production-quality yet. See the Capability Matrix for the current status of each component.