Basic Python Compiler

Basic Python Compiler is a lightweight compiler implementation for a Python-like high-level programming language. The project explores how source code moves through the core stages of a compiler, from lexical analysis and parsing to semantic validation and execution.

Rather than relying on an existing Python interpreter, the project defines its own language syntax, semantics, validation rules, and execution flow, providing a practical foundation for understanding compiler construction and language implementation.

High-level programming languages hide a large amount of complexity behind a simple programming experience. A statement such as:

result = factorial(5)

requires multiple stages of interpretation before a computer can execute it.

The challenge was to build a compact compiler that could:

• Define and recognize a controlled programming syntax
• Convert source code into structured representations
• Detect syntactic and semantic errors
• Resolve variables, functions, and scopes
• Validate program meaning before execution
• Execute valid programs through a dedicated runtime

The project therefore focuses on making the complete compilation pipeline explicit and understandable.

The project was intentionally designed as a basic compiler rather than a complete Python implementation.

Key constraints included:

• Supporting a defined subset of Python-like syntax
• Keeping the compiler lightweight and understandable
• Separating compilation stages cleanly
• Providing meaningful diagnostics for invalid programs
• Handling scope and symbol resolution explicitly
• Avoiding dependence on Python's native execution model
• Maintaining an architecture that could be extended with additional language features later

The goal was not to reproduce the entire Python language, but to build a coherent end-to-end compiler pipeline.

The compiler was designed as a modular pipeline in which every stage has a clearly defined responsibility.

Source code first passes through lexical analysis, where raw characters are transformed into tokens. These tokens are then consumed by a parser that constructs an Abstract Syntax Tree (AST) representing the structure of the program.

The AST is subsequently analyzed semantically to validate identifiers, scopes, function usage, and other language rules. Once validation succeeds, the resulting representation is passed to the runtime layer for execution.

This separation keeps syntax, semantics, and execution independent and makes the compiler easier to extend and debug.

The compiler follows a staged architecture:

Source Code → Lexer → Parser → AST → Semantic Analyzer → Intermediate Representation → Runtime / Executor → Output

Lexer

Converts source characters into structured tokens such as:

• Keywords
• Identifiers
• Operators
• Literals
• Parentheses
• Delimiters

Parser

Consumes the token stream and constructs an Abstract Syntax Tree, representing relationships between expressions, statements, functions, conditions, and operations.

Semantic Analyzer

Validates the meaning of the program after syntactic correctness has been established.

It handles concepts such as:

• Symbol resolution
• Scope management
• Function definitions
• Function calls
• Variable references
• Type consistency
• Return semantics

Intermediate Representation

Provides a structured representation between analysis and execution, allowing the compiler to evolve without tightly coupling the parser directly to the runtime.

Runtime / Executor

Executes validated programs using runtime state such as:

• Call frames
• Variables
• Function scopes
• Expression evaluation
• Control flow
• Program output

Diagnostics

Compiler errors are treated as a dedicated concern, allowing invalid source code to produce understandable feedback instead of failing unpredictably during execution.

The intelligence of the compiler is not based on machine learning. It comes from the language-processing logic implemented across its analysis stages.

The main intelligence layer consists of:

Lexical Intelligence

The lexer understands the language's vocabulary and transforms raw source text into meaningful tokens.

Structural Intelligence

The parser determines how individual tokens combine into valid language constructs and builds the corresponding AST.

Semantic Intelligence

The semantic analyzer determines whether syntactically valid programs actually make sense within the language rules.

For example, it can reason about:

• Whether a variable exists
• Whether a function can be called
• Which scope contains an identifier
• Whether a return statement is valid
• Whether expressions satisfy expected constraints

Runtime Intelligence

The executor maintains program state while evaluating expressions, resolving variables, managing call frames, and following control flow.

Together, these layers transform static source text into executable behavior.

1. Pipeline-Based Compiler Design

Each compiler phase has a clear contract with the next phase. This prevents parsing, validation, and execution logic from becoming tightly coupled.

2. AST as the Central Structural Representation

Using an Abstract Syntax Tree makes the language structure explicit and gives later stages a stable representation to work with.

3. Dedicated Semantic Analysis

Semantic validation is kept separate from parsing. This allows the parser to focus on grammatical structure while the semantic layer focuses on meaning.

4. Explicit Symbol and Scope Management

Variables and functions are resolved through explicit symbol information rather than relying on implicit runtime behavior.

5. Dedicated Runtime State

Execution uses structured runtime state and call frames, making recursive functions and nested execution contexts easier to reason about.

6. Modular Extensibility

The architecture allows additional language constructs, operators, types, and runtime capabilities to be introduced without redesigning the entire compiler.

7. Diagnostics as a First-Class Concern

Error handling is considered part of the compiler architecture rather than an afterthought, improving debugging and developer experience.

Building the Basic Python Compiler involved several challenges that required careful separation of language design, parsing, semantic validation, and runtime execution.

• Designing a consistent language specification — Defining a practical Python-like syntax and semantics while keeping the language small enough to implement and reason about clearly.
• Building a reliable lexer — Converting raw source code into an accurate token stream while correctly handling keywords, identifiers, literals, operators, delimiters, and invalid input.
• Constructing the Abstract Syntax Tree — Translating tokens into a structured AST while preserving operator precedence, nesting, function definitions, conditional logic, expressions, and return statements.
• Separating syntax from semantics — A program can be syntactically valid while still being semantically incorrect. Keeping these concerns separate required explicit validation rules and compiler-stage boundaries.
• Managing scopes and symbols — Correctly resolving variables and functions across global and local scopes, particularly when dealing with function parameters and recursive calls.
• Designing the runtime execution model — Maintaining execution state, call frames, variables, function calls, and return values without relying on Python's native execution engine.
• Handling meaningful compiler diagnostics — Detecting errors at the appropriate stage and presenting them in a way that makes the underlying source-code problem understandable.
• Maintaining extensibility — Designing the compiler so that additional language constructs and runtime capabilities can be introduced without tightly coupling the lexer, parser, semantic analyzer, and executor.

These challenges made the project more than a simple interpreter exercise: they required designing a coherent end-to-end language implementation pipeline, from source code all the way to execution.

The project resulted in a complete educational compiler pipeline capable of taking Python-like source code through the major stages of language implementation:

Source → Tokens → AST → Semantic Validation → Runtime Execution → Output

It demonstrates the complete lifecycle of a programming language implementation while keeping the individual compiler components understandable and independently extensible.

The project also establishes a strong foundation for future capabilities such as richer type systems, additional control-flow constructs, bytecode generation, optimization passes, improved diagnostics, and a more sophisticated runtime.