🧪 Liquid State Transformer | Continuous-Time Auditable Neural Network Model

An auditable, mathematically formal, continuous-time Transformer implementation integrating criticality theory with the robust dynamics of Liquid State Machines to solve complex temporal modeling.

📑 Table of Contents

• Overview
• Key Features
• Architecture / Repository Structure
• Installation & Setup
• Usage
• Issues & Support
• Contributing
• License

🚀 Overview

This repository marries the exact mathematical formulations defined in "Formal Algorithms for Transformers" (Mary Phuong & Marcus Hutter, arXiv:2207.09283) with the rich, non-linear dynamics of Liquid State Machines (LSM) and criticality theory.

$ python generate_plots.py
[INFO] Initiating burn-in phase for criticality alignment...
[INFO] Liquid State Transformer successfully reached critical threshold.
[INFO] Training on Lorenz96 non-linear dynamics...
[SUCCESS] Generated 3D liquid state criticality landscapes.

🌟 Key Features

• Auditable Formal Transformer: Implements exact causal multi-head attention, layer normalization, and MLP components exactly as described by Hutter et al., completely eliminating approximation errors.
• Liquid Criticality Dynamics: Enforces sophisticated mathematical criticality constraints across transformer layers using continuous-time metric evaluation.
• Custom High-Performance Kernels: Built with the helion framework, featuring online softmax (FlashAttention-style) kernel auto-tuning for mathematically pure, highly parallel sequence processing.
• Chaos System Evaluation: Evaluated on ultra-high-resolution Lorenz96—a highly non-linear chaotic dynamic system—demonstrating the model's capacity to represent and predict complex temporal data.
• Formal Verification: Includes an extensive LaTeX-based mathematical certification system ensuring implementation correctness.

🏗️ Architecture / Repository Structure

The Auditable Kernel

The implementation of attention completely eschews PyTorch's native approximate or fast attention mechanisms in favor of an explicit, auditable loop structure. It maps directly to:

• Algorithm 2: Masking
• Algorithm 3/4: Formal Multi-Head Attention via stable Online Softmax.
• Algorithm 10: Complete Formal D-Transformer.

The Continuous Liquid State

By wrapping the formal transformer in continuous bounds, the model achieves the dynamic stability of a reservoir computer (LSM) combined with the representational capacity of a deep transformer. A specialized 17-dimensional metric vector ($M_0$ through $M_{16}$) tracks Jacobian eigenvalues, trace variants, and attention entropy to keep the system at the edge of chaos.

Repository Files

• auditable_transformer.py: Core framework implementing exact, auditable kernels.
• liquid_state_transformer.py: The continuous model bridging formal transformers with ODE evaluations.
• generate_plots.py: Orchestrates training on the Lorenz96 system and visualizations.
• certification.tex: LaTeX source for formal mathematical certification.
• extract.py: Utility script for diagram extraction.
• recreate_env.sh: Shell script to set up the environment.
• Makefile: Build tool for certification docs.

💻 Installation & Setup

Ensure you have a modern GPU-accelerated environment with Python 3.10+. The framework depends on standard scientific libraries alongside the custom helion kernel engine.

To set up the environment and install all dependencies, run the provided script:

chmod +x recreate_env.sh
./recreate_env.sh

💡 Usage

Training and Visualization (Lorenz96 Dynamics)

To train the model on the chaotic Lorenz96 system and visualize the resulting 3D liquid state criticality landscapes:

python generate_plots.py

Note

This script performs an initial "burn-in" phase to align weights with optimal critical metrics before executing the main training loop.

Compiling the Certification Document

To generate the mathematical certification and proof document (certification.pdf), ensure you have a standard TeX Live distribution installed:

make

🐛 Issues & Support

If you encounter any issues while setting up the environment, running the models, or compiling the LaTeX documentation, please open an issue in the repository. We aim to support researchers utilizing our mathematically formal framework.

🤝 Contributing

We welcome contributions from the community to enhance the Liquid State Transformer and its theoretical foundations!

1. Fork the repository.
2. Create a new branch (git checkout -b feature/your-feature-name).
3. Commit your changes (git commit -m 'Add some feature').
4. Push to the branch (git push origin feature/your-feature-name).
5. Open a Pull Request.

Please ensure all formal verifications are updated if core architectures are modified.

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

---

Built for absolute mathematical clarity in continuous-time modeling. Check out my other repositories for more open-source projects.