This project is inspired by the Deep Red film trilogy from the creators of "Iron Sky". It aims to create a custom LLM model that has only knowledge up to July 1969 - the launch date of the fictional Mars mission in the movie - and responds in a style that aligns with a "Soviet utopia" setting controlled by the fictional "chess playing" AI.
These steps document how to perform a Model Training from Scratch procedure using a continued pre-training approach for temporal and thematic alignment.
- How to setup an AMD "Strix Halo" device (Fedora) — primary development and training system; includes automated setup script
- How to setup an optional second NVIDIA device (A4000) — dedicated GPU for training/inference; includes automated setup script
- How to set up the Wikipedia MCP server and data pipeline — extraction, indexing, search, and MCP server for Wikipedia content
- How to extract year-based historical topics from Wikipedia — enriched event data for temporal training
- How to augment Wikipedia with temporal metadata — YAGO/Wikidata parsing, normalization, and database augmentation for time-period filtering
- How to retrieve Project Gutenberg literature — thematically relevant books for training data
- How to prepare the chess training corpus — chess content retrieval, PGN conversion, and corpus preparation
- How to augment the chess game corpus - corpus augmentation to create custom chess-game content for the Deep Red persona
- How to tokenize and prepare the training corpus — tokenization, shuffling, and train/val splitting for continued pre-training
- How to train the Deep Red model — continued pre-training on the temporally-filtered pre-1969 corpus using dev (SmolLM2-360M) or prod (TinyLlama-1.1B) profiles
- How to train Deep Red on Gemma-3 (SFT) — parallel supervised fine-tuning track using Gemma-3-4B-IT / 12B-IT via TRL
SFTTrainer, mirroring the kyuz0/amd-strix-halo-llm-finetuning setup - How to use the generated GGUF models with LM Studio — loading, testing, and comparing trained model checkpoints in LM Studio
- How to evaluate checkpoints and plan recovery — independent 1969 probe bank, corpus contamination audit, GPU-accelerated trajectory evaluation across archived checkpoints, and the evidence-driven plan for temporal and persona training
- How to run the Phase 2 rebuild — step-by-step runbook for corpus generation, auditing, training and gating, with measured generator selection and throughput
- Prototypes
- DeepRed-gemma-4b-2026-05-23-5-final.gguf - see corpus and model details
- gemma-4b-balanced-v1-small-1500-final.gguf - see balanced run details and examples
- gemma-4b-temporal-v1-10d-final.gguf - temporal-cutoff run (reintroduces
retain/unlearnat the 1969-07-20 cutoff); see run details and examples.
- Production
- gemma-4b-temporal-v1-10d-2-final.gguf - Full 10-day follow-up calibrated from previous run: 2026-06-17 runbook (2.32 GB)
- Intermediate checkpoints (10%, 25%, 50%, 75%):
Evaluation note: a 1,377-generation evaluation across 17 archived checkpoints found that none of these models meets the project goal. The temporal checkpoints suppress modern facts only by refusing broadly — they also refuse many pre-1969 questions — while the balanced and prototype checkpoints retain modern knowledge entirely. Results, measurements, and the revised training plan are in the evaluation and recovery plan.
Prebuilt chess augmentation archives can be downloaded directly from:
https://www.ferzkopp.net/Data/chess_games.jsonl.gzhttps://www.ferzkopp.net/Data/augmented_chess_games.jsonl.gz
Each URL uses the pattern https://www.ferzkopp.net/Data/[filename].
For augmentation workflow details, see the full guide:
documentation/ChessAugmentation-Setup.md.
/documentation- Setup guides and planning documents for the project/evaluation- Model registry and independent probe bank used by the 1969 evaluation harness/notebooks- Jupyter notebooks for testing embeddings and OpenSearch functionality/patches- System patches (network driver fix for AMD Strix Halo for older kernels)/scripts- Python scripts for Wikipedia extraction/indexing, temporal augmentation (YAGO/Wikidata), Gutenberg and chess content retrieval, MCP server, and system setup/services- Systemd service files for automated startup (inference servers, MCP server, OpenSearch, web GUI)/tests- Unit tests for the evaluation harness/webapp- React-based web interface for Wikipedia search with Vite configuration