Skip to content
BranchesTags

Repository files navigation

People of the Medieval Levant — OUTREMER

CI Docker

A proof-of-concept pipeline for AI-assisted prosopography of the medieval Levant (Crusades era, 11th–14th centuries). Part of a collaborative research project by Jochen Burgtorf (Cal State Fullerton), Tobias Hodel (University of Bern), and Laura Morreale (Harvard / independent scholar).

Status: proof of concept. The pipeline runs end-to-end. All LLM calls route through the local GPUStack instance at gpustack.unibe.ch — no external third-party LLM calls (an optional Mistral OCR fallback exists for scanned PDFs, off unless mistralai is installed and MISTRAL_API_KEY is set).

Architecture

Layer 1 — LLM extraction. Reads historical texts (PDF or plain text) and extracts person-like signals: names, titles, epithets, roles, collective groups. Uses GPUStack-hosted models (Qwen3-30B-A3B for extraction, Qwen3-VL for scanned-PDF OCR, MiniMax-M2.7 for orchestration). Falls back to heuristic regex NER when GPUStack is unavailable, and optionally to Mistral OCR for scans.

Which engine produced the published data. GPUStack sits behind the university network: GitHub Actions cannot reach it and every extraction chunk returns 403 Forbidden. The nightly run therefore degrades to the heuristic regex extractor, and the data published to GitHub Pages is heuristic output, not Qwen3 output. Model-quality figures require a run from inside the network. Each document records what actually produced it in extraction_engine (gpustack / mixed / heuristic, with chunk counts), the run report aggregates it under extraction.documents_by_engine, and a degraded run emits a CI warning. Until 2026-07-30 this degradation was silent and the output was labelled gpustack regardless.

Layer 2 — KG linking. Fuzzy-matches extracted mentions against a curated authority file of known crusader persons. Returns ranked candidates with confidence scores and flags ambiguous or multi-candidate matches.

Results are published as a static GitHub Pages site with a Human-in-the-Loop review UI — scholars can accept, reject, or flag individual candidate links and export their decisions as JSON.

Repository structure

outremer/
├── data/
│   ├── raw/                       Source texts (.pdf, .txt)
│   ├── peerage_pre1500_export/    Wikidata peerage data (pre-1500 persons)
│   ├── entity_feedback.json       Filtered noisy entities
│   └── decisions.json             Human adjudication decisions
├── scripts/
│   ├── config.py                  GPUStack configuration (reads .env.gpustack)
│   ├── llm_client.py              Thin OpenAI-compatible GPUStack client
│   ├── run_pipeline.py            Main pipeline entry point
│   ├── extract_persons.py         Layer 1: extraction via GPUStack or regex fallback
│   ├── wikidata_reconcile.py      Layer 2: KG linking
│   ├── export_peerage_pre1500.py  Wikidata peerage export (QID → CSV)
│   └── install-triplestore.sh     Canonical Fuseki/GraphDB installer
├── scrapers/                      Historical web scrapers
├── bib/                           BibTeX output
├── docs/                          Living documentation
│   ├── LOCAL_LLM_ADAPTATION_PLAN.md   Full Epic 1–8 roadmap
│   ├── EPIC4_HBLS_MCP.md              HBLS MCP server docs
│   └── archive/                       Stale/historical docs
├── site/                          Static site (deployed to GitHub Pages)
│   ├── index.html
│   ├── app.js                     Explorer + H-i-t-L adjudication UI
│   └── data/                      Generated per-document JSON
├── .github/workflows/
│   ├── pipeline.yml               Runs extraction + linking on push / nightly
│   └── pages.yml                  Deploys site/ to GitHub Pages
├── requirements.txt
└── README.md

Setup

git clone https://github.com/thodel/outremer.git
cd outremer

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

GPUStack configuration

Copy .env.gpustack template (or create manually):

# All LLM calls route to GPUStack on gpustack.unibe.ch
GPUSTACK_BASE_URL=https://gpustack.unibe.ch/v1
GPUSTACK_API_KEY=your-token-here

# Optional local ATR recognition gateway
ATR_GATEWAY_URL=http://localhost:8200
ATR_API_KEY=your-gateway-token
ATR_HTTP_TIMEOUT=300

# Model names (check GPUStack dashboard for exact names)
EXTRACTION_MODEL=qwen3-30b-a3b-instruct
EXTRACTION_SEED=42
ORCHESTRATOR_MODEL=minimax-m2.7
QWEN3_VL_MODEL=qwen3-vl-30b-a3b-instruct

# OCR engine: qwen3-vl (GPUStack, default) or mistral (legacy fallback)
OCR_ENGINE=qwen3-vl

.env.gpustack is git-ignored. Without it, config.py uses sensible defaults (tei endpoint, no API key required for public models).

Running the pipeline

source .venv/bin/activate

# Standard run (uses .env.gpustack if present)
python scripts/run_pipeline.py --input-dir data/raw

# With GPUStack API key
export GPUSTACK_API_KEY=your-token
python scripts/run_pipeline.py --input-dir data/raw

# Language hint for multilingual sources
python scripts/run_pipeline.py --input-dir data/raw --language la    # Latin
python scripts/run_pipeline.py --input-dir data/raw --language ar    # Arabic
# Supported: la, fro (Old French), ar, el (Greek), de (Middle High German)

# Sync human adjudication into feedback memory
python scripts/run_pipeline.py --input-dir data/raw \
  --entity-feedback-path data/entity_feedback.json \
  --review-decisions-path data/decisions.json

# All options
python scripts/run_pipeline.py --help

OCR engines:

Engine How it works Speed Cost
qwen3-vl (default) GPUStack Qwen3-VL; falls back to Mistral if empty and available Fast Free (local)
mistral Mistral API only (legacy; pip install -e '.[ocr-mistral]' + MISTRAL_API_KEY) Fast Paid

Output: site/data/*.json, site/bib/*.bib, bib/*.bib.

Every processed document also produces a canonical evidence-first artifact at data/evidence/<document-id>.evidence.json. These records separate immutable source snapshots and passages from extracted mentions, assertions, identity hypotheses, and generation provenance. They are validated against the evidence-first JSON Schema and SHACL shapes before being published. Invalid canonical output fails the document run; the existing site/data JSON remains available as a temporary compatibility format.

The pipeline mirrors validated artifacts to site/evidence/ for the unified Explorer. Select a document in site/explorer.html to review source passages, assertions, identity hypotheses, candidate scores, and generation provenance beside the legacy link review. Documents without an evidence artifact continue to use the legacy interface. site/evidence-review.html remains available as a compatibility entry point and uses the same renderer and local review store.

Tests

pip install -r requirements-dev.txt
pytest tests -q

Evaluation

The evaluation/ package measures pipeline quality against gold fixtures, so prompt/model changes are judged by numbers rather than impressions. Gold is seeded from Human-in-the-Loop adjudications (data/decisions.json): scholar decisions become regression tests.

# Score the committed fixture snapshots (offline; also runs in CI)
python -m evaluation.harness

# Score the *current* site/data output — run after a pipeline change
python -m evaluation.harness --live

# Regenerate fixtures after new adjudications arrive
python -m evaluation.build_fixture

Extraction quality is measured separately, against the full-gold fixture (mode: "full", currently munro only, DRAFT pending scholar validation). Because CI cannot reach GPUStack, this scores the heuristic extractor — the engine that actually produces the published corpus. Rewriting it on 2026-07-30 moved munro from P 0.047 / R 0.318 / F1 0.082 to P 0.568 / R 0.955 / F1 0.712 (individual persons only; collectives are excluded from the person gold by design and scored separately). Recall is now the strong side: 21 of 22 gold persons are found. Measuring Qwen3 itself needs a run from inside the university network.

Key metric: linking agreement — of the pairs scholars reviewed, how many does the responsible system's top proposal agree with. Adjudications cover two systems, each judged against its own output: the authority-file linker (AUTH:CR… ids) and Wikidata reconciliation (wikidata:Q… ids). Pinned-seed baseline 2026-07-18: combined 0.9155 over 71 pairs (authority 0.8909 over 55, wikidata 1.0 over 16). Before seed pinning, unchanged code produced an observed combined band of approximately 0.8873–0.9296 (63–66 correct of 71); point differences inside that range must not be presented as improvements. The pinned baseline follows the #44 gold repair (two wrong-person accepts re-adjudicated to reject) and the #45 authority additions (Godfrey of Bouillon, Robert II of Flanders, Ralph of Caen). Residual misses are dominated by extraction drift, not linking — see issue #42.

Read agreement against the null baseline, not on its own. Agreement rewards proposing what scholars accepted and not proposing what they rejected. The authority gold is 7 accepts against 48 rejects, so a linker that proposes nothing at all already scores 48/55 = 0.873 on authority and 0.9014 combined. Measured authority agreement is 0.891 — a lift of just +0.018 over silence. The harness therefore reports null, lift, and accept_rate per system, and CI gates on --min-lift against the weakest segment; a combined threshold cannot fail, because Wikidata's accept-only gold (lift +1.0) masks any authority collapse. Until the gold repair (#98 — six of seven accepts link the wrong person) and gold growth (#36) land, treat the authority figure as uninformative about linker quality.

Where a backend does not honour EXTRACTION_SEED, generate repeated live outputs and evaluate them as a band:

python -m evaluation.harness --live --repeat 5 \
  --repeat-command "python scripts/run_pipeline.py"

Repeated gates use the lowest observed agreement, not the mean, so sampling variance cannot make a regression appear to pass.

Reviewing results

GitHub Pages: Auto-deployed on every push to main. https://thodel.github.io/outremer/

Locally:

cd site && python3 -m http.server 8080
# open http://localhost:8080

H-i-t-L workflow:

  1. Select a document from the dropdown and click Load.
  2. Extracted Persons panel lists all detected mentions with confidence scores.
  3. Links panel shows candidate matches ranked by fuzzy score (green = high, yellow = medium, red = low).
  4. Click ✅ Accept, ❌ Reject, or 🚩 Flag for each candidate.
  5. Filter bar focuses on unreviewed or flagged items.
  6. Export decisions downloads adjudications as outremer-decisions-YYYY-MM-DD.json.

Decisions are persisted in browser localStorage — survive page refreshes, scoped per-document.

From export to pipeline impact — the feedback round-trip:

Scholar decisions flow back into the pipeline via a JSON file that the pipeline reads on the next run:

Explorer review → [Export decisions JSON] → data/decisions.json → pipeline run → entity_feedback.json

Steps to close the loop:

  1. In the Explorer, click Export decisions — a file like outremer-decisions-2026-07-10.json downloads.

  2. Save it to the repo as data/decisions.json (or any path you pass with --review-decisions-path).

  3. Run the pipeline with both flags:

    python scripts/run_pipeline.py --input-dir data/raw \
  --entity-feedback-path data/entity_feedback.json \
  --review-decisions-path data/decisions.json

  4. The pipeline will:

    • Validate the decisions file (aborts with a clear error report if the schema is invalid).
    • Tally accept/reject votes per name (cross-reviewer deduplication is automatic).
    • Move names with ≥2 reject votes from different reviewers into blocked_terms.
    • Move names with ≥1 accept vote (and accept ≥ reject) into allow_terms.
    • Write the updated data/entity_feedback.json.

Validation before running:

Catch schema errors before the pipeline runs:

python3 -m scripts.validate_decisions data/decisions.json

Valid decisions show ✅ N entries, N valid, 0 errors. Invalid files abort with a line-by-line error report.

Conflict detection:

If two different reviewers disagree on the same person in the same document (one accept, one reject), the pipeline logs a conflict warning. Conflicts do not block processing — the vote threshold determines the outcome.

Schema for decisions.json:

[
  {
    "doc_id":     "rileysmith-motivesearliestcrusaders-1983-92cc17aaccd3",
    "person":     "Baldwin I",
    "decision":   "accept",          // accept | reject | not_a_person | wrong_era | is_group
    "client_id":  "anon-abc123xyz",  // optional, auto-generated per browser
    "comment":    "confirmed match", // optional
    "submitted_at": "2026-07-10T09:00:00Z"  // optional, ISO 8601
  }
]

Accept/reject votes are aggregated per normalised name across all entries with the same doc_id + person. The canonical name stored in entity_feedback.json is taken from the first occurrence.

Authority file

scripts/outremer_index.json contains curated gold-standard person entries. Each entry:

  • authority_id — unique identifier (e.g. AUTH:CR1)
  • preferred_label — canonical name
  • variants — alternate spellings and forms
  • normalized — pre-computed lowercase/accent-stripped forms
  • name — parsed name components (given, toponym, regnal, epithet)
  • provenance.source_files — source attribution

The linker matches against all variant forms using rapidfuzz token-sort ratio (≥ 60% = candidate; ≥ 90% = high confidence).

HBLS MCP Integration

The HBLS (Historisches Biographisches Lexikon der Schweiz) MCP server runs on tei at http://localhost:8003. Use it to cross-reference extracted persons against HBLS biographical data:

# Quick check
curl "http://localhost:8003/mcp/search?q=Habsburg&limit=3"

# Full API reference
curl "http://localhost:8003/mcp"

See docs/EPIC4_HBLS_MCP.md for full API reference.

GitHub Actions

Workflow Trigger What it does
pipeline.yml push to main, nightly 02:00 UTC, manual Runs run_pipeline.py, commits site/data/, site/bib/ back to main
pages.yml push to main, manual Deploys site/ to GitHub Pages

For pipeline.yml secrets, add GPUSTACK_API_KEY under Settings → Secrets and variables → Actions.

Project context

People of the Medieval Levant is a collaborative digital humanities project exploring how generative AI and Knowledge Graphs can enable a more inclusive prosopography of the Crusades era — one that goes beyond the traditional elite focus to encompass non-Western actors, women, refugees, artisans, and unnamed collectives.

Led by Jochen Burgtorf (medieval history), Tobias Hodel (digital humanities / AI), and Laura Morreale (medieval cultural contact).

The pipeline treats ambiguity as data rather than error. Mismatches between the LLM layer and the KG layer are diagnostic signals — they reveal name collisions, missing entities, or outdated assumptions. Scholarly adjudication through the H-i-t-L interface is where historical interpretation happens.

About

No description, website, or topics provided.

Resources

Readme
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages