Add jenner-init-mcp server for laser-init docs - #39
Open
jonathanhhb wants to merge 9 commits into
Open
Conversation
Fourth RAG MCP server following the existing core/measles/generic pattern.
Serves laser-init documentation (five geodata microservices + browser
choropleth webapp + laser-generate CLI + laser_config.yaml) on port 9768.
Server
- main_init.py: FastMCP server loading vectorstore_init, exposing
search / generate_code / version_info. System prompt grounds the
generator in laser-init's webservice-first workflow:
* gadm-service / geoboundaries: GET /boundaries/{ISO}/{level}
* worldpop-service: POST /aggregate/{ISO}?year=YYYY
* unwpp-service: GET /pyramid/{ISO}?year=YYYY
* unocha-service: GET /crisis/{ISO}
Rules: preserve nodeid mapping, stream the aggregate response, never
redownload GADM shapefiles / WorldPop rasters directly, use the
laser-generate CLI where applicable, and point users at the client.html
webapp for interactive population previews.
- Anchor queries hit the microservice API surface, laser-generate CLI,
laser_config.yaml, and the (previously undocumented) client.html webapp.
- Dockerfile.init: python:3.12-slim + requirements.txt + vectorstore_init.
- docker-compose.yml: jenner-init-mcp service on PORT_INIT (default 9768).
Ingest
- Consumes the combined_mkdocs.md produced by laser-init's `make docs-jenner`
target (feat/geodata-services branch). Override via INIT_MD=<path>.
Makefile
- ingest-init / build-init / run-init / stop-init / logs-init / push-init
- test-init: 5-prompt smoke suite runner (no validators yet - init depends
on running microservices; automate once those are in CI).
- ingest-all / build-all / run-all include the new server.
- register-local adds jenner-init-mcp to Claude Code.
Tests
- tests/prompt_test_suite_init/: 5 smoke prompts covering:
p01 single-service gadm HTTP API
p02 multi-service composition (gadm -> worldpop streaming)
p03 laser-generate CLI invocation
p04 browser webapp launch + laser_config.yaml (documents client.html)
p05 worldpop prewarm endpoint + error handling
Docs
- CLAUDE.md: promote the server table from 3 to 4 rows; add a short note
explaining that laser-init is upstream of laser-generic / laser-measles.
- .gitignore: vectorstore_init/, tests/prompt_test_suite_init/output/,
tests/prompt_test_suite_generic/output/.
Verified against a local server built from the current
laser-init combined_mkdocs.md; all 5 prompts return grounded, runnable
code with the anchor-query-augmented retrieval.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The `combined_mkdocs.md` this server ingests is produced by laser-init's `make docs-jenner` target, which currently only exists on the `feat/geodata-services` branch (not yet merged to laser-init main). Call this out prominently in CLAUDE.md and next to the INIT_MD default in the Makefile so reviewers and future ingest runs know they need that branch checked out until it lands upstream. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
parallel_test_runner.py's built-in validator skips prompts whose generated
code did not execute successfully (returncode != 0). For init that gate
never opens: init generation targets HTTP microservices that usually are
not running in the CI environment, so every prompt trips the exit-code
guard and the runner reports either PASS-by-NO_CHECK or execution failure.
Introduce a parallel static-check track that runs against the generated
.py file directly, regardless of whether execution succeeded. Checks are
regex-based and enforce the architectural rules baked into the server's
system prompt: microservice-first, correct endpoint paths, streaming for
/aggregate, `laser-generate` CLI over Python re-implementation, and so on.
Files
- tests/prompt_test_suite_init/validation/_helpers.py
load_generated_code() (accepts .py or .out path), has_any/has_all/missing
regex helpers, strip_comments_and_strings(), ok/fail exit conventions.
- tests/prompt_test_suite_init/validation/p{01..05}_check.py
One validator per smoke prompt. Each checks positive signals
(endpoint hit, ISO referenced, feature-count printed, streaming
iterator present, laser-generate command surfaced, client.html
referenced, timing + error handling in a BRA/IND/RUS loop) and
negative signals (no direct GADM downloads, no raster bypass, no
Python re-implementation of the CLI pipeline).
- tests/prompt_test_suite_init/run_static_checks.py
Standalone runner. Auto-discovers the most recent
output/parallel_*/ directory, picks the latest attempt per prompt,
invokes each pNN_check.py. Also accepts --output-dir and explicit
--code label=path pairs for ad-hoc runs.
- Makefile: `check-init-static` target that runs the standalone script.
- tests/prompt_test_suite_init/README.md rewritten to explain the
two-phase (generate + check) workflow and the reason the built-in
validator does not fire for this suite.
Verified against hand-crafted good and bad samples (5/5 goods pass,
targeted-bad samples fail on the specific defect); details in README.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The AKS-deployed laser-init microservices (gadm-service at 48.200.52.126,
worldpop-service at 4.155.140.158) are live and public, so the init test
suite can now exercise real endpoints instead of skipping stdout checks.
Prompts (p01, p02, p03, p05)
- Reference the AKS URLs directly. The purpose is to test whether the LLM
emits the correct endpoint PATHS (/boundaries/, /aggregate/, /prewarm/)
and correct request/response handling, not to test its ability to guess
a host+port. Host+port is deployment config; endpoint semantics is what
the MCP server should get right.
- p04 unchanged: the webapp prompt is inherently local-dev-oriented, and
its response is prose + shell commands rather than a runnable script.
Validators (p01, p02, p05)
- Extend each `pNN_check.py` with a stdout phase that fires when a `.out`
file exists and the recorded exit code is 0:
p01: NGA admin-1 must report a feature count in [20, 60]
(37 in current GADM, generous window for revisions).
p02: >= 5 name+number lines; top row's max number >= 100_000
(Nairobi ~4.4M in 2020).
p05: One line per BRA/IND/RUS with either a numeric timing or ERROR;
all three isos must be reported; timings non-negative.
- On non-zero exit code, the stdout phase fails loudly with the exit code
so a static-pass does not silently save a broken run.
- p03, p04 remain static-only — their responses are prose + shell, not
executable Python, so a runner-side exit code guard would always trip.
Helpers
- `load_stdout()`, `exit_code()`, `find_numbers()` in `_helpers.py`.
- `load_stdout()` returns None when no `.out` file exists (standalone
`run_static_checks.py` calling convention), so the same check works for
both online (parallel_test_runner) and offline (standalone) modes.
Docs
- README.md now leads with the deployment-assumption note; adds a
dedicated "Testing prose responses" section that explains why p03/p04
cannot be stdout-validated, what tighter static rules compensate, and
what would be needed for full runtime validation of prose responses
(an LLM-judge validator, or an integration test that runs the
commands and inspects the resulting artifacts).
- .gitignore: exclude __pycache__/ and *.pyc from the repo.
Verified: all 5 hand-crafted good samples pass; targeted-bad samples fail
on their specific defect (no plausible feature count for p01, missing
name+number lines for p02, missing ISO report for p05, non-zero exit
code caught).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Two bugs surfaced when running the suite against live AKS microservices:
1. run_static_checks.py::latest_attempt_for used `_iter\d+_\d+\.py$` but
parallel_test_runner writes stems with an underscore-embedded timestamp
(`%Y%m%d_%H%M%S%f`), producing filenames like
`p01_iter1_20260714_204058566556.py` with two `_<digits>` blocks. The
check reported "MISSING" for every prompt. Relax to `_iter\d+_[\d_]+\.py$`.
2. p05_check.py's stdout regex was too permissive: `^.*\bRUS\b.*?...` with
greedy `.*` caught the RUS substring inside the /prewarm/RUS?year=2020
URL that AKS returns in its error-detail body, so it reported RUS as
completing in "2020s" instead of ERROR. Anchor the match at the start
of the line — `^\s*<ISO>\s*:\s*<timing|ERROR>` — which mirrors the
format the prompt asks for and avoids body-text substring matches.
Verified with the AKS run in
tests/prompt_test_suite_init/output/parallel_2026-07-14_20-40-43/:
- Standalone checker now finds all 5 attempts and reports 5/5 PASS
- p05 correctly reports {'BRA': '0.01', 'IND': '0.01', 'RUS': 'ERROR'}
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
First AKS-live run had two false-PASS cases:
- p02: generated code executed successfully but reported top-5 Kenya
counties as Mandera South / West / North / East / Daadab at 400-560k.
Kenya's real top admin-2 is Nairobi County at ~4.4M in 2020; the
MCP-generated code misread the worldpop /aggregate streaming response
shape and picked up wrong numbers. The old max-population threshold
(100k) was too low to catch this.
- p04: generated code hardcoded SERVE_DIR = Path("services/worldpop"),
which does not exist in the runner's per-attempt cwd. The script
printed "Error: expected directory ... to exist" and exited 0 —
slipping past both the runner's returncode gate and the static-only
check.
Fixes
- p02 stdout phase:
* Require "Nairobi" (case-insensitive) to appear in stdout — the
strongest ground-truth-anchored assertion available without an
oracle for this dataset.
* Raise the max-population floor to 1_000_000. Any real Kenya
top-5 clears this trivially (Nairobi alone is 4.4M); any
MCP-produced misinterpretation that surfaces sub-county
fragments cannot.
* Failure message lists what DID appear so debugging is direct.
- p04 stdout phase (new):
* Reject sad-path signals: `^Error:`, "expected directory ... to
exist", "Run this from the repository root", "FileNotFoundError",
"does not exist". These are the classic Python exit-0 failure
modes that a hardcoded-path launch script produces.
* Positive signal is optional — pure-prose responses that produce
empty stdout are still acceptable.
Verified against the last live run (output/parallel_2026-07-14_20-40-43):
- p01 PASS (unchanged) — 37 features for NGA
- p02 FAIL — "top-5 does not contain Nairobi; got ['Mandera South', ...]"
- p03 PASS (unchanged) — laser-generate command surfaced
- p04 FAIL — "stdout contains failure-path signal /expected...to exist/"
- p05 PASS (unchanged) — BRA/IND OK, RUS ERROR (raster too large)
The FAILs are correct: they surface real MCP-server output defects that
need to be fixed separately (worldpop response-shape docs and prohibition
on hardcoding repo-relative paths in launch scripts).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Adds a new “jenner-init-mcp” RAG MCP server that indexes laser-init docs and provides search + generate_code, plus Docker/Makefile wiring and a 5-prompt smoke suite with static validators.
Changes:
- Introduces
jenner-init-mcp(port 9768) with its own FAISS vectorstore and laser-init–specific system prompt/anchor retrieval. - Extends Docker Compose + Make targets to ingest/build/run/log/push/test the new server alongside core/measles/generic.
- Adds
tests/prompt_test_suite_init/with prompts, validators, and a standalone static-check runner.
Reviewed changes
Copilot reviewed 18 out of 19 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
main_init.py |
New laser-init MCP server implementation (FAISS load, retrieval, search/generate_code/version_info). |
Dockerfile.init |
New Docker image definition for jenner-init-mcp. |
docker-compose.yml |
Adds jenner-init-mcp service and port wiring. |
Makefile |
Adds ingest-init, build-init, run-init, stop-init, logs-init, push-init, test-init, check-init-static, and integrates into *-all and register-local. |
CLAUDE.md |
Updates server architecture docs/table to include init server and dependency note. |
.gitignore |
Ignores init/generic outputs and vectorstores. |
tests/prompt_test_suite_init/README.md |
Documents the init smoke suite and its static/stdout validation approach. |
tests/prompt_test_suite_init/run_static_checks.py |
Standalone runner to execute init validators against generated code. |
tests/prompt_test_suite_init/validation/_helpers.py |
Shared helper utilities for init validators. |
tests/prompt_test_suite_init/validation/p01_check.py |
Validator for gadm-service boundaries fetch prompt. |
tests/prompt_test_suite_init/validation/p02_check.py |
Validator for gadm→worldpop streamed aggregation prompt. |
tests/prompt_test_suite_init/validation/p03_check.py |
Validator for laser-generate CLI invocation prompt. |
tests/prompt_test_suite_init/validation/p04_check.py |
Validator for webapp launch instructions prompt. |
tests/prompt_test_suite_init/validation/p05_check.py |
Validator for worldpop prewarm + error handling prompt. |
tests/prompt_test_suite_init/prompts/p01_prompt.txt |
Prompt text for NGA admin-1 boundaries via gadm-service. |
tests/prompt_test_suite_init/prompts/p02_prompt.txt |
Prompt text for KEN admin-2 + streamed worldpop aggregation. |
tests/prompt_test_suite_init/prompts/p03_prompt.txt |
Prompt text for laser-generate CLI command for IND admin-2 2020. |
tests/prompt_test_suite_init/prompts/p04_prompt.txt |
Prompt text for launching/serving client.html webapp with Docker services. |
tests/prompt_test_suite_init/prompts/p05_prompt.txt |
Prompt text for worldpop prewarm for BRA/IND/RUS (2020) with timing + errors. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+68
to
+70
| # Stdout phase — expect exactly-or-approximately 5 lines carrying a name + a | ||
| # population-like number. Kenya admin-2 has 47 counties; the top 5 should all | ||
| # be > 100k population (Nairobi ~4.4M in 2020). |
Collaborator
Author
There was a problem hiding this comment.
Already addressed in 2610bbe (part of the p02 admin-level-1 rewrite). The comment now reads: Kenya admin-1 has 47 counties; the top 5 should all be well above 1M (Nairobi ~4.7M in 2020, Mandera ~2.6M). — matches the >= 1_000_000 threshold in the assertion.
Comment on lines
+92
to
+93
| - **p02**: Assert >=5 lines containing both a name (letters) and a number. | ||
| Top-line's max number must be >=100,000 (Nairobi is ~4.4M in 2020). |
Collaborator
Author
There was a problem hiding this comment.
Already addressed in 2610bbe. The README p02 stdout-phase description now reads: Top-5 must include Nairobi (Kenya's most populous admin-1 county at ~4.7M in 2020) and the max reported population must clear 1M. — matches the validator behavior.
p04 asks how to launch the browser choropleth webapp — a workflow whose correct answer is instructions, not runnable code (open client.html in a browser after the microservices are up). Any script the MCP generates either serves the client and blocks forever, or hard-codes services/ paths that don't exist in the per-attempt test cwd and exits 1. Add a sibling empty sentinel `pNN_prompt.no_execute` convention to the parallel runner. When present, `one_attempt` skips subprocess execution and validates the generated `.py` file text directly. Validators drop their stdout phase gracefully via load_stdout's existing "returns None when no .out sibling" behavior. Add the marker for p04; README documents the convention. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The Nairobi anchor was always factually wrong at admin-2. Kenya's admin-2
in GADM has 300 sub-counties, none named "Nairobi" — the capital is split
into 17 sub-counties (Embakasi East, Dagoretti South, etc.) and the actual
admin-2 top-5 by WorldPop aggregation is Mandera's geographically huge
rural sub-counties (Mandera South ~560k, Mandera West ~447k, etc.). No
admin-2 unit anywhere in Kenya clears 1M in the WorldPop 2020 data.
Ground truth confirmed by hitting the live services directly:
- GET /boundaries/KEN/1 → 47 admin-1 counties
- POST /aggregate/KEN?year=2020 → {nodeid: pop}
- Top 5 by pop: Nairobi 4.7M, Mandera 2.6M, Nakuru 2.3M, Bungoma 2.05M,
Kiambu 1.86M — matches the assertion in p02_check.py cleanly.
Test rewritten at admin-1 exercises the same architectural surface
(gadm boundaries fetch, worldpop /aggregate POST, nodeid→name join, top-N
ranking) with a correct ground-truth anchor. Suite is now 5/5.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
YeChen-IDM
reviewed
Jul 15, 2026
|
|
||
| top_n_signals = [ | ||
| r"\bsorted\s*\(", r"\.sort\s*\(", r"\.nlargest\s*\(", | ||
| r"\bheapq\b", r"\[:5\]", r"top", r"most\W*populous", |
Collaborator
There was a problem hiding this comment.
recommend to use "\btop\b" for a whole word search
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The
combined_mkdocs.mdthis server ingests is produced by laser-init'smake docs-jennertarget, which currently lives on thefeat/geodata-servicesbranch of laser-init and has NOT been merged to laser-initmainyet.Practical implications:
make ingest-initrequires a local checkout of that laser-init branch (or a manualINIT_MD=<path>override pointing at its docs output).Called out in
CLAUDE.md(server-table section) and next to theINIT_MDdefault inMakefileso it stays visible to future ingest runs.Summary
jenner-init-mcp(port 9768), that serves laser-init documentation, following the existing core/measles/generic pattern.laser-generateCLI.Server design
main_init.pymirrorsmain_generic.pystructure but grounds the generator in laser-init's webservice-first workflow. Key rules baked into the system prompt:nodeidmapping end-to-end (gadm boundaries → worldpop aggregation).POST /aggregateas a streamed JSON response.laser-generateCLI where it covers the pipeline; don't re-implement in raw Python.services/{gadm,worldpop}/client.htmlfor interactive previews, with URLs read fromlaser_config.yaml.Anchor queries hit the microservice API surface, the CLI,
laser_config.yaml, and the previously-undocumented client.html webapp so retrieval is stable regardless of user phrasing.Ingest source
INIT_MDdefaults to../laser-init/dist/combined_mkdocs.md, produced by laser-init'smake docs-jennertarget on thefeat/geodata-servicesbranch. Override on the command line:make ingest-init INIT_MD=/path/to/combined_mkdocs.md.Tests
Two-phase workflow: generate (via MCP runner) + check (static + stdout validators).
tests/prompt_test_suite_init/— 5 smoke prompts:laser-generateCLI invocationno_execute)Prompts p01/p02/p03/p05 embed the live AKS microservice URLs (gadm-service
http://48.200.52.126, worldpop-servicehttp://4.155.140.158) — seetests/prompt_test_suite_init/README.mdfor the deployment-assumption note. Host+port is deployment config; the suite tests whether the LLM emits correct endpoint paths and request/response handling.Static validators run against the generated
.pyfile text; stdout validators run against captured execution output.make check-init-staticruns the standalone validator against the latest output regardless of exit code — useful when microservices aren't reachable and the runner's built-in exit-0 gate would otherwise skip everything.Prose-only prompts (
no_executemarker)Launching the choropleth webapp (p04) has no purely-runnable answer — the correct workflow is "start the microservices, open
client.htmlin a browser." Any script the MCP produces either serves the client and blocks forever, or hard-codes paths that don't exist in the per-attempt test cwd.The parallel runner now honors a sibling empty sentinel file
pNN_prompt.no_executealongsidepNN_prompt.txt. When present, the runner writes the generated.pybut skips execution, and validates static regex rules against the code text directly. Validators use_helpers.load_stdout()which returnsNonewhen no.outsibling exists, so their stdout phase becomes a no-op. Seetests/prompt_test_suite_init/README.mdandtests/parallel_test_runner.py::one_attemptfor the wiring.p02: admin-level 1 (Nairobi is admin-1, not admin-2)
p02 was originally written at admin-level 2 with a "top-5 must include Nairobi" ground-truth assertion. Every live-AKS run failed that assertion — investigation of the actual endpoint responses proved the MCP-generated code was correct and the assertion was factually wrong: GADM admin-2 for Kenya has 300 sub-counties, none named "Nairobi" (the capital splits into 17 sub-counties like Embakasi East, Dagoretti South, etc.). At admin-2 the true top-populous units are Mandera's geographically huge rural sub-counties; no admin-2 unit clears 1M in WorldPop 2020 data.
Ground truth verified by hitting the live services directly:
GET /boundaries/KEN/1returns 47 admin-1 counties.POST /aggregate/KEN?year=2020returns{nodeid: population}as a flat JSON object.Rewritten at admin-1, p02 exercises the same architectural surface (gadm boundaries fetch → worldpop
/aggregatePOST →nodeid→namejoin → top-N ranking) against a correct ground-truth anchor. Suite is now 5/5 locally.Files changed
main_init.py,Dockerfile.init— server + imagedocker-compose.yml— new service onPORT_INIT(default 9768)Makefile— ingest-init / build-init / run-init / stop-init / logs-init / push-init / test-init / check-init-static, plus wiring into ingest-all / build-all / run-all / register-local / cleanCLAUDE.md— promote the server table from 3 → 4 rows; note that laser-init is upstream of the other projects + the branch-dependency banner.gitignore—vectorstore_init/, tests output dir,__pycache__/tests/prompt_test_suite_init/— 5 prompts, README, static + stdout validators, standalonerun_static_checks.pytests/parallel_test_runner.py—no_executemarker support for prose-only promptsTest plan
make -n {ingest,build,run,stop,logs,test}-initandmake -n check-init-staticall resolve and expand correctlydocker compose config --serviceslists all four servicesmake test-init LABEL=aks-live— 5/5 pass (p03 needed one retry through the built-in error-feedback loop; all other prompts pass on attempt 1)feat/geodata-services, thenmake ingest-init INIT_MD=<your path>+make run-init+make test-init🤖 Generated with Claude Code