Providers as packages, adapter triggers, plan cost, policy v1 schema

.github/workflows/ci.yml

@@ -22,7 +22,7 @@ jobs:
   tests:
     uses: QuietFlare/ci-workflows/.github/workflows/python-test.yml@v1
     with:
-      test-command: 'python3 -m unittest discover -s tests'
+      test-command: 'make dev test'
       # 3.9 is what macOS ships, so it is the floor people actually hit.
       python-versions: '["3.9", "3.11", "3.13"]'
 
@@ -48,7 +48,7 @@ jobs:
           python-version: "3.12"
 
       - name: Install the driver
-        run: pip install -r requirements.txt
+        run: pip install -r requirements.txt && make dev
 
       - name: Run the full suite against Postgres
         env:

CHANGELOG.md

@@ -14,6 +14,23 @@ follow [Semantic Versioning](https://semver.org/).
 - ADR 0008: a bundle verifies against something it does not control.
 - ADR 0009: an effective date is an instant, and the gate has a now.
 
+### Changed
+
+- One shipped policy table, `v1`, with five dimensions: `contribution`,
+  `storage`, `scope` (`exclusive` or `shared`), `released` and `mode`
+  (`remove` or `trace`). Release is asked before existence, and a corrected
+  subject's separable part is regenerated rather than purged (R5). The
+  earlier tables and the names `exclusive`, `terminal` and `because` are
+  gone.
+- Plan items are `clew_plan_version` 2: `scope`, `released`, `mode`,
+  `reason` for the rule's rationale and `evidence` for how the class was
+  found. The printed plan carries a column header.
+- `--assertions` files list released artifacts under `released`.
+- `clew evidence build`: `--out` is optional and defaults to
+  `<trigger>-<date>`.
+- The plan header says "removal of", not "withdrawal of".
+- `clew providers` names each provider's package on Python 3.9 too.
+
 ### Fixed
 
 - `clew impact`: evidence chains come from one breadth-first pass over the
@@ -24,7 +41,6 @@ follow [Semantic Versioning](https://semver.org/).
   withheld. Directory outputs are now found under `--results` by name.
 - `clew impact`: a subject that matches no task tag exits non-zero instead
   of reporting zero affected tasks.
-<<<<<<< HEAD
 - `clew reclaim`: a task is `FAILED` only when the engine recorded a
   failure. Every extractor now maps its engine's status word (`Done`,
   `SUCCEEDED`, `skipped`, `CACHED`) to one of `COMPLETED`, `FAILED`,
@@ -92,7 +108,6 @@ follow [Semantic Versioning](https://semver.org/).
   are skipped by the bundle store.
 - `eventlog.append`: `recorded_at` is the database server's clock, read in
   the appending transaction. Callers can no longer supply it.
-=======
 - `clew extract-work`: refuses when any task's work directory is missing,
   naming them; `--allow-partial` writes the graph with the gap as a
   `coverage` note. A cleaned tree used to give an empty graph and exit 0.
@@ -127,7 +142,6 @@ follow [Semantic Versioning](https://semver.org/).
   donor table without `--subject` now reads the samplesheet.
 - `clew extract-work` refuses a six-character hash prefix that matches two
   work directories instead of merging them.
->>>>>>> f360682 (Attribute subjects longest first, surface graph limits in impact)
 
 ## [0.4.0] - 2026-09-07
 

CONTRIBUTING.md

@@ -13,15 +13,14 @@ and it improves by contact with practitioners.
 
 After that, in rough order:
 
-- A domain adapter for a pipeline you use. See `clew/domains/`. A new
-  adapter is usually a few dozen lines on top of `nfcore.py`, plus a
-  regression test pinning real numbers from a real run.
+- A domain adapter for a pipeline you use, or an extractor for an engine
+  Clew cannot read. Both are one subclass in your own package, found by
+  name. [docs/providers.md](docs/providers.md) walks through each with
+  examples. A regression test pinning real numbers from a real run is
+  what makes an adapter trustworthy.
 - A bug report with a graph. A wrong blast radius is the most serious class
   of bug here, above all one that reports something as unaffected when it
   is not. Attach the graph JSON if you can share it.
-- An extractor for another engine or provenance format. Every extractor
-  emits the same graph JSON, so `clew/extract/horus.py` is a good
-  model.
 
 ## Ground rules for code
 
@@ -39,10 +38,21 @@ After that, in rough order:
   decides. An auditor asking why something was flagged must get a policy
   version, hashes, and a re-run that agrees.
 
-Run the suite before opening a pull request:
+The engine and the six providers under `providers/` are separate
+distributions. Install all seven editable first, or nothing registers:
 
 ```bash
-python3 -m unittest discover -s tests
+make dev
+```
+
+Pass `PYTHON=` if the first `python3` on your PATH is not the one `clew`
+runs under.
+
+Run every suite before opening a pull request. The engine's tests are in
+`tests/`; each provider's are in its own `tests/`, beside its fixtures:
+
+```bash
+make test
 ```
 
 ## Licensing of contributions

Makefile

@@ -0,0 +1,16 @@
+# The engine and its providers are separate distributions. Install them all
+# editable to work on the checkout; the tests need every provider present.
+# PYTHON must be the interpreter clew runs under: make PYTHON=... if it is
+# not the first python3 on PATH.
+PYTHON ?= python3
+PROVIDERS := $(wildcard providers/*)
+
+dev:
+	$(PYTHON) -m pip install -e . $(addprefix -e ,$(PROVIDERS))
+
+# The engine's suite, then each provider's, from its own tests/.
+test:
+	$(PYTHON) -m unittest discover -s tests
+	@for p in $(PROVIDERS); do echo "== $$p"; $(PYTHON) -m unittest discover -s $$p/tests || exit 1; done
+
+.PHONY: dev test

README.md

@@ -20,9 +20,13 @@ A clew is the ball of thread Ariadne gave Theseus. You follow it back out.
 ## Install
 
 ```bash
-pip install clew-lineage
+pip install "clew-lineage[all]"
 ```
 
+That is the engine plus every provider it ships. `[nextflow]`,
+`[snakemake]`, `[cromwell]`, `[horus]`, `[dnanexus]` or `[latch]` installs
+one, and the bare `clew-lineage` is the engine alone.
+
 ```bash
 clew demo
 ```
@@ -32,8 +36,8 @@ nf-core/sarek run that ships with the package.
 
 ## What it answers
 
-**Something upstream went bad.** A reference update, a broken container, a
-withdrawn sample.
+**Something upstream went bad.** A reference update, a broken container, an
+input that turned out wrong.
 
 ```bash
 clew impact --graph graph.json --container gatk4
@@ -82,12 +86,17 @@ One command per engine turns a run into a graph.
 
 | Engine | Command |
 |---|---|
-| Nextflow, including Seqera Platform | `clew extract-store --store .lineage --run <run> --json-out graph.json` |
-| Snakemake | `clew extract-snakemake --workdir . --json-out graph.json` |
-| Cromwell and WDL, including Terra | `clew extract-cromwell --metadata metadata.json --json-out graph.json` |
-| Horus, through [horus-lineage](https://github.com/QuietFlare/horus-lineage) | `clew extract-horus --run-dir <run> --json-out graph.json` |
-| DNAnexus | `clew extract-dnanexus --analysis <id> --json-out graph.json` |
-| Latch | `clew extract-latch --execution <id> --json-out graph.json` |
+| Nextflow, including Seqera Platform | `clew extract nextflow --store .lineage --run <run> --json-out graph.json` |
+| Snakemake | `clew extract snakemake --workdir . --json-out graph.json` |
+| Cromwell and WDL, including Terra | `clew extract cromwell --metadata metadata.json --json-out graph.json` |
+| Horus, through [horus-lineage](https://github.com/QuietFlare/horus-lineage) | `clew extract horus --run-dir <run> --json-out graph.json` |
+| DNAnexus | `clew extract dnanexus --analysis <id> --json-out graph.json` |
+| Latch | `clew extract latch --execution <id> --json-out graph.json` |
+
+`clew extract` lists every engine it knows, and `clew providers` shows
+every adapter and extractor installed with the package each came from.
+An engine or pipeline that is not there is one subclass away: see
+[providers](docs/providers.md).
 
 Or skip the file: `reclaim`, `drift` and `digest` take `--runs` pointing
 at the engine's own record, a `.lineage` store or a horus-lineage root,
@@ -108,7 +117,8 @@ no credentials. A gate can block a run before it starts.
 
 [Storage](docs/storage.md), [event log](docs/event-log.md),
 [policy](docs/policy.md), [evidence](docs/evidence.md), [gate](docs/gate.md),
-[auditor surfaces](docs/auditors.md), [architecture](docs/architecture.md).
+[auditor surfaces](docs/auditors.md), [architecture](docs/architecture.md),
+[providers](docs/providers.md) for adding your own pipeline or engine.
 
 ## Status
 

clew/__main__.py

@@ -13,7 +13,7 @@
     "demo": ("clew.demo",
              "the shipped sample run: three triggers, one engine"),
     "impact": ("clew.questions.impact",
-               "what a withdrawal, defect or update reaches, and what to do"),
+               "what a removal, defect or update reaches, and what to do"),
     "gate": ("clew.questions.gate",
              "block a run whose inputs the log says are not usable"),
     "reclaim": ("clew.questions.reclaim",
@@ -30,26 +30,26 @@
                   "one self-contained HTML page over sealed bundles"),
     "mcp": ("clew.views.mcp_server",
             "read-only MCP server over sealed bundles, for auditors"),
+    "extract": ("clew.extract",
+                "build a graph from an engine's record: clew extract <engine>"),
+    "providers": ("clew.providers",
+                  "every domain and extractor installed, and the package each came from"),
     "stitch": ("clew.extract.stitch",
                "join run graphs where one run consumed another's outputs"),
     "digest": ("clew.extract.digest",
                "hash a run's files once, for graphs without content digests"),
-    "extract-store": ("clew.extract.nextflow_store",
-                      "build a graph from the engine's native lineage store"),
-    "extract-crate": ("clew.extract.rocrate",
-                      "build a graph from a Workflow Run RO-Crate"),
-    "extract-work": ("clew.extract.nextflow_work",
-                     "build a graph from work/ symlinks, any engine version"),
-    "extract-horus": ("clew.extract.horus",
-                      "build a graph from a horus-lineage run directory"),
-    "extract-dnanexus": ("clew.extract.dnanexus",
-                         "build a graph from a DNAnexus analysis"),
-    "extract-latch": ("clew.extract.latch",
-                      "build a graph from a Latch execution"),
-    "extract-cromwell": ("clew.extract.cromwell",
-                         "build a graph from Cromwell workflow metadata"),
-    "extract-snakemake": ("clew.extract.snakemake",
-                          "build a graph from Snakemake's metadata store"),
+}
+
+# The names extractors had before `clew extract <engine>`. Still accepted.
+ALIASES = {
+    "extract-store": "nextflow",
+    "extract-work": "nextflow-work",
+    "extract-crate": "ro-crate",
+    "extract-horus": "horus",
+    "extract-dnanexus": "dnanexus",
+    "extract-latch": "latch",
+    "extract-cromwell": "cromwell",
+    "extract-snakemake": "snakemake",
 }
 
 
@@ -66,12 +66,14 @@ def usage():
 def main(argv=None):
     argv = list(sys.argv[1:] if argv is None else argv)
     if argv and argv[0] in ("-V", "--version"):
-        from clew import __version__
-        print(f"clew {__version__}")
+        from importlib.metadata import version
+        print(f"clew {version('clew-lineage')}")
         return 0
     if not argv or argv[0] in ("-h", "--help"):
         print(usage())
         return 0
+    if argv[0] in ALIASES:
+        argv = ["extract", ALIASES[argv[0]]] + argv[1:]
     if argv[0] not in COMMANDS:
         print(f"clew: unknown command {argv[0]!r}\n\n{usage()}",
               file=sys.stderr)

clew/contracts/__init__.py

@@ -0,0 +1,8 @@
+"""from clew.contracts import Adapter, Extractor, Trigger"""
+
+from .adapter import Adapter
+from .extractor import Extractor
+from .registry import discover
+from .trigger import REMOVE, TRACE, Mode, Trigger
+
+__all__ = ["Adapter", "Extractor", "Trigger", "Mode", "TRACE", "REMOVE", "discover"]

clew/contracts/adapter.py

@@ -0,0 +1,39 @@
+"""A domain is what a site knows about one pipeline. docs/providers.md has worked examples."""
+
+from .registry import Provider
+
+
+class Adapter(Provider):
+    group = "clew.adapters"
+
+    # kind name -> Trigger. What can go wrong here, in this pipeline's own
+    # words, and where each enters. Empty is valid: the engine's own kinds
+    # still apply.
+    triggers = {}
+
+    # reference files: triggers in their own right, never owned by anyone
+    load_bearing_inputs = ()
+
+    def contribution(self, graph, task_hash, kind):
+        """
+        Optional. The class of this task's output with respect to one value of
+        `kind` being removed: SEPARABLE, REGENERABLE or IRREDUCIBLE. None keeps
+        the engine's evidence-based answer. The plan records that the adapter said so.
+        """
+        return None
+
+    def pending(self):
+        """
+        Optional. Triggers recorded at this site and not yet asked:
+        [{"kind": "batch", "value": "B017", "asserted_by": "qa", "date": "2026-09-10"}]
+        With any returned, `clew impact --pipeline X` and no trigger answers each.
+        """
+        return []
+
+
+def check(trigger):
+    """Problems with one trigger record; empty when usable."""
+    if not isinstance(trigger, dict):
+        return ["not an object"]
+    return [f"{f} missing or empty" for f in ("kind", "value")
+            if not isinstance(trigger.get(f), str) or not trigger[f]]