Record runs as a store-agnostic record with a PROV view

.github/workflows/tests.yml

@@ -0,0 +1,53 @@
+# SPDX-License-Identifier: MPL-2.0
+# SPDX-FileCopyrightText: 2026 SECORO AG (secoro.uni-bremen.de)
+# Author: Vamsi Kalagaturu
+
+name: Tests (Python 3.12 and 3.14)
+
+on:
+  push:
+    branches: [main, dev]
+  pull_request:
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python-version: ["3.12", "3.14"]
+    services:
+      mariadb:
+        image: mariadb:11
+        env:
+          MARIADB_DATABASE: rec_test
+          MARIADB_USER: rec
+          MARIADB_PASSWORD: rec
+          MARIADB_ROOT_PASSWORD: root
+        ports:
+          - 3306:3306
+        options: >-
+          --health-cmd="healthcheck.sh --connect --innodb_initialized"
+          --health-interval=10s
+          --health-timeout=5s
+          --health-retries=5
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/checkout@v4
+        with:
+          repository: secorolab/metamodels
+          ref: main
+          path: metamodels
+      - uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python-version }}
+      - run: sudo apt-get update
+      - run: sudo apt-get install --yes libmariadb-dev
+      - run: pip install -e ".[mariadb,dev]"
+      - run: pytest -q
+        env:
+          MARIADB_USER: rec
+          MARIADB_PASSWORD: rec
+          MARIADB_HOST: 127.0.0.1
+          MARIADB_PORT: 3306
+          REC_TEST_MARIADB_DATABASE: rec_test
+          REC_METAMODELS_DIR: metamodels

README.md

@@ -1,40 +1,175 @@
 # Robot Execution Catalog (REC)
 
-REC is a tool that aims to make data in robot executions [FAIR](https://www.go-fair.org/fair-principles/). 
-It is designed to configure, manage, organize and consolidate (meta)data used or generated by robots executing tasks or tests.
+REC makes the data of robot executions [FAIR](https://www.go-fair.org/fair-principles/): it records
+each run's provenance in [PROV](https://www.w3.org/TR/prov-o/), on the
+[rec and prov-extension vocabularies](https://secorolab.github.io/metamodels/).
 
 ## Installation
 
-In your terminal, go to where you have cloned this repository and install it in editable mode:
-
 ```shell
-pip install -e .
+pip install -e .              # file stores
+pip install -e ".[mariadb]"   # and the MariaDB store
 ```
 
-## Examples
+## Recording a run
 
-The current skeleton of the code allows you to run two examples: The `MariaDBObserver` and a `Run`.
+A run is a `Run` subclass; `main` does the work and logs what it uses and generates.
 
-### Observers
+```python
+from rec.observer import Observer
+from rec.run import Run
+from rec.stores.file_store import FileStore
 
-An Observer is an interface to a type of data storage. For now, we have added a `MariaDBObserver` with some basic functionality. To test it you can run the following in your terminal:
 
-```shell
-python rec/observers/mariadb_observer.py
+class PickPlaceRun(Run):
+    def main(self):
+        self.add_agent("https://example.org/agent/arm", "Agent", name="arm")
+        self.log_repositories([{"name": "controller", "url": "https://example.org/lab/controller", "commit": "0123abcd"}])
+        self.add_resource("config/robot.yaml")
+        self.add_artefact("results/trajectory.csv")
+        self.log_scalar("final-position-error", 0.004)
+        return "placed"
+
+
+observer = Observer(FileStore("runs/prov"))
+PickPlaceRun(observers=[observer], run_id="pick-place-1").run()
+observer.close()
 ```
 
-### Creating a run
+`run()` records the start, runs `main`, and ends the run as `passed`, `failed` (`main` raised) or
+`error` (interrupted). What `main` returns is kept as the run's result.
 
-The [example](examples/decentral_run.py) shows how to create a run object, attach an observer, and define the `main` function of your tests. You can test its basic functionality by running the script:
+### Steps of a run
 
-```shell
-python examples/decentral_run.py
+A step is an activity of a prov-extension kind (`Execution`, `Assurance`, `Transformation`,
+`Generalization`, `Specification`) that is `dcterms:isPartOf` the run. Files name the step that
+used or generated them:
+
+```python
+calibration = "https://example.org/calibration/1"
+self.add_activity(calibration, "Execution", "https://example.org/agent/arm", start, end)
+self.add_resource("config/robot.yaml", usage_activity=calibration)
+self.add_artefact("results/calibration.json", gen_activity=calibration)
+```
+
+### Queueing and cancelling
+
+State lives in the store, so any `Run` with the same id sees and changes the same run:
+
+```python
+DrawerRun(observers=[observer], run_id="drawer-2").queue()
+
+# elsewhere, e.g. a scheduler with its own connection
+Run(observers=[Observer(MariaDBStore())], run_id="drawer-2").cancel()
 ```
 
+| State when cancelled | Result |
+|---|---|
+| `queued` | `canceled` |
+| `in-progress` | `canceling`; `main` sees `self.state is State.CANCELING`, returns, and the run ends `canceled` |
+| anything else | `RuntimeError` |
 
-## Connecting to MariaDB
+The store decides every step under a lock (`flock` for files, `SELECT ... FOR UPDATE` for
+MariaDB): only a new run is queued, only a new or queued run starts.
 
-Create an `.env` file at the root of this repository and update the contents with your credentials and the IP of the server where its deployed:
+## Stores
+
+| Store | Keeps a run as |
+|---|---|
+| `FileStore(dir)` | its PROV document, `<run_id>.ld.json` |
+| `FileStore(dir, fmt="json")` | the `RunRecord` as `<run_id>.json` |
+| `MariaDBStore(db_name, prefix)` | a row of `<prefix>_runs` and a row per list item, see [mariadb.sql](rec/stores/mariadb.sql) |
+
+All keep the same record. A run may have several observers, one store each:
+
+```python
+stores = [FileStore("runs/prov"), FileStore("runs/json", fmt="json"), MariaDBStore()]
+run = ScrewingRun(observers=[Observer(store) for store in stores], run_id="screw-1")
+run.run()
+
+graphs = [provenance.graph(store.load("screw-1")) for store in stores]
+assert all(isomorphic(graphs[0], graph) for graph in graphs)
+```
+
+### One artefact in each store
+
+```python
+self.add_artefact("results/trajectory.csv", sha256="9f86d0...0a08", size_bytes=2048)  # logged from /home/lab
+```
+
+`FileStore(dir)`, `<run_id>.ld.json` (shown as Turtle; `run:` is `https://secoro.uni-bremen.de/rec/run/`):
+
+```turtle
+<run:pick-place-1/file/%2Fhome%2Flab/results%2Ftrajectory.csv>
+    a prov:Entity ;
+    prov:atLocation [ a prov-ext:RelativeLocation ;
+                      prov-ext:rel-path "results/trajectory.csv"^^xsd:anyURI ;
+                      prov:atLocation <file:///home/lab/> ] ;
+    prov:wasGeneratedBy run:pick-place-1 ;
+    prov:qualifiedGeneration [ a prov:Generation ;
+                               prov:activity run:pick-place-1 ;
+                               prov:atTime "2026-10-15T09:30:00+00:00"^^xsd:dateTime ] ;
+    spdx:checksum [ a spdx:Checksum ;
+                    spdx:algorithm spdx:checksumAlgorithm_sha256 ;
+                    spdx:checksumValue "9f86d0...0a08"^^xsd:hexBinary ] ;
+    dcat:byteSize "2048"^^xsd:nonNegativeInteger .
+```
+
+`FileStore(dir, fmt="json")`, `<run_id>.json`:
+
+```json
+"artefacts": [
+  {
+    "file": {"path": "results/trajectory.csv", "root": "/home/lab", "title": null,
+             "sha256": "9f86d0...0a08", "size_bytes": 2048},
+    "activity": null,
+    "time": "2026-10-15T09:30:00+00:00"
+  }
+]
+```
+
+`MariaDBStore()`, a row of `logs_artefacts` (`run` is the run's `id` in `logs_runs`; `activity` is
+`''` for the run itself):
+
+| run | path | root | title | sha256 | size_bytes | activity | time |
+|---|---|---|---|---|---|---|---|
+| 1 | results/trajectory.csv | /home/lab | NULL | 9f86d0...0a08 | 2048 | | 2026-10-15 09:30:00.000000 |
+
+A value one store could not give back is refused when logged: a NaN or infinite metric, or an
+item logged twice (`rec.record.KEYS`). Any object with `load`, `edit`, `run_ids` and `close` is a
+store (`rec.stores.Store`).
+
+## The PROV graph
+
+```python
+from rec import provenance
+
+g = provenance.graph(store.load("pick-place-1"))      # RunRecord -> rdflib Graph
+record = provenance.record(g, "pick-place-1")         # and back
+```
+
+```turtle
+<https://secoro.uni-bremen.de/rec/run/pick-place-1>
+    a prov-ext:Execution, oslc_auto:AutomationResult ;
+    oslc_auto:state oslc_auto:complete ;
+    oslc_auto:verdict oslc_auto:passed ;
+    prov:used <.../repository/controller>, ... ;
+    prov:wasAssociatedWith <https://example.org/agent/arm> ;
+    rec:result "placed" .
+```
+
+A run id is RFC 3986 unreserved characters (letters, digits, `-`, `.`, `_`, `~`); a run given none
+gets a UUID.
+
+## Examples
+
+```shell
+python examples/file_store_run.py      # PROV and JSON files
+python examples/mariadb_store_run.py   # queue, cancel by id, run
+python examples/all_stores_run.py      # one run in all three stores
+```
+
+The MariaDB examples need the `mariadb` extra and an `.env` at the repository root:
 
 ```dotenv
 MARIADB_USER="user"
@@ -43,6 +178,19 @@ MARIADB_HOST="localhost"
 MARIADB_PORT=3306
 ```
 
+## Tests
+
+```shell
+pip install -e ".[dev]"
+pytest
+```
+
+The conformance tests validate runs against the shapes of a
+[metamodels](https://github.com/secorolab/metamodels) checkout beside this repository or at
+`REC_METAMODELS_DIR`. MariaDB tests need the `mariadb` extra and a disposable database named by
+`REC_TEST_MARIADB_DATABASE`.
+
 ## Acknowledgments
 
-This tool is inspired by [sacred](https://github.com/IDSIA/sacred/tree/master), a tool to organize experiments for Machine Learning. We borrow small pieces of code under the [MIT License](https://opensource.org/license/MIT), but have rewritten and simplified the framework for robotics. 
+Inspired by [sacred](https://github.com/IDSIA/sacred/tree/master); small pieces of its code are
+used under the [MIT License](https://opensource.org/license/MIT).

examples/all_stores_run.py

@@ -0,0 +1,38 @@
+"""Record one run in all three stores at once -- PROV document, JSON record, MariaDB tables -- and check they give back the same graph.
+
+Needs the `mariadb` extra and the connection settings in `.env` (see the README).
+"""
+
+from rdflib.compare import isomorphic
+
+from rec import provenance
+from rec.observer import Observer
+from rec.run import Run
+from rec.stores.file_store import FileStore
+from rec.stores.mariadb_store import MariaDBStore
+
+
+class ScrewingRun(Run):
+    def main(self):
+        self.add_agent("https://example.org/agent/arm", "Agent", name="arm")
+        self.log_dependencies([{"name": "rdflib", "version": "7.6.0"}])
+        self.add_resource("config/screw.yaml")
+        self.add_artefact("results/torque.csv")
+        # One count per screw: a step per attempt, not per sample.
+        for screw, attempts in enumerate([1, 2, 1]):
+            self.log_scalar("attempts", attempts, screw)
+        self.log_scalar("safety-stops", 0)
+        return 3
+
+
+if __name__ == "__main__":
+    stores = [FileStore("runs/prov"), FileStore("runs/json", fmt="json"), MariaDBStore(db_name="logbook", prefix="logs")]
+    observers = [Observer(store) for store in stores]
+    run = ScrewingRun(observers=observers, run_id="screw-1")
+    run.run()
+
+    prov, records, tables = (provenance.graph(store.load(run.id)) for store in stores)
+    assert isomorphic(prov, records) and isomorphic(prov, tables)
+    print(prov.serialize(format="turtle"))
+    for observer in observers:
+        observer.close()

examples/file_store_run.py

@@ -0,0 +1,32 @@
+"""Record a run as a PROV document and as a JSON record in directories, and read it back."""
+
+from rec.observer import Observer
+from rec.run import Run
+from rec.stores.file_store import FileStore
+
+
+class PickPlaceRun(Run):
+    def main(self):
+        self.add_agent("https://example.org/agent/arm", "Agent", name="arm")
+        self.log_repositories([{"name": "controller", "url": "https://example.org/lab/controller", "commit": "0123abcd"}])
+        self.log_dependencies([{"name": "rdflib", "version": "7.6.0"}])
+        self.add_resource("config/robot.yaml")
+        self.add_artefact("results/trajectory.csv")
+        # The value the verdict is judged on; the trajectory itself is the artefact.
+        self.log_scalar("final-position-error", 0.004)
+        return "placed"
+
+
+if __name__ == "__main__":
+    prov = FileStore("runs/prov")
+    records = FileStore("runs/json", fmt="json")
+    observers = [Observer(prov), Observer(records)]
+    run = PickPlaceRun(observers=observers, run_id="pick-place-1")
+    run.run()
+    for observer in observers:
+        observer.close()
+
+    print(f"PROV document: {prov.path(run.id)}")
+    record = records.load(run.id)
+    assert record is not None
+    print(record.state, record.verdict, [(metric.name, metric.step, metric.value) for metric in record.metrics])