feat: reproducible scripted recordings in CI, with termshow check --json and a workflow recipe

README.md

@@ -411,7 +411,7 @@ Beyond `init`, `build`, and `preview`, Great Docs includes a full suite of quali
 | `great-docs termshow render` | Render SVG frames without a full site build |
 | `great-docs termshow export` | Export a recording as one animated SVG for READMEs |
 | `great-docs termshow validate` | Check a recording and its script for mistakes |
-| `great-docs termshow check` | Re-run scripted steps and report recordings whose output has changed |
+| `great-docs termshow check` | Re-run scripted steps and report recordings whose output has changed (`--json` for CI) |
 | `great-docs termshow assets` | Copy the web player for pages built without Quarto |
 | `great-docs termshow import-cast` | Import an asciinema `.cast` file |
 | `great-docs termshow import-tape` | Convert a VHS `.tape` file to a script with `steps:` |

docs/user_guide/41-terminal-recordings.qmd

@@ -1161,6 +1161,10 @@ The `record` section sets up the shell:
 - `cwd` is the directory to start in, relative to the script.
 - `env` sets environment variables. The shell starts with a plain `$ ` prompt and no history
   file, so nothing you record ends up in your own history.
+- `inherit_env` (`false`) passes on your whole environment. By default the shell only gets a few
+  variables from it, such as `PATH`, `HOME`, and `LANG`, so the recording doesn't pick up
+  whatever your shell profile, editor, or CI service has set. That's what keeps a recording
+  made on your machine the same as one made in CI. List anything else the steps need in `env`.
 - `typing_delay` (0.05 seconds) and `typing_jitter` (0.3) set the typing speed and how much it
   varies. `delay` on a `type`, `run`, or `key` step overrides the speed for that step.
 - `timeout` (10 seconds) is how long to wait for a prompt or a pattern. `timeout` on a `run` or
@@ -1245,6 +1249,101 @@ the next marker isn't. A `marker` between commands compares each command's outpu
 next one pushes it away, and a taller terminal (`record.rows`) helps with commands that print a
 lot. The check runs at the recording's size, whatever the script's `record` section says now.
 
+### Keeping demos up to date in CI
+
+This GitHub Actions workflow does two jobs. On every pull request it checks the demos, and the
+job fails if one has drifted, with the diff in its log. Once a week, or when you run it by hand,
+it records the drifted demos again and opens a pull request with the new recordings, so a
+release that changes your tool's output leads to a demo update you only have to review:
+
+```{.yaml filename=".github/workflows/demos.yml"}
+name: Terminal demos
+
+on:
+  pull_request:
+  schedule:
+    - cron: "0 6 * * 1"
+  workflow_dispatch:
+
+jobs:
+  check:
+    if: github.event_name == 'pull_request'
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+      - run: pip install great-docs .
+      - run: great-docs termshow check docs/assets/*.termshow.yml
+
+  update:
+    if: github.event_name != 'pull_request'
+    runs-on: ubuntu-latest
+    permissions:
+      contents: write
+      pull-requests: write
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+      - run: pip install great-docs .
+      - run: great-docs termshow check docs/assets/*.termshow.yml --update
+      # For recordings that a README shows as an animated SVG
+      - run: |
+          for recording in docs/assets/*.termshow; do
+            great-docs termshow export "$recording"
+          done
+      - uses: peter-evans/create-pull-request@v7
+        with:
+          branch: update-terminal-demos
+          commit-message: Record the terminal demos again
+          title: Update the terminal demos
+          body: "`great-docs termshow check --update` found demos whose output has changed."
+```
+
+`pip install great-docs .` installs your package along with Great Docs, so the steps run the
+version of your tool in the pull request. The `update` job can only open a pull request if the
+repository allows it: turn on **Allow GitHub Actions to create and approve pull requests** under
+**Settings → Actions → General**. Both jobs fail if a step fails, such as a command that isn't
+installed on the runner.
+
+A recording made on your Mac and checked on a Linux runner can differ in ways that have nothing
+to do with your tool: `sh` is a different shell on each, and paths and some commands' output
+differ too. Two things help. Set `record.shell` to `bash --norc` in each script, so the same
+shell runs everywhere. And let the `update` job make the recordings, so the ones you commit come
+from the same kind of machine that checks them.
+
+To use the results in another tool, or in a job summary, add `--json`. The output is one JSON
+object, and the exit status is the same as without it:
+
+```bash
+great-docs termshow check docs/assets/*.termshow.yml --json > drift.json
+```
+
+```json
+{
+  "version": 1,
+  "summary": {"up_to_date": 0, "drifted": 1, "updated": 0, "failed": 0},
+  "results": [
+    {
+      "script": "docs/assets/install-guide.termshow.yml",
+      "recording": "docs/assets/install-guide.termshow",
+      "status": "drifted",
+      "compared": 3,
+      "differences": [
+        {"at": "marker 'Verify'", "diff": ["--- install-guide.termshow, at marker 'Verify'", "..."]}
+      ]
+    }
+  ]
+}
+```
+
+`status` is `up_to_date`, `drifted`, `updated` (with `--update`), or `failed`. A failed result
+has an `error` message instead of `differences`, and a `screen` with the screen's text when a
+step timed out.
+
 ## Importing Existing Recordings
 
 ### From asciinema

docs/user_guide/42-termshow-file-formats.qmd

@@ -223,13 +223,17 @@ them as written when it saves.
 | `rows` | `24` | Terminal height |
 | `cwd` | the script's directory | Working directory, relative to the script's directory |
 | `env` | (none) | Environment variables to set. Values may be strings, numbers, or `true`/`false` |
+| `inherit_env` | `false` | Start the shell with the whole current environment, not just the variables listed below |
 | `typing_delay` | `0.05` | Seconds between typed characters |
 | `typing_jitter` | `0.3` | How much each delay varies, as a fraction of it (0 to 1) |
 | `timeout` | `10` | Seconds to wait for the prompt or a `wait_for` pattern |
 | `prompt_pattern` | `\$ $` | Regular expression (Python syntax) that matches the shell's prompt |
 
-The shell starts with `PS1="$ "`, `TERM=xterm-256color`, and no history file. `env` can change
-any of these. `COLUMNS` and `LINES` are removed so the terminal size is `cols` by `rows`.
+Only `PATH`, `HOME`, `USER`, `LOGNAME`, `LANG`, `LC_*`, `TMPDIR`, `TZ`, and `VIRTUAL_ENV` are
+taken from the current environment, unless `inherit_env` is `true`. `SHELL` is set to the shell
+being recorded. The shell then gets `PS1="$ "`, `TERM=xterm-256color`, and no history file, and
+`env` can change any of these. With `inherit_env`, `COLUMNS` and `LINES` are still removed so
+the terminal size is `cols` by `rows`, and `SHELL` is left as it is.
 
 Each step is one of these words or single-action mappings:
 

great_docs/_term_player/schemas/termshow-script.schema.json

@@ -408,10 +408,15 @@
           "type": "string"
         },
         "env": {
-          "description": "Environment variables to set on top of the current environment.",
+          "description": "Environment variables to set for the shell.",
           "type": "object",
           "additionalProperties": { "type": ["string", "number", "boolean"] }
         },
+        "inherit_env": {
+          "description": "Start the shell with the whole current environment. By default it only gets PATH, HOME, USER, LOGNAME, LANG, LC_*, TMPDIR, TZ, and VIRTUAL_ENV, so the recording doesn't depend on what your shell, editor, or CI service has set.",
+          "type": "boolean",
+          "default": false
+        },
         "typing_delay": {
           "description": "Seconds between typed characters.",
           "type": "number",

great_docs/_term_player/steps.py

@@ -14,6 +14,7 @@
 import re
 import select
 import shlex
+import shutil
 import sys
 import time
 from collections.abc import Callable
@@ -61,6 +62,9 @@
     "HISTFILE": "",
     "BASH_SILENCE_DEPRECATION_WARNING": "1",
 }
+# Kept from the current environment unless `record.inherit_env` is set, so a recording doesn't
+# depend on what your own shell, editor, or CI service has set. `LC_*` variables are kept too.
+KEPT_ENV = ("PATH", "HOME", "USER", "LOGNAME", "LANG", "TMPDIR", "TZ", "VIRTUAL_ENV")
 # Removed from the environment, since they'd override the terminal size
 _SIZE_VARS = ("COLUMNS", "LINES")
 
@@ -81,6 +85,22 @@ def key_sequence(name: str) -> str | None:
     return None
 
 
+def shell_env(settings: RecordSettings, environ: dict[str, str] | None = None) -> dict[str, str]:
+    """The environment the shell starts with.
+
+    Only `KEPT_ENV` and `LC_*` are taken from `environ` (the current environment), unless
+    `settings.inherit_env` is set. `DEFAULT_ENV` and the script's `record.env` go on top.
+    """
+    environ = dict(os.environ if environ is None else environ)
+    if settings.inherit_env:
+        env = {k: v for k, v in environ.items() if k not in _SIZE_VARS}
+    else:
+        env = {k: v for k, v in environ.items() if k in KEPT_ENV or k.startswith("LC_")}
+    env.update(DEFAULT_ENV)
+    env.update(settings.env)
+    return env
+
+
 def screen_text(emulator: TerminalEmulator) -> str:
     """The screen's text, with trailing spaces removed from each row and blank rows at the end
     left out."""
@@ -108,6 +128,7 @@ class RecordSettings:
     rows: int = 24
     cwd: Path | None = None
     env: dict[str, str] = field(default_factory=dict)
+    inherit_env: bool = False
     typing_delay: float = 0.05
     typing_jitter: float = 0.3
     timeout: float = 10.0
@@ -324,9 +345,10 @@ def start(self) -> None:
             argv = shlex.split(self.settings.shell)
         except ValueError as e:
             raise StepsError(f"can't read record.shell: {e}") from None
-        env = {k: v for k, v in os.environ.items() if k not in _SIZE_VARS}
-        env.update(DEFAULT_ENV)
-        env.update(self.settings.env)
+        env = shell_env(self.settings)
+        if not self.settings.inherit_env:
+            # Programs that start a subshell use the one being recorded
+            env.setdefault("SHELL", shutil.which(argv[0]) or argv[0])
         winsize = struct.pack("HHHH", self.settings.rows, self.settings.cols, 0, 0)
 
         pid, fd = pty.fork()

great_docs/cli.py

@@ -4614,7 +4614,16 @@ def _has_steps(script: str) -> bool:
 )
 @click.option("--update", is_flag=True, help="Re-record the recordings that have drifted")
 @click.option("--shell", type=str, default=None, help="Run the steps in this shell instead")
-def term_check(scripts: tuple[str, ...], ignore: tuple[str, ...], update: bool, shell: str | None):
+@click.option(
+    "--json", "as_json", is_flag=True, help="Print the results as JSON, for CI and other tools"
+)
+def term_check(
+    scripts: tuple[str, ...],
+    ignore: tuple[str, ...],
+    update: bool,
+    shell: str | None,
+    as_json: bool,
+):
     """Check that recordings made from steps still match what the steps produce.
 
     Each script's steps are run again, as `record --steps` would, and the
@@ -4629,10 +4638,15 @@ def term_check(scripts: tuple[str, ...], ignore: tuple[str, ...], update: bool,
     Exits with status 1 if any recording has drifted or a step failed. With
     --update, recordings that have drifted are recorded again instead.
 
+    With --json, the results are printed as one JSON object instead, with
+    the same exit status. Progress still goes to stderr.
+
     Examples:
       great-docs termshow check docs/assets/*.termshow.yml
       great-docs termshow check docs/assets/install.termshow --ignore 'v\\d+\\.\\d+'
+      great-docs termshow check docs/assets/*.termshow.yml --json > drift.json
     """
+    import json
     import re
 
     from ._term_player.check import CheckError, check_script
@@ -4647,54 +4661,80 @@ def term_check(scripts: tuple[str, ...], ignore: tuple[str, ...], update: bool,
                 f"{pattern!r} isn't a valid regular expression ({e})", param_hint="--ignore"
             ) from None
 
-    drifted = failed = 0
+    results = []
     for script in scripts:
         script_path = Path(script)
         if script_path.name.endswith(".termshow"):
             script_path = script_path.with_name(script_path.name + ".yml")
         click.echo(f"Checking {script_path}", err=True)
+        entry = {"script": script_path.as_posix(), "recording": None}
         try:
             result = check_script(script_path, ignore=ignore, shell=shell)
-        except CheckError as e:
-            click.echo(f"✗ {script_path}: {e}")
-            failed += 1
-            continue
-        except StepsError as e:
-            click.echo(f"✗ {script_path}: {e}")
-            if e.screen:
-                click.echo("\nThe screen at the time:")
-                for line in e.screen.splitlines():
-                    click.echo(f"  │ {line}")
-            failed += 1
+        except (CheckError, StepsError) as e:
+            entry |= {"status": "failed", "error": str(e)}
+            if isinstance(e, StepsError) and e.screen:
+                entry["screen"] = e.screen
+            results.append(entry)
+            if not as_json:
+                _echo_check_failure(entry)
             continue
 
-        screens = f"{result.snapshots} screen{'' if result.snapshots == 1 else 's'}"
-        if not result.drifted:
-            click.echo(f"✓ {result.recording_path} is up to date ({screens} compared)")
-            continue
-        drifted += 1
-        click.echo(f"✗ {result.recording_path} has drifted:\n")
-        for difference in result.differences:
-            for line in difference.lines:
-                color = {"+": "green", "-": "red", "@": "cyan"}.get(line[:1])
-                if line.startswith(("+++", "---")):
-                    color = None
-                click.secho(f"  {line}", fg=color)
-            click.echo()
-        if update:
+        entry |= {
+            "recording": result.recording_path.as_posix(),
+            "status": "drifted" if result.drifted else "up_to_date",
+            "compared": result.snapshots,
+            "differences": [{"at": d.where, "diff": d.lines} for d in result.differences],
+        }
+        if result.drifted and update:
             _write_termshow(result.recording, result.recording_path)
-            click.echo(f"↻ Recorded {result.recording_path} again")
-
-    if len(scripts) > 1:
+            entry["status"] = "updated"
+        results.append(entry)
+        if not as_json:
+            _echo_check_result(entry)
+
+    counts = {
+        status: sum(r["status"] == status for r in results)
+        for status in ("up_to_date", "drifted", "updated", "failed")
+    }
+    if as_json:
+        click.echo(json.dumps({"version": 1, "summary": counts, "results": results}, indent=2))
+    elif len(scripts) > 1:
+        changed = counts["drifted"] + counts["updated"]
         click.echo(
-            f"\n{len(scripts) - drifted - failed} of {len(scripts)} up to date"
-            + (f", {drifted} {'updated' if update else 'drifted'}" if drifted else "")
-            + (f", {failed} couldn't be checked" if failed else "")
+            f"\n{counts['up_to_date']} of {len(scripts)} up to date"
+            + (f", {changed} {'updated' if update else 'drifted'}" if changed else "")
+            + (f", {counts['failed']} couldn't be checked" if counts["failed"] else "")
         )
-    if failed or (drifted and not update):
+    if counts["failed"] or counts["drifted"]:
         raise SystemExit(1)
 
 
+def _echo_check_failure(entry: dict) -> None:
+    click.echo(f"✗ {entry['script']}: {entry['error']}")
+    if entry.get("screen"):
+        click.echo("\nThe screen at the time:")
+        for line in entry["screen"].splitlines():
+            click.echo(f"  │ {line}")
+
+
+def _echo_check_result(entry: dict) -> None:
+    recording = entry["recording"]
+    if entry["status"] == "up_to_date":
+        screens = f"{entry['compared']} screen{'' if entry['compared'] == 1 else 's'}"
+        click.echo(f"✓ {recording} is up to date ({screens} compared)")
+        return
+    click.echo(f"✗ {recording} has drifted:\n")
+    for difference in entry["differences"]:
+        for line in difference["diff"]:
+            color = {"+": "green", "-": "red", "@": "cyan"}.get(line[:1])
+            if line.startswith(("+++", "---")):
+                color = None
+            click.secho(f"  {line}", fg=color)
+        click.echo()
+    if entry["status"] == "updated":
+        click.echo(f"↻ Recorded {recording} again")
+
+
 @click.command("assets")
 @click.argument("dest", type=click.Path(file_okay=False))
 def term_assets(dest: str) -> None: