Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
54 changes: 31 additions & 23 deletions .github/workflows/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ Stored the app credentials in `wandb/docs`:

The workflows use `actions/create-github-app-token@v3` to create short-lived installation tokens from these credentials.

Workflows that push back to a same-repo PR branch with this token (instead of the default workflow `GITHUB_TOKEN`) include **Compress Images** (`calibreapp-image-actions.yml`), **Build CSS** (`build-css.yml`), and **Knowledgebase Nav** (`knowledgebase-nav.yml`). That way downstream `pull_request` checks (for example **Validate MDX**) still run on the automation commit.
Workflows that push back to a same-repo PR branch with this token (instead of the default workflow `GITHUB_TOKEN`) include **Compress Images** (`calibreapp-image-actions.yml`), **Build CSS** (`build-css.yml`), and **Knowledgebase Nav** (`knowledgebase-nav.yml`). That way downstream `pull_request` checks (for example **Doc quality**) still run on the automation commit.

## Action pinning

Expand Down Expand Up @@ -151,46 +151,54 @@ git commit -m "Sync code examples from docs-code-eval"
git push
```

## Readability delta
## Doc quality

**Workflow**: `readability-delta.yml`
**Workflow**: `doc-quality.yml`

Posts an informational, **non-blocking** PR comment describing how the PR affects the readability of the English docs it changes (DOCS-2626). It reports the *delta* (before/after) for well-established formulas (Flesch-Kincaid grade, Flesch reading ease, Gunning fog, SMOG), word-weighted across the changed pages, plus an optional AI-agent-comprehension rating from a W&B Inference LLM judge.
One workflow, one runner job, one upserted PR comment for the three checkout-based doc-quality checks. It replaces the former `validate-mdx.yml`, `linkcheck-pr.yml`, and `readability-delta.yml` workflows. The job id stays `validate-mdx` so the branch-protection required check keeps its name.

### Triggers
The three checks:

- **Pull request**: `opened`, `synchronize`, `reopened` on PRs that touch `**/*.mdx`
- **Manual**: `workflow_dispatch` (writes the report to the job summary instead of a comment)
1. **MDX validation** (**blocking**): `mint validate` + `mint broken-links` via `scripts/mdx-validation/validate-mdx-mintlify.sh`, using the Mintlify CLI from npm (cached ~4 days). `mint broken-links` owns *internal* links because it resolves them against the real route table (docs.json navigation and redirects).
2. **External link check** (informational): `lychee` over the changed `.mdx` files only, external URLs only (`lychee.toml` excludes `^file://`). Runs on plain `pull_request` from the git checkout — it does not wait for the Mintlify preview deployment (the old `deployment_status` trigger was vestigial scheduling and carried the DOCS-3041 wrong-PR-resolution bug).
3. **Readability delta** (informational, DOCS-2626): reports how the PR affects the readability of the English docs it changes — the *delta* (before/after) for well-established formulas (Flesch-Kincaid grade, Flesch reading ease, Gunning fog, SMOG), word-weighted across the changed pages, plus an optional AI-agent-comprehension rating from a W&B Inference LLM judge. Localized content under `ja/`, `ko/`, and `fr/` is skipped.

### What it does
Only MDX validation can fail the job; the other two never fail a PR. All three results land in a single PR comment identified by the `<!-- doc-quality-report -->` marker (status table plus collapsed details). A migration step deletes leftover comments from the replaced workflows (`<!-- lychee-link-checker-comment -->` and `<!-- readability-delta-report -->` markers) on open PRs.

1. Diffs the PR base and head, scoring each changed English `.mdx` file (localized content under `ja/`, `ko/`, and `fr/` is skipped).
2. Extracts narrative prose and scores it with `textstat` via the analyzer in the `coreweave/docs-skills` submodule (`.claude/scripts/_readability.py`).
3. Optionally runs the AI agent comprehension judge (W&B Inference) when `WANDB_API_KEY` is set.
4. Upserts a single PR comment identified by the `<!-- readability-delta-report -->` marker.
### Triggers

The check **never fails** a PR. If scoring is unavailable it posts a brief notice and exits successfully.
- **Pull request**: the default activity types (`opened`, `synchronize`, `reopened`), deliberately with **no `paths` filter** — a paths filter would leave the required check pending on PRs it skips. A scope-gate `git diff` inside the job fast-exits instead when a PR touches no Mintlify-relevant files (`.mdx`, `.json`, `.yaml`/`.yml`, images, or the validation script).
- **Push**: to `main` (full validation)
- **Manual**: `workflow_dispatch` (writes the report to the job summary instead of a comment). Dispatch runs the full MDX validation only: the link and readability checks diff against `main`, so on `main` itself the change list is empty and they skip. The old `linkcheck-pr.yml` dispatch fell back to a full-tree external-link sweep in that case; that behavior was **not** carried over — the closest substitute is `linkcheck-prod.yml`, which checks the live site's sitemap (monthly cron, also manually dispatchable).

### Configuration

- **Python**: 3.11
- **Node**: 20 (Mintlify CLI), **Python**: 3.11 (readability)
- **Permissions**: `contents: read`, `pull-requests: write`
- **Report glue**: `scripts/readability/pr_report.py`
- **Scoring logic**: `.claude/scripts/_readability.py` and `_docs_eval_lib.py` (submodule)
- **Validation script**: `scripts/mdx-validation/validate-mdx-mintlify.sh`
- **Link-check config**: `lychee.toml` (repo root)
- **Readability report glue**: `scripts/readability/pr_report.py`
- **Readability scoring logic**: `.claude/scripts/_readability.py` and `_docs_eval_lib.py` (submodule)

### Authentication

- The main checkout uses the default `GITHUB_TOKEN`.
- The private, cross-org `coreweave/docs-skills` submodule is initialized in a separate step with the `DOCENGINE_TOKEN` secret (the same `x-access-token` credential used for the `gitsubmodule` ecosystem in `.github/dependabot.yml`; it rotates ~every 30 days and needs no `wandb/docs` scope).
- The main checkout, the lychee rate-limit token, and PR comments use the default `GITHUB_TOKEN`.
- The private, cross-org `coreweave/docs-skills` submodule is initialized in a separate step with the `DOCENGINE_TOKEN` secret (the same `x-access-token` credential used for the `gitsubmodule` ecosystem in `.github/dependabot.yml`; it rotates ~every 30 days and needs no `wandb/docs` scope). Readability only.
- The AI agent comprehension judge calls W&B Inference with the `WANDB_DOCS_INFERENCE_API_KEY` secret (a W&B API key whose entity has Inference credits), passed to the scorer as `WANDB_API_KEY`. When that secret is absent, the deterministic `textstat` delta still runs.

### Forks

Fork PRs have no access to repo secrets, so the first step detects a fork, posts an Actions notice, and makes the whole job a no-op (still reporting success). Forks are uncommon in `wandb/docs` and coreweave repos cannot use forks at all.
Fork PRs have no access to repo secrets and get a read-only `GITHUB_TOKEN`. MDX validation and the external link check need no secrets, so they still run (and validation still blocks). The readability delta skips (it needs the private submodule and the Inference key), and the report goes to the workflow step summary instead of a PR comment. Forks are uncommon in `wandb/docs` and coreweave repos cannot use forks at all.

### Related workflows (not consolidated)

- `mintlify-deployment-preview.yml` needs the deployment's `environment_url`, so it stays on `deployment_status`.
- `linkcheck-prod.yml` checks the live site on a monthly cron.
- The PinGuard pair and the commit-pushing bots (Compress Images, Build CSS, Knowledgebase Nav) use the `wandb-docs-pr-writer` App token — a different trust category that should not share a job with fork-facing validation.

### Related Files

- **Report glue**: `scripts/readability/pr_report.py`
- **Dependencies**: `scripts/readability/requirements.txt`
- **Tests**: `scripts/readability/tests/`
- **Documentation**: `scripts/readability/README.md`
- **Readability report glue**: `scripts/readability/pr_report.py`
- **Readability dependencies**: `scripts/readability/requirements.txt`
- **Readability tests**: `scripts/readability/tests/`
- **Readability documentation**: `scripts/readability/README.md`
2 changes: 1 addition & 1 deletion .github/workflows/build-css.yml
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ jobs:
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
with:
# App token rather than the default GITHUB_TOKEN so the rebuild
# commit triggers downstream workflows (for example Validate MDX).
# commit triggers downstream workflows (for example Doc quality).
# Fork PRs are already excluded by the job-level `if` above.
token: ${{ steps.app-token.outputs.token }}
ref: ${{ github.head_ref }}
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/calibreapp-image-actions.yml
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,6 @@ jobs:
avifQuality: '75'
minPctChange: '10'
# GitHub App token with contents read-write. Unlike the default workflow `GITHUB_TOKEN`,
# commits pushed with this token trigger other workflows (for example Validate MDX).
# commits pushed with this token trigger other workflows (for example Doc quality).
# Fork PRs are already excluded by the job `if` above.
GITHUB_TOKEN: ${{ steps.app-token.outputs.token }}
Loading
Loading