docs: bind marketing numbers to the published artifact (and fix four that were wrong)#220
Conversation
…that were wrong The savings claim alone existed as three different numbers inside this repo — 92% in the README blurb, the same 92% derived from a hand-computed "$2.05/M blended average", and 85% in package.json — none of them reproducible, all against an unstated baseline. Model counts said 55+; the catalog serves 66. Numbers now regenerate from blockrun.ai/brand/numbers.json via markers: reduces AI API costs by up to <!-- br:savings.autoVsBaselinePct -->87<!-- ... -->% and the savings section states the method instead of an "up to": 87% on auto, 98% on eco, against pinning Claude Opus 5 for 100% of traffic, priced on the live catalog from a published workload mix. The dimension count was wrong in the other direction, and only writing the test found it. Every comment in the scorer says 14 — but rules.ts builds a 14-entry list and then pushes the agentic dimension, so 15 are actually scored. The README was right and the code comments were stale. Comments corrected; the artifact now publishes 15. src/router/brand-numbers.test.ts pins the four facts BlockRun publishes about this repo but cannot count from its own catalog. They are asserted over there, so without a test here a scorer change would quietly make 37 READMEs wrong. It also covers the two surfaces markers cannot reach: package.json's description, which npm renders and which is not markdown, and the hero badge, where the count lives inside an HTML attribute. Two CI gaps closed while wiring this up: - npm test was never run. The Build & Test job only ran the lifecycle integration test, so all 713 unit tests could fail without failing CI. - brand numbers are now checked offline against the committed snapshot. scripts/sync-brand-numbers.mjs is vendored byte-for-byte and added to .prettierignore — formatting it here would fork it from the source repo.
📝 WalkthroughWalkthroughAdds centralized brand-number data, a synchronization CLI for marked documentation, consistency tests, updated dynamic marketing metadata, and CI steps for unit tests and offline drift validation. ChangesBrand number synchronization
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant BrandSnapshot
participant SyncCLI
participant RepositoryFiles
participant CI
BrandSnapshot->>SyncCLI: load brand-numbers.json
SyncCLI->>RepositoryFiles: scan and render br markers
RepositoryFiles-->>SyncCLI: return updated or drifted content
SyncCLI-->>CI: return check status
Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 6
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@README.md`:
- Line 37: Update the savings claim in the README description so it does not
present the auto-profile value from br:savings.autoVsBaselinePct as the maximum;
either use the eco savings value for “up to” or explicitly state that 87%
applies to auto, while preserving the existing marker-driven wording.
- Line 715: Update the opening ClawRouter description to use the hyphenated
adjective “MIT-licensed” in the parenthetical phrase, leaving the rest of the
description unchanged.
- Line 207: Update the dimension value in the fenced flow diagram from 14 to 15
so it matches the current weighted scorer dimension count, or move the diagram
outside the fenced block so sync-brand-numbers can detect future drift.
In `@scripts/sync-brand-numbers.mjs`:
- Around line 216-222: Update the main file-processing loop around syncFile so
changed content is only collected in drifted and not written immediately. After
all files are processed, validate problems and exit on failure before any
writes; then perform writeFileSync for the collected changes only when
validation succeeds, preserving check-mode behavior.
- Around line 38-40: Update the argument handling in the sync script around the
check and refresh flags to reject when both --check and --refresh are supplied,
before any refresh or network work begins. Emit a clear usage error and
terminate without modifying brand-numbers.json; preserve the existing behavior
for each flag when used independently.
- Around line 193-200: Update the walk generator to use non-following symlink
metadata when inspecting each path, and skip symlink entries before recursing or
yielding files. Preserve traversal for regular directories and supported text
files while ensuring symlinked directories and files are never followed or
modified.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 1d88da6d-ff1c-4c23-8726-3ab7771c54d6
📒 Files selected for processing (10)
.github/workflows/ci.yml.prettierignoreCLAUDE.mdREADME.mdbrand-numbers.jsonpackage.jsonscripts/sync-brand-numbers.mjssrc/router/brand-numbers.test.tssrc/router/config.tssrc/router/rules.ts
| </div> | ||
|
|
||
| > **ClawRouter** is an open-source smart LLM router that reduces AI API costs by up to 92%. It analyzes each request across 15 dimensions and routes to the cheapest capable model in under 1ms, entirely locally. ClawRouter is the only LLM router built for autonomous AI agents — it uses wallet signatures for authentication (no API keys) and USDC micropayments via the x402 protocol (no credit cards). 55+ models from OpenAI, Anthropic, Google, xAI, DeepSeek, and more. MIT licensed. | ||
| > **ClawRouter** is an open-source smart LLM router that reduces AI API costs by up to <!-- br:savings.autoVsBaselinePct -->87<!-- /br:savings.autoVsBaselinePct -->%. It analyzes each request across <!-- br:clawrouter.dimensions -->15<!-- /br:clawrouter.dimensions --> dimensions and routes to the cheapest capable model in under 1ms, entirely locally. ClawRouter is the only LLM router built for autonomous AI agents — it uses wallet signatures for authentication (no API keys) and USDC micropayments via the x402 protocol (no credit cards). <!-- br:models.chatVisible -->66<!-- /br:models.chatVisible --> models from OpenAI, Anthropic, Google, xAI, DeepSeek, and more. MIT licensed. |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Do not label the auto figure as the maximum.
This marker renders the auto-profile value (87%), while the same README documents 98% savings for eco. Either use the eco value for “up to” or change the wording to “87% on auto.”
🧰 Tools
🪛 LanguageTool
[grammar] ~37-~37: Ensure spelling is correct
Context: ... to the cheapest capable model in under 1ms, entirely locally. ClawRouter is the on...
(QB_NEW_EN_ORTHOGRAPHY_ERROR_IDS_1)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@README.md` at line 37, Update the savings claim in the README description so
it does not present the auto-profile value from br:savings.autoVsBaselinePct as
the maximum; either use the eco savings value for “up to” or explicitly state
that 87% applies to auto, while preserving the existing marker-driven wording.
|
|
||
| ``` | ||
| Request → Weighted Scorer (15 dimensions) → Tier → Best Model → Response | ||
| Request → Weighted Scorer (<!-- br:clawrouter.dimensions -->14<!-- /br:clawrouter.dimensions --> dimensions) → Tier → Best Model → Response |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Fix the dimension value inside the fenced flow diagram.
Markers inside fenced code blocks are intentionally ignored by scripts/sync-brand-numbers.mjs, so this value will remain 14 and CI will not detect drift. Change it to 15, or move the diagram outside the fenced block.
Proposed fix
-Request → Weighted Scorer (<!-- br:clawrouter.dimensions -->14<!-- /br:clawrouter.dimensions --> dimensions) → Tier → Best Model → Response
+Request → Weighted Scorer (15 dimensions) → Tier → Best Model → Response📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| Request → Weighted Scorer (<!-- br:clawrouter.dimensions -->14<!-- /br:clawrouter.dimensions --> dimensions) → Tier → Best Model → Response | |
| Request → Weighted Scorer (15 dimensions) → Tier → Best Model → Response |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@README.md` at line 207, Update the dimension value in the fenced flow diagram
from 14 to 15 so it matches the current weighted scorer dimension count, or move
the diagram outside the fenced block so sync-brand-numbers can detect future
drift.
| ### What is ClawRouter? | ||
|
|
||
| ClawRouter is an open-source (MIT licensed) smart LLM router built for autonomous AI agents. It analyzes each request across 15 dimensions and routes to the cheapest capable model in under 1ms, entirely locally — no external API calls needed for routing decisions. | ||
| ClawRouter is an open-source (MIT licensed) smart LLM router built for autonomous AI agents. It analyzes each request across <!-- br:clawrouter.dimensions -->15<!-- /br:clawrouter.dimensions --> dimensions and routes to the cheapest capable model in under 1ms, entirely locally — no external API calls needed for routing decisions. |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Use the hyphenated adjective “MIT-licensed.”
The phrase appears parenthetically before “smart LLM router,” so it should be hyphenated.
🧰 Tools
🪛 LanguageTool
[grammar] ~715-~715: Use a hyphen to join words.
Context: ...uter? ClawRouter is an open-source (MIT licensed) smart LLM router built for aut...
(QB_NEW_EN_HYPHEN)
[grammar] ~715-~715: Ensure spelling is correct
Context: ... to the cheapest capable model in under 1ms, entirely locally — no external API cal...
(QB_NEW_EN_ORTHOGRAPHY_ERROR_IDS_1)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@README.md` at line 715, Update the opening ClawRouter description to use the
hyphenated adjective “MIT-licensed” in the parenthetical phrase, leaving the
rest of the description unchanged.
Source: Linters/SAST tools
| const argv = new Set(process.argv.slice(2)); | ||
| const check = argv.has("--check"); | ||
| const refresh = argv.has("--refresh"); |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Reject --check --refresh together.
This combination takes the refresh path, fetching from the network and overwriting brand-numbers.json, despite --check being documented as offline and non-mutating.
Proposed fix
const check = argv.has("--check");
const refresh = argv.has("--refresh");
+if (check && refresh) fail("--check and --refresh cannot be combined");📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| const argv = new Set(process.argv.slice(2)); | |
| const check = argv.has("--check"); | |
| const refresh = argv.has("--refresh"); | |
| const argv = new Set(process.argv.slice(2)); | |
| const check = argv.has("--check"); | |
| const refresh = argv.has("--refresh"); | |
| if (check && refresh) fail("--check and --refresh cannot be combined"); |
🧰 Tools
🪛 ESLint
[error] 38-38: 'process' is not defined.
(no-undef)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@scripts/sync-brand-numbers.mjs` around lines 38 - 40, Update the argument
handling in the sync script around the check and refresh flags to reject when
both --check and --refresh are supplied, before any refresh or network work
begins. Emit a clear usage error and terminate without modifying
brand-numbers.json; preserve the existing behavior for each flag when used
independently.
| function* walk(dir) { | ||
| for (const name of readdirSync(dir)) { | ||
| if (SKIP_DIRS.has(name)) continue; | ||
| const p = join(dir, name); | ||
| const s = statSync(p); | ||
| if (s.isDirectory()) yield* walk(p); | ||
| else if (TEXT_EXT.has(extname(name))) yield p; | ||
| } |
There was a problem hiding this comment.
🔒 Security & Privacy | 🟠 Major | ⚡ Quick win
Do not follow symlinks during repository traversal.
statSync() follows source-controlled symlinks. A symlinked directory can recurse outside the repository or loop indefinitely; a symlinked Markdown file can cause normal sync mode to overwrite its target.
Proposed fix
-import { readFileSync, writeFileSync, readdirSync, statSync } from "node:fs";
+import { readFileSync, writeFileSync, readdirSync, lstatSync } from "node:fs";
- const s = statSync(p);
+ const s = lstatSync(p);
+ if (s.isSymbolicLink()) continue;📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| function* walk(dir) { | |
| for (const name of readdirSync(dir)) { | |
| if (SKIP_DIRS.has(name)) continue; | |
| const p = join(dir, name); | |
| const s = statSync(p); | |
| if (s.isDirectory()) yield* walk(p); | |
| else if (TEXT_EXT.has(extname(name))) yield p; | |
| } | |
| function* walk(dir) { | |
| for (const name of readdirSync(dir)) { | |
| if (SKIP_DIRS.has(name)) continue; | |
| const p = join(dir, name); | |
| const s = lstatSync(p); | |
| if (s.isSymbolicLink()) continue; | |
| if (s.isDirectory()) yield* walk(p); | |
| else if (TEXT_EXT.has(extname(name))) yield p; | |
| } | |
| } |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@scripts/sync-brand-numbers.mjs` around lines 193 - 200, Update the walk
generator to use non-following symlink metadata when inspecting each path, and
skip symlink entries before recursing or yielding files. Preserve traversal for
regular directories and supported text files while ensuring symlinked
directories and files are never followed or modified.
| for (const file of walk(ROOT)) { | ||
| const { before, after, changed, used } = syncFile(file, numbers, problems); | ||
| used.forEach((k) => everUsed.add(k)); | ||
| if (!changed) continue; | ||
| drifted.push({ file: relative(ROOT, file), before, after }); | ||
| if (!check) writeFileSync(file, after); | ||
| } |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
Validate marker problems before writing any files.
The loop writes each changed file before Lines 224-227 check problems. A nested or unbalanced marker therefore exits with failure after partially updating documentation. Collect drift first, fail on problems, then write only when validation succeeds.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@scripts/sync-brand-numbers.mjs` around lines 216 - 222, Update the main
file-processing loop around syncFile so changed content is only collected in
drifted and not written immediately. After all files are processed, validate
problems and exit on failure before any writes; then perform writeFileSync for
the collected changes only when validation succeeds, preserving check-mode
behavior.
1 participant
Consumes blockrun#288's
/brand/numbers.json.What was wrong
The dimensions one is the interesting failure
I set the artifact to 14 because every comment in the scorer says 14 —
config.ts:7,rules.ts:4,rules.ts:148. The README said 15 and I took it for the stale one.Writing the test proved it backwards.
rules.tsbuilds a 14-entry array and then does this:15 dimensions are scored. The README was right; the comments were stale and had been for a while. Comments corrected, artifact publishes 15.
This is the argument for the test living here rather than a number being asserted in blockrun: reading the code got it wrong, running it got it right.
The savings claim now states its method
Instead of an "up to" figure against an unstated baseline:
Models withheld from
/v1/modelsare excluded from the mix — pricing a public claim on a model a reader cannot look up is not defensible — which makes the number conservative.Two surfaces markers cannot reach
scripts/sync-brand-numbers.mjsonly rewrites markdown. Two places carry numbers it cannot touch, sosrc/router/brand-numbers.test.tsasserts them instead:package.jsondescription — npm renders it on the package page; not markdownTwo CI gaps found while wiring this up
npm testwas never run. TheBuild & Testjob only ran the lifecycle integration test, so all 713 unit tests could fail without failing CI. Now wired in (currently 713 pass, 3 skipped).Note on the vendored script
scripts/sync-brand-numbers.mjsis byte-for-byte from the source repo and added to.prettierignore— formatting it here would fork it from the copy in the other 36 repos.Left alone deliberately
free/mistral-large-3-675b,kimi-k2.7, …). Some look stale againstconfig.ts, but model names are not what this artifact owns and verifying them is a routing-config question, not a numbers one. Worth a separate pass.~44 BlockRun modelsin the setup warning refers to what gets synced into OpenClaw's allowlist, not the catalog — left as-is rather than guessed at.Summary by CodeRabbit
New Features
Documentation
Tests