feat(app): add civitai app metrics <slug> for owner-only App analytics

[ZacxDev]

What

Adds civitai app metrics <slug> — the authenticated owner's analytics for one of
their own App Blocks: installs, runs + Buzz spent, Buzz purchased, and API
engagement. Registered under newAppCmd() alongside status / withdraw.

This is CLI plumbing only — no server change. The blocks.getMyAppAnalytics
tRPC proc already exists and is live on civitai.com; nothing here adds or alters
a server-side route. The slug → appBlockId resolution reuses the existing
GET /api/v1/blocks/submissions client (appapi.Lister), the same route
civitai app status reads — no new resolution endpoint was introduced.

$ civitai app metrics gen-matrix --from 2026-05-01 --to 2026-08-03
App:          gen-matrix
Window:       2026-05-01 00:00 UTC → 2026-08-03 00:00 UTC
Granularity:  week

Installs
  Total   12
  Active  9

Runs
  Count       20
  Buzz spent  65

Buzz purchased
  Purchases  3
  Buzz       15000
  Gross      $14.97

Engagement
  API calls     26
  Active users  2
  Error rate    0

  Top scopes:
    ai:write:budgeted  20

  Top endpoints:
    /api/v1/blocks/me  4

The three correctness requirements, and how each is tested

Each of these is a case where the obvious implementation prints something that
looks like data but isn't. They are the reason the command is shaped this way.

1. The printed window comes from the response, never the flags

The server defaults to the last 30 days, clamps any request to 366 days,
and returns zeros for data that exists outside that window — an app with 20
runs in mid-June reads 0 under the default window. So a zero is meaningless
without the period it covers, and echoing the requested window would be a lie
whenever the server clamped it. The renderer prints range.from / range.to /
range.granularity straight from the payload.

Tested by TestAppMetricsClampedRangeEchoesResponseNotFlags: the CLI requests
--from 2020-01-01, the fake server answers with a range clamped to
2025-08-03, and the test asserts the output contains the clamped start and
does not contain 2020-01-01. TestAppMetricsRendersVerifiedPayload and
TestAppMetricsGenuineZeroRendersWithWindow additionally pin that the window is
present on both a populated and an all-zero read.

2. "Not entitled" is distinguished from "genuinely zero"

The proc sits behind the appBlocksAuthor feature flag (moderators-only by
default) and, when the caller doesn't match it or doesn't own the app, answers
HTTP 200 with every counter zeroed — flagged only by the payload's
notOwned boolean. Rendering that would present a permission failure as a real,
empty dashboard, which is the primary failure mode here. On notOwned the
command renders nothing and returns an actionable error naming
civitai whoami and civitai app status <slug>.

Tested by TestAppMetricsNotOwnedRefusesDashboard, which asserts both halves:
the error text, and that stdout contains none of Installs / Runs /
Buzz purchased / Engagement / Granularity. Its complement,
TestAppMetricsGenuineZeroRendersWithWindow, feeds all-zero counts with
notOwned:false and asserts the dashboard does render (with its window) and is
not reported as a permission problem — so the two cases can't collapse into
each other.

3. A 403 names the fix instead of saying "forbidden"

The proc declares no requiredScope, so it defaults to full scope: an OAuth
login token gets a 403 where a personal API key succeeds. The 403 branch says to
create a key and run civitai login --token <key>, and points at
civitai whoami to see which credential is active. Per AGENTS.md item 4, no
scope bitmask is vendored — this is purely the HTTP status plus guidance.

Tested by TestAppMetricsForbiddenAsksForPersonalAPIKey, which asserts the
error contains 403, personal API key, civitai login --token and OAuth —
not merely that some error occurred.

Also fixed while building it

A literal null tRPC payload ({"result":{"data":{"json":null}}}) unmarshals
cleanly into the zero struct and would have rendered as an all-zero dashboard
over an empty window — the exact silent-empty failure requirement 2 is about. It
is now rejected as malformed. This was caught by a test written before the code
handled it, and watched fail first.

Flags

• --from / --to — a bare YYYY-MM-DD (interpreted as midnight UTC, so
  the window is reproducible regardless of where the CLI runs) or a full RFC3339
  timestamp, normalised to UTC on the wire. A malformed value or an inverted
  window is an ErrUsage → exit 2, decided client-side before any request
  (asserted: the tests check no HTTP call was made). Omitting both bounds omits
  them from the wire so the server's own 30-day default applies.
• --json — passes the server's unwrapped payload through verbatim (not
  re-marshalled through the CLI's structs), so scripts keep every field
  including notOwned and the per-bucket series arrays the human view omits.

Test coverage

internal/cmd/app_metrics_test.go drives the command end-to-end through
NewRootCmd() + SetArgs against an httptest server serving both routes
(submissions + tRPC) — no live network. 31 test cases including subtests:

Path	Test
happy path, verified-shape fixture	TestAppMetricsRendersVerifiedPayload
--json passthrough (unwrapped, no rendered text)	TestAppMetricsJSONPassthrough
notOwned:true → no dashboard	TestAppMetricsNotOwnedRefusesDashboard
genuine zero + window printed	TestAppMetricsGenuineZeroRendersWithWindow
server clamps the range	TestAppMetricsClampedRangeEchoesResponseNotFlags
403 → personal-API-key guidance	TestAppMetricsForbiddenAsksForPersonalAPIKey
401 → civitai login	TestAppMetricsUnauthorizedPointsAtLogin
no token configured	TestAppMetricsMissingTokenErrors
slug with no submissions	TestAppMetricsUnknownSlugIsActionableNotFound
all appBlockId: null → "no approved block yet"	TestAppMetricsNoApprovedBlockYet
newest row null, older row approved	TestAppMetricsPicksNewestNonNullAppBlockID
malformed envelope ×5 (non-JSON, no data, null, wrong type, empty)	TestAppMetricsMalformedEnvelopeIsCleanError
--from/--to parsing ×3 (date, RFC3339, offset→UTC)	TestAppMetricsWindowFlagsAccepted
bad window ×3 → exit 2, no request made	TestAppMetricsBadWindowIsUsageError
both bounds omitted when unset	TestAppMetricsNoWindowFlagsOmitsBothBounds
resolution failure surfaces	TestAppMetricsSubmissionsErrorSurfaces
formatting units	TestUSDFromCents, TestUTCStamp

Every error-path test asserts the specific message or behaviour, not merely
that an error occurred.

Mutation sweep — 13/13 mutants killed. Each guard was broken on purpose and
confirmed to go red on its own assertion (not a different guard's): the window
echo, the notOwned branch (two mutants — one that skips it, one that renders
before erroring, so both halves of the assertion are load-bearing), the 403
message, the null-payload rejection, the resolver's fall-through to the
approved row, both asUsageError tags, the inverted-window check, date-only
parsing, --json passthrough, and both formatting helpers. The sweep was run
with a green baseline before and after, and the tree verified restored.

Verification

$ make ci
go mod tidy
go vet ./...
go test ./...
ok  	github.com/civitai/cli	0.016s
ok  	github.com/civitai/cli/cmd/civitai	0.023s
ok  	github.com/civitai/cli/internal/antipattern	0.006s
ok  	github.com/civitai/cli/internal/appapi	0.194s
ok  	github.com/civitai/cli/internal/auth	0.006s
ok  	github.com/civitai/cli/internal/cmd	7.112s
ok  	github.com/civitai/cli/internal/config	0.005s
ok  	github.com/civitai/cli/internal/devtunnel	0.027s
ok  	github.com/civitai/cli/internal/dnsprobe	0.457s
ok  	github.com/civitai/cli/internal/manifest	0.005s
ok  	github.com/civitai/cli/internal/pkgzip	0.045s
ok  	github.com/civitai/cli/internal/scaffold	0.025s
ok  	github.com/civitai/cli/internal/scaffold/cmd/bump-pins	0.005s
ok  	github.com/civitai/cli/internal/ui	0.003s
ok  	github.com/civitai/cli/internal/validate	0.021s
ok  	github.com/civitai/cli/pkg/civitai	0.128s
go build ...

Counted rather than read off the exit code: 1311 === RUN, 1309 --- PASS,
0 --- FAIL, 2 --- SKIP, 0 panics, 16 packages ok. Both skips
(TestScanDirFromEnv, TestScaffoldPinsSatisfyPublished) are pre-existing and
environment-gated, unrelated to this change. gofmt -s -l . prints nothing.

The built binary was also driven against a local fake server for the three
user-visible paths: the render above (exit 0), notOwned (error, no dashboard,
exit 1), and --from last-tuesday (usage error, exit 2).

Docs

README gets the command-table row and an App metrics section (the sample
output above is real CLI output, not hand-written), covering the three
requirements plus the data caveat: engagement counts only authenticated,
scope-gated API calls, so an app with no scoped API surface shows real
installs and revenue with a flat engagement section — expected, not a bug. The
same caveat is in the command's Long help, and the human output prints a note
when apiCalls == 0.

Note for review

errorRate is rendered verbatim rather than as a percentage. Its unit isn't
determinable from the verified payload (the observed value is 0), and
multiplying by 100 on a guess would be a silent 100× error in a user-facing
figure. If the server's contract says it's a 0–1 ratio, converting it is a
one-line follow-up.

🤖 Generated with Claude Code

[ZacxDev]

Audit fixes pushed — 07731cf

Three findings addressed. Nothing was taken on faith: every new assertion was watched go red under a deliberate mutation before being trusted.

1. errorRate rendered as a raw ratio 🟡

engagement.errorRate is a 0–1 ratio, settled from the server source rather than guessed — app-analytics.service.ts:276 computes:

const errorRate = apiCalls > 0 ? errorCount / apiCalls : 0;

The human view printed it verbatim, so a real app read Error rate 0.02040816326530612. It now goes through a pctFromRatio helper at one decimal place. Observed from the built binary against a local fake server serving that exact live payload:

  Error rate    2.0%

and an exact zero reads 0.0% (not 0.0000%, not NaN). --json is untouched — the same run confirms it still emits "errorRate": 0.02040816326530612 verbatim.

2. The exit-code contract was untested 🟡 (the important one)

The audit stripped the classification while leaving every error message byte-identical, and the whole TestAppMetrics* suite stayed green — so README + the taxonomy at root.go:205-216 promised 401/403 → exit 3 and 404 → exit 4 with nothing pinning it.

Fixed by adding errors.Is assertions alongside the existing message assertions (none were weakened or restructured), mirroring how the usage-error test already pins ErrUsage:

test	new assertion
TestAppMetricsUnauthorizedPointsAtLogin	errors.Is(err, civitai.ErrUnauthorized) — exit 3
TestAppMetricsForbiddenAsksForPersonalAPIKey	errors.Is(err, civitai.ErrUnauthorized) — exit 3
TestAppMetricsUnknownSlugIsActionableNotFound	errors.Is(err, civitai.ErrNotFound) — exit 4
TestAppMetricsServer404ClassifiesNotFound (new)	errors.Is(err, civitai.ErrNotFound) — exit 4

The last one is new because the two not-found paths are tagged differently: the unknown-slug guard uses an explicit civitai.Tag, while an analytics-query 404 is tagged by analyticsError's TagStatus deferral. One test could not cover both.

3. --json fails open on notOwned 🟡

Behaviour deliberately unchanged — raw passthrough at exit 0 is the point of the flag, and the test at app_metrics_test.go pinning it is left alone. Documented instead, in the --json flag help and in README.md: a notOwned: true payload is passed through with every counter zeroed and still exits 0, so civitai app metrics <slug> --json | jq .runs.count returns 0 for an app you can't see, and a script must branch on notOwned rather than trusting the counts.

Bonus: a rendering bug found while writing that doc string

pflag's UnquoteUsage lifts the first back-quoted span out of a usage string and uses it as the flag's value name. Mentioning a back-quoted `notOwned: true` payload therefore rendered the boolean as:

--json notOwned: true   emit the raw analytics payload …

i.e. the help told you to pass an argument to a flag that takes none. Back-quotes dropped; TestAppMetricsJSONFlagRendersAsABoolean now pins it.

---

Verification

Mutation matrix — each mutation applied to the source, suite re-run, failure message recorded, source restored. M0 is the no-mutation control (proves the harness can observe a pass); M2 and M3 are exactly the two mutations that previously stayed green.

#	mutation	result	failing assertion
M0	(control — none)	green	—
M1	pctFromRatio → FormatFloat(…, 'g', -1, 64)	red ×3	metrics output missing "0.0%" · error rate 0.25 should render as 25.0% · errorRate 0.020408… should render as 2.0% · the human view must not print the raw ratio
M2	defer func() { err = civitai.TagStatus(status, err) }() → no-op	red ×3	a 401 must classify as ErrUnauthorized (exit 3), got *errors.errorString · same for 403 · a 404 must classify as ErrNotFound (exit 4)
M3	drop civitai.Tag(civitai.ErrNotFound, …) on the unknown-slug guard	red ×1	an unknown slug must classify as ErrNotFound (exit 4), got *errors.errorString
M4	remove the notOwned sentence from the --json help	red ×1	metrics help missing "notowned"
M5	pctFromRatio drops the *100	red ×1	pctFromRatio(0.25) = "0.2%", want "25.0%"
M6	reintroduce back-quotes in the --json usage string	red ×1	--json picked up a value name from a back-quoted usage span

Under M2 and M3 only the new classification assertions fire — every message assertion still passes, which is the whole point: those mutations are invisible to message-shape tests.

Full suite (go test ./... -count=1 -v, counted rather than read off the exit code):

=== RUN   : 1315
--- PASS  : 1313
--- FAIL  : 0
--- SKIP  : 2      (TestScanDirFromEnv, TestScaffoldPinsSatisfyPublished — pre-existing, environment-gated)
panic: test timed out : 0

gofmt -s -l . prints nothing, go vet ./... clean, go build ./... ok.

README sample — the Error rate line in the civitai app metrics sample block was regenerated by running the built binary against a local fake server (payload matching the sample's other figures: 1 error in 26 calls), not hand-edited. It now reads Error rate 3.8%.

No live credentials were used; both binary checks ran against a throwaway loopback server. No force-push, not merged.

[ZacxDev]

Closed the last audit finding: a real-but-tiny error rate no longer reads 0.0%

Pushed 9315070.

The finding. The previous round's pctFromRatio (one-decimal percentage) collapsed every ratio below 0.0005 to 0.0% — byte-identical to a genuine zero. An app with 5,000 API calls and 2 errors (errorRate: 0.0004) printed Error rate 0.0%, i.e. as having no errors at all. That band is exactly where a healthy, high-traffic app sits, so the information loss landed on the population this command is most useful for. The raw passthrough it replaced did distinguish the two.

The fix. Anything nonzero that would otherwise round to zero now renders <0.1%:

pct := strconv.FormatFloat(r*100, 'f', 1, 64)
if r > 0 && pct == "0.0" {
    return "<0.1%"
}
return pct + "%"

The branch is decided on the rendered string, not a hard-coded 0.0005 constant, so the threshold and the formatter cannot drift apart at the float64 boundary. Constraints held: an exact 0 still prints 0.0%; everything at or above the threshold keeps the existing one-decimal form; --json is untouched and still passes the server's raw ratio through verbatim (the pinning test is unchanged and still green).

Also addressed the 🟢 on the same function: the doc comment claimed only that "an exact zero reads 0.0%" — true of inexact near-zeros too, which is what let this slip. It now states the real behaviour and pins the [0,1] precondition to the server invariant it rests on: errorCount counts a strict subset of the rows apiCalls counts (same block_scope_invocations filter plus statusCode >= 400, service lines 241-256), and the divide is guarded by apiCalls > 0 — so negative / >1 / NaN / Inf are not producible by this server, and the function deliberately does not defend against them.

Test gap that let it ship

TestPctFromRatio covered {0, 0.0204, 0.25, 0.005, 1} and never entered the sub-0.0005 band. It now covers the band (0.0004, 1e-5, math.SmallestNonzeroFloat64) and both sides of the boundary (math.Nextafter(0.0005, 0) → <0.1%, 0.0005 → 0.1%). Two new tests:

• TestPctFromRatioTinyRateIsNotZero — the discriminating one. It asserts nothing about what the small rendering is, only that a nonzero rate can never produce the same string as zero. Reverting the fix fails it.
• TestAppMetricsTinyErrorRateRendersDistinctly — the same regression end-to-end through the command: the dashboard shows <0.1% while --json still emits 0.0004.

Verification

Mutation matrix — every new assertion was watched fail, with its own error message, and the source restored byte-identically afterwards (sha256 match). Control run before the sweep was green (14/14), so the harness can report both colours.

Mutant	Result	Which new assertions went red
control (unmutated)	14 pass / 0 fail	—
M1 revert to bare FormatFloat(r*100,'f',1,64) (the exact regression)	7 fail	all 4 band cases, TinyRateIsNotZero (all 4 ratios), the e2e dashboard assertions
M2 drop the r > 0 guard	3 fail	exact zero stays 0.0% + the TinyRateIsNotZero zero-precondition Fatalf
M3 shift the band to pct == "0.1"	8 fail	all band cases and exactly the 0.0005 boundary (0.0005 wrongly became <0.1%) — pins both sides
M4 disable the --json passthrough branch	2 fail	both --json assertions in the new e2e test (positive control proving that half is wired to real output)

M2 and M3 exist because M1 alone leaves the exact-zero case and the upper boundary untested — they'd pass with the guard broken in the other direction.

Gate. make ci green, gofmt -s -l . empty. Counted, not read from the exit code: RUN 1327 / PASS 1325 / FAIL 0 / SKIP 2, no panic: test timed out. Baseline before the change was 1315/1313/0/2, so +12 = 10 new TestPctFromRatio subtests + 2 new tests. The 2 skips are the pre-existing env-gated TestScanDirFromEnv and TestScaffoldPinsSatisfyPublished.

Live rendering — the built binary (not the test harness) driven against a local fake server, since there are no live credentials here:

server errorRate	human Error rate	--json
0	0.0%	0
0.0004	<0.1%	0.0004
1e-5	<0.1%	1e-05
0.0005	0.1%	0.0005
0.02040816326530612	2.0% (unchanged)	0.02040816326530612
0.25	25.0% (unchanged)	0.25

The harness carried a negative control (an unknown slug must exit non-zero — it exits 4), so these rows are known to be real round-trips rather than a silently-stubbed path.

README. The sample block's Error rate 3.8% is unaffected and still matches what the binary prints for that ratio; the surrounding paragraph gained one sentence noting that only a genuine zero reads 0.0%. Its other numbers were left alone — it's an illustrative composite, not a capture.

🤖 Generated with Claude Code

https://claude.ai/code/session_01858ymA3tEJQi83435u7npi