feat(experiments): target-scoped PLP ranking, so any PLP is testable

[hugo-ccabral]

Makes the PLP ranking A/B capability generic, so any collection-driven PLP can be tested instead of one hand-wired page. Design: 04_engineering/platform/experiments/README.md in context (contract 1 updated in a companion PR).

The hole this closes

Contract 1 published {key, variants} with no target — even though the control plane's experiments table has modelled target_kind + a target id from the start. So the runtime received an experiment with no way to know which of a site's PLPs it belonged to.

The only way left to scope it was to hardcode a page in site code. That is what the first cut of the consuming PR did, and it is actively wrong: FARM Rio has three CMS pages sharing one loader — /produtos (2258), /bazar (2259), /produtos/vestido (2247) — each curating a different product set. An arm precomputed for one of them applied to all three serves Bazar's shoppers the main PLP's catalogue.

It also could not reach most PLPs at all: 1936 of FARM Rio's pages call intelligentSearch/productListingPage.ts straight from their CMS block and never touch site code, 953 of them passing a hardcoded productClusterIds.

What changes

packages/blocks — ExperimentDefinition gains optional targetKind + target, and a resolveExperimentForTarget(kind, target) lookup joins the existing key lookup. The assignment moves into a shared decide() so the two cannot drift.

Both fields are optional, so an experiment without them resolves by key exactly as before — no behaviour change for anything published today.

The assignment stays keyed on experiment.key, never the target: the cookie and the analytics join identify the cohort, not the surface. The target is deliberately not encoded into the key (plp-ranking:2258) — that forces a splitByChar before every GROUP BY, the same objection contract 4 raises against gluing experiment and variant.

packages/apps-vtex — vtexProductListingPage resolves the incoming productClusterIds facet and swaps it in place when an arm targets that collection.

• Replaces, never appends. Two collections OR'd together are neither model's ranking, and the filter-chip and pagination hrefs are built from that same facet array — an appended facet leaks into every link on the page, where a control-arm visitor opening a shared link inherits the other arm's collection while still being tagged control.
• Inert without a published experiment for that exact collection: nothing recorded, facets returned untouched. Exposure therefore always means "could actually be affected", which the analysis requires.

Why in apps-vtex and not in each site

It is the single entry point every collection-driven PLP already flows through, so it reaches the 1936 pages no site-side change can. It is also sync-proof: FARM Rio's .deco blocks are regenerated from the Fresh site — 143 sync: .deco content from farmrio commits landed during one review cycle — so a block-level opt-in prop would simply be overwritten. Every VTEX deco site gets the capability for free.

A nice consequence: FARM Rio's region loader picks a Sul-specific collection for PR/RS/SC visitors. Because matching is by target, those visitors simply don't match an experiment aimed at the default collection — regional shoppers are excluded from a ranking test with no special-casing anywhere.

Tests

31 passed in packages/blocks/src/sdk/experiments.test.ts — 5 new, covering: the right experiment wins per surface (/bazar never inherits /produtos' arm), an untargeted surface returns null and records no assignment, kind alone doesn't match, untargeted contract-1 documents still resolve by key, and the cookie is keyed on the experiment key rather than the target.

No pre-existing test regressed from the decide() extraction. packages/blocks and packages/apps-vtex both typecheck clean.

🤖 Generated with Claude Code

---

Summary by cubic

Makes PLP ranking experiments target-scoped so any collection-driven PLP can be tested instead of one hardwired page. Previously, experiments resolved by key with no target, forcing a hardcoded page that applied one PLP's arm to every PLP sharing the loader.

• ExperimentDefinition gains optional targetKind and target; experiments without them resolve by key with no behavior change.
• New resolveExperimentForTarget lookup assigns per surface, sharing assignment logic with the existing key lookup.
• vtexProductListingPage swaps in the assigned collection for the search only, replacing — never appending — the productClusterIds facet; the page's own links keep the original facets.
• Duplicate experiment keys are dropped at read time, keeping the first, so colliding experiments can't thrash the assignment cookie.
• Untargeted PLPs record no assignment and return facets untouched, so exposure always means the visitor could actually be affected.

^{Written for commit 3d5360a. Summary will update on new commits.}

[hugo-ccabral]

Verdict: BLOCK — the swap is in-place, but every outbound link on the page is serialised from the swapped array, so the arm's collection escapes into filter.productClusterIds and gets appended on the next request. That is the exact leak the PR says replacing-not-appending prevents; it just happens one hop later. (round 1)

Blocking

• packages/apps-vtex/src/loaders/intelligentSearch/productListingPage.ts:439 — facets is reassigned to the swapped array, and that same array builds every href on the page: toFilter(facets, ...) at :512 (chip URLs via filtersToSearchParams, :190-198) and the pagination loop at :523-526. Both emit filter.<key>=<value> for every facet, including the collection one. So a treatment visitor's "page 2" and filter-chip links carry ?filter.productClusterIds=412.

  Trigger. Treatment visitor on /produtos (CMS block hardcodes productClusterIds=2258, arm payload 412). They click any filter chip or page 2 → the URL now carries filter.productClusterIds=412. Open that URL as a control visitor (shared link, or the same visitor re-rolled to control when the control plane advances a ramp weight and the fingerprint goes stale):

  1. :409-416 appends {productClusterIds, 412} from the URL — the dedupe at :412 requires key and value to match, and 2258 !== 412.
  2. applyPlpRankingExperiment findIndexes the first collection facet (index 0, value 2258), resolves the control arm, writes 2258 back into index 0.
  3. Result: [{pCI,2258},{pCI,412}] → facetPath productClusterIds/2258/productClusterIds/412.

  A visitor tagged control is served both arms' collections combined — "neither model's ranking", exactly as the PR body argues against. The in-place swap prevents the append inside this function; it does not prevent the append the link itself causes on the following request.

  Two follow-ons from the same root: (a) findIndex at :361 only ever swaps the first collection facet, so once a page carries two the second is left untouched; (b) crawlers have no cookie, get assigned, and the crawled pagination hrefs carry the arm collection — those indexed URLs outlive the experiment, and once collection 412 is retired they render an empty PLP.

  Fix: keep the pre-swap array for link construction. const queryFacets = await applyPlpRankingExperiment(facets); — use queryFacets only for toFacetPath at :460 (and therefore the IS calls), and leave the original facets for toFilter at :512 and the pagination loop at :523. The facets.length === 0 gate and filtersFromPageTypes reassignment above are unaffected either way. → fix here

Worth fixing

• packages/blocks/src/sdk/experiments.ts:293 + productListingPage.ts:368 — the exposure is recorded before it is known that anything can change. decide() pushes the assignment as soon as an experiment matches (kind, target); the loader then bails when variant.payload.collectionId is missing and returns the facets untouched. The assignment is already in the bag by then, so it lands in deco_segment and fragments the cache key via __abf for a render that is a no-op. Trigger: an arm published with payload: {} — the natural way to express a control arm as "no change". This is the same exposure bias the PR is explicitly guarding against, so it deserves the same guard: require a usable payload before enrolling (either check collectionId before calling resolve, or have the control plane reject a plp_ranking arm without one). → file an issue

• packages/blocks/src/sdk/experiments.ts:313 — keying the cookie on experiment.key is right, but nothing rejects two experiments sharing a key with different targets and different weight vectors. stickyDecide stores one entry per name, so their fingerprints alternate: /produtos writes pct=F1, /bazar sees a mismatch and re-rolls to F2, back to /produtos re-rolls again. Stickiness is gone and every PLP request writes a fresh cookie, which also means every one of them bypasses the cache. Publisher-side validation (unique key per site) is the cheap fix; not this PR's job. → file an issue

Checked, fine

• decide() is a faithful extraction. name: experiment.key is identical to the old experimentKey because the lookup was by key; the fingerprint, the accepts re-roll on a retired variant id, and the recorded → stored → roll precedence are all byte-for-byte the old body. An untargeted contract-1 doc still takes find(e => e.key === ...) and behaves as before — confirmed by the test at experiments.test.ts:394.
• Ordering. :439 runs after CMS props, the URL filter.* merge, and the ?map= merge, and before the pageTypes fallback. The swap cannot change array length, so the facets.length === 0 gate at :443 and !facets.length && !query at :455 are untouched. It runs before facets = filtersFromPageTypes(pageTypes) at :453, but filtersFromPageTypes (client.ts:639) only emits keys from pageTypeToMapParam — category/department/brand, never productClusterIds — so no reachable PLP is missed today. Worth remembering if that mapping ever gains a Collection case.
• Cost / failure modes. loadConfig (experiments.ts:340-351) memoises the in-flight promise on the RequestContext bag, so N loaders share one kv.get, and readExperimentConfig returns null for an absent binding, an unset key, a non-array experiments, and any throw — a KV outage degrades to "no experiment", never a failed PLP. The memoised promise cannot reject. findIndex < 0 short-circuits before the resolve, so a non-collection PLP never reads KV at all.
• Exposure on the null path. decide returns at !experiment?.variants?.length before touching recorded, so an untargeted surface pushes nothing.
• Cache safety. deco_segment is not in DEFAULT_SAFE_COOKIES (workerEntry.ts:771), so a freshly-assigned visitor's response carries a non-safe Set-Cookie and bypasses the cache instead of poisoning the shared no-cookie entry; a returning visitor's non-empty segmentCacheToken sets __abf (workerEntry.ts:1392-1397) and cdnCacheableServerFn fails closed at :1274. No cross-arm bleed on this axis.

[hugo-ccabral]

Triaged all three. Pushed in a1f9c2e.

1. Facets leaking into the page's own links — fixed, blocking, and you were right

I verified it rather than taking it on faith: toFilter(facets, …) at :512 and the pagination loop at :523-526 both serialise filter.<key>=<value> from the same array I was swapping in place. So the arm's collection went into every chip and pagination href, and your replay is exactly right — the URL dedupe matches on key and value, so 2258 !== 412 doesn't collapse, and a control visitor ends up querying both collections while tagged control.

That is precisely the leak the PR body claims replacing-not-appending prevents. It just happened one request later, via the link. Embarrassing, and a good catch.

Fixed as you suggested: the swap produces a separate queryFacets used only for toFacetPath; the page's own links keep the original facets. I also moved it below the page-types fallback rather than above it — facets is reassigned from filtersFromPageTypes at :453, so the old position was operating on a pre-fallback array. You noted filtersFromPageTypes never emits productClusterIds today, so nothing changes now, but the new position is correct by construction instead of by coincidence.

Both follow-ons you flagged dissolve with it: nothing pins a crawlable pagination URL to an arm's collection, and the single-findIndex limit stops mattering for links.

2. Duplicate key — fixed in-PR, not filed

I moved this out of your "worth fixing" bucket because this PR is what creates the risk: target scoping is what makes several concurrent experiments per site normal, so a repeated key goes from near-impossible to plausible. Fixing it elsewhere would be shipping the hazard and filing the cleanup.

readExperimentConfig now drops repeats, keeping the first. Losing one arm of a mis-published pair beats corrupting every assignment on the site — your fingerprint-alternation analysis is right, and the cache consequence (__abf changing on every navigation) is arguably worse than the analytics one. Two tests cover it.

3. Exposure recorded before the payload check — filed, not fixed here: decocms/context#721

Partly disagreeing on where it belongs. Recording at match time is correct: that is the moment the visitor is enrolled, and a control arm legitimately renders identically to the unenrolled path. Cache fragmentation for a control arm is inherent to A/B testing, not a defect.

What is genuinely wrong is a payload: {} reaching KV at all — and the runtime can't distinguish that from a deliberate publish, because contract 1 defines payload as opaque to it. Guessing would hide the publisher's bug. So it belongs in EXPERIMENT_ACTIVE_PUBLISH, and #721 asks for payload-shape, weight-sum, (targetKind, target) uniqueness and key uniqueness validation at publish time.

Also corrected — the PR body's own numbers were wrong

The doc reviewer on decocms/context#720 caught me mixing units. Recounted by files:

page blocks calling the framework PLP loader	959 of 1035
of those, with a hardcoded productClusterIds	958
on regionListingPage	3
on site/loaders/plp.ts	0

So it's 958 of 959, not "953 of 1936". The real figure argues the case harder than the one I printed.

33/33 tests pass; blocks and apps-vtex both typecheck clean.