feat(materializer): daily rollup mode 'on' — the fold leaves the span transaction (#55 step 4) - #63
Merged
Merged
Conversation
… transaction (#55 step 4) MATERIALIZER_DAILY_ROLLUP_MODE=on: the daily watermarked refresh maintains lake.signals_latest itself and the per-pass fold (captureRollupDelta + foldSignalsRollup — the three lake.signals scans that dominate span cost) is off. On the first boot after shadow, the validated one-row-per-key shadow table is PROMOTED into lake.signals_latest in one transaction, with a post-swap cardinality check guarding the exact silent-delete failure mode that corrupted the fold-era table (2026-08-08: 823,520 rows over 7,666 keys; the promote discards them), and the shadow table dropped only after verification. Under mode on: seeds target the live table with transactional clear (never a DROP of a serving table); FlushRollup's signals recompute is diverted to the late set (bounded heals only — an unbounded recompute would break the watermark's disjoint count split); RecomputeRollup errors with a pointer to the watermark-drop reseed; the shadow-only diff and prune paths are gated off. Pair with LAKE_ROLLUP_DAILY_SERVING=true on the query fleet. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This was referenced Aug 10, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The flip.
MATERIALIZER_DAILY_ROLLUP_MODE=onmakes the daily watermarked refresh maintainlake.signals_latestitself and turns the per-pass fold off — the threelake.signalsscans that made span cost grow with the day's file count leave the span transaction entirely. This is the payoff #55 exists for, and after the 2026-08-08 duplicate-corruption finding it is also the remediation.The promote (shadow → on, first boot)
The validated shadow table — proven one-row-per-key by the cardinality probe (7,666 = 7,666, zero dups) while the fold-era live table carries 823,520 rows over the same keys — is swapped into
lake.signals_latestin one transaction: readers see the old table until commit, the clean one after, never a dropped or partial serving table. A post-swap cardinality check guards the exact failure mode that caused the corruption (a DELETE silently removing nothing under din's concurrent compaction): if the result isn't one-row-per-key, the promote failed regardless of what the transaction reported, and it retries next pass — the shadow table is dropped only after verification.Mode-on rules
FlushRollup's signals recompute is diverted (backfills hand subjects to the late set for bounded heals — an unbounded recompute would fold rows past the watermark and break the summaries union's disjoint count split).RecomputeRolluperrors with a pointer to the watermark-drop reseed.Rollout (with DIMO-Network/dimo-node PR to follow)
Materializer:
MATERIALIZER_DAILY_ROLLUP_MODE=on. Query fleet:LAKE_ROLLUP_DAILY_SERVING=true(the summaries union, exact under the daily rollup — dq#61). Watch: the promote log line +dq_materializer_daily_rollup_table_rows{side="live"}rows==keys, span durations dropping,dq_materializer_phase_seconds{phase="commit"}flattening. Revert: mode back toshadow(a fresh shadow seed rebuilds the evidence table; the fold resumes maintaining live) — note the fold-era table is gone either way, which is the point.Step 5 (delete the fold code, span back to 16, retire scaffolding) follows once this soaks.
Tests: the promote heals a manufactured duplicate-corrupted live table and drops the shadow only after verification; fold-off proven (decode leaves the rollup untouched until refresh); refresh-maintained live table equals a full recompute via a mode-off oracle; fresh-install seeding. Full suite + lint green.
Refs #55.
🤖 Generated with Claude Code