perf(cloud): move project bytes once through shared content storage

docs/deploy/aws-lambda.mdx

@@ -87,7 +87,7 @@ See the [CLI reference](/packages/cli#hyperframes-lambda) for full flag document
 
 #### Pre-staging a project with `sites create`
 
-Re-rendering the same project tree on every `lambda render` call re-tars and re-uploads it each time. For tight inner loops (CI smoke jobs, prompt iteration in a demo flow), pre-stage the project once and reuse the upload:
+Every `lambda render` call without `--site-id` hashes the project tree and checks each file against the bucket. For tight inner loops (CI smoke jobs, prompt iteration in a demo flow), pre-stage the project once and reuse the upload:
 
 ```bash
 hyperframes lambda sites create ./my-project
@@ -97,7 +97,15 @@ hyperframes lambda render ./my-project --site-id=a1b2c3d4e5f6g7h8 \
   --width 1920 --height 1080 --wait
 ```
 
-The `siteId` is content-addressed via a SHA-256 of the project tree; re-running `sites create` on an unchanged tree skips the upload via a `HeadObject` short-circuit. Pass the same `--site-id` to as many `lambda render` calls as you like — they all reuse the one S3 PUT.
+The `siteId` is content-addressed via a SHA-256 of the project tree. A site is a small `sites/<siteId>/files.json` that lists each file's path and SHA-256. The files themselves live once per bucket under `cas/sha256/`, shared by every site and render, so a file another project already uploaded is never uploaded again. Pass the same `--site-id` to as many `lambda render` calls as you like.
+
+The stack's bucket expires objects under `cas/sha256/` 30 days after they were last written (the `ExpireSharedContent` lifecycle rule). Each render or `sites create` that reuses a file at least 7 days old rewrites it in place, so files in use never expire. If a site's files did expire, `lambda render --site-id` re-uploads them from the project directory. `renderToLambda` given a `siteHandle` but no `projectDir` stops with `SITE_FILES_MISSING` before starting; run `sites create` again to restore them.
+
+#### Upgrading from a version before shared content storage
+
+- **Redeploy the stack before upgrading the CLI or SDK that starts renders.** A stack from an earlier version cannot read `files.json` sites. The CLI and `renderToLambda` check the state machine first and stop with `STACK_TOO_OLD` before uploading anything; run `hyperframes lambda deploy` (or redeploy your SAM template or CDK construct from the same version) and render again.
+- **Existing site ids keep working.** When `sites/<siteId>/files.json` is missing, the render uses that site's `project.tar.gz` from the earlier version. If neither exists the render stops with `SITE_NOT_FOUND`, naming both keys; run `sites create` again.
+- **Every project uploads once more.** The site id scheme changed (it now also covers symlinked files), so `sites create` or the first `lambda render` of each project after the upgrade returns a new site id and uploads its files once.
 
 ### Path 2 — Direct SAM deploy
 
@@ -166,6 +174,8 @@ hyperframes lambda policies role
 hyperframes lambda policies validate ./infra/iam/hyperframes-deploy.json
 ```
 
+Code that calls the SDK's `renderToLambda` with its own role needs `states:DescribeStateMachine` on the state machine besides `states:StartExecution` and S3 read/write on the bucket: each render reads the definition to confirm the stack is new enough.
+
 The generated documents grant `Resource: "*"` for the CLI's required action set. After your first successful deploy you can narrow `Resource` to the deployed ARNs — predictable per the CloudFormation outputs above. Adopters running the CLI in CI typically check the policy doc into source control and run `policies validate` as a pre-deploy step to catch drift.
 
 ## Cost shape
@@ -220,6 +230,14 @@ If progress doesn't advance for >10 minutes, check the Step Functions execution
 
 The render bucket is created with CloudFormation `Retain` on delete — `hyperframes lambda destroy` (or `sam delete`) tears the function + state machine down but the bucket survives. This is intentional: it protects final-rendered MP4s from being lost when you re-deploy. To fully reclaim storage, empty + delete the bucket via the AWS console / `aws s3 rb`.
 
+### `STACK_TOO_OLD`
+
+The stack was deployed by an earlier version that cannot read this version's sites. Redeploy it (`hyperframes lambda deploy`), then render again. See [Upgrading](#upgrading-from-a-version-before-shared-content-storage).
+
+### `SITE_FILES_MISSING` or `SITE_NOT_FOUND`
+
+`SITE_FILES_MISSING`: some of the site's files expired from `cas/sha256/` after 30 days unused. Pass the project directory to `lambda render`, or run `sites create` again. `SITE_NOT_FOUND`: the bucket has no site under that id in either format; run `sites create` again and use the id it prints.
+
 ## Current limits
 
 - There is no completion webhook. Poll with `hyperframes lambda progress` or
@@ -231,8 +249,8 @@ The render bucket is created with CloudFormation `Retain` on delete — `hyperfr
 - The distributed AWS path is SDR-only.
 - Each render chunk extracts only the video frames it shows, so the plan step
   no longer grows with video length. On a 10,240 MB function, a 26-minute
-  576×1024 24 fps H.264 clip plans in about 20 seconds. Each chunk downloads
-  the whole source file of every video it shows. Variable-frame-rate and HDR
+  576×1024 24 fps H.264 clip plans in about 20 seconds. Each chunk reads
+  only the byte ranges of each video it shows, straight from the bucket. Variable-frame-rate and HDR
   sources are the exception: the plan step still extracts every frame of those,
   inside Lambda's 15-minute limit, in 2-minute segments, so
   `FFMPEG_PROCESS_TIMEOUT_MS` applies to each segment, not to the whole clip.

docs/deploy/gcp-cloud-run.mdx

@@ -72,7 +72,8 @@ use H.264 or H.265. Distributed rendering is currently SDR-only.
 ## Reuse an upload
 
 Projects are content-addressed. Upload an unchanged project once, then reuse
-its site ID for later renders:
+its site ID for later renders. Site IDs from earlier versions keep working, but
+the ID scheme changed, so each project uploads once more under a new ID:
 
 ```bash
 hyperframes cloudrun sites create ./my-project

docs/deploy/migrating-to-hyperframes-lambda.mdx

@@ -10,7 +10,7 @@ If you're already running a different framework that deploys a serverless video
 | In your current framework you call... | In HyperFrames you call...                                       | Notes                                                                                        |
 | ------------------------------------- | ---------------------------------------------------------------- | -------------------------------------------------------------------------------------------- |
 | One-shot deploy command               | `hyperframes lambda deploy`                                      | Builds `packages/aws-lambda/dist/handler.zip` and runs `sam deploy`. Idempotent.             |
-| One-shot site upload                  | `hyperframes lambda sites create ./project`                      | Content-addressed S3 key — re-uploads of an unchanged tree are skipped via a HeadObject 200. |
+| One-shot site upload                  | `hyperframes lambda sites create ./project`                      | Content-addressed file list; files already in the bucket's shared store are skipped. |
 | Trigger a render                      | `hyperframes lambda render ./project --width 1920 --height 1080` | Returns immediately with a `renderId`; add `--wait` to stream per-chunk progress.            |
 | Poll render progress                  | `hyperframes lambda progress <renderId>`                         | Includes accrued cost in the same response.                                                  |
 | Tear down                             | `hyperframes lambda destroy`                                     | The S3 bucket is `Retain`'d — documented in the deploy guide.                                |

docs/packages/cli.mdx

@@ -1983,8 +1983,9 @@ hyperframes lambda deploy \
 
 ### `lambda sites create <projectDir>`
 
-Tars and uploads the project to S3 under a content-addressed key, and returns a
-`siteId` you can reuse. A re-render of the same tree then skips the upload.
+Uploads the project's files to the bucket's shared content store, skipping any
+file already there, and returns a `siteId` you can reuse. A re-render of the
+same tree then skips the upload.
 
 ```bash
 hyperframes lambda sites create ./my-project

examples/aws-lambda/template.yaml

@@ -130,6 +130,11 @@ Resources:
             # Users keep the final mp4 (different key prefix); the rest
             # can age out after a week to keep storage costs flat.
             ExpirationInDays: 7
+          - Id: ExpireSharedContent
+            Status: Enabled
+            Prefix: cas/sha256/
+            # Project files and plan artifacts by SHA-256, shared across renders.
+            ExpirationInDays: 30
       Tags:
         - Key: Project
           Value: !Ref ProjectName
@@ -283,6 +288,7 @@ Resources:
                   - PLAN_ARTIFACT_DIGEST_MISMATCH
                   - FORMAT_NOT_SUPPORTED_IN_DISTRIBUTED
                   - CHUNK_EXCEEDS_WORKER_TIMEOUT
+                  - SITE_MANIFEST_INVALID
                 MaxAttempts: 0
               # Another render on the same function can still take the reserved slots; wait minutes.
               - ErrorEquals: [Lambda.TooManyRequestsException]
@@ -340,6 +346,7 @@ Resources:
                   - FORMAT_NOT_SUPPORTED_IN_DISTRIBUTED
                   - CHUNK_EXCEEDS_WORKER_TIMEOUT
                   - ChromeBinaryUnavailableError
+                  - SITE_MANIFEST_INVALID
                 MaxAttempts: 0
               - ErrorEquals: [Lambda.TooManyRequestsException]
                 IntervalSeconds: 5

packages/aws-lambda/README.md

@@ -77,10 +77,15 @@ infrastructure upgrade. Before upgrading an application that calls
    same new package version.
 3. Resume renders, then upgrade the application/SDK dependency.
 
-Older state machines can default missing protocol fields to v1 or lack v2
-branches, while the new SDK sends explicit v2. If infrastructure cannot be
-redeployed first, keep the application on its previous package version or
-pass `planProtocol: "v1"` explicitly until the redeploy is complete.
+A site is a `sites/<id>/files.json` file list that an older stack cannot
+read, under either plan protocol. Before it uploads anything,
+`renderToLambda` reads the state machine definition
+(`states:DescribeStateMachine`) and fails with `STACK_TOO_OLD` when the
+stack predates that format. If infrastructure cannot be redeployed first,
+keep the application on its previous package version until the redeploy is
+complete. Site ids from earlier versions keep working after the redeploy:
+when `sites/<id>/files.json` is missing, the render uses that site's
+`project.tar.gz`.
 
 ## Chrome runtime
 

packages/aws-lambda/build.mjs

@@ -2,10 +2,11 @@
 /**
  * Build script for @hyperframes/aws-lambda (public OSS package).
  *
- * Bundles each subpath barrel via esbuild → dist/, then emits .d.ts via tsc.
+ * Bundles the subpath barrels in one esbuild pass → dist/, then emits .d.ts via tsc.
  *
  * Subpaths (each gets its own dist entry so adopters that import one path
- * don't load the others' transitive graphs at module-load time):
+ * don't load the others' transitive graphs at module-load time; code two
+ * entries share is emitted once, in dist/_chunks/):
  *
  *   .         (the umbrella barrel: handler + sdk + cdk types re-exported)
  *   ./handler (the Lambda runtime entry — what `scripts/build-zip.ts` ZIPs)
@@ -33,6 +34,7 @@ const sharedOpts = {
     "@aws-sdk/client-lambda",
     "@aws-sdk/client-s3",
     "@aws-sdk/client-sfn",
+    "@aws-sdk/s3-request-presigner",
     "@hyperframes/producer",
     "@hyperframes/producer/distributed",
     "@sparticuz/chromium",
@@ -45,12 +47,18 @@ const sharedOpts = {
   ],
 };
 
-await Promise.all([
-  build({ ...sharedOpts, entryPoints: ["src/index.ts"], outfile: "dist/index.js" }),
-  build({ ...sharedOpts, entryPoints: ["src/handler.ts"], outfile: "dist/handler.js" }),
-  build({ ...sharedOpts, entryPoints: ["src/sdk/index.ts"], outfile: "dist/sdk/index.js" }),
-  build({ ...sharedOpts, entryPoints: ["src/cdk/index.ts"], outfile: "dist/cdk/index.js" }),
-]);
+await build({
+  ...sharedOpts,
+  entryPoints: {
+    index: "src/index.ts",
+    handler: "src/handler.ts",
+    "sdk/index": "src/sdk/index.ts",
+    "cdk/index": "src/cdk/index.ts",
+  },
+  outdir: "dist",
+  splitting: true,
+  chunkNames: "_chunks/[name]-[hash]",
+});
 
 // esbuild doesn't emit .d.ts. tsc does, with a build-only tsconfig that
 // drops the workspace `paths` overrides so `@hyperframes/producer` resolves
@@ -62,4 +70,4 @@ execSync(
   { stdio: "inherit" },
 );
 
-console.log("[Build] Complete: dist/{index,handler,sdk/index,cdk/index}.js + .d.ts");
+console.log("[Build] Complete: dist/{index,handler,sdk/index,cdk/index}.js + dist/_chunks + .d.ts");

packages/aws-lambda/package.json

@@ -72,6 +72,7 @@
     "@aws-sdk/client-lambda": "^3.1146.0",
     "@aws-sdk/client-s3": "^3.1146.0",
     "@aws-sdk/client-sfn": "^3.1146.0",
+    "@aws-sdk/s3-request-presigner": "3.1146.0",
     "@hyperframes/producer": "workspace:^",
     "@sparticuz/chromium": "148.0.0",
     "ffmpeg-static": "^5.2.0",

packages/aws-lambda/src/cdk/HyperframesRenderStack.contract.test.ts

@@ -15,6 +15,7 @@ import { tmpdir } from "node:os";
 import { join } from "node:path";
 import { App, Stack } from "aws-cdk-lib";
 import { Template } from "aws-cdk-lib/assertions";
+import { CONTENT_STORE_PREFIX, CONTENT_STORE_RETENTION_DAYS } from "../contentStore.js";
 import { HyperframesRenderStack } from "./HyperframesRenderStack.js";
 
 // CDK synth is slow on cold start and reached 32s under full-workspace CI
@@ -61,7 +62,7 @@ describe("HyperframesRenderStack — contract", () => {
     });
   });
 
-  it("provisions exactly one S3 bucket with PublicAccessBlockConfiguration and a 7-day intermediates lifecycle", () => {
+  it("provisions exactly one S3 bucket with PublicAccessBlockConfiguration and a 7-day intermediates and 30-day shared content lifecycle", () => {
     const t = DEFAULT_TEMPLATE;
     t.resourceCountIs("AWS::S3::Bucket", 1);
     t.hasResourceProperties("AWS::S3::Bucket", {
@@ -79,6 +80,12 @@ describe("HyperframesRenderStack — contract", () => {
             Prefix: "renders/",
             ExpirationInDays: 7,
           },
+          {
+            Id: "ExpireSharedContent",
+            Status: "Enabled",
+            Prefix: `${CONTENT_STORE_PREFIX}/`,
+            ExpirationInDays: CONTENT_STORE_RETENTION_DAYS,
+          },
         ],
       },
     });

packages/aws-lambda/src/cdk/HyperframesRenderStack.snapshot.test.ts

@@ -26,6 +26,12 @@ import { join } from "node:path";
 import { App, Stack } from "aws-cdk-lib";
 import { Template } from "aws-cdk-lib/assertions";
 import { parse as parseYaml } from "yaml";
+import {
+  CONTENT_STORE_PREFIX,
+  CONTENT_STORE_REFRESH_AFTER_DAYS,
+  CONTENT_STORE_RETENTION_DAYS,
+} from "../contentStore.js";
+import { SITE_MANIFEST_INVALID } from "../siteManifest.js";
 import { HyperframesRenderStack } from "./HyperframesRenderStack.js";
 
 // CDK synth + Template.fromStack is slow on cold start in CI (~5-8s on
@@ -84,6 +90,7 @@ const EXPECTED_NON_RETRYABLE_ERRORS = new Set([
   "PLAN_ARTIFACT_DIGEST_MISMATCH",
   "FORMAT_NOT_SUPPORTED_IN_DISTRIBUTED",
   "ChromeBinaryUnavailableError",
+  "SITE_MANIFEST_INVALID",
 ]);
 
 function doSynth(reservedConcurrency?: number): {
@@ -213,6 +220,16 @@ describe("HyperframesRenderStack — snapshot", () => {
     }
   });
 
+  it("lists the error the SDK checks for before rendering in every SAM and CDK plan task", () => {
+    for (const definition of [SYNTHED.definition, readSamDefinition()]) {
+      for (const plan of [getV1TaskStates(definition).Plan, getV2TaskStates(definition).PlanV2]) {
+        const errors = new Set<string>();
+        collectNonRetryableErrors(plan, errors);
+        expect(errors.has(SITE_MANIFEST_INVALID)).toBe(true);
+      }
+    }
+  });
+
   it("keeps SAM and CDK terminal classifiers identical for every v2 Lambda task", () => {
     const cdkTasks = getV2TaskStates(SYNTHED.definition);
     const samTasks = getV2TaskStates(readSamDefinition());
@@ -267,6 +284,26 @@ describe("HyperframesRenderStack — snapshot", () => {
     expect(v2).not.toContain("PlanS3Uri");
   });
 
+  it("keeps the SAM bucket lifecycle identical to CDK's, shared content on the transport's terms", () => {
+    const cdkBucket = Object.values(SYNTHED.template.findResources("AWS::S3::Bucket"))[0] as {
+      Properties: { LifecycleConfiguration: unknown };
+    };
+    const samBucket = requireRecordProperty(readSamResources(), "RenderBucket", "SAM RenderBucket");
+    const samProperties = requireRecordProperty(samBucket, "Properties", "SAM bucket properties");
+    expect(samProperties.LifecycleConfiguration).toEqual(
+      cdkBucket.Properties.LifecycleConfiguration,
+    );
+    expect(CONTENT_STORE_REFRESH_AFTER_DAYS * 2).toBeLessThan(CONTENT_STORE_RETENTION_DAYS);
+    expect(cdkBucket.Properties.LifecycleConfiguration).toMatchObject({
+      Rules: expect.arrayContaining([
+        expect.objectContaining({
+          Prefix: `${CONTENT_STORE_PREFIX}/`,
+          ExpirationInDays: CONTENT_STORE_RETENTION_DAYS,
+        }),
+      ]),
+    });
+  });
+
   it("sends the same payload from every Lambda task as the SAM template", () => {
     // CDK drops null-valued payload fields when it renders the definition (#5259), so compare
     // what each deployed task sends rather than the construct source.

packages/aws-lambda/src/cdk/HyperframesRenderStack.ts

@@ -30,6 +30,7 @@ import * as s3 from "aws-cdk-lib/aws-s3";
 import * as sfn from "aws-cdk-lib/aws-stepfunctions";
 import * as tasks from "aws-cdk-lib/aws-stepfunctions-tasks";
 import { Construct } from "constructs";
+import { CONTENT_STORE_PREFIX, CONTENT_STORE_RETENTION_DAYS } from "../contentStore.js";
 
 /** Construction-time props for {@link HyperframesRenderStack}. */
 export interface HyperframesRenderStackProps {
@@ -93,6 +94,12 @@ export class HyperframesRenderStack extends Construct {
           prefix: "renders/",
           expiration: Duration.days(7),
         },
+        {
+          id: "ExpireSharedContent",
+          enabled: true,
+          prefix: `${CONTENT_STORE_PREFIX}/`,
+          expiration: Duration.days(CONTENT_STORE_RETENTION_DAYS),
+        },
       ],
     });
 
@@ -218,6 +225,7 @@ export class HyperframesRenderStack extends Construct {
       "FORMAT_NOT_SUPPORTED_IN_DISTRIBUTED",
       "CHUNK_EXCEEDS_WORKER_TIMEOUT",
       "ChromeBinaryUnavailableError",
+      "SITE_MANIFEST_INVALID",
     ];
     const NON_RETRYABLE_CHUNK = [
       "FFMPEG_VERSION_MISMATCH",