Skip to content

feat: storage streaming - #22

Merged
darwin67 merged 10 commits into
mainfrom
storage/stream
Aug 6, 2026
Merged

darwin67 merged 10 commits into
mainfrom
storage/stream

Conversation

@darwin67

@darwin67 darwin67 commented Aug 6, 2026 •

Copy link
Copy Markdown
Member

Summary

  • add memory, local filesystem, and S3-compatible paste storage backends
  • run SeaweedFS as the development S3 service through Docker Compose
  • stream raw API uploads through bounded private spool files while calculating size and SHA-256
  • store UTF-8/NUL-free textual pastes up to 8 KiB inline in PostgreSQL and use blob storage for larger or binary content
  • persist normalized content types and safely represent binary bodies in the API, Rust client, CLI, and LiveView
  • harden upload finalization, read integrity, cleanup, and raw browser responses

Storage and upload behavior

Paste bodies are limited to 1 MiB by default. Raw request bodies are read in bounded 64 KiB chunks into private temporary files while size and SHA-256 are calculated.

Bodies are stored inline in PostgreSQL only when they:

  • are at or below TEXTBIN_INLINE_PASTE_BYTES (8 KiB by default)
  • have a textual content type
  • contain valid UTF-8 without NUL bytes

All other bodies use the configured local or S3-compatible backend. Development uses SeaweedFS through Docker Compose; simple self-hosting can select local filesystem storage.

Safety and durability

  • synchronize local file contents, renames, directory creation, deletion, and temporary-file cleanup before acknowledging operations
  • journal pending blob uploads in PostgreSQL so interrupted uploads can be recovered after process or host failure
  • atomically coordinate cleanup claims with paste insertion to prevent committed objects from being deleted
  • register active spool files by owner process and periodically remove only stale, unowned files
  • verify persisted size and SHA-256 whenever paste content is read
  • disable redirects and retries for manually signed one-shot S3 streams
  • limit persisted content types to the database boundary

Raw browser responses only allow passive text/plain and application/json content inline. HTML, XML, SVG, JavaScript, binary, and other user-provided types are attachments. Responses include X-Content-Type-Options: nosniff and a sandboxed CSP.

API and CLI compatibility

Text responses retain the existing data string. Binary responses use:

{
  \"data\": null,
  \"data_base64\": \"/wAB\",
  \"data_encoding\": \"base64\",
  \"content_type\": \"application/octet-stream\"
}

The Rust client now exposes paste data as bytes and preserves the text/binary discriminator. The workspace version is bumped to 0.2.0 to document this deliberate pre-1.0 compatibility change. CLI raw output preserves arbitrary bytes exactly.

Database changes

  • add storage metadata and location fields to pastes
  • add normalized content_type
  • add pending_uploads cleanup journal with claim leases

CI and verification

GitHub Actions exercises memory, local filesystem, and real S3-compatible storage through a matrix. SeaweedFS is only started for the S3 matrix entry.

Validated locally with:

  • mix precommit — 268 tests
  • memory backend — 268 tests
  • local backend — 268 tests
  • SeaweedFS/S3 backend — 268 tests
  • cargo test --workspace — 53 tests
  • make lint — Credo and Clippy pass

A final Oracle review returned SHIP with no remaining blockers.

@github-actions github-actions Bot added the feat label Aug 6, 2026
@darwin67
darwin67 marked this pull request as ready for review August 6, 2026 07:21
@darwin67
darwin67 merged commit 8bd82cc into main Aug 6, 2026
9 checks passed
@darwin67
darwin67 deleted the storage/stream branch August 6, 2026 07:22
@chaba2-bot chaba2-bot Bot mentioned this pull request Aug 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant