Skip to content

feat: Add offline media download pipeline and vertical video player feed - #13

Closed
DriftyPencil wants to merge 1 commit into
sidinsearch:mainfrom
DriftyPencil:feat/offline-media-downloads
Closed

DriftyPencil wants to merge 1 commit into
sidinsearch:mainfrom
DriftyPencil:feat/offline-media-downloads

Conversation

@DriftyPencil

Copy link
Copy Markdown

This PR introduces a comprehensive offline media asset-management pipeline, enabling the application to download, store, sync, and play physical media (.mp4) locally on mobile devices without an active network connection.

Key Architectural Changes

1. Backend Extensions (/backend)

  • Media Scraping & Storage: Updated worker pipelines (Instagrapi / yt-dlp) to fetch and save raw physical .mp4 payloads into a dedicated local server directory (/media).
  • Database Updates: Altered the SQLite database schema to support local_filename and media_file_size tracking.
  • Static File Serving: Exposed a secure static endpoint at /api/v1/media/{filename} to serve files to client devices.

2. Mobile App Sync & Caching (/superbrain-app)

  • Background Asset Downloader: Created a TypeScript manager service powered by expo-file-system to fetch assets over network handshakes and commit them directly to FileSystem.documentDirectory.
  • State Mapping: Integrated a local storage layer mapping to track local_uri file paths alongside existing post metadata.

3. UX & Offline Playback Componentry

  • Zero-Buffer Local Feed: Introduced an infinite vertical scrolling view using FlatList with strict viewabilityConfig performance tunings (windowSize={3}) to handle full-screen media recycling without memory leaks.
  • Smart Source Switching: The video renderer checks for the presence of a verified local_uri mapping. It seamlessly switches playback to the file:// scheme when offline, dropping back to the remote network URL only as a backup.
  • Storage Dashboard: Added a utility menu displaying localized disk footprint calculations with actions to purge physical caches while retaining AI-generated text metadata.

Verification & Testing

  • Verified background chunk downloading on physical devices running iOS and Android via Expo.
  • Simulated total network blackouts (Airplane Mode) to confirm seamless local playback, file looping, and client-side fuzzy search operations through cached datasets.

@sidinsearch

sidinsearch commented Sep 1, 2026

Copy link
Copy Markdown
Owner

Holding as experimental — not for beta/main yet

Thanks for the substantial contribution, @DriftyPencil. The architecture is solid — vertical FlatList with windowSize={3}, smart file:// ↔ remote URL switching, parallel download limit, auth on the media endpoint, path-traversal guard. The code is well-factored.

But this PR is being held as experimental for now. I created a tracking branch at experimental/video-playback (based on current main) so the work isn't lost — it has all your changes plus the latest main fixes. Nothing will merge into beta/main until the concerns below are addressed.


Why not merge yet

1. Backend storage growth is unbounded (the big one)

backend/media/ will grow forever. Every saved Instagram reel + YouTube video is a multi-MB MP4 on the server's disk. After 1,000 posts with 50MB videos = ~50GB of MP4s with no cleanup policy. The client has a StorageManager purge UI but the server has nothing.

What's needed before this lands:

  • A cleanup cron / startup sweep that purges MP4s whose shortcode is no longer in the DB
  • OR a max-age policy (e.g., drop MP4s not requested in 30 days)
  • OR a hard disk quota that evicts oldest-first

Without this, running the beta image in production for a week is a disk-fill waiting game.

2. No backend tests for the new endpoint

Only test_media_metadata.py was added (74 lines). The new /api/v1/media/{filename} endpoint has zero direct tests:

  • Auth (no token → 401, wrong token → 401)
  • Path traversal (../../../etc/passwd → 400)
  • Missing file → 404
  • Valid file → 200 + correct video/mp4 content-type

These are easy to add and should be required for an endpoint that streams user-uploaded files.

3. No client tests for the new components

OfflineMediaManager.ts, ReelsFeed.tsx, StorageManager.tsx — no Jest/RNTL coverage. The smart source-switching logic in particular deserves at least a couple of unit tests (local URI present → file://, local URI absent → remote URL, file missing → fallback + re-download trigger).

4. yt-dlp is ~50MB added to the image

Acceptable, but worth calling out. If the offline feature is opt-in, consider making the download step a separate worker process so the main API container stays slim.


What I'd suggest as the path forward

Hold PR #13 open. Use experimental/video-playback as the discussion branch. Re-push your changes there (or ask me to copy them) and iterate on:

  1. Add a cleanup policy for backend/media/ — the most critical gap
  2. Add tests for the media endpoint + at least the source-switching logic
  3. Document the storage implications in the PR description (rough estimate of disk usage at N posts)

Once those land and a maintainer signs off, we can promote from experimental/video-playbackbetamain with the standard release pipeline.


Things that are already good ✅

  • Path-traversal guard on the streaming endpoint
  • Auth required on /api/v1/media/{filename}
  • Concurrent download limit (2) prevents bandwidth chaos
  • windowSize={3} on the FlatList keeps vertical scroll memory-bounded
  • Smart source switching falls back gracefully when local file is missing
  • DB columns local_filename + media_file_size are indexed in the right places
  • The new OfflineMediaManager is a clean separation between storage and UI

Bottom line: feature is genuinely useful, code is well-written, but it needs a server-side cleanup story before it's safe to ship to beta. Take your time on this — it's the right kind of work to do carefully.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants