Web client: embedded server + TypeScript client + React UI#38
Merged
Conversation
…ent staged buffer Output state was reset to zero every frame in the render loop, so ZMQ SetOutput* writes were overwritten at the next vblank before list_lines could read them back. This made test_vtl_set_output_line fail intermittently on Jetson when a real render loop was running. Replace the per-frame-reset pending_outputs[] field in both render loops and the null backend with a persistent VtlState::staged[] buffer: - VtlState gains `staged: [u64; MAX_BANKS]`, `commit_staged()`, `set_staged_bit()`, and `set_staged_bank()`. The old `write_outputs`/`write_outputs_immediate`/`flush_outputs` API is removed. - Each render loop copies staged out before advance_animations, passes the copy as the output accumulator, then writes it back — preserving the one-frame cascade-prevention guarantee (animations cannot trigger each other in the same iteration). - ZMQ SetOutput* handlers write through set_staged_bit/set_staged_bank, updating staged and shm atomically so list_lines reflects the change immediately without waiting for the next [A]. - Add 9 unit tests in server/tests/vtl_staging.rs covering persistence, animation trigger lines, explicit clear, and cascade prevention. - Wrap all e2e VTL tests in try/finally so cleanup runs even on assertion failure; add a session-level fixture to clear stale named lines on connect. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01G4wdxVhGBJ7yt1kDQpGkfV
Three hot-path cleanups in the render/ZMQ threads: - ipc: decode REP requests off a Cow<[u8]>. Single-frame messages (the common case) now decode with zero copies instead of copying the payload twice via into_vec().flat_map(to_vec).collect(). - render/render_frame: iterate stimuli via iter_mut() + match instead of collecting keys into a fresh Vec each frame and re-looking-up each handle 3-5 times. Removes the per-frame allocation and the redundant HashMap lookups. - scene/state: reuse a scratch buffer for the animation-handle snapshot in advance_animations instead of allocating a Vec every tick. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01G4wdxVhGBJ7yt1kDQpGkfV
Reorganize the debug overlay into focusable groups toggled by F1–F7 with a backtick master toggle, multiple visible at once. Esc closes a dialog or hides the overlay and never quits. Add DRM character input so dialog text fields are typeable on the mouseless rig, keyboard-driven create dialogs for stimuli (rect/circle/ellipse/grating) and animations, an animations list with arm/disarm/trigger, and focusable VTL fire buttons. Photodiode/wireframe move into the System group; demo spawn (D) only fires while the overlay is hidden. WIP: windowed visual pass not yet done. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01G4wdxVhGBJ7yt1kDQpGkfV
…owed DRM error - Panels now lay out horizontally based on visible-count column positions (320px slots) instead of absolute group index, eliminating gaps and overlap. Each panel gets a default_width(310) constraint on first show. - F-key press surrenders egui's current focus before the new panel claims it, so Tab events queued in the same batch navigate the newly focused panel. - Plain F1–F7: show_group (always makes visible + focuses; idempotent). Shift+F1–F7: hide_group (hides panel; auto-hides overlay when none remain). AppKey::SelectGroup replaced by ShowGroup / HideGroup on both backends. - --windowed now exits with a clear error when DRM mode would be selected (no DISPLAY / WAYLAND_DISPLAY set). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
clip_rect is the physical panel boundary, which sits outside the content inner_margin. max_rect is the content area start, so the old StrokeKind::Inside border overlapped the first content column. Also switches group_frame to Frame::side_top_panel so inner_margin is correct, and drops the redundant custom stroke. Co-Authored-By: Claude <noreply@anthropic.com> Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019eqNJz6vowBYWeBRJBUQzZ
Add an optional browser control surface that shares the scene/vtl Arcs with
the render and ZMQ threads and reuses SceneState::handle_request, so it adds
no per-frame render cost.
- proto: snapshot.proto adds SceneSnapshot + CommandLogEntry (transport-agnostic,
reuses existing query responses); wired into build.rs.
- scene/command.rs: extract query_stimulus_response() (shared with QueryStimulus)
and add single-pass SceneState::build_snapshot(&self, vtl) (read-lock only).
- server/src/web: axum server on its own thread/runtime with two WS endpoints —
/ws (Request→Response, same dispatch as ZMQ) and /events (SceneSnapshot push
via a ~30Hz broadcast pump that skips work when no client is connected).
- main.rs: spawn web thread + clean shutdown; seed runtime.screen_size from
rig-config (default 1920x1080) so the null renderer reports a screen size.
- Optional at three levels (precedence CLI ?? rig-config ?? default):
Cargo feature `web` (default-on; --no-default-features compiles it out),
rig-config [web] { enabled, port }, and CLI --no-web / --web-port.
Server side only; React frontend and e2e harness still to come.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012mos2p97Z5gZbdYhBbsSCP
Snapshot stimuli (and QueryStimulus results) now carry the u32 server handle alongside the UUID id. The web map needs the handle to address mutations like SetPosition when dragging a stimulus for receptive-field mapping. The shared query_stimulus_response() builder already receives the handle, so both the snapshot and the QueryStimulus path populate it. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012mos2p97Z5gZbdYhBbsSCP
Hand-written client mirroring the Python client's layering: generated protobuf-es stubs stay private under src/_proto (gitignored, regen with `npm run gen`) and never leak into user code. Public API exposes domain types and namespaced sub-clients only. - types.ts / errors.ts: Vec2, Color, StimulusHandle, ErrorCode + typed errors (mirrors response.py / exceptions.py). - transport.ts: internal WS plumbing — CommandTransport (/ws, serialised REQ/REP) and EventTransport (/events, SceneSnapshot push). - stimuli.ts / system.ts: ShapesClient (createRect/createCircle), generic mutations (setEnabled/setPosition/delete), SystemClient (queryServerInfo). - snapshot.ts: SceneSnapshot/StimulusView domain mapping (incl. handle). - connection.ts: Connection wires sub-clients + central error mapping (the `send` mirrors Python's _send); index.ts re-exports the public surface. - tests/e2e.test.ts: vitest harness spawning `vstimd --null`, driving the public API (no proto), asserting command/snapshot/RF-mapping round-trips. Frontend (React UI) still to come; this is the client library + test harness. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012mos2p97Z5gZbdYhBbsSCP
Vite + React app on top of the web client library. - useScene hook: owns the Connection, holds the latest SceneSnapshot as the read model (subscribes to /events), exposes the connection for commands. - StimulusMap: canvas reconstruction of the screen from snapshot geometry (origin centre, +y up). Drag a stimulus to move it — optimistic local position with setPosition coalesced to one message per animation frame; the next snapshot reconciles. This is the manual receptive-field mapping path. - StimuliPanel: list with enable/delete and quick create (rect/circle). - App: connection status, server info header, map + panel layout. - vite.config proxies /ws and /events to the server in dev; in production the bundle is same-origin (to be embedded in the server via rust-embed). Builds clean (tsc + vite); the null-mode e2e harness stays green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012mos2p97Z5gZbdYhBbsSCP
With `--features embed-ui`, bake the built client/web/dist bundle into the binary (rust-embed) and serve it at `/` with an SPA fallback, so the whole control surface ships as one self-contained executable (Jetson deployment). - New opt-in feature `embed-ui = ["web", "dep:rust-embed"]`; rust-embed uses the mime-guess feature for correct content types. - Router: with the feature, `fallback(static_handler)` serves assets / index; without it, the previous placeholder `/` page (so a plain `cargo build` does not require the frontend to be built). Build the UI first: `cd client/web && npm run build`, then `cargo build --release --features embed-ui`. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012mos2p97Z5gZbdYhBbsSCP
Lets test harnesses (e.g. the Playwright browser e2e) run an isolated vstimd --null on a non-default ZMQ port so it never collides with a real server on 5555. Mirrors the existing --web-port flag. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012mos2p97Z5gZbdYhBbsSCP
Real-browser smoke test of the React UI, the one path not covered by the node client e2e: app boots, connects, renders, creates a stimulus, and a canvas drag drives a position change (manual receptive-field mapping). - playwright.config.ts spins up its own isolated backend (vstimd --null --web-port 8138 --zmq-port 5566) + a Vite dev server pinned to 127.0.0.1, so it never touches a real server on 5555/8080. - beforeEach resets the scene via the same TS client (system.deleteAll, added here) so tests are independent of accumulated server state. - vitest.config.ts scopes vitest to tests/ so the two suites don't overlap. Run: cd client/web && npm run test:ui (after: npx playwright install chromium) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012mos2p97Z5gZbdYhBbsSCP
Extend the existing build-once/share-artifact CI: the rust job already uploads the vstimd binary; two new jobs consume it (needs: [rust]) so the web tests run against the same prebuilt --null server with no Rust toolchain in those jobs. - web-client: npm ci, buf codegen, tsc --noEmit, node WS e2e (npm run test:e2e). - web-browser-e2e: + Playwright Chromium, npm run test:ui; uploads the playwright-report artifact on failure. - Both pass VSTIMD_BIN=<downloaded binary> so the harnesses skip cargo. - buf is now a dev dep (@bufbuild/buf) so `npm run gen` is self-contained in CI. - playwright.config / e2e.test.ts honour VSTIMD_BIN (fall back to cargo run locally). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012mos2p97Z5gZbdYhBbsSCP
joschaschmiedt
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a browser-based control surface for vstimd, covering what the egui overlay does plus an interactive map for positioning stimuli and low-latency manual receptive-field (RF) mapping (drag a stimulus with the mouse). It runs as an axum HTTP+WebSocket thread inside the existing process, sharing the same Arc<RwLock> and reusing SceneState::handle_request — so it adds no per-frame render-thread cost and no duplicated command logic. The egui overlay, ZMQ client, and web UI all drive one scene.
Server-side is complete and tested headlessly; the React UI is functional (map + stimuli panel). It's wired into CI with both node and real-browser e2e.
Architecture
Browser (React + protobuf-es)
├── /ws command channel: Request → Response (same dispatch as ZMQ)
└── /events state channel: SceneSnapshot push (~30 Hz broadcast)
vstimd process
├── render thread (unchanged; egui overlay still works)
├── zmq-server thread (unchanged)
└── web-server thread (new: axum, shares scene/vtl Arcs, reuses handle_request)
Two endpoints, each one message type (no envelope, no multiplexing). SceneSnapshot is transport-agnostic (reuses existing query messages; could later feed a ZMQ PUB).
Server (server/src/web/, snapshot.proto)
TypeScript client (client/web/src/)
React UI (client/web/src/app/)
Testing
Status / follow-ups