Skip to content

Latest commit

 

History

History
69 lines (42 loc) · 2.98 KB

README.md

File metadata and controls

69 lines (42 loc) · 2.98 KB

StreamEmbed mascot StreamEmbed

Real-time Indexing from Pixels for Screenshot Retrieval

PyPI Dataset License

A question typed over a rolling stream of past screenshots, and the past screen StreamEmbed finds for it

StreamEmbed finds a past screen from a question such as "Which smartphone model did I add to my cart?".

It embeds a screenshot every 5 s into an index on your laptop. Since it runs while you work, it should take only a small share of the laptop's time and memory. StreamEmbed cuts the time per screenshot from 2.3 s to 0.6–0.8 s by exploiting the redundancy in the stream of screenshots:

  • Spatial redundancy: blocks that repeat a neighbor are dropped (spatial pruning).
  • Temporal redundancy: blocks unchanged since the previous screenshot reuse their computation (temporal reuse).
  • A Consistency adapter keeps the embedding close to that of the full screenshot.

On an Apple M2 Pro it embeds a screenshot 2.8–3.7× faster than Qwen3-VL-Embedding-2B and keeps over 98% of its Recall@10 on ScreenMemEval.

Our paper

demo.mp4

Run it on your laptop 🚀

From the terminal

uv tool install streamembed
streamembed start --folder {folder}        # index the screenshots saved in {folder}, then capture every 5 s
streamembed search "Which smartphone model did I add to my cart?"

From Python

uv pip install streamembed
python run.py
usage.mp4

Ask Claude Code about your screen 💬

mcp_server.py gives Claude Code a tool that searches your screen history and looks at the screens it finds. Add it with

claude mcp add streamembed -- uv run --python 3.12 --with streamembed --with mcp python /path/to/StreamEmbed/mcp_server.py

Then ask, for example, "Which smartphone did I add to my cart yesterday? Check my screen history." The server searches the index that streamembed start keeps. While that service runs, it embeds the question with the service's model; otherwise it loads the model on the first search (20–60 s).

Reproduce the paper

reproduce/ has the code that reproduces the paper's tables from ScreenMemEval with the streamembed library.

Library

The source of the streamembed package is in library/.

License

Apache-2.0. The model weights and the dataset have their own licenses.