2D WebMCP

This project explores a shared way for agents to read and change 2D surfaces, such as whiteboards, diagrams, and node editors.

The goal is a small common schema. Each app can keep product-specific details in named extensions. The schema is not a replacement for an app's native file format.

What's here

• CONTEXT.md defines the shared terms and explains what belongs in an app adapter.
• rfcs/0001-semantic-2d-surface-interaction.md is the draft proposal.
• schemas contains the draft JSON Schemas. They are RFC files, not a published package.
• docs/research/shared-mcp-output-schema.md explains common ways to share MCP output schemas.
• docs/research/2d-surface-schema-tool-mapping.md compares the schema with popular whiteboard and diagram tools.
• evals/tool-communication tests how descriptions, output schemas, and useful errors affect tool use.
• evals/integrated-surface compares the shared Snapshot and Receipt with a minimal generic editor baseline and tests full, compact, and omitted WebMCP return contracts using the real demo app.

Run the tests and evals

npm install
npm test
npm run eval:tool-schemas -- --list
npm run eval:integrated -- --list
npm run eval:integrated:smoke
npm run eval:langsmith:integrated -- --help
npm run eval:release -- --help

Live model evals need an API key and model name:

OPENAI_API_KEY=... npm run eval:integrated -- \
  --model YOUR_MODEL \
  --trials 5

Set LANGSMITH_TRACING=true and LANGSMITH_API_KEY to trace ordinary live eval runs. The eval entry points load a repository-root .env automatically; values already exported by the shell take precedence. Traces default to the 2d-webmcp-evals project; override it with LANGSMITH_PROJECT.

For one shareable, matched evidence bundle, run npm run eval:release -- --model YOUR_MODEL --trials 5. It traces the same attempts to LangSmith and writes README.md, report.md, results.jsonl, trials.jsonl, and manifest.json to one ignored eval-artifacts/<run-id>/ directory. It does not publish anything remotely.

Eval recordings are saved in eval-results/. Git ignores this directory.

What belongs in this project

This repository owns the shared terms, schemas, research, and evals.

Apps own their document models, UI code, and adapters. An adapter maps native app data to the shared schemas.

The draft 0.1 schemas use stable URNs. A later release can add public HTTPS addresses without changing the self-contained schemas used by MCP tools.

Substantial focus anchors

lib/substantialFocusAnchors.mjs is a small browser-side adapter for directing assistive technology to meaningful visual context after an agent changes a page. It discovers headings, native and ARIA controls (including div[role=button]), links, labelled images, landmarks, and sufficiently large text blocks. It excludes hidden, inert, and disabled content.

It does not put every item in the sequential Tab order. When focus is requested for a semantic but non-native target—such as a heading—it adds tabindex="-1" and calls focus(). That is the appropriate programmatic-focus pattern: keyboard users do not have to tab through every paragraph, while a screen reader has a real DOM focus target. A div with role="button", tabindex, or an inline onclick is also treated as a control; applications can supply a classify(element) function for custom canvases, charts, cards, or framework-specific controls that do not advertise useful DOM semantics.

import {
  createSubstantialFocusAnchorManager,
  registerWebMCPFocusTools,
} from "./lib/substantialFocusAnchors.mjs";

const anchors = createSubstantialFocusAnchorManager(document);

// An agent first calls the discovery tool and then may select one returned ID.
if (document.modelContext?.registerTool) {
  registerWebMCPFocusTools(anchors);
}

The optional WebMCP registration exposes accessibility_list_substantial_focus_anchors and accessibility_focus_substantial_anchor. The first is read-only; the second changes only document focus and returns a receipt-like result. A Chrome extension may use the same manager in a content-script or main-world bridge; the extension should keep one manager per document/frame and clear it on navigation.