jsonb writes fail with "unsupported Unicode escape sequence" when model output contains U+0000 or a lone UTF-16 surrogate

[hackdion]

Bug Description

A single NUL character or lone UTF-16 surrogate in an agent response aborts the whole session run, losing every lesson generated so far in that run.

PgAgentSessionStore.appendTreeEntry and the event projection in packages/@openmaic/storage/src/agent-session/pg.ts pass JSON.stringify(...) output straight into a jsonb parameter (encodeJson). PostgreSQL rejects two escape families that JSON.stringify emits verbatim, with SQLSTATE 22P05:

escape in serialized JSON	PostgreSQL jsonb	note
\u0000 (NUL)	rejected, 22P05	occasionally appears in model output
lone UTF-16 surrogate (\ud800)	rejected, 22P05	half a character; often a truncated emoji
\u0001, \u007f and other control chars	accepted	no handling needed
paired surrogate (\ud83d\ude00, i.e. emoji)	accepted	must not be stripped

Because the failing write is treated as a critical entry write, the runner aborts the run and the session ends failed — in our case destroying 3–4 generated lessons per occurrence, repeatedly, during batch course generation.

Steps to Reproduce

1. Run the Pro workbench against a PostgreSQL-backed agent session store.
2. Make a session whose model output contains a NUL character or a lone surrogate (deterministic version: send a user message whose content includes \u0000).
3. server.log then shows:

[ERROR] [AgentRunner] session <id>: event write failed error: unsupported Unicode escape sequence
[ERROR] [AgentRunner] session <id>: entry write failed error: unsupported Unicode escape sequence
[ERROR] [AgentRunner] session <id>: failed error: unsupported Unicode escape sequence

Direct proof that these escapes are what PostgreSQL refuses:

select '{"t":"a\\u0000b"}'::jsonb;        -- ERROR: unsupported Unicode escape sequence (22P05)
select '{"t":"a\\ud800b"}'::jsonb;        -- ERROR: unsupported Unicode escape sequence (22P05)
select '{"t":"\\ud83d\\ude00"}'::jsonb;   -- OK

Expected Behavior

Session entries whose text contains such characters should still persist — the offending code unit dropped or replaced with U+FFFD — instead of failing the entire run.

Actual Behavior

The jsonb write raises unsupported Unicode escape sequence; the runner treats it as a fatal critical-write error and fails the session, discarding all in-flight work for that run.

Deployment Method

Local / self-hosted next start on a standalone build (output: 'standalone'), with the embedded PostgreSQL 16.11 managed by the repo.

OpenMAIC Version

1.0.2 (commit 98765db). Also present in 1.0.0 (commit 1e10f60); CHANGELOG.md does not mention it and I found no existing issue.

Affected Area

Model / provider integration → actually Agent runtime / storage: @openmaic/storage PostgreSQL backends.

Browser

Not applicable (server-side).

Operating System

macOS 27.0 (arm64), Node v24.18.1.

Relevant Logs

See the three [AgentRunner] lines above; server.log also records the PostgreSQL statement context for the failed parameter.

Suggested Fix

Sanitize string values while serializing, via the JSON.stringify replacer. Four jsonb write helpers have the same blind spot — three encodeJson (packages/@openmaic/storage/src/{agent-session,document,runtime}/pg.ts) plus encodeMeta (packages/@openmaic/storage/src/asset/pg.ts):

function sanitizeJsonString(_key: string, value: unknown): unknown {
  if (typeof value !== 'string') return value;
  let out = '';
  let last = 0;
  for (let i = 0; i < value.length; i += 1) {
    const code = value.charCodeAt(i);
    let drop = false;
    if (code === 0) drop = true;
    else if (code >= 0xd800 && code <= 0xdbff) {                  // high surrogate
      const next = value.charCodeAt(i + 1);
      if (next >= 0xdc00 && next <= 0xdfff) { i += 1; continue; }  // valid pair: keep
      drop = true;
    } else if (code >= 0xdc00 && code <= 0xdfff) drop = true;     // lone low surrogate
    if (drop) { out += value.slice(last, i); last = i + 1; }
  }
  return out + value.slice(last);
}

function encodeJson(value: unknown, label: string): string {
  try {
    const encoded = JSON.stringify(value === undefined ? null : value, sanitizeJsonString);
    if (encoded === undefined) throw new TypeError('value is not JSON-serializable');
    return encoded;
  } catch (error) {
    throw new Error(`@openmaic/storage: ${label} is not JSON-serializable`, { cause: error });
  }
}

Two notes for whoever picks this up, both learned the hard way:

• Do the check on the in-memory string via a replacer, not with a regex over the serialized JSON. A regex over serialized text cannot distinguish a real lone-surrogate escape from the literal characters \ + ud800 in the content, and stripping those produces invalid JSON (22P02) — a worse failure than the original bug. We hit exactly that with a first attempt.
• \u0000 can alternatively be replaced with U+FFFD instead of dropped, if you prefer keeping a placeholder.

[wyuc]

Closed by #1499 (released on main at cfae106): every jsonb write in @openmaic/storage now goes through a shared encoder that replaces U+0000 and unpaired surrogates with U+FFFD, so a NUL or lone surrogate in model output no longer aborts the run. Scope note: the plain TEXT columns on the agent-session table (session title, initial prompt, error) are not covered by this fix — a NUL there still fails with SQLSTATE 22021 before any JSON encoding — and that case is tracked separately in #1502.