Skip to content

storage: fix SQLite 3.53.4/3.54.0 CLI compatibility for init rollback and busy diagnostics (#1645) - #1654

Merged
fujibee merged 5 commits into
mainfrom
fix/1645-sqlite-cli-strict-init-busy
Oct 20, 2026
Merged

fujibee merged 5 commits into
mainfrom
fix/1645-sqlite-cli-strict-init-busy

Conversation

@fujibee

@fujibee fujibee commented Oct 17, 2026

Copy link
Copy Markdown
Owner

What

SQLite 3.53.4 and 3.54.0's CLI broke two assumptions agmsg's storage layer made about the older CLI (#1645):

  1. storage_init (scripts/drivers/storage/sqlite.sh) sent its whole schema transaction as a single multi-statement command-line argument, assuming -bail would stop at the first error before the trailing PRAGMA user_version stamp. On the newer CLI a failing statement no longer stops that argument form, so a broken init could still stamp the current revision. Fix: feed the same statements to sqlite3 on stdin instead (every top-level statement already ends its own physical line in this SQL, which is what the CLI's -bail honors reliably on every version tested).
  2. A lock-timeout giveup reports exit code 1 instead of 5 on the newer CLI, bypassing agmsg_sqlite()'s busy-specific diagnostic. Fix: agmsg_sqlite() now also classifies busy by matching "database is locked" in stderr (the same text match the recording variant already used) and normalizes the return code to 5 when the raw exit code did not already say so.

Survey of other -bail call sites

Checked every -bail use under scripts/ for the same "single command-line argument carrying more than one statement" shape:

Site Form Affected?
sqlite.sh storage_init (line ~165) was: multi-statement argv yes — fixed here
sqlite.sh _sqlite_exec_stdin (line 73) stdin, one statement per line no
sqlite.sh storage_send (line ~337) stdin, one statement per line no
sqlite.sh storage_import (line ~618) stdin, one statement per line no
sqlite-sync.sh _sqlite_sync_commit_chunk (line 66) stdin, one statement per line no
sqlite-sync.sh ack/pull batch via _sqlite_exec_stdin (lines 959, 1652) stdin, one statement per line no
sqlite-sync.sh apply-page sql_file (lines 1419, 2021) file piped on stdin, one statement per line no

Also found 4 sites that build a multi-statement BEGIN IMMEDIATE ... COMMIT block and pass it as a single argv string without -bail at all (sqlite-sync.sh lines ~431, ~520, ~1829, ~1858 — schema migration and the resync/read-block/read-unblock assert blocks). These never relied on -bail's stop-at-first-error contract in the first place; they validate the outcome via a CHECK-constrained temp-table assert plus the exit code, a different design unaffected by this CLI change. Left as-is.

Testing

Existing tests/test_storage.bats cases only (no new tests):

  • storage: a failing init batch leaves the OLD revision, never a new stamp over a broken schema (#1001)
  • storage: a call that gives up after the busy timeout says so on stderr, and a quiet store says nothing (#1001)

Both confirmed green on this machine's SQLite 3.51.0 and, separately, on SQLite 3.53.4 (downloaded standalone from sqlite.org into a scratch directory and prepended to PATH only for that test run — nothing installed system-wide). Full tests/test_storage.bats (31 cases) also green on both versions.

… and busy diagnostics (#1645)

storage_init sent its schema transaction as a single multi-statement
command-line argument and assumed -bail would stop at the first error
before the trailing PRAGMA user_version. On SQLite 3.53.4 and 3.54.0's
CLI, a failing statement inside that argument no longer stops the
batch: the schema stamp lands over a broken transaction. Feeding the
same statements to sqlite3 on stdin (one top-level statement ending
its own physical line, same shape already used elsewhere in this
file) restores the stop-at-first-error contract on every supported
CLI version.

Separately, a lock-timeout giveup reports exit code 1 instead of 5 on
the same CLI versions, bypassing agmsg_sqlite()'s busy-specific
diagnostic (the recording variant already classified busy/failed by
matching stderr text rather than trusting the exit code). agmsg_sqlite()
now normalizes its own return code to 5 whenever stderr says "database
is locked" and the raw exit code did not already say so, so every
existing caller that checks for 5 keeps working.

Reproduced and fixed against both SQLite 3.51.0 (this machine's
version) and 3.53.4 (downloaded standalone, not installed system-wide)
on the two existing tests/test_storage.bats cases; no new tests added.
Review follow-up on #1654: mktemp/cat/rm on every agmsg_sqlite call
added 3 extra forked processes per call on this hot path (every
send/receive/watch.sh cycle), measured at tens to hundreds of ms per
process on Windows -- the same cost #1627/#1630/#1652 are reducing
elsewhere, and a per-call temp file is also what #1580 just finished
cleaning up stray copies of. Replaced with the standard fd-juggle
(3>&1 on the outer group, then 1>&3/2>&1 inside the command
substitution) so stdout still streams straight to the caller and
stderr lands in a shell variable, with no process beyond sqlite3/sed
forked. rc capture (pipefail against sqlite3, not sed) is unchanged.

tests/test_watch_process_count.bats: 46 forked processes per idle
poll cycle, unchanged from main.
@fujibee

fujibee commented Oct 17, 2026

Copy link
Copy Markdown
Owner Author

Updated per review: agmsg_sqlite() no longer uses mktemp/cat/rm to capture stderr (3 extra forked processes per call on this hot path). Rewritten with the standard fd-juggle (3>&1 outer, 1>&3/2>&1 inside the command substitution) — stdout streams straight through, stderr lands in a shell variable, no extra process beyond sqlite3/sed.

Verified:

  • Existing 2 tests/test_storage.bats cases, and the full 31-case file, green on both SQLite 3.51.0 and 3.53.4 (same throwaway setup as before).
  • Negative control: disabling just the busy-text normalization re-reds the busy test on 3.53.4 at the expected assertion.
  • tests/test_watch_process_count.bats: 46 forked processes per idle poll cycle, identical to main (checked against a clean origin/main worktree) — no increase.

The busy-text check added for #1645 matched "database is locked"
anywhere in stderr, so an ordinary error that happens to mention that
text in a table/column name or an echoed statement was misclassified
as a lock timeout. Reproduced: a SELECT against a table literally
named "database is locked" reported "no such table: database is
locked" and got treated as busy.

Replaced the substring test with _agmsg_sqlite_err_is_busy(), matched
against the exact whole lines sqlite3 itself emits for a real lock
timeout, collected by reproducing one on both argv and stdin input and
on both SQLite 3.51.0 and 3.53.4. The counter-example above (and the
same shape on 3.53.4) stays rc=1, confirmed by hand -- no new test
added. Regression coverage stays the existing tests/test_storage.bats
busy case (#1001), green on both CLI versions.
@fujibee

fujibee commented Oct 17, 2026

Copy link
Copy Markdown
Owner Author

Fixed the whole-line match gap: `agmsg_sqlite()`'s busy check now uses `_agmsg_sqlite_err_is_busy()`, matched against the exact whole lines sqlite3 itself emits for a real lock timeout (collected by reproducing one on both argv and stdin input, on both SQLite 3.51.0 and 3.53.4), instead of a bare substring test.

Counter-example checked by hand, not added as a test: a `SELECT` against a table literally named `database is locked` reports "no such table: database is locked" and stays rc=1 on both CLI versions after the fix (it previously misclassified as busy, rc=5).

Verified:

  • Existing 2 tests/test_storage.bats cases, and the full 31-case file, green on both SQLite 3.51.0 and 3.53.4.
  • The counter-example above reproduced by hand on both versions, rc=1 unchanged.

_agmsg_sqlite_err_is_busy()'s per-line patterns anchor on `$` (end of
line). Windows' sqlite3.exe ends stderr lines with \r\n, so a trailing
\r sat between "locked" and that anchor and every real busy line
missed, silently falling back to the non-busy rc. Stripped with a
builtin substitution (${1//$'\r'/}) used only for the match itself;
the stderr re-emitted to the caller is untouched.

Verified by hand, no new test: a real busy line matches whether it
carries \r\n, is one of several lines in a multi-line stderr, or is
plain LF; the existing table-named-"database is locked" counter-example
still does not match, CRLF or not.
@fujibee

fujibee commented Oct 17, 2026

Copy link
Copy Markdown
Owner Author

Fixed: _agmsg_sqlite_err_is_busy()'s per-line patterns anchor on `$`, and Windows' sqlite3.exe ends stderr lines with \r\n, so the trailing \r sat between "locked" and the anchor and a real busy line went unmatched. Stripped with a builtin substitution (`${1//$'\r'/}`) used only for the match itself — no external command added, and the stderr re-emitted to the caller is untouched.

Checked by hand, no new test:

  • plain LF busy line: matches
  • the same line with a trailing \r (CRLF): matches
  • the busy line embedded as one of several lines in a multi-line stderr (LF and CRLF): matches both
  • the table-named-"database is locked" counter-example from the prior round: still does not match, with or without \r

Existing 2 tests/test_storage.bats cases, and the full 31-case file, still green on both SQLite 3.51.0 and 3.53.4.

The stdin fix for #1645 fed the schema SQL through `printf | agmsg_sqlite
-bail`, a pipe whose right-hand side runs in a subshell -- so the escape
probe's memo set by agmsg_sqlite_warm above it dies with that subshell,
same as every other piped call in this codebase, which is why they all
carry their own warm call directly above the pipe. tests/test_remote_sync.bats's
#462 scan checks for exactly that: a warm within two lines of a
`| agmsg_sqlite`. storage_init's warm sat before the SQL string, ~100
lines above the actual pipe, so the scan counted it as an unwarmed pipe
and both of CI's #462 cases went red.

A heredoc redirect (`<<SQL ... SQL`) runs in the CURRENT shell, not a
subshell, so there is no memo to lose and no warm call is needed here at
all -- removing the pipe removes the test's premise rather than working
around it. The delimiter is left unquoted so ${_AGMSG_STORAGE_SCHEMA_REV}
still expands; checked that nothing else in the block contains a `$` or
backtick for that same unquoted expansion to catch.

tests/test_remote_sync.bats #47/#48 (the #462 scan and its mutation
control) and the two tests/test_storage.bats #1645 cases green; full
test_storage.bats (31) and test_remote_sync.bats (55) green. test_storage
cases confirmed on both SQLite 3.51.0 and 3.53.4 again.
@fujibee

fujibee commented Oct 17, 2026

Copy link
Copy Markdown
Owner Author

Fixed the CI red: storage_init's schema SQL now goes through a heredoc redirect (`<<SQL ... SQL`), not a `printf | agmsg_sqlite -bail` pipe. A pipe's right-hand side runs in a subshell, so the escape probe's memo dies with it — same as every other piped call in this codebase, which is why they all carry `agmsg_sqlite_warm` directly above their pipe. This one's warm sat ~100 lines above the actual pipe (before the SQL string began), so tests/test_remote_sync.bats's #462 scan (warm within two lines of a `| agmsg_sqlite`) counted it as unwarmed and both #462 cases went red.

A heredoc runs in the current shell, so there's no subshell to lose the memo to and no warm call needed at all — this removes the pipe (and the test's premise) rather than working around it. Delimiter is unquoted so `${_AGMSG_STORAGE_SCHEMA_REV}` still expands; confirmed nothing else in the block has a `$` or backtick for that to catch.

Verified:

@fujibee
fujibee merged commit a19bb20 into main Oct 20, 2026
27 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant