Repository navigation
storage: fix SQLite 3.53.4/3.54.0 CLI compatibility for init rollback and busy diagnostics (#1645) - #1654
Conversation
… and busy diagnostics (#1645) storage_init sent its schema transaction as a single multi-statement command-line argument and assumed -bail would stop at the first error before the trailing PRAGMA user_version. On SQLite 3.53.4 and 3.54.0's CLI, a failing statement inside that argument no longer stops the batch: the schema stamp lands over a broken transaction. Feeding the same statements to sqlite3 on stdin (one top-level statement ending its own physical line, same shape already used elsewhere in this file) restores the stop-at-first-error contract on every supported CLI version. Separately, a lock-timeout giveup reports exit code 1 instead of 5 on the same CLI versions, bypassing agmsg_sqlite()'s busy-specific diagnostic (the recording variant already classified busy/failed by matching stderr text rather than trusting the exit code). agmsg_sqlite() now normalizes its own return code to 5 whenever stderr says "database is locked" and the raw exit code did not already say so, so every existing caller that checks for 5 keeps working. Reproduced and fixed against both SQLite 3.51.0 (this machine's version) and 3.53.4 (downloaded standalone, not installed system-wide) on the two existing tests/test_storage.bats cases; no new tests added.
Review follow-up on #1654: mktemp/cat/rm on every agmsg_sqlite call added 3 extra forked processes per call on this hot path (every send/receive/watch.sh cycle), measured at tens to hundreds of ms per process on Windows -- the same cost #1627/#1630/#1652 are reducing elsewhere, and a per-call temp file is also what #1580 just finished cleaning up stray copies of. Replaced with the standard fd-juggle (3>&1 on the outer group, then 1>&3/2>&1 inside the command substitution) so stdout still streams straight to the caller and stderr lands in a shell variable, with no process beyond sqlite3/sed forked. rc capture (pipefail against sqlite3, not sed) is unchanged. tests/test_watch_process_count.bats: 46 forked processes per idle poll cycle, unchanged from main.
|
Updated per review: agmsg_sqlite() no longer uses mktemp/cat/rm to capture stderr (3 extra forked processes per call on this hot path). Rewritten with the standard fd-juggle ( Verified:
|
The busy-text check added for #1645 matched "database is locked" anywhere in stderr, so an ordinary error that happens to mention that text in a table/column name or an echoed statement was misclassified as a lock timeout. Reproduced: a SELECT against a table literally named "database is locked" reported "no such table: database is locked" and got treated as busy. Replaced the substring test with _agmsg_sqlite_err_is_busy(), matched against the exact whole lines sqlite3 itself emits for a real lock timeout, collected by reproducing one on both argv and stdin input and on both SQLite 3.51.0 and 3.53.4. The counter-example above (and the same shape on 3.53.4) stays rc=1, confirmed by hand -- no new test added. Regression coverage stays the existing tests/test_storage.bats busy case (#1001), green on both CLI versions.
|
Fixed the whole-line match gap: `agmsg_sqlite()`'s busy check now uses `_agmsg_sqlite_err_is_busy()`, matched against the exact whole lines sqlite3 itself emits for a real lock timeout (collected by reproducing one on both argv and stdin input, on both SQLite 3.51.0 and 3.53.4), instead of a bare substring test. Counter-example checked by hand, not added as a test: a `SELECT` against a table literally named `database is locked` reports "no such table: database is locked" and stays rc=1 on both CLI versions after the fix (it previously misclassified as busy, rc=5). Verified:
|
_agmsg_sqlite_err_is_busy()'s per-line patterns anchor on `$` (end of
line). Windows' sqlite3.exe ends stderr lines with \r\n, so a trailing
\r sat between "locked" and that anchor and every real busy line
missed, silently falling back to the non-busy rc. Stripped with a
builtin substitution (${1//$'\r'/}) used only for the match itself;
the stderr re-emitted to the caller is untouched.
Verified by hand, no new test: a real busy line matches whether it
carries \r\n, is one of several lines in a multi-line stderr, or is
plain LF; the existing table-named-"database is locked" counter-example
still does not match, CRLF or not.
|
Fixed: _agmsg_sqlite_err_is_busy()'s per-line patterns anchor on `$`, and Windows' sqlite3.exe ends stderr lines with \r\n, so the trailing \r sat between "locked" and the anchor and a real busy line went unmatched. Stripped with a builtin substitution (`${1//$'\r'/}`) used only for the match itself — no external command added, and the stderr re-emitted to the caller is untouched. Checked by hand, no new test:
Existing 2 tests/test_storage.bats cases, and the full 31-case file, still green on both SQLite 3.51.0 and 3.53.4. |
The stdin fix for #1645 fed the schema SQL through `printf | agmsg_sqlite -bail`, a pipe whose right-hand side runs in a subshell -- so the escape probe's memo set by agmsg_sqlite_warm above it dies with that subshell, same as every other piped call in this codebase, which is why they all carry their own warm call directly above the pipe. tests/test_remote_sync.bats's #462 scan checks for exactly that: a warm within two lines of a `| agmsg_sqlite`. storage_init's warm sat before the SQL string, ~100 lines above the actual pipe, so the scan counted it as an unwarmed pipe and both of CI's #462 cases went red. A heredoc redirect (`<<SQL ... SQL`) runs in the CURRENT shell, not a subshell, so there is no memo to lose and no warm call is needed here at all -- removing the pipe removes the test's premise rather than working around it. The delimiter is left unquoted so ${_AGMSG_STORAGE_SCHEMA_REV} still expands; checked that nothing else in the block contains a `$` or backtick for that same unquoted expansion to catch. tests/test_remote_sync.bats #47/#48 (the #462 scan and its mutation control) and the two tests/test_storage.bats #1645 cases green; full test_storage.bats (31) and test_remote_sync.bats (55) green. test_storage cases confirmed on both SQLite 3.51.0 and 3.53.4 again.
|
Fixed the CI red: storage_init's schema SQL now goes through a heredoc redirect (`<<SQL ... SQL`), not a `printf | agmsg_sqlite -bail` pipe. A pipe's right-hand side runs in a subshell, so the escape probe's memo dies with it — same as every other piped call in this codebase, which is why they all carry `agmsg_sqlite_warm` directly above their pipe. This one's warm sat ~100 lines above the actual pipe (before the SQL string began), so tests/test_remote_sync.bats's #462 scan (warm within two lines of a `| agmsg_sqlite`) counted it as unwarmed and both #462 cases went red. A heredoc runs in the current shell, so there's no subshell to lose the memo to and no warm call needed at all — this removes the pipe (and the test's premise) rather than working around it. Delimiter is unquoted so `${_AGMSG_STORAGE_SCHEMA_REV}` still expands; confirmed nothing else in the block has a `$` or backtick for that to catch. Verified:
|
What
SQLite 3.53.4 and 3.54.0's CLI broke two assumptions
agmsg's storage layer made about the older CLI (#1645):storage_init(scripts/drivers/storage/sqlite.sh) sent its whole schema transaction as a single multi-statement command-line argument, assuming-bailwould stop at the first error before the trailingPRAGMA user_versionstamp. On the newer CLI a failing statement no longer stops that argument form, so a broken init could still stamp the current revision. Fix: feed the same statements to sqlite3 on stdin instead (every top-level statement already ends its own physical line in this SQL, which is what the CLI's-bailhonors reliably on every version tested).agmsg_sqlite()'s busy-specific diagnostic. Fix:agmsg_sqlite()now also classifies busy by matching"database is locked"in stderr (the same text match the recording variant already used) and normalizes the return code to 5 when the raw exit code did not already say so.Survey of other
-bailcall sitesChecked every
-bailuse underscripts/for the same "single command-line argument carrying more than one statement" shape:sqlite.shstorage_init(line ~165)sqlite.sh_sqlite_exec_stdin(line 73)sqlite.shstorage_send(line ~337)sqlite.shstorage_import(line ~618)sqlite-sync.sh_sqlite_sync_commit_chunk(line 66)sqlite-sync.shack/pull batch via_sqlite_exec_stdin(lines 959, 1652)sqlite-sync.shapply-pagesql_file(lines 1419, 2021)Also found 4 sites that build a multi-statement
BEGIN IMMEDIATE ... COMMITblock and pass it as a single argv string without-bailat all (sqlite-sync.shlines ~431, ~520, ~1829, ~1858 — schema migration and the resync/read-block/read-unblock assert blocks). These never relied on-bail's stop-at-first-error contract in the first place; they validate the outcome via aCHECK-constrained temp-table assert plus the exit code, a different design unaffected by this CLI change. Left as-is.Testing
Existing
tests/test_storage.batscases only (no new tests):storage: a failing init batch leaves the OLD revision, never a new stamp over a broken schema (#1001)storage: a call that gives up after the busy timeout says so on stderr, and a quiet store says nothing (#1001)Both confirmed green on this machine's SQLite 3.51.0 and, separately, on SQLite 3.53.4 (downloaded standalone from sqlite.org into a scratch directory and prepended to
PATHonly for that test run — nothing installed system-wide). Fulltests/test_storage.bats(31 cases) also green on both versions.