Run Composio normalisation and storage on a driver that is not TinyCortex (#18 §B3) - #41
Conversation
Issue tinyhumansai#18 §B3: "Payload normalisers are pure `Value -> Value` transforms with no engine dependency. Move them back into a `tinymemory-sync` crate — so a non-TinyCortex engine gets Composio sync for free." They were not in this workspace at all. They lived inside the TinyCortex engine, and `tinymemory-core` reached in through `tinycortex::memory::sync::composio::providers::normalize::*` to use them. A host binding a different memory engine therefore could not have Composio sync, despite none of this code caring which engine is bound. That is the coupling §B3 names, and it ran through the engine rather than around it. `tinymemory-sync` is fifteen files and 2,598 lines, depending on `serde_json`, two logging facades, and `chrono`. It links no engine, no storage, no async runtime — and no contract either. Two things found while moving, both stated rather than smoothed over. The crate is not quite the pure function of its input that §B3 describes. `format_email_local_time` renders in `chrono::Local`, so it reads the host's timezone, and `notion::now_ms` reads the clock. Both are deliberate upstream — the agent presents local times without doing UTC arithmetic, and Notion payloads carry no ingestion timestamp — and the raw UTC field is preserved alongside, so sorting and deduplication stay UTC-based. Documented at the crate root and at each function rather than left for someone whose output moves when they change `TZ`. The two logging facades are also inherited: `gmail_post_process` traces through `tracing`, `slack_post_process` through `log`. Preserved rather than unified, because §B3 is a move and swapping a facade changes where a host's log lines surface — a behaviour change hiding inside a relocation. The move is otherwise verbatim, with four exceptions, all forced by this workspace's lint configuration being stricter than the engine's gate reached: two `unwrap`s removed by checking presence immutably before fetching mutably, one `if let ... else { return None }` rewritten as `?`, and one `unwrap` in `ensure_object` turned into a scoped `expect` with the invariant spelled out — the case `AGENTS.md` explicitly permits. Doc links pointing at engine-internal paths are unlinked to prose, since this crate deliberately cannot see them. Acceptance, measured: `cargo tree -p tinymemory-sync` links zero of `tinycortex`, `rusqlite`, `tinymemory-core`, `tinymemory-api`. Core no longer names the engine's normalisers anywhere. The engine keeps its copy until tinyhumansai/tinycortex removes it; that side is a companion change, and the module is dead code there — its only remaining references are four doc links. Refs tinyhumansai#18 (§B3)
…rtex Issue tinyhumansai#18 §B3's stated purpose — "so a non-TinyCortex engine gets Composio sync for free" — asserted rather than assumed, and the first half of §B5's acceptance test. The extraction in the previous commit is only worth something if the result is reachable without an engine. This drives a raw Composio Gmail payload through `tinymemory_sync::gmail_post_process` and stores each normalised message into a bound provider, in a file whose dependencies are the facade, the conformance reference driver, and the sync crate. It names no engine, and before §B3 it could not have compiled: the normalisers lived inside TinyCortex, and `tinymemory-core` reached in to use them. Run against two drivers rather than one. "A non-TinyCortex engine" is a claim about drivers in general, not about whichever one happened to be convenient, so the identical path runs against the reference driver and against the null driver — whose retention semantics are the opposite, and whose empty read is asserted for that reason. Provenance is asserted too. The payload came off somebody's inbox, so it is stored `ExternalSync`, and the read-back checks the driver did not launder it to `Internal`. That is the one failure the taint argument exists to prevent, and it is worth pinning on the path where external content actually arrives. What this does not prove is written into the module docs rather than left to be inferred: §B5's full acceptance drives a live Composio API through the sync pipeline, and that pipeline still sits behind engine-owned state (§B1, §B2). What is testable today is that the transform and the storage tier have no engine between them, which is the part §B3 was responsible for. The first draft of this test failed, which was useful: the fixture used the reshaped field names rather than the upstream ones, so the normaliser emitted nothing and the assertion caught an empty result rather than a wrong one. The fixture now uses the upstream shape (`messageId`, `sender`, `messageTimestamp`) and the assertions the reshaped one (`id`), which is the transform under test. Refs tinyhumansai#18 (§B3, toward §B5 and acceptance criterion 6)
|
Important Review available on request
Reviews should be triggered manually for repositories with fewer than 10 stars. Select Trigger review above or comment ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Both are places where a guard I added was weaker than the claim it was meant to
support.
**The awkward-content assertion had too low a floor.** §E1's run against
TinyCortex found that the engine refuses empty content, which the contract
explicitly permits — `MemoryCore::store` documents `Invalid` "for caller input
the driver rejects" — so the assertion was corrected to allow a refusal. The
guard against that becoming vacuous was `accepted > 0`, which is too weak: a
driver accepting only `empty` and refusing unicode, large and newlines would
have passed.
The floor is now `unicode` specifically. Refusing `empty` is documented
validation and refusing `large` is a defensible size limit, but refusing
ordinary UTF-8 is a broken driver — and unicode is the case where mangling
shows at all, since truncation and re-encoding are invisible on ASCII. All
seven drivers still pass.
**`tinymemory-sync` was unguarded.** That crate exists because it has no engine
behind it (§B3): the Composio normalisers lived inside TinyCortex, and a host
binding a different engine could not run them. Nothing enforced that after the
extraction. A dependency added two crates away would put the coupling back
silently — the build would stay green and the property would just stop being
true.
`dependency-budget.sh` now reports the crate (20 crates today) and fails if it
reaches `tinycortex`, `rusqlite`, `libsqlite`, `tinymemory-core` or
`tinymemory-api`. Verified in both directions: it passes now, and injecting
`tinymemory-api = { path = "../api" }` makes it fire and name the offender.
Refs tinyhumansai#18 (§B3, §E1)
tinycortex#149 landed as a squash (8401346b), discarding the branch head this pin pointed at; 34cbb6c is diverged from tinycortex main rather than an ancestor of it. The merged commit is also the one that deletes the 33 duplicated files under api/src/, which is the state this stack depends on.
How this change flows1 changed behaviour across 4 relationships. 3 surrounding behaviours are shown (60 graph nodes walked). 21 further behaviours left out to keep the diagram readable. flowchart LR
n0["assert_awkward_content_round_trips<br/>changed"]:::changed
n1["MemoryProvider"]:::impacted
n2["push"]:::impacted
n3["ingest_into"]:::impacted
n0 -->|uses| n1
n0 -->|calls| n2
n3 -->|calls| n2
n3 -->|tests| n2
classDef changed fill:#0d4429,stroke:#238636,color:#e6edf3
classDef impacted fill:#161b22,stroke:#6e7681,color:#c9d1d9
classDef flagged fill:#5a1e02,stroke:#d93f0b,color:#ffffff
classDef blocking fill:#67060c,stroke:#f85149,color:#ffffff
Green: changed behaviour. Grey: surrounding behaviour. Arrows name the call, use, implementation, or test relationship. Orange: has findings. Red: has a finding that blocks the merge. |
Summary
#40 extracted the Composio normalisers into an engine-neutral crate. This asserts that the result is actually reachable without an engine, which is the only thing that makes the extraction worth anything.
A raw Composio Gmail payload goes through
tinymemory_sync::gmail_post_process, and each normalised message is stored into a bound provider. The test file's entire dependency list is the facade, the conformance reference driver, and the sync crate — it names no engine, and before §B3 it could not have compiled.That is §B3's stated purpose ("so a non-TinyCortex engine gets Composio sync for free") turned into something that fails if it stops being true.
Two drivers, not one
"A non-TinyCortex engine" is a claim about drivers in general, not about whichever one was convenient. So the identical path runs against:
Provenance is asserted on the path where it matters
The payload came off somebody's inbox, so it is stored
ExternalSyncand the read-back checks the driver did not launder it toInternal. That is the single failure the taint argument exists to prevent, and this is the path where externally-sourced content actually arrives.What this does not prove
Written into the module docs rather than left to be inferred.
§B5's full acceptance drives a live Composio API through the sync pipeline, and that pipeline still sits behind engine-owned state (§B1, §B2). What is testable today is that the transform and the storage tier have no engine between them — the part §B3 was responsible for.
The first draft failed, usefully
The fixture used the reshaped field names rather than the upstream ones, so the normaliser emitted nothing:
It caught an empty result rather than a wrong one. The fixture now uses the upstream shape (
messageId,sender,messageTimestamp) and the assertions use the reshaped one (id) — which is precisely the transform under test, and would not have been exercised by the original.Public API changes
None. One integration test, and a dev-dependency on
tinymemory-sync.Validation
cargo fmt --all -- --checkcargo clippy --all-targets --all-features -- -D warningscargo build --all-targets --all-featurescargo test --all-featuresRUSTDOCFLAGS="-D warnings" cargo doc --no-deps --all-features./scripts/ci/dependency-budget.shcargo test --libNote on §B4
I started on §B4 ("
core/src/sources/types.rsstops being a TinyCortex re-export shim") and stopped, because it is not the repoint it appears to be.SourceItemandSourceKindexist in both the contract and the engine, and they are different types wearing the same names:SourceItemitem_id, title, content, mime, url, updated_at_ms, tagsid, title, updated_at_ms, …SourceKindChat, Email, Document— a content classComposio, Conversation, Folder, GithubRepo, TwitterQuery— an originRepointing the shim would silently change semantics. The engine's
sourcesmodule is extractable in principle — 27 files, 6,045 lines, no SQL — but it depends onMemoryConfig(7 sites),MemoryEngineResult/MemoryError(5) andRawKind(2), which is the §A3 boundary in another guise. Flagging rather than guessing.Related
Part of #18 (§B3), and the first movement on acceptance criterion 6.