Skip to content

Run Composio normalisation and storage on a driver that is not TinyCortex (#18 §B3) - #41

Merged
YellowSnnowmann merged 5 commits into
tinyhumansai:mainfrom
YellowSnnowmann:feat/18-b3-sync-on-a-foreign-driver
Aug 18, 2026
Merged

Run Composio normalisation and storage on a driver that is not TinyCortex (#18 §B3)#41
YellowSnnowmann merged 5 commits into
tinyhumansai:mainfrom
YellowSnnowmann:feat/18-b3-sync-on-a-foreign-driver

Conversation

@YellowSnnowmann

Copy link
Copy Markdown
Contributor

Stacked on #40. Once that merges this reduces to one commit.

Summary

#40 extracted the Composio normalisers into an engine-neutral crate. This asserts that the result is actually reachable without an engine, which is the only thing that makes the extraction worth anything.

A raw Composio Gmail payload goes through tinymemory_sync::gmail_post_process, and each normalised message is stored into a bound provider. The test file's entire dependency list is the facade, the conformance reference driver, and the sync crate — it names no engine, and before §B3 it could not have compiled.

That is §B3's stated purpose ("so a non-TinyCortex engine gets Composio sync for free") turned into something that fails if it stops being true.

Two drivers, not one

"A non-TinyCortex engine" is a claim about drivers in general, not about whichever one was convenient. So the identical path runs against:

  • the reference driver — which retains, and is asserted to round-trip the normalised body
  • the null driver — whose retention semantics are the opposite, and whose empty read is asserted for exactly that reason

Provenance is asserted on the path where it matters

The payload came off somebody's inbox, so it is stored ExternalSync and the read-back checks the driver did not launder it to Internal. That is the single failure the taint argument exists to prevent, and this is the path where externally-sourced content actually arrives.

What this does not prove

Written into the module docs rather than left to be inferred.

§B5's full acceptance drives a live Composio API through the sync pipeline, and that pipeline still sits behind engine-owned state (§B1, §B2). What is testable today is that the transform and the storage tier have no engine between them — the part §B3 was responsible for.

The first draft failed, usefully

The fixture used the reshaped field names rather than the upstream ones, so the normaliser emitted nothing:

assertion `left == right` failed: one message was written
  left: []
 right: ["msg-1"]

It caught an empty result rather than a wrong one. The fixture now uses the upstream shape (messageId, sender, messageTimestamp) and the assertions use the reshaped one (id) — which is precisely the transform under test, and would not have been exercised by the original.

Public API changes

None. One integration test, and a dev-dependency on tinymemory-sync.

Validation

Command Result
cargo fmt --all -- --check pass
cargo clippy --all-targets --all-features -- -D warnings pass
cargo build --all-targets --all-features pass
cargo test --all-features 1219 passed, 0 failed
RUSTDOCFLAGS="-D warnings" cargo doc --no-deps --all-features pass
./scripts/ci/dependency-budget.sh pass
module crate cargo test --lib pass — 35

Note on §B4

I started on §B4 ("core/src/sources/types.rs stops being a TinyCortex re-export shim") and stopped, because it is not the repoint it appears to be.

SourceItem and SourceKind exist in both the contract and the engine, and they are different types wearing the same names:

contract engine
SourceItem item_id, title, content, mime, url, updated_at_ms, tags id, title, updated_at_ms, …
SourceKind Chat, Email, Document — a content class Composio, Conversation, Folder, GithubRepo, TwitterQuery — an origin

Repointing the shim would silently change semantics. The engine's sources module is extractable in principle — 27 files, 6,045 lines, no SQL — but it depends on MemoryConfig (7 sites), MemoryEngineResult/MemoryError (5) and RawKind (2), which is the §A3 boundary in another guise. Flagging rather than guessing.

Related

Part of #18 (§B3), and the first movement on acceptance criterion 6.

Issue tinyhumansai#18 §B3: "Payload normalisers are pure `Value -> Value` transforms with
no engine dependency. Move them back into a `tinymemory-sync` crate — so a
non-TinyCortex engine gets Composio sync for free."

They were not in this workspace at all. They lived inside the TinyCortex engine,
and `tinymemory-core` reached in through
`tinycortex::memory::sync::composio::providers::normalize::*` to use them. A
host binding a different memory engine therefore could not have Composio sync,
despite none of this code caring which engine is bound. That is the coupling
§B3 names, and it ran through the engine rather than around it.

`tinymemory-sync` is fifteen files and 2,598 lines, depending on `serde_json`,
two logging facades, and `chrono`. It links no engine, no storage, no async
runtime — and no contract either.

Two things found while moving, both stated rather than smoothed over.

The crate is not quite the pure function of its input that §B3 describes.
`format_email_local_time` renders in `chrono::Local`, so it reads the host's
timezone, and `notion::now_ms` reads the clock. Both are deliberate upstream —
the agent presents local times without doing UTC arithmetic, and Notion payloads
carry no ingestion timestamp — and the raw UTC field is preserved alongside, so
sorting and deduplication stay UTC-based. Documented at the crate root and at
each function rather than left for someone whose output moves when they change
`TZ`.

The two logging facades are also inherited: `gmail_post_process` traces through
`tracing`, `slack_post_process` through `log`. Preserved rather than unified,
because §B3 is a move and swapping a facade changes where a host's log lines
surface — a behaviour change hiding inside a relocation.

The move is otherwise verbatim, with four exceptions, all forced by this
workspace's lint configuration being stricter than the engine's gate reached:
two `unwrap`s removed by checking presence immutably before fetching mutably,
one `if let ... else { return None }` rewritten as `?`, and one `unwrap` in
`ensure_object` turned into a scoped `expect` with the invariant spelled out —
the case `AGENTS.md` explicitly permits. Doc links pointing at engine-internal
paths are unlinked to prose, since this crate deliberately cannot see them.

Acceptance, measured: `cargo tree -p tinymemory-sync` links zero of
`tinycortex`, `rusqlite`, `tinymemory-core`, `tinymemory-api`. Core no longer
names the engine's normalisers anywhere.

The engine keeps its copy until tinyhumansai/tinycortex removes it; that side is
a companion change, and the module is dead code there — its only remaining
references are four doc links.

Refs tinyhumansai#18 (§B3)
…rtex

Issue tinyhumansai#18 §B3's stated purpose — "so a non-TinyCortex engine gets Composio sync
for free" — asserted rather than assumed, and the first half of §B5's
acceptance test.

The extraction in the previous commit is only worth something if the result is
reachable without an engine. This drives a raw Composio Gmail payload through
`tinymemory_sync::gmail_post_process` and stores each normalised message into a
bound provider, in a file whose dependencies are the facade, the conformance
reference driver, and the sync crate. It names no engine, and before §B3 it
could not have compiled: the normalisers lived inside TinyCortex, and
`tinymemory-core` reached in to use them.

Run against two drivers rather than one. "A non-TinyCortex engine" is a claim
about drivers in general, not about whichever one happened to be convenient, so
the identical path runs against the reference driver and against the null
driver — whose retention semantics are the opposite, and whose empty read is
asserted for that reason.

Provenance is asserted too. The payload came off somebody's inbox, so it is
stored `ExternalSync`, and the read-back checks the driver did not launder it to
`Internal`. That is the one failure the taint argument exists to prevent, and it
is worth pinning on the path where external content actually arrives.

What this does not prove is written into the module docs rather than left to be
inferred: §B5's full acceptance drives a live Composio API through the sync
pipeline, and that pipeline still sits behind engine-owned state (§B1, §B2).
What is testable today is that the transform and the storage tier have no engine
between them, which is the part §B3 was responsible for.

The first draft of this test failed, which was useful: the fixture used the
reshaped field names rather than the upstream ones, so the normaliser emitted
nothing and the assertion caught an empty result rather than a wrong one. The
fixture now uses the upstream shape (`messageId`, `sender`, `messageTimestamp`)
and the assertions the reshaped one (`id`), which is the transform under test.

Refs tinyhumansai#18 (§B3, toward §B5 and acceptance criterion 6)
@coderabbitai

coderabbitai Bot commented Aug 18, 2026

Copy link
Copy Markdown

Important

Review available on request

  • 🔍 Trigger review

Reviews should be triggered manually for repositories with fewer than 10 stars. Select Trigger review above or comment @coderabbitai review to review the latest changes. For a full review, comment @coderabbitai full review.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: b1712256-5b49-48d9-898a-75cbf8ddc461


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@tinysweeper tinysweeper Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tinysweeper found nothing blocking. Approving.

$0.0000 · 0 in / 0 out

@tinysweeper tinysweeper Bot added the priority: p3 Whenever. Cosmetic, a nicety, or a cleanup with no user visible effect. label Aug 18, 2026
Both are places where a guard I added was weaker than the claim it was meant to
support.

**The awkward-content assertion had too low a floor.** §E1's run against
TinyCortex found that the engine refuses empty content, which the contract
explicitly permits — `MemoryCore::store` documents `Invalid` "for caller input
the driver rejects" — so the assertion was corrected to allow a refusal. The
guard against that becoming vacuous was `accepted > 0`, which is too weak: a
driver accepting only `empty` and refusing unicode, large and newlines would
have passed.

The floor is now `unicode` specifically. Refusing `empty` is documented
validation and refusing `large` is a defensible size limit, but refusing
ordinary UTF-8 is a broken driver — and unicode is the case where mangling
shows at all, since truncation and re-encoding are invisible on ASCII. All
seven drivers still pass.

**`tinymemory-sync` was unguarded.** That crate exists because it has no engine
behind it (§B3): the Composio normalisers lived inside TinyCortex, and a host
binding a different engine could not run them. Nothing enforced that after the
extraction. A dependency added two crates away would put the coupling back
silently — the build would stay green and the property would just stop being
true.

`dependency-budget.sh` now reports the crate (20 crates today) and fails if it
reaches `tinycortex`, `rusqlite`, `libsqlite`, `tinymemory-core` or
`tinymemory-api`. Verified in both directions: it passes now, and injecting
`tinymemory-api = { path = "../api" }` makes it fire and name the offender.

Refs tinyhumansai#18 (§B3, §E1)
tinycortex#149 landed as a squash (8401346b), discarding the branch head
this pin pointed at; 34cbb6c is diverged from tinycortex main rather than
an ancestor of it. The merged commit is also the one that deletes the 33
duplicated files under api/src/, which is the state this stack depends on.
@tinysweeper

tinysweeper Bot commented Aug 18, 2026

Copy link
Copy Markdown

How this change flows

1 changed behaviour across 4 relationships. 3 surrounding behaviours are shown (60 graph nodes walked). 21 further behaviours left out to keep the diagram readable.

flowchart LR
  n0["assert_awkward_content_round_trips<br/>changed"]:::changed
  n1["MemoryProvider"]:::impacted
  n2["push"]:::impacted
  n3["ingest_into"]:::impacted
  n0 -->|uses| n1
  n0 -->|calls| n2
  n3 -->|calls| n2
  n3 -->|tests| n2
  classDef changed fill:#0d4429,stroke:#238636,color:#e6edf3
  classDef impacted fill:#161b22,stroke:#6e7681,color:#c9d1d9
  classDef flagged fill:#5a1e02,stroke:#d93f0b,color:#ffffff
  classDef blocking fill:#67060c,stroke:#f85149,color:#ffffff
Loading

Green: changed behaviour. Grey: surrounding behaviour. Arrows name the call, use, implementation, or test relationship. Orange: has findings. Red: has a finding that blocks the merge.

tinysweeper 0.1.0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

priority: p3 Whenever. Cosmetic, a nicety, or a cleanup with no user visible effect.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant