Repository navigation
Integrations should not own document text: shrink the side-car to ids + metadata #427
Description
Summary
The four integration wrappers persist a JSON side-car holding the full document/node text alongside the .tvim index. turbovec should own indexing and search; the frameworks should own documents. The side-car should shrink to ids + metadata only, and the text should go.
This is the structural fix behind #425 (side-car 5.5x larger than the index at 100k docs) — text is what makes it large.
Why this is safe: all four frameworks support resolving text separately
Validated empirically, not from documentation. Wheel built with maturin build --release, installed non-editable into a fresh Python 3.11.12 venv; framework versions langchain-core 1.5.3 / langchain-classic 1.0.8, llama-index-core 0.14.23, haystack-ai 3.0.0, agno 2.8.6.
| Framework | Resolution step | Status |
|---|---|---|
| LangChain | MultiVectorRetriever takes a docstore and calls docstore.mget(ids) on our hits |
first-class, works today |
| LlamaIndex | StorageContext.docstore owns text; SimpleVectorStore itself sets stores_text = False |
first-class |
| Haystack | AutoMergingRetriever / SentenceWindowRetriever accept a different document_store than the one that searched |
first-class |
| Agno | none — user subclasses and overrides search |
hand-rolled |
Confirmed running: MultiVectorRetriever(vectorstore=TurboQuantVectorStore(...), docstore=InMemoryStore(), id_key="doc_id") retrieves documents that provably came from the docstore; LocalFileStore likewise. AutoMergingRetriever(document_store=parent_store).run(documents=turbovec_hits) returned the parent's full text while every document held in turbovec had content=None.
What has to change
Deleting the JSON writer alone is not enough — three read paths reconstruct results from the side-car, so the in-memory duplication would simply remain undisk'd:
-
LlamaIndex —
query()hard-requires its own copy of the node:metadata_dict_to_node(data["node_dict"])atturbovec-python/python/turbovec/llama_index.py:544, called from:848. Blanking the payload raisesValueError: Node content not found in metadata dict.Needsadd()to stop storing node payloads andquery()to returnVectorStoreQueryResult(nodes=None, ids=..., similarities=...), matchingSimpleVectorStore. A ~40-line prototype overturbovec.IdMapIndexwas verified end to end: no vector-store JSON written at all, retrieval returns full text with the correct top hit.- Accepted cost:
VectorStoreIndex.from_vector_store(vs)raisesValueError: Cannot initialize from a vector store that does not store text.This is by design and is whatSimpleVectorStoredoes too. - Note the shipped class already accepts
stores_text=Falseas a constructor arg, but still writesdefault__vector_store.nodes.json(2862 B) containing the text — so text is duplicated on disk today even when the caller opts out.
- Accepted cost:
-
Agno — correctness bug, independent of this work.
TurboQuantVectorDb._dedup_by_contenthashesmd5(doc.content). With blank content asearch(limit=3)silently returns 1 result. Should dedup on id, not content. -
LangChain — no vector-in door.
add_textsembeds exactly what it stores, andadd_embeddingsdoes not exist (sorted(m for m in dir(vs) if m.startswith("add")) == ['add_documents', 'add_texts']). Passingpage_content=""does not raise — it goes semantically dead: all three test documents embedded identically, every score exactly0.2286243885755539, wrong document ranked first. Addingadd_embeddings(embeddings, ids, metadatas)makes text-free a supported path rather than a trick. -
Haystack — no change required.
write_documentsacceptscontent=None, andembedding_retrievalranked correctly with all content blank (p11.004 vsp20.1243).
What stays
Metadata stays in the side-car. LangChain's dict and callable filters and Haystack's filter_documents both walk the local document table, so metadata cannot leave without breaking filtering. Metadata is small, and it is arguably index-adjacent anyway.
Also worth capturing when this is done: turbovec derives its own ids (_derive_doc_id = md5(f"{base_id}_{content_hash}")), so a caller-supplied id like p1 becomes 225df170ad3940d474744dbc0de92068. Any external store must key on doc.name or metadata rather than the id the user passed in — that needs documenting. Separately, Agno's content_hash_exists() is in-memory only, so with no durable turbovec state the skip-if-already-ingested check always reports "not present" and re-ingests every run; that state needs a home.
Scope
Breaking only for callers that rely on turbovec handing back text it stored. Not a major version.
Activity
github-actions commentedon Jul 30, 2026
Claude encountered an error after 1s —— View job
I'll analyze this and get back to you.
RyanCodrai commentedon Jul 30, 2026
Set aside for a maintainer decision: this is a design/restructure call, not a defect with a single correct fix. Flagged during an automated issue-fixing pass on 2026-07-30 rather than patched, because point fixes here risk entrenching the structure the issue is questioning.
RyanCodrai commentedon Jul 31, 2026
/summary
tq-bot commentedon Jul 31, 2026
Plain-English summary: When turbovec is used through one of the four framework integrations (LangChain, LlamaIndex, Haystack, Agno), it currently saves a full copy of every document's text in a JSON file next to its search index. That copy is redundant — the frameworks already have their own places to keep documents — and it's what makes that file balloon to several times the size of the index itself (#425). This issue proposes that turbovec stop storing text entirely and keep only document ids and metadata, with each framework's own document store supplying the text when results come back. The author tested all four frameworks and confirmed they can each resolve text on their own, so the change is safe in principle.
Simply deleting the file isn't enough, though, because parts of the code read text back out of it. The LlamaIndex integration would fail outright without its stored copy, the Agno integration has a separate bug where it accidentally collapses results that have no text, and LangChain would need a new way to accept precomputed embeddings so callers aren't forced to hand over text. Haystack needs no changes. Metadata has to stay because filtering depends on it. The change would only break users who rely on turbovec returning the text it stored, and the issue has been set aside for a maintainer to decide, since it's a design restructuring rather than a straightforward bug fix.
RyanCodrai commentedon Jul 31, 2026
@q-turbovec-bot please investigate whether this is possible. Ideally people will be able to use the 4 integrations as a build your own setup where people can choose turboquant as the index backend and the rest is handled by an existing functionality. Let me know to what degree this is possible
tq-bot commentedon Jul 31, 2026
Claude finished @RyanCodrai's task in 2m 21s —— View job
Todos
Verdict: possible for all four, but to very different degrees today
I read all four wrappers end to end (
langchain.py,llama_index.py,haystack.py,agno.py) plus_persist.py, and code-verified every claim in the issue body. The short answer: the "turboquant as the index backend, framework owns the documents" composition is architecturally sound for all four frameworks, works out of the box for two of them (Haystack fully, LangChain with a caveat), needs a moderate wrapper change for LlamaIndex, and needs both a bug fix and a hand-rolled pattern for Agno.add_embeddings()stores_text=Falseinadd()/query()/persist()Haystack — works today, zero changes
write_documentsonly requiresdoc.embeddingto be non-None (haystack.py:265-270);content=Nonepasses straight through and_reconstruct(haystack.py:1020) returns it as-is. Haystack's retrievers (AutoMergingRetriever,SentenceWindowRetriever) accept a differentdocument_storethan the one that searched, so the pattern "content-free documents + embeddings intoTurboQuantDocumentStore, full text in anInMemoryDocumentStore" is first-class framework functionality. The issue's empirical result (correct ranking with all content blank) is consistent with the code: ranking runs entirely on the index, and content is never consulted on the retrieval path. The side-car still gets written, but withcontent: nullit's already just ids + metadata.One caveat: metadata filtering does read the local table (
filter_documentsand theembedding_retrievalallowlist athaystack.py:646-649both walk_u64_to_doc), so metadata must keep living in the side-car — which matches the issue's "metadata stays" scope.LangChain — the retrieval side is first-class; the ingestion side has no vector-in door
MultiVectorRetriever(vectorstore=..., docstore=..., id_key=...)resolving hits throughdocstore.mget(ids)is stock LangChain and works with our store today. The gap is exactly what the issue says: the only write paths areadd_texts/add_documents(langchain.py:258,:323), which embed exactly the text they store. There is no way to hand the store a precomputed embedding, so a caller who wants text-free storage has two bad options — pass real text (it gets stored in_docsand dumped to the side-car, recreating the bloat) or passpage_content=""(every document embeds identically, search goes semantically dead; nothing raises). Anadd_embeddings(embeddings, ids, metadatas)method is a small, purely additive change —_store_texts_and_vectors(langchain.py:357) already takes vectors directly, so it's mostly a new public entry point storing""(or a sentinel) as text. Dict/callable filters walkself._docs(langchain.py:592-601), so metadata again stays local.LlamaIndex — the framework supports it first-class; our wrapper doesn't honor it yet
This is the interesting one. The framework's own pattern is exactly what you want: a store that declares
stores_text=FalsemakesVectorStoreIndexkeep nodes inStorageContext.docstoreand resolve query-result ids back to full nodes there —SimpleVectorStoreitself works this way. Our class even acceptsstores_text=Falseas a constructor kwarg (it's a pydantic field,llama_index.py:213). But the flag is currently cosmetic:add()unconditionally stores the full node payload including text —node_to_metadata_dict(node, remove_text=False, ...)atllama_index.py:348-350.query()unconditionally reconstructs nodes from that copy viametadata_dict_to_nodein_reconstruct_node(llama_index.py:548, called from:829/:852— the issue's:544/:848refs have drifted a few lines).persist()writes the whole_nodesmap to*.nodes.json(llama_index.py:961-980), text included, even when the caller passedstores_text=False.So the composition functions today (retrieval returns full text — from our copy), but the duplication in memory and on disk remains. Making
stores_text=Falsereal means: skip thenode_dictpayload inadd(), returnVectorStoreQueryResult(nodes=None, ids=..., similarities=...)fromquery(), and drop node payloads frompersist(). The accepted costs are the same onesSimpleVectorStoreimposes:VectorStoreIndex.from_vector_store(vs)raises, andget_nodes()can no longer return populated nodes in that mode. Moderate, well-scoped change.Agno — the only one with no framework-side mechanism
Agno's
VectorDbprotocol has no "resolve text elsewhere" hook;search()buildsDocuments straight from the side-car payload (agno.py:785-814). Two concrete blockers for a text-free setup:_dedup_by_content(agno.py:767-783) keys onmd5(doc.content). With blank content, every hit hashes identically andsearch(limit=3)silently returns 1 result. (Withcontent=Noneit wouldAttributeErroron.encode().) Deduping ondoc.idfixes it and is worth doing independently of this issue._derive_doc_id(agno.py:349-353) turns a caller-supplied id intomd5(f"{base_id}_{content_hash}"), so an external text store can't key on the id the user passed in — it has to key ondoc.nameor metadata. That's a documentation item, not a code change (the derivation matches LanceDb's contract deliberately).One correction to the issue body: the claim that
content_hash_exists()state has no durable home isn't quite right under this proposal._content_hashesis rebuilt from the side-car payloads on load (agno.py:1169-1174), andcontent_hashis metadata-sized — so as long as the shrunk ids+metadata side-car keeps thecontent_hashfield (it should), the skip-if-already-ingested check survives restarts withpath=set. It only breaks if turbovec keeps no state at all, which is beyond what this issue proposes.Cross-cutting: what must stay in the side-car
Metadata genuinely cannot leave. All four wrappers resolve filters by walking the local table before the kernel search: LangChain
_search_vector, LlamaIndex_resolve_allowed_handles/_filters_match, Haystackfilter_documents/embedding_retrieval, Agno_meta_matches. The kernel only understands handle allowlists, so metadata is index-adjacent by construction. The shrunk side-car is ids + handles + metadata (+ Agno'scontent_hash/name/content_idbookkeeping fields) — which is exactly the issue's proposed end state, and removes the text that makes #425's file 5.5x the index.Bottom line
The "build your own" vision holds: Haystack needs a docs page, LangChain needs one small additive method, LlamaIndex needs its existing
stores_text=Falseflag wired through three methods, and Agno needs a one-line dedup fix plus an honest "subclass and overridesearch" recipe. None of it is a major version — only callers who rely on turbovec handing back text it stored would notice, and for LlamaIndex only when they opt intostores_text=False. Happy to implement any of these in follow-up if you want to sequence it — the LangChainadd_embeddingsand the Agno dedup fix are independent and low-risk, the LlamaIndex change is the substantive one.