Incremental extract dies at merge with KeyError: 'id' — attach_hyperedges reads persisted hyperedges that the writer never gave an id (0.9.41)

[otterswimware]

Summary

attach_hyperedges (graphify/export.py:165) reads previously-persisted hyperedges with a hard subscript h["id"], but nothing on the write path requires a hyperedge to carry an id. The semantic (LLM) extractor routinely emits hyperedges as {"nodes": [...], "type": "...", "attributes": {...}} with no id, and build.py persists them verbatim. The result: the first graphify extract on a corpus succeeds (there is no prior graph.json to read back), and every incremental re-extract afterward dies at the merge step with KeyError: 'id', writing nothing.

Because the first run works, this reads as flaky extraction rather than a deterministic bug. It is deterministic.

Version: graphifyy 0.9.41 (verified against the installed package). Backend: --backend ollama (qwen3:8b). macOS.

Traceback

File "graphify/cli.py", line 3873, in dispatch_command
    G = _build_merge(...)
File "graphify/build.py", line 1749, in build_merge
    attach_hyperedges(G, carried)
File "graphify/export.py", line 165, in attach_hyperedges
    seen_ids = {h["id"] for h in existing}
               ~^^^^^^
KeyError: 'id'

Root cause

The asymmetry is visible inside the single function:

def attach_hyperedges(G: nx.Graph, hyperedges: list) -> None:
    existing = G.graph.get("hyperedges", [])
    seen_ids = {h["id"] for h in existing}          # line 165 — hard subscript on the PERSISTED set
    for h in hyperedges:
        if h.get("id") and h["id"] not in seen_ids: # line 167 — .get() on the INCOMING set
            existing.append(h)
            seen_ids.add(h["id"])
    G.graph["hyperedges"] = existing

existing is whatever was written to graph.json on a prior run. The write path (build.py:1249, G.graph["hyperedges"] = kept_hyperedges) stores LLM-emitted hyperedges without ever requiring an id, so a persisted graph can — and in practice does — contain id-less hyperedges. On the next run, line 165 iterates that persisted set with h["id"] and raises.

In my doc graph, 183 of 234 persisted hyperedges had no id (all of type: "project" / type: "module", member lists present, id absent). AST-only code graphs are unaffected — they carry zero hyperedges; only the semantic path creates them.

Reproduction

1. graphify extract <corpus> --backend ollama --model qwen3:8b --out out on a doc corpus whose LLM output includes project/module-type hyperedges. Succeeds.
2. Run the same command again (incremental). Dies at merge with the traceback above.

Minimal, no LLM needed — exercises the exact failing call:

import json, networkx as nx
from graphify.export import attach_hyperedges

he = [{"nodes": ["a", "b"], "type": "project", "attributes": {}}]  # id-less, as the LLM emits
G = nx.DiGraph(); G.graph["hyperedges"] = he
attach_hyperedges(G, he)   # KeyError: 'id'

Suggested fix

Make the read tolerant, symmetric with the write and with the loop directly below it — line 165 should skip id-less persisted entries the same way line 167 already does for incoming ones:

seen_ids = {h["id"] for h in existing if h.get("id")}

That stops the crash but still silently drops id-less hyperedges from dedup. The more complete fix is to stamp a deterministic, content-derived id at the point of persistence (build.py, where kept_hyperedges is assigned), so every hyperedge that reaches graph.json carries one and dedup across incremental runs stays correct. Deriving the id from sha256({type, sorted(nodes)}) keeps it stable across runs so existing id-dedup continues to work.

Workaround (for anyone hitting this now)

Pre-stamp ids on the persisted graph.json before each re-extract:

import hashlib, json, os
d = json.load(open("out/graphify-out/graph.json"))
for holder in (d, d.get("graph") or {}):
    for h in holder.get("hyperedges", []) or []:
        if isinstance(h, dict) and not h.get("id"):
            key = json.dumps({"t": h.get("type",""), "n": sorted(h.get("nodes",[]) or [])}, sort_keys=True)
            h["id"] = "he_" + hashlib.sha256(key.encode()).hexdigest()[:16]
tmp = "out/graphify-out/graph.json.tmp"
json.dump(d, open(tmp,"w"), separators=(",",":")); os.replace(tmp, "out/graphify-out/graph.json")

Related but distinct from the closed hyperedge-merge issues (#2484 relabel/compose, #2485 wrong-slot read + revalidation wipeout, #2486 TypeError on object-member hyperedges): those concern merge-graphs/label/cluster-only and object-vs-string members. This one is an id-less hyperedge crashing attach_hyperedges dedup on the ordinary incremental extract path, still present in 0.9.41.

[safishamsi]

Fixed in v0.9.45 (on PyPI now) via #2776, thanks @ousamabenyounes. Closing.