[Feature]: Add deterministic project knowledge boundaries and cited full-text search

[roomote-community]

Problem

Roomote stores relevant project sources but cannot define a safe repository-scoped knowledge boundary or retrieve them as cited project context.

Proposed behavior

Add immutable knowledge-policy versions with repository/branch/source/date scope, include/exclude rules, authority, transcript policy, retention, and limits. Discover proposed boundaries from ignore files, generated/vendor/binary/size/secret detection, docs, CI/tests/build config, and code symbols. Ingest authorized versioned documents/chunks and provide PostgreSQL full-text retrieval with citations.

Reuse repository/environment/task/artifact authorization, Git/PR facts, .gitignore, .rooignore, repository instructions, PostgreSQL, and background jobs. Exclude raw transcripts and secrets by default.

Acceptance criteria

• Authorization and boundary filters run before ranking.
• Scope expansion requires approval.
• Deleted or unauthorized sources are tombstoned immediately.
• Results include immutable source/revision citations.
• Conflicts preserve authority/freshness rather than silently merging.
• Standard PostgreSQL provides a complete lexical mode.

Dependencies

• [Feature]: Add the project planning domain model and authorization foundation #1037
• [Feature]: Create structured project plans from existing Roomote plan artifacts #1039
• [Feature]: Add the project event timeline and audited control commands #1043

[roomote-community]

I've analyzed this issue and here's my proposed implementation plan:

1. Implement this after the project foundation in [Feature]: Add the project planning domain model and authorization foundation #1037 and consume its project membership, repository grants, and configurable permissions directly. Use [Feature]: Create structured project plans from existing Roomote plan artifacts #1039's immutable approved plan versions as an optional authoritative source and [Feature]: Add the project event timeline and audited control commands #1043's command/event ledger for policy approvals and scope changes; do not create parallel project authorization, approval, or audit models in this feature.
2. Add shared knowledge-domain types in packages/types/src/project-knowledge.ts for source kinds, branch/date scope, include/exclude rules, authority, transcript policy, retention/size limits, discovery findings, ingestion state, conflicts, and immutable citations. Recommend a v1 boundary of project-linked repositories plus explicitly approved structured project sources; raw task_messages, task artifacts, credentials, and secret-like content remain denied unless a later policy version explicitly adds a supported source class.
3. Extend packages/db/src/schema.ts with additive tables for immutable knowledge-policy versions and their normalized repository/source rules, policy approvals, source revisions, documents, chunks, ingestion runs, and tombstones. Tie every row to the [Feature]: Add the project planning domain model and authorization foundation #1037 project/repository grant model; identify repository content by repository ID, commit SHA, blob/object ID, path, and line/byte range; identify non-repository content by an immutable owning version. Generate the next Drizzle migration and expose all query/mutation helpers only through @roomote/db/server, preserving the N-1 rollback guarantee.
4. Add a server-only policy service under packages/db/src/lib/project-knowledge/. Draft edits create a new policy version; approved versions are append-only. Before activation, diff the candidate's effective repositories, refs, paths, source classes, dates, retention, and limits against the active version. Any widening must be submitted through [Feature]: Add the project event timeline and audited control commands #1043's authorized approval command and event; narrowing may activate directly and must synchronously tombstone newly excluded rows in the same control-plane transaction.
5. Build deterministic boundary discovery as a proposal generator, not an automatic permission expander. At a pinned commit, combine .gitignore and other ignore files with generated/vendor/binary/size checks, secret scanning, repository instructions, docs, CI/test/build manifests, and symbol inventory. Persist each finding with its detector, reason, source revision, and proposed rule, then require an approved policy version before ingestion. Apply a hard non-overridable secret/credential rejection before content persistence, with raw transcripts excluded by default.
6. Add a dedicated BullMQ ingestion/reconciliation queue following apps/bullmq/src/snapshot-queue.ts and apps/bullmq/src/jobs/snapshot.ts patterns for deterministic job IDs, bounded concurrency, retries, persisted progress, and reconciliation. Keep product policy and database access out of apps/worker; collect content at an exact provider ref through a narrow source-control adapter, re-check the active policy before storing each batch, chunk deterministically, and make (policy version, source revision, document identity, chunk ordinal/content hash) idempotent.
7. Make revocation safe independently of asynchronous indexing. Repository unlink, permission loss, policy narrowing, source deletion, or rewritten refs must first mark affected revisions/documents/chunks tombstonedAt in a short authorized database transaction and append the [Feature]: Add the project event timeline and audited control commands #1043 audit event; only then enqueue physical purge/reindex work. Search and citation reads must always exclude tombstoned or no-longer-authorized rows, while retaining minimal immutable metadata needed for audit according to the approved retention policy.
8. Implement PostgreSQL lexical retrieval in a package-owned query helper using a stored tsvector, a GIN index, and standard websearch_to_tsquery/ts_rank_cd primitives. Structure the query around an eligible-chunks relation that joins the caller's current project grant, project-repository boundary, active policy version, allowed source/revision/date scope, and non-tombstoned state before computing rank. Add deterministic tie-breakers and stable cursor pagination; no vector service is required for this complete lexical mode.
9. Return each result as chunk text/snippet plus an immutable citation containing project/source IDs, policy version, repository identity, commit SHA, blob/object ID, path, and line/byte range (or the equivalent immutable artifact/plan version). Keep competing sources separate: attach authority and observed/source timestamps, rank by explicit policy authority and freshness only after authorization filtering, and return conflict metadata rather than synthesizing or silently merging contradictory text.
10. Expose authorized policy preview/approval, ingestion status, and search commands through a focused apps/web/src/trpc/commands/project-knowledge/ module and the protected router. Add a run-token tool to apps/api/src/handlers/mcp/roomote.ts only after deriving the run's project and repository scope server-side; do not inherit the broader cross-task artifact-read rule in apps/api/src/handlers/artifacts/auth.ts. The tool should accept a query and limits, never caller-supplied grants, and return citation-ready results for agent context.
11. Add real-Postgres tests for immutable versions and citations, project/repository isolation, grant revocation, policy expansion approval, synchronous tombstones, default transcript/secret exclusion, idempotent ingestion, stale-job policy rechecks, authority/freshness conflicts, FTS behavior and pagination, and citations surviving later revisions. Add BullMQ concurrency/retry/reconciliation tests, API/MCP user-token and run-token authorization tests, and migration/rollback validation with the existing package-scoped checks and deploy/ci/upgrade-compatibility.sh.
12. Document policy semantics, supported source classes, default exclusions, citation format, retention/tombstone behavior, and PostgreSQL-only search operation in apps/docs/. A project Knowledge UI can then build on the project shell from the dependency work to show discovered boundary proposals, policy-version diffs requiring approval, ingestion health, and cited search results without making UI state authoritative.

This approach makes the authorization boundary and immutable policy version part of every persisted and queried record, removes revoked content from eligibility synchronously, and uses background jobs only for expensive discovery/indexing and eventual physical cleanup. It also keeps the first complete retrieval mode deployable on Roomote's existing PostgreSQL stack while leaving source adapters and later ranking modes additive.

Please tag @roomote if you'd like me to implement this, or reply with feedback on the plan.