Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion REFERENCES.md
Original file line number Diff line number Diff line change
Expand Up @@ -94,7 +94,7 @@ This recursive approach demonstrates the power of RAG (Retrieval-Augmented Gener
- **Structured Error Handling** - Consistent error propagation and handling

**Files using these concepts:**
- `src/domain/exceptions.ts` - Exception hierarchy implementation
- `src/domain/exceptions/` - Modular exception hierarchy (base `ConceptRAGError` with domain-specific subtypes)

## Testing Patterns

Expand Down
2 changes: 1 addition & 1 deletion USAGE.md
Original file line number Diff line number Diff line change
Expand Up @@ -533,7 +533,7 @@ Default: `~/.concept_rag`

Custom:
```bash
export CONCEPT_RAG_DB="/custom/path"
export DATABASE_URL="/custom/path"
```

### MCP Server Configuration
Expand Down
6 changes: 4 additions & 2 deletions docs/architecture/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ This section covers the technical architecture of Concept-RAG.
| `src/tools/` | MCP tool implementations (10 tools) |
| `src/wordnet/` | WordNet integration and strategies |
| `docs/` | MkDocs documentation site |
| `docs/architecture/` | Architecture Decision Records |
| `docs/architecture/` | Architecture deep-dives and decision records (ADRs) |
| `scripts/` | Maintenance and diagnostic utilities |
| `prompts/` | LLM prompt templates |

Expand All @@ -26,4 +26,6 @@ This section covers the technical architecture of Concept-RAG.
- **[Database Schema](../database-schema.md)** - LanceDB table structures
- **[Stage Cache](../stage-cache-structure.md)** - Intermediate processing cache

For architectural decisions and their rationale, see the **[ADRs](../architecture/adr0001-typescript-nodejs-runtime.md)** section.
Architectural decisions and their rationale are recorded as Architecture Decision
Records (ADRs) in the repository under `docs/architecture/`. ADRs are point-in-time
engineering artifacts and are not published to this site.
17 changes: 10 additions & 7 deletions docs/architecture/bm25-keywords.md
Original file line number Diff line number Diff line change
Expand Up @@ -114,15 +114,16 @@ flowchart TB

## Integration in Hybrid Search

BM25 is one of four signals in the hybrid ranking system:
BM25 is one of five signals in the hybrid ranking system:

```mermaid
flowchart TB
subgraph Signals["Search Signals"]
Vector["Vector Similarity<br/>35%"]
BM25["BM25 Keywords<br/>30-35%"]
Concept["Concept Matching<br/>15-20%"]
WN["WordNet Expansion<br/>15%"]
Vector["Vector Similarity<br/>30-35%"]
BM25["BM25 Keywords<br/>20-30%"]
Title["Title / Name Match<br/>0-40%"]
Concept["Concept Matching<br/>0-20%"]
WN["WordNet Expansion<br/>10-15%"]
end

subgraph Combined["Hybrid Score"]
Expand All @@ -132,6 +133,7 @@ flowchart TB

Vector --> Combine
BM25 --> Combine
Title --> Combine
Concept --> Combine
WN --> Combine
Combine --> Results
Expand Down Expand Up @@ -249,7 +251,8 @@ The hybrid approach preserves these benefits while adding keyword precision.

## Related Documentation

- [ADR-0006: Hybrid Search Strategy](adr0006-hybrid-search-strategy.md) — Design decision
- [WordNet Enrichment](wordnet-enrichment.md) — Synonym expansion
- [ADR-0010: Query Expansion](adr0010-query-expansion.md) — Term weighting

The hybrid-search and query-expansion design decisions are recorded in ADRs
(ADR-0006, ADR-0010), kept in the repository under `docs/architecture/`.

5 changes: 3 additions & 2 deletions docs/architecture/seeding-architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -135,6 +135,7 @@ flowchart TB

- [Getting Started](../getting-started.md) — Quick start guide with seeding commands
- [Stage Cache](../stage-cache-structure.md) — Intermediate caching during seeding
- [ADR-0013: Incremental Seeding](adr0013-incremental-seeding.md) — Design decision for incremental processing
- [ADR-0044: Seeding Modularization](adr0044-seeding-script-modularization.md) — Script architecture

The incremental-seeding and seeding-modularization design decisions are recorded
in ADRs (ADR-0013, ADR-0044), kept in the repository under `docs/architecture/`.

18 changes: 10 additions & 8 deletions docs/architecture/wordnet-enrichment.md
Original file line number Diff line number Diff line change
Expand Up @@ -83,15 +83,16 @@ Expanded terms receive lower weights than original terms to maintain search prec

### Integration in Hybrid Search

WordNet contributes to the 4-signal hybrid search ranking:
WordNet contributes to the 5-signal hybrid search ranking:

```mermaid
flowchart TB
subgraph Signals["Search Signals"]
Vector["Vector Similarity<br/>35%"]
BM25["BM25 Keywords<br/>35%"]
Concept["Concept Matching<br/>15%"]
WN["WordNet Expansion<br/>15%"]
Vector["Vector Similarity<br/>30-35%"]
BM25["BM25 Keywords<br/>20-30%"]
Title["Title / Name Match<br/>0-40%"]
Concept["Concept Matching<br/>0-20%"]
WN["WordNet Expansion<br/>10-15%"]
end

subgraph Score["Final Ranking"]
Expand All @@ -101,6 +102,7 @@ flowchart TB

Vector --> Combine
BM25 --> Combine
Title --> Combine
Concept --> Combine
WN --> Combine
Combine --> Results
Expand Down Expand Up @@ -223,7 +225,7 @@ python3 -c "from nltk.corpus import wordnet as wn; print(f'✅ WordNet: {len(lis

## Related Documentation

- [ADR-0008: WordNet Integration](adr0008-wordnet-integration.md) — Design decision
- [ADR-0010: Query Expansion](adr0010-query-expansion.md) — Query expansion strategy
- [ADR-0006: Hybrid Search](adr0006-hybrid-search-strategy.md) — Multi-signal ranking
The WordNet-integration, query-expansion, and hybrid-search design decisions are
recorded in ADRs (ADR-0008, ADR-0010, ADR-0006), kept in the repository under
`docs/architecture/`.

7 changes: 3 additions & 4 deletions docs/database-schema.md
Original file line number Diff line number Diff line change
Expand Up @@ -577,12 +577,11 @@ These performance caches remain for CPU/DB optimization:

## Related Documentation

- [ADR-0002: LanceDB Vector Storage](architecture/adr0002-lancedb-vector-storage.md)
- [ADR-0009: Three-Table Architecture](architecture/adr0009-three-table-architecture.md)
- [ADR-0027: Hash-Based Integer IDs](architecture/adr0027-hash-based-integer-ids.md)
- [ADR-0043: Schema Normalization](architecture/adr0043-schema-normalization.md)
- Domain Models: `src/domain/models/`
- Schema Validators: `src/infrastructure/lancedb/utils/schema-validators.ts`

Schema design decisions are recorded in ADRs (ADR-0002, ADR-0009, ADR-0027,
ADR-0043), kept in the repository under `docs/architecture/`.
- Lexical Linking: `scripts/link_related_concepts.ts`
- Derived Fields Regeneration: `scripts/rebuild_derived_names.ts`
- Seeding Script: `hybrid_fast_seed.ts` (supports `--parallel N` for parallel concept extraction)
6 changes: 3 additions & 3 deletions docs/development.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ This fork extends the original [lance-mcp](https://github.com/adiom-data/lance-m
- 📚 **Formal concept model** - Rigorous definition ensuring semantic matching and disambiguation
- 🧠 **Enhanced concept extraction** - 80-150+ concepts per document (Claude Sonnet 4.5)
- 🌐 **WordNet semantic enrichment** - Synonym expansion and hierarchical navigation
- 🔍 **Multi-signal hybrid ranking** - Vector + BM25 + title + concept + WordNet (4-signal scoring)
- 🔍 **Multi-signal hybrid ranking** - Vector + BM25 + title + concept + WordNet (5-signal scoring)
- 📖 **Large document support** - Multi-pass extraction for >100k token documents
- ⚡ **Parallel concept extraction** - Process up to 25 documents concurrently with shared rate limiting
- 🔁 **Resumable seeding** - Checkpoint-based recovery from interrupted runs
Expand All @@ -31,7 +31,7 @@ src/
├── infrastructure/ # External integrations
│ ├── lancedb/ # Database adapters (normalized schema v7)
│ ├── embeddings/ # Embedding service
│ ├── search/ # Hybrid search with 4-signal scoring
│ ├── search/ # Hybrid search with 5-signal scoring
│ ├── resilience/ # Circuit breaker, bulkhead, timeout patterns
│ ├── checkpoint/ # Resumable seeding with progress tracking
│ ├── cli/ # Progress bar display utilities
Expand Down Expand Up @@ -67,7 +67,7 @@ Catalog Chunks Concepts Categories
└─────────┴─────────┴─────────┘
Hybrid Search Engine
(Vector + BM25 + Concepts + WordNet)
(Vector + BM25 + Title + Concepts + WordNet)
```

### Four-Table Normalized Schema
Expand Down
23 changes: 15 additions & 8 deletions docs/how-it-works.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ flowchart TB
subgraph Search["🔍 Hybrid Search"]
Vector[Vector Similarity]
BM25[BM25 Keywords]
Title[Title / Name Match]
ConceptMatch[Concept Matching]
WordNet[WordNet Expansion]
Rank[Weighted Ranking]
Expand All @@ -52,11 +53,13 @@ flowchart TB
Extract --> Categories

Catalog --> Vector
Catalog --> Title
Chunks --> Vector
Concepts --> ConceptMatch

Vector --> Rank
BM25 --> Rank
Title --> Rank
ConceptMatch --> Rank
WordNet --> Rank

Expand Down Expand Up @@ -98,16 +101,20 @@ Vector embeddings are generated for:

### 5. Hybrid Search

Queries are scored using four signals:
Queries are scored by combining five signals:

| Signal | Weight | Purpose |
|--------|--------|---------|
| Vector Similarity | 35% | Semantic meaning match |
| BM25 Keywords | 35% | Exact term matching |
| Concept Matching | 15% | Extracted concept overlap |
| WordNet Expansion | 15% | Synonym and hypernym matching |
| Signal | Purpose |
|--------|---------|
| Vector Similarity | Semantic meaning match |
| BM25 Keywords | Exact term matching |
| Title / Name Match | Document title or concept-name relevance |
| Concept Matching | Extracted concept overlap |
| WordNet Expansion | Synonym and hypernym matching |

Results are combined using weighted ranking for optimal retrieval accuracy.
Signal weights are tuned per search type — for example, document (catalog) search
weights them 30% / 25% / 20% / 15% / 10%, while passage and concept searches use
different profiles. See the [API Reference](api-reference.md#scoring-weights) for the
full per-tool weighting table.

### 6. Gap Detection (Elbow Method)

Expand Down
4 changes: 3 additions & 1 deletion docs/stage-cache-structure.md
Original file line number Diff line number Diff line change
Expand Up @@ -262,6 +262,8 @@ This ensures cache files are never partially written, even if the process is kil

## Related Documentation

- [ADR-0048: Stage Caching](architecture/adr0048-stage-caching.md) - Architecture decision record
- [Database Schema](database-schema.md) - LanceDB table structures

The stage-caching design decision is recorded in ADR-0048, kept in the repository
under `docs/architecture/`.

65 changes: 9 additions & 56 deletions mkdocs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -105,67 +105,20 @@ nav:
- Seeding Architecture: architecture/seeding-architecture.md
- BM25 Keywords: architecture/bm25-keywords.md
- WordNet Enrichment: architecture/wordnet-enrichment.md
- Activity & Skill Interface: architecture/skills-interface.md
- Database Schema: database-schema.md
- Stage Cache: stage-cache-structure.md

- ADRs:
- "ADR-0001: TypeScript Runtime": architecture/adr0001-typescript-nodejs-runtime.md
- "ADR-0002: LanceDB Storage": architecture/adr0002-lancedb-vector-storage.md
- "ADR-0003: MCP Protocol": architecture/adr0003-mcp-protocol.md
- "ADR-0004: RAG Architecture": architecture/adr0004-rag-architecture.md
- "ADR-0005: PDF Processing": architecture/adr0005-pdf-document-processing.md
- "ADR-0006: Hybrid Search": architecture/adr0006-hybrid-search-strategy.md
- "ADR-0007: Concept Extraction": architecture/adr0007-concept-extraction-llm.md
- "ADR-0008: WordNet Integration": architecture/adr0008-wordnet-integration.md
- "ADR-0009: Three-Table Architecture": architecture/adr0009-three-table-architecture.md
- "ADR-0010: Query Expansion": architecture/adr0010-query-expansion.md
- "ADR-0011: Multi-Model Strategy": architecture/adr0011-multi-model-strategy.md
- "ADR-0012: OCR Fallback": architecture/adr0012-ocr-fallback-tesseract.md
- "ADR-0013: Incremental Seeding": architecture/adr0013-incremental-seeding.md
- "ADR-0014: Multi-Pass Extraction": architecture/adr0014-multi-pass-extraction.md
- "ADR-0015: Formal Concept Model": architecture/adr0015-formal-concept-model.md
- "ADR-0016: Layered Architecture": architecture/adr0016-layered-architecture-refactoring.md
- "ADR-0017: Repository Pattern": architecture/adr0017-repository-pattern.md
- "ADR-0018: Dependency Injection": architecture/adr0018-dependency-injection-container.md
- "ADR-0019: Vitest Framework": architecture/adr0019-vitest-testing-framework.md
- "ADR-0020: TypeScript Strict Mode": architecture/adr0020-typescript-strict-mode.md
- "ADR-0021: Performance Optimization": architecture/adr0021-performance-optimization-vector-search.md
- "ADR-0022: HybridSearchService": architecture/adr0022-hybrid-search-service-extraction.md
- "ADR-0023: SQL Injection Prevention": architecture/adr0023-sql-injection-prevention.md
- "ADR-0024: Multi-Provider Embeddings": architecture/adr0024-multi-provider-embeddings.md
- "ADR-0025: Document Loader Factory": architecture/adr0025-document-loader-factory.md
- "ADR-0026: EPUB Support": architecture/adr0026-epub-format-support.md
- "ADR-0027: Hash-Based IDs": architecture/adr0027-hash-based-integer-ids.md
- "ADR-0028: Category Storage": architecture/adr0028-category-storage-strategy.md
- "ADR-0029: Category Search Tools": architecture/adr0029-category-search-tools.md
- "ADR-0030: Auto-Extracted Categories": architecture/adr0030-auto-extracted-categories.md
- "ADR-0031: Eight Tools Strategy": architecture/adr0031-eight-specialized-tools-strategy.md
- "ADR-0032: Tool Selection Guide": architecture/adr0032-tool-selection-guide.md
- "ADR-0033: BaseTool Abstraction": architecture/adr0033-basetool-abstraction.md
- "ADR-0034: Error Handling": architecture/adr0034-comprehensive-error-handling.md
- "ADR-0035: Test Suite Expansion": architecture/adr0035-test-suite-expansion.md
- "ADR-0036: Configuration Centralization": architecture/adr0036-configuration-centralization.md
- "ADR-0037: Functional Validation": architecture/adr0037-functional-validation-layer.md
- "ADR-0038: Dependency Rules": architecture/adr0038-dependency-rules-enforcement.md
- "ADR-0039: Observability": architecture/adr0039-observability-infrastructure.md
- "ADR-0040: Result/Option Types": architecture/adr0040-result-option-types.md
- "ADR-0041: Advanced Caching": architecture/adr0041-advanced-caching.md
- "ADR-0042: System Resilience": architecture/adr0042-system-resilience-patterns.md
- "ADR-0043: Schema Normalization": architecture/adr0043-schema-normalization.md
- "ADR-0044: Seeding Modularization": architecture/adr0044-seeding-script-modularization.md
- "ADR-0045: API Key Preflight": architecture/adr0045-api-key-preflight-check.md
- "ADR-0046: Document Classification": architecture/adr0046-document-type-classification.md
- "ADR-0047: Architecture Review": architecture/adr0047-architecture-review-dec-2025.md
- "ADR-0048: Stage Caching": architecture/adr0048-stage-caching.md
- "ADR-0049: Incremental Summaries": architecture/adr0049-incremental-category-summaries.md
- "ADR-0050: MkDocs Site": architecture/adr0050-mkdocs-material-documentation-site.md
- "ADR-0051: API Documentation": architecture/adr0051-api-documentation-consolidation.md
- "ADR-0052: Docs Restructure": architecture/adr0052-documentation-site-restructure.md
- "ADR-0053: Meta Content Detection": architecture/adr0053-meta-content-detection.md
- "ADR-0054: Debug Flag Deprecation": architecture/adr0054-deprecate-debug-input-schema.md

- FAQ: faq.md

# Engineering artifacts kept in the repo but excluded from the published site.
# ADRs are point-in-time decision records, not user documentation. development.md
# is the README-linked developer guide and uses repo-relative links that only
# resolve when browsed on GitHub, not on the rendered site.
exclude_docs: |
architecture/adr*.md
development.md

# Plugins
plugins:
- search
Loading