Rag pipeline completed

backend/README.md

@@ -0,0 +1,143 @@
+# RAG Pipeline Backend
+
+A Python backend for Retrieval-Augmented Generation (RAG) using Docling for document processing and LlamaIndex for the RAG pipeline.
+
+## Features
+
+- **Document Processing**: Uses Docling for superior document parsing and chunking
+- **RAG Pipeline**: LlamaIndex for efficient indexing and querying
+- **Vector Storage**: ChromaDB for persistent vector storage
+- **API**: FastAPI with async support
+- **Multiple Formats**: Supports PDF, DOCX, TXT, MD files
+
+## Quick Start
+
+### 1. Install Dependencies
+
+```bash
+cd backend
+pip install -r requirements.txt
+```
+
+### 2. Set up Ollama (Required for LLM)
+
+Install and start Ollama:
+```bash
+# Install Ollama (see https://ollama.ai for installation instructions)
+# Then pull a model (e.g., llama2)
+ollama pull llama2
+```
+
+### 3. Configure Environment
+
+Create a `.env` file:
+```bash
+# Copy and modify the example
+cp .env.example .env
+```
+
+Key configuration options:
+- `LLM_BASE_URL`: Ollama server URL (default: http://localhost:11434)
+- `DEFAULT_MODEL`: Ollama model name (default: llama2)
+- `EMBEDDING_MODEL`: HuggingFace embedding model
+- `CHUNK_SIZE`: Document chunk size for processing
+
+### 4. Start the Server
+
+```bash
+python main.py
+```
+
+The API will be available at `http://localhost:8000`
+
+## API Endpoints
+
+### Upload Documents
+```bash
+POST /api/v1/upload
+```
+Upload documents for processing and indexing.
+
+### Query Documents
+```bash
+POST /api/v1/query
+```
+Query the RAG system with a question.
+
+### Get Status
+```bash
+GET /api/v1/status
+```
+Get system status and index statistics.
+
+## Integration with Frontend
+
+The backend is designed to work with your existing Electron app. Update your frontend to use these endpoints:
+
+```typescript
+// Example API calls
+const uploadDocuments = async (files: FileList) => {
+  const formData = new FormData();
+  Array.from(files).forEach(file => formData.append('files', file));
+
+  const response = await fetch('http://localhost:8000/api/v1/upload', {
+    method: 'POST',
+    body: formData
+  });
+
+  return response.json();
+};
+
+const queryDocuments = async (question: string) => {
+  const response = await fetch('http://localhost:8000/api/v1/query', {
+    method: 'POST',
+    headers: { 'Content-Type': 'application/json' },
+    body: JSON.stringify({ question })
+  });
+
+  return response.json();
+};
+```
+
+## Why Docling?
+
+Docling excels at:
+- **Intelligent Chunking**: Preserves document structure and context
+- **Format Support**: Handles complex PDFs, DOCX with formatting
+- **Metadata Extraction**: Preserves important document metadata
+- **AI-Optimized**: Designed specifically for AI/ML workflows
+
+## Architecture
+
+```
+Frontend (Electron) → FastAPI Backend → Docling (Processing) → LlamaIndex (RAG) → Vector Store (Chroma)
+                                     ↓
+                                 Ollama (LLM)
+```
+
+## Troubleshooting
+
+### Common Issues
+
+1. **Ollama Connection Error**
+   - Ensure Ollama is running: `ollama serve`
+   - Check the base URL in configuration
+
+2. **Model Not Found**
+   - Pull the model: `ollama pull llama2`
+   - Verify model name in configuration
+
+3. **Memory Issues**
+   - Reduce chunk size and batch size
+   - Use smaller embedding models
+
+4. **File Processing Errors**
+   - Check file format support
+   - Verify file size limits
+
+## Performance Optimization
+
+- Use GPU for embeddings if available
+- Adjust chunk size based on your documents
+- Consider using smaller, faster models for development
+- Use local vector stores for better performance

backend/config.py

@@ -0,0 +1,62 @@
+"""Configuration settings for the RAG backend."""
+
+import os
+from pathlib import Path
+from typing import Optional
+from pydantic_settings import BaseSettings
+
+class Settings(BaseSettings):
+    # API Configuration
+    HOST: str = "localhost"
+    PORT: int = 8000
+    API_PREFIX: str = "/api/v1"
+
+    # Storage paths
+    BASE_DIR: Path = Path(__file__).parent
+    UPLOAD_DIR: Path = BASE_DIR / "uploads"
+    INDEX_DIR: Path = BASE_DIR / "indexes"
+    TEMP_DIR: Path = BASE_DIR / "temp"
+
+    # Document processing
+    MAX_FILE_SIZE: int = 50 * 1024 * 1024  # 50MB
+    SUPPORTED_FORMATS: list = [".pdf", ".docx", ".txt", ".md"]
+
+    # Chunking configuration
+    CHUNK_SIZE: int = 500  # Reduced from 1000 to prevent long context
+    CHUNK_OVERLAP: int = 50  # Reduced from 200 proportionally
+
+    # Embedding configuration
+    EMBEDDING_MODEL: str = "BAAI/bge-small-en-v1.5"
+    EMBEDDING_DEVICE: str = "cpu"
+
+    # LLM configuration
+    LLM_BASE_URL: str = "http://localhost:11434"  # Ollama default
+    DEFAULT_MODEL: str = "qwen2.5:3b-instruct-q4_K_M"  # Smaller default model to fit in typical RAM; frontend can override
+    ALLOW_MODEL_OVERRIDE: bool = True  # Allow frontend to specify model per request
+    OLLAMA_DEBUG: Optional[int] = None
+
+    # Vector store configuration
+    VECTOR_STORE_TYPE: str = "chroma"  # Options: chroma, faiss
+    SIMILARITY_TOP_K: int = 2  # Reduced from 5 to 2 to prevent CUDA OOM errors
+
+    # CORS settings
+    ALLOWED_ORIGINS: list = [
+        "http://localhost:3000",
+        "http://localhost:5173",
+        "http://127.0.0.1:3000",
+        "http://127.0.0.1:5173",
+        "*"
+    ]
+
+    class Config:
+        env_file = ".env"
+        case_sensitive = True
+
+    def __init__(self, **kwargs):
+        super().__init__(**kwargs)
+        # Create directories if they don't exist
+        for directory in [self.UPLOAD_DIR, self.INDEX_DIR, self.TEMP_DIR]:
+            directory.mkdir(parents=True, exist_ok=True)
+
+# Global settings instance
+settings = Settings()

backend/document_processor.py

@@ -0,0 +1,134 @@
+"""Document processing using Docling for superior chunking and extraction."""
+
+import logging
+from pathlib import Path
+from typing import List, Dict, Any
+
+# Optional heavy dependencies – we fall back to dummy objects so static analysis
+# doesn't fail on machines where they are not installed (e.g. CI without GPU).
+try:
+    from docling.document_converter import DocumentConverter  # type: ignore
+    from docling.datamodel.document import DoclingDocument  # type: ignore
+except ModuleNotFoundError:  # pragma: no cover
+    DocumentConverter = DoclingDocument = object  # type: ignore
+
+try:
+    from llama_index.core.schema import Document as LlamaDocument  # type: ignore
+    from llama_index.core.node_parser import SentenceSplitter  # type: ignore
+except ModuleNotFoundError:  # pragma: no cover
+    LlamaDocument = SentenceSplitter = object  # type: ignore
+
+from config import settings
+
+logger = logging.getLogger(__name__)
+
+class DocumentProcessor:
+    """Handles document processing with Docling and chunking for RAG."""
+
+    def __init__(self):
+        self.converter = DocumentConverter()
+        self.sentence_splitter = SentenceSplitter(
+            chunk_size=settings.CHUNK_SIZE,
+            chunk_overlap=settings.CHUNK_OVERLAP,
+            separator=" "
+        )
+
+    async def process_document(self, file_path: Path) -> Dict[str, Any]:
+        """Process a document using Docling and return structured data."""
+        try:
+            logger.info("Processing document: %s", file_path)
+
+            # Use Docling to process the document
+            result = self.converter.convert(str(file_path))
+
+            # Extract text from the converted document
+            doc = result.document
+            text_content = doc.export_to_markdown() if hasattr(doc, 'export_to_markdown') else ""
+
+            # Get document metadata
+            metadata = {
+                "filename": file_path.name,
+                "file_path": str(file_path),
+                "file_size": file_path.stat().st_size,
+                "file_type": file_path.suffix.lower(),
+            }
+
+            # Add Docling-specific metadata if available
+            if hasattr(result, 'pages'):
+                metadata['num_pages'] = len(result.pages) if result.pages else 0
+
+            logger.info("Successfully processed document: %s", file_path.name)
+
+            return {
+                "text": text_content,
+                "metadata": metadata,
+                "success": True
+            }
+
+        except Exception as e:
+            logger.error("Error processing document %s: %s", file_path, str(e))
+            return {
+                "text": "",
+                "metadata": {"filename": file_path.name, "error": str(e)},
+                "success": False
+            }
+
+    async def chunk_document(self, text: str, metadata: Dict[str, Any]) -> List[LlamaDocument]:
+        """Chunk processed document text into LlamaIndex documents."""
+        try:
+            if not text.strip():
+                logger.warning("Empty text provided for chunking")
+                return []
+
+            document = LlamaDocument(text=text, metadata=metadata)
+
+            nodes = self.sentence_splitter.get_nodes_from_documents([document])
+
+            chunked_docs = []
+            for i, node in enumerate(nodes):
+                chunk_metadata = metadata.copy()
+                chunk_metadata.update({
+                    "chunk_id": i,
+                    "chunk_size": len(node.text),
+                    "source_document": metadata.get("filename", "unknown")
+                })
+
+                chunked_docs.append(LlamaDocument(
+                    text=node.text,
+                    metadata=chunk_metadata
+                ))
+
+            logger.info("Created %d chunks from document", len(chunked_docs))
+            return chunked_docs
+
+        except Exception as e:
+            logger.error("Error chunking document: %s", str(e))
+            return []
+
+    async def process_and_chunk(self, file_path: Path) -> List[LlamaDocument]:
+        """Process and chunk a document in one step."""
+        # Process the document
+        result = await self.process_document(file_path)
+
+        if not result["success"]:
+            logger.error(f"Failed to process document: {file_path}")
+            return []
+
+        # Chunk the processed text
+        return await self.chunk_document(result["text"], result["metadata"])
+
+    def is_supported_format(self, file_path: Path) -> bool:
+        """Check if the file format is supported."""
+        return file_path.suffix.lower() in settings.SUPPORTED_FORMATS
+
+    async def process_directory(self, directory: Path) -> List[LlamaDocument]:
+        """Process all supported documents in a directory."""
+        all_documents = []
+
+        for file_path in directory.rglob("*"):
+            if file_path.is_file() and self.is_supported_format(file_path):
+                documents = await self.process_and_chunk(file_path)
+                all_documents.extend(documents)
+
+        logger.info(f"Processed {len(all_documents)} total chunks from directory: {directory}")
+        return all_documents

backend/indexes/0e2a8c52-ab38-467e-8363-fe9c3ef702b3/length.bin

@@ -0,0 +1,6 @@
+, if it
+                exists, and that will be located in `pretrained_model_name_or_path/speaker_embeddings_directory`.
+            speaker_embeddings_directory (`str`, *optional*, defaults to `"speaker_embeddings/"`):
+                The name of the folder in which the speaker_embeddings arrays will be saved.
+            push_to_hub (`bool`, *optional*, defaults to `False`):
+                Whe