Develop

API.md

@@ -86,18 +86,81 @@ curl http://127.0.0.1:8765/api/v1/search \
 | `GET` | `/api/v1/sources` | List user-approved source folders |
 | `POST` | `/api/v1/sources` | Add a source folder |
 | `DELETE` | `/api/v1/sources/{source_id}` | Remove a source and its indexed memory |
+| `GET` | `/api/v1/ignore-rules` | List protected and user ignore rules |
+| `POST` | `/api/v1/ignore-rules` | Create an ignore rule |
+| `PATCH` | `/api/v1/ignore-rules/{rule_id}` | Update a user ignore rule |
+| `DELETE` | `/api/v1/ignore-rules/{rule_id}` | Remove a user ignore rule |
+| `POST` | `/api/v1/ignore-rules/test` | Explain whether a path is ignored |
 | `POST` | `/api/v1/index/start` | Start or return the active background indexing job |
 | `GET` | `/api/v1/index/status` | Read indexing progress |
 | `POST` | `/api/v1/index/pause` | Request indexing pause |
 | `POST` | `/api/v1/index/resume` | Resume paused indexing |
 | `POST` | `/api/v1/index/stop` | Request indexing stop |
+| `POST` | `/api/v1/watch/start` | Start source-folder synchronization |
+| `GET` | `/api/v1/watch/status` | Read watcher state and recent activity |
+| `POST` | `/api/v1/watch/stop` | Request watcher shutdown |
 | `POST` | `/api/v1/search` | Search indexed local memory |
 | `POST` | `/api/v1/ask` | Return an answer with structured sources |
 | `POST` | `/api/v1/ask/stream` | Stream answer text as server-sent events |
 | `DELETE` | `/api/v1/chat/sessions/{session_id}` | End an in-memory chat session |
 | `GET` | `/api/v1/documents/{file_id}` | Inspect an indexed file and its text chunks |
 | `POST` | `/api/v1/actions/open` | Open a validated indexed file in its default OS app |
 
+## Watch mode
+
+Start the local watcher after sources have been added:
+
+```bash
+curl -X POST http://127.0.0.1:8765/api/v1/watch/start \
+  -H "Authorization: Bearer $TOKEN"
+```
+
+The API and CLI use the same watcher service and shared SQLite state. Starting through either interface creates one detached local worker that remains active after the launching client or terminal exits. Repeated start requests return the current active watcher instead of creating duplicates. Status includes watched source paths, queued jobs, the current file, latest event and indexing timestamps, and recent file errors. Stop requests are cooperative and take effect after the current file operation finishes.
+
+The watcher only observes enabled sources already approved through OpenMind. It does not modify or delete user files; a deleted source file only causes its OpenMind metadata and searchable chunks to be removed.
+
+## Ignore rules
+
+Client applications can manage the same indexing rules used by the CLI, scanner, and watcher. Create a global rule:
+
+```json
+{
+  "type": "extension",
+  "value": ".mp4",
+  "enabled": true,
+  "scope": "global",
+  "source_id": null,
+  "reason": "Video files"
+}
+```
+
+For a source-specific rule, set `scope` to `source` and provide an existing `source_id`. Supported types are `path`, `folder_name`, `file_name`, `extension`, `pattern`, `source_type`, `max_file_size`, and `hidden_files`.
+
+Test a path without indexing it:
+
+```json
+{
+  "path": "/Users/example/Documents/private/receipt.pdf",
+  "source_id": "src_0123456789ab"
+}
+```
+
+The response is explainable:
+
+```json
+{
+  "ignored": true,
+  "matched_rule": {
+    "id": "ign_0123456789abcdef",
+    "type": "path",
+    "value": "/Users/example/Documents/private",
+    "reason": "Private folder"
+  }
+}
+```
+
+System rules are visible but protected from updates and deletion. Adding or enabling a rule immediately removes matching content from LanceDB and marks its file record as skipped; original files are never changed. After disabling or deleting a user rule, start indexing to include newly eligible files.
+
 ## Sources
 
 Add a folder:
@@ -109,7 +172,7 @@ Add a folder:
 }
 ```
 
-OpenMind resolves the path and rejects missing files, non-directory paths, and duplicate sources. Removing a source also removes its file records and searchable chunks from OpenMind, but never deletes the original folder or files. Source removal returns the number of file records and memory chunks removed and is refused while an indexing job is active.
+OpenMind resolves the path and rejects missing files, non-directory paths, and duplicate sources. Removing a source also removes its file records and searchable chunks from OpenMind, but never deletes the original folder or files. Source removal returns the number of file records and memory chunks removed and is refused while indexing or watch mode is active.
 
 ```json
 {

CHANGELOG.md

@@ -4,7 +4,14 @@ User-facing changes for each OpenMind Core release.
 
 ## Unreleased
 
+- No unreleased changes.
+
+## 0.0.7 - 2026-07-22
+
+- Added ignore rules through the CLI and API, with protected defaults, source scopes, path testing, and shared indexing and Watch Mode behavior.
+- Added background watch mode to keep enabled source folders synchronized after files are created, changed, moved, or deleted, with shared CLI and local API controls.
 - Each chat message now retrieves fresh context for its own question, including when the topic changes during a session.
+- Source code, HTML, and JSON are no longer available through the indexing pipeline; existing indexed memory for those formats is removed without touching the original files.
 
 ## 0.0.6 - 2026-07-19
 

FEATURES.md

@@ -61,17 +61,26 @@ Current boundaries:
 - User-approved folders only.
 - Re-adding an existing source reports that it is already registered.
 - Recursive folder scanning.
-- Ignores noisy folders:
-  - `.git`
-  - `node_modules`
-  - `venv`
-  - `.venv`
-  - `.env`
-  - `__pycache__`
-  - `dist`
-  - `build`
-  - `.cache`
-  - hidden folders
+
+### Ignore Rules
+
+- First-class SQLite-backed rules shared by indexing and Watch Mode.
+- CLI and authenticated API management use the same engine methods.
+- Global and source-specific scopes.
+- Rule types:
+  - exact path
+  - folder name
+  - file name
+  - extension
+  - glob pattern
+  - source type
+  - maximum file size
+  - hidden files and folders
+- Explainable path testing reports the matching rule and reason.
+- Protected system rules cover dependencies, generated output, caches, temporary downloads, local databases, environment files, and private keys.
+- System rules are visible and cannot be disabled or removed.
+- Adding or enabling a rule immediately removes matching searchable chunks while preserving original files.
+- Disabling or removing a user rule makes matching files eligible for the next index run.
 
 ### File Support
 
@@ -82,7 +91,6 @@ Supported indexed formats:
 - `.pdf`
 - `.docx`
 - `.csv`
-- `.html`
 - `.png`
 - `.jpg`
 - `.jpeg`
@@ -91,7 +99,9 @@ Supported indexed formats:
 - `.tif`
 - `.tiff`
 
-OpenMind is document-first by default. Source code, JSON config, package metadata, app asset catalogs, and other low-level project internals are not indexed unless a future opt-in mode is added.
+OpenMind is document-first. Source code, HTML, JSON config, package metadata, app asset catalogs, and other low-level project internals are not supported indexed formats. Markdown remains the supported format for high-level project documentation.
+
+OpenMind removes stale indexed memory for unsupported formats during startup without deleting or modifying the original files.
 
 Image files are indexed by generating text descriptions through a local vision model endpoint and embedding that text like any other document chunk.
 
@@ -103,7 +113,6 @@ Image files are indexed by generating text descriptions through a local vision m
 - Automatic scanned-PDF OCR fallback with local RapidOCR + ONNX Runtime.
 - DOCX extraction with `python-docx`.
 - CSV extraction with `pandas`.
-- HTML extraction with BeautifulSoup.
 - Image description extraction through a local LM Studio vision model.
 - Optional image OCR text extraction with RapidOCR.
 - Searchable image metadata extraction for dimensions, format, EXIF, and safe image info fields.
@@ -218,6 +227,23 @@ Image files are indexed by generating text descriptions through a local vision m
 - Pause/stop take effect after the current file finishes.
 - Progress is capped at `100.0%`.
 
+### Watch Mode
+
+- `openmind watch`
+- `openmind watch status`
+- `openmind watch stop`
+- Runs as a detached local worker until explicitly stopped.
+- CLI and API controls share the same worker lifecycle and SQLite state.
+- Watches enabled, user-approved source folders only.
+- Runs a lightweight catch-up scan before listening for live changes.
+- Uses the cross-platform watchdog backend.
+- Debounces repeated filesystem events and waits for files to stabilize.
+- Queues create, re-index, and delete work in SQLite.
+- Treats a move as deletion of the old path followed by indexing the new path.
+- Removes deleted-file metadata from SQLite and chunks from LanceDB.
+- Records per-file failures and continues processing later changes.
+- Exposes authenticated start, status, and stop controls through the local API.
+
 ### Developer Logs
 
 - Structured OpenMind logs:
@@ -228,6 +254,7 @@ Image files are indexed by generating text descriptions through a local vision m
   - `openmind dev logs`
   - `openmind dev logs --no-follow --lines 40`
   - `openmind dev logs --log all`
+  - `openmind dev logs --log watch`
   - `openmind dev logs --lm-studio`
 - LM Studio log mode runs:
   - `lms log stream`
@@ -245,6 +272,7 @@ Image files are indexed by generating text descriptions through a local vision m
 - Status and model-provider inspection.
 - Model listing, validated selection, and loading.
 - Source listing, creation, and removal.
+- Ignore-rule listing, creation, update, removal, and path testing.
 - Start, status, pause, resume, and stop indexing operations.
 - Search responses with paths, snippets, scores, metadata, and stable file IDs.
 - Source-grounded synchronous Ask responses.
@@ -258,16 +286,15 @@ Image files are indexed by generating text descriptions through a local vision m
 ### Test Data
 
 - `data/` folder with local indexing fixtures.
-- Includes text, Markdown, JSON, CSV, HTML, JavaScript, PDF, PNG, and JPEG samples.
-- Supported document-first formats, PDFs, PNGs, and JPEGs can be indexed.
+- Includes supported document and image samples plus source/config fixtures that verify excluded formats stay unindexed.
 
 ## Known Limits
 
 - Pause and stop cannot interrupt a file already inside extraction or an LM Studio embedding request.
 - Image indexing requires a local vision model served through LM Studio.
 - Default scanned-PDF OCR can be installed through `uv`; optional OCRmyPDF mode still needs local OCRmyPDF, Tesseract, and Ghostscript.
 - No persistent chat history yet.
-- No file watcher yet.
+- Watch mode survives terminal closure but must be restarted after a machine reboot or process termination.
 - No ranking tuning beyond vector search.
 - No hybrid keyword/vector search yet.
 - No UI.
@@ -323,7 +350,6 @@ Image files are indexed by generating text descriptions through a local vision m
 ### Local Service Extensions
 
 - Background worker process management.
-- File watcher for incremental indexing.
 - Optional event stream for indexing progress.
 
 ### Future Providers