🚀 ScanX Expense

AI-powered Intelligent Document Processing (IDP) System for Bills & Invoices

Extract structured information from bills, automatically categorize expenses, store them in a database, and interact with your expenses using natural language.

⭐ Overview

Managing receipts manually is time-consuming.

ScanX Expense automates the complete workflow by combining:

Computer Vision
OCR
Local LLMs
SQLite
Streamlit

The system converts unstructured invoice images into structured records and allows users to chat with their expense database.

🎯 Project Goals

✔ Process invoices automatically

✔ Extract structured data

✔ Categorize expenses

✔ Store in database

✔ Query using Natural Language

✔ Run completely offline using Ollama

✨ Features

📷 Image Processing

Image cleaning
Noise removal
Grayscale conversion
Thresholding
Batch processing

🔍 OCR

Reads receipts
Supports multiple invoices
Batch OCR
Generates raw text

🤖 AI Parsing

Local LLM extracts

Invoice Number
Vendor
Customer
Date
Amount
Total
Line Items

Returns structured JSON.

🧠 AI Categorization

Each line item is classified into

Food
Grocery
Travel
Logistic
Utilities
Drinks
Other

💾 Database

Stores

invoice
line items
totals
categories

using SQLite.

📊 Dashboard

Interactive Streamlit dashboard

Supports

Search
Filters
Invoice lookup
Category lookup

💬 Chat with Bills

Ask questions like

How much did I spend on food?

Show invoices from Amazon.

Highest spending category.

The LLM generates SQL automatically.

🏗 System Architecture

               Bill Images
                    │
                    ▼
         Image Preprocessing
              (OpenCV)
                    │
                    ▼
              OCR Engine
          (Tesseract OCR)
                    │
                    ▼
          Raw OCR Text Output
                    │
                    ▼
        ┌─────────────────────┐
        │ Agent 1 (Ollama)    │
        │ Invoice Extraction  │
        └─────────────────────┘
                    │
                    ▼
        Structured Invoice JSON
                    │
                    ▼
        ┌─────────────────────┐
        │ Agent 2 (Ollama)    │
        │ Expense Category    │
        └─────────────────────┘
                    │
                    ▼
             SQLite Database
                    │
                    ▼
        ┌─────────────────────┐
        │ Streamlit Dashboard │
        └─────────────────────┘
                    │
                    ▼
        ┌─────────────────────┐
        │ Agent 3 (Ollama)    │
        │ NL → SQL            │
        └─────────────────────┘
                    │
                    ▼
             Chat with Bills

📂 Project Structure

ScanX_Expense/

│
├── bill_images/
│
├── image_cleaning.py
├── ocr_processor.py
├── parser.py
├── data_insertion.py
├── table_creation.py
│
├── ollama1.py
├── ollama2.py
├── ollama3.py
│
├── prompt1.py
├── prompt2.py
├── prompt3.py
│
├── frontend1.py
├── frontend2.py
│
├── extracted_text.txt
├── ocr_master.db
│
└── README.md

⚙ Tech Stack

Layer	Technology
Language	Python
Image Processing	OpenCV
OCR	Tesseract OCR
LLM	Ollama
Model	Gemma 3 / Phi-3
Database	SQLite
Dashboard	Streamlit
Data	Pandas

🚀 Installation

Clone

git clone https://github.com/<your-username>/ScanX_Expense.git

cd ScanX_Expense

Install Dependencies

pip install -r requirements.txt

Install Tesseract

Download

https://github.com/UB-Mannheim/tesseract/wiki

Update

pytesseract.tesseract_cmd = r"YOUR_PATH"

Install Ollama

ollama pull gemma3:4b

or

ollama pull phi3:3.8b

▶ Running the Project

Step 1

Put receipts inside

bill_images/

Step 2

Image Cleaning

python image_cleaning.py

Step 3

OCR

python ocr_processor.py

Step 4

Create Database

python table_creation.py

Step 5

Extract + Categorize + Store

python data_insertion.py

Step 6

Launch Dashboard

streamlit run frontend2.py

🤖 AI Agents

Agent 1

Invoice Extraction

Input

OCR Text

Output

{
Invoice_No,
Vendor,
Customer,
Description,
Amount
}

Agent 2

Expense Categorization

Input

Description

Output

Food

Travel

etc.

Agent 3

Natural Language → SQL

Input

How much did I spend on food?

Output

SELECT SUM(amount)

📊 Database

Table

ocr_line_items

Stores

Invoice Number
Vendor
Customer
Amount
Category
Description
Total
Source Image

)

⚠ Current Limitations

Uses Tesseract OCR, which struggles with noisy or low-quality receipts.
OCR accuracy drops on handwritten notes, skewed images, and faded receipts.
Image preprocessing is basic and may not handle all receipt conditions.
Local LLMs can occasionally return malformed JSON that requires validation.
SQLite is suitable for local usage but not for multi-user or production deployments.

🚀 Future Roadmap

🔍 OCR Improvements

Replace Tesseract with PaddleOCR
Add TrOCR support
EasyOCR benchmark
Google Vision OCR support
Azure Document Intelligence integration

🖼 Image Processing

CLAHE enhancement
Deskew receipts automatically
Perspective correction
Adaptive thresholding
Morphological filtering
Shadow removal
Reflection removal
Blur detection
Receipt edge detection
Auto crop

🤖 AI

Multi-Agent Workflow
LLM Output Validation
Retry Strategy
JSON Schema Validation
Confidence Score
Vendor Normalization
Duplicate Bill Detection

📊 Database

PostgreSQL support
MongoDB support
Vector Database
Semantic Search
Embeddings

💬 Chatbot

RAG Pipeline
Memory
Charts
Spending Trends
Monthly Reports
PDF Export
Excel Export

🌐 Web App

User Authentication
Multi-user Support
REST API
Docker Deployment
Cloud Deployment

🤝 Contributions

Contributions are welcome!

If you'd like to improve OCR accuracy, image preprocessing, LLM prompts, or the dashboard, feel free to open an issue or submit a pull request.

📜 License

MIT License

Name		Name	Last commit message	Last commit date
__pycache__		__pycache__
bill_images		bill_images
image_cleaning_one_folder		image_cleaning_one_folder
OCR_processor.py		OCR_processor.py
README.md		README.md
data_insertion.py		data_insertion.py
extracted_text.txt		extracted_text.txt
frontend1.py		frontend1.py
frontend2.py		frontend2.py
image_cleaning.py		image_cleaning.py
ocr_master.db		ocr_master.db
ollama1.py		ollama1.py
ollama2.py		ollama2.py
ollama3.py		ollama3.py
parser.py		parser.py
parser2.py		parser2.py
prompt1.py		prompt1.py
prompt2.py		prompt2.py
prompt3.py		prompt3.py
table_creation.py		table_creation.py

Latest commit

History

Folders and files

Repository files navigation

🚀 ScanX Expense

⭐ Overview

🎯 Project Goals

✨ Features

📷 Image Processing

🔍 OCR

🤖 AI Parsing

🧠 AI Categorization

💾 Database

📊 Dashboard

💬 Chat with Bills

🏗 System Architecture

📂 Project Structure

⚙ Tech Stack

🚀 Installation

Clone

Install Dependencies

Install Tesseract

Install Ollama

▶ Running the Project

Step 1

Step 2

Step 3

Step 4

Step 5

Step 6

🤖 AI Agents

Agent 1

Agent 2

Agent 3

📊 Database

⚠ Current Limitations

🚀 Future Roadmap

🔍 OCR Improvements

🖼 Image Processing

🤖 AI

📊 Database

💬 Chatbot

🌐 Web App

🤝 Contributions

📜 License

⭐ If you found this project useful, consider giving it a star on GitHub!

About

Topics

Resources

Stars

Watchers

Forks

Contributors

Languages