GPT-OSS Vision Project

A vision-language AI pipeline combining LLaVA for image analysis with GPT-OSS for advanced reasoning capabilities.

Overview

This project implements a two-stage vision pipeline:

1. LLaVA Vision Analysis - Processes images and generates detailed descriptions
2. GPT-OSS Reasoning - Applies chain-of-thought reasoning to the visual analysis

Features

• 🔍 Image analysis using LLaVA 7B vision model
• 🧠 Advanced reasoning with GPT-OSS 20B language model
• 🖥️ GPU monitoring and temperature management
• 📝 Comprehensive logging
• 💾 JSON output for analysis results
• 🔄 Interactive CLI mode

Requirements

• Python 3.8+
• Ollama installed and running
• Required Ollama models:
  • llava:7b - Vision model
  • gpt-oss:20b - Language model

Installation

1. Clone the repository:

git clone https://github.com/YOUR_USERNAME/gptoss-vision-project.git
cd gptoss-vision-project

2. Install dependencies:

pip install pillow requests

3. Pull required Ollama models:

ollama pull llava:7b
ollama pull gpt-oss:20b

4. Start Ollama server:

ollama serve

Usage

Interactive Mode

python scripts/gptoss_llava_pipeline.py

This launches an interactive CLI where you can:

1. Analyze images by providing their path
2. Ask specific questions about images
3. View detailed analysis results

Programmatic Usage

from scripts.gptoss_llava_pipeline import GPTOSSLLaVAPipeline

pipeline = GPTOSSLLaVAPipeline()
result = pipeline.analyze_image("path/to/image.jpg", question="What is happening in this image?")

print(result["vision_analysis"])
print(result["gptoss_reasoning"])

Project Structure

gptoss-vision-project/
├── scripts/
│   ├── config.py              # Configuration settings
│   ├── gptoss_llava_pipeline.py  # Main pipeline
│   ├── utils.py               # Utility functions
│   └── test_llava.py          # Test scripts
├── datasets/
│   ├── images/                # Input images
│   └── captions/              # Generated captions
├── checkpoints/               # Model checkpoints
├── logs/                      # Log files
└── outputs/                   # Analysis results (JSON)

Configuration

Edit scripts/config.py to customize:

• Ollama host and port
• Model names
• Hardware limits (VRAM, temperature)
• Inference parameters (temperature, top_p, max tokens)

Hardware Requirements

• GPU with 8GB+ VRAM recommended
• 16GB+ system RAM
• Models run in 4-bit quantization mode for efficiency

License

MIT License