Azure CEO is a multi-agent AI system designed to automate marketing workflows using large language models, retrieval-augmented generation (RAG), and structured experimentation pipelines.
The system demonstrates how LLMs can be connected to real-world data and backend services to support reliable, data-driven decision-making.
This architecture can be extended to use cases such as text-to-SQL systems, enterprise copilots, and data-driven decision support tools.
This project focuses on building an end-to-end AI system that integrates:
- Multi-agent orchestration (Semantic Kernel)
- Retrieval-augmented generation across structured and unstructured data
- Backend API services (FastAPI)
- Evaluation and experimentation workflows (A/B/n testing)
- Safety and compliance layers for production use
It is designed to reflect real-world AI system architecture rather than isolated prompt-based applications.
The system is composed of several coordinated layers:
-
Orchestration Layer
Multi-agent coordination using Semantic Kernel (Strategy Lead, Data Segmenter, Content Generator, Compliance Agent, etc.) -
Retrieval Layer
Hybrid RAG using Azure AI Search for grounding, citation handling, and hallucination mitigation
Supports both structured (SQL-style reasoning) and unstructured data retrieval -
Backend API Layer
FastAPI-based services for handling requests, agent coordination, and pipeline execution -
Evaluation & Experimentation Layer
A/B/n testing, variant generation, and structured evaluation workflows -
Safety & Compliance Layer
Input validation, Azure Content Safety integration, and rule-based guardrails
- Multi-agent system with shared memory and sequential task execution
- Hybrid retrieval across structured and unstructured data sources
- Async RAG pipelines with grounding and citation support
- Built-in experimentation framework for evaluating outputs
- Modular plugin architecture for extensibility
- Telemetry and monitoring integration (Azure Monitor, OpenTelemetry)
- AI / Orchestration: Semantic Kernel, Azure OpenAI
- Retrieval: Azure AI Search (Hybrid RAG)
- Backend: FastAPI
- Infrastructure: Docker, Azure
- Monitoring: OpenTelemetry, Azure Monitor
Most LLM projects focus on prompting.
This project focuses on systems.
It explores how to:
- Connect LLMs to real data sources
- Build backend services that support AI workflows
- Design modular, production-oriented AI architectures
- Evaluate and iterate on model outputs in structured ways
- Deeper integration with structured data sources (SQL / warehouse systems)
- Enhanced evaluation pipelines and scoring frameworks
- Expanded agent roles and orchestration strategies