SOP Agent (UC-native) — Generic Agent, SOP-per-Version

Agents & Process Automation 101 was the why and Levels of Autonomy (LoA) for Enterprise Automation was the dial, this is the wiring you can ship, now tightened to match how you’ll actually deploy: one generic agent, SOP packaged with each model version, and the caller chooses the version (e.g., @Champion/@Challenger or separate endpoints). Tools are Unity Catalog (UC) functions only. Policy lives in YAML. We finish with a standardised Assurance Gate and real telemetry.

TL;DR

• Agent ≠ SOP: Ship one generic agent; each model version bundles its SOP. Caller picks the version (alias/endpoint).
• Tools = UC functions: Reads → UC SQL FUNCTION. Actions → Model-as-Function (MLflow pyfunc + Serving). The agent never calls raw APIs or MCP directly.
• Planner on rails: LLM plans one step at a time using only allow-listed UC tools from the SOP, executes, observes, reflects.
• Guardrails without extra boxes: Structure/PII via UC FUNCTIONS; policy thresholds as CEL expressions inside the SOP.
• Assurance Gate: Verify (UC) → policy (CEL) → MLflow judges (groundedness/relevance/safety) → auto-finalize or HITL.
• Ops that scale: MLflow Tracing → OpenTelemetry; aliases for Champion/Challenger rollout; clean lineage because SOP is versioned with the model.

What’s in this repo

• agent/sop_agent_langchain.py: Generic LangChain agent that reads an SOP YAML, builds UC-backed tools, runs, and applies the Assurance Gate (UC guardrail → CEL → MLflow judges).
• register/register_agent_langchain_uc.py: Registers the agent as an MLflow model in UC; the SOP path is tagged and bundled with the version.
• invoke/invoke_champion.py: Small CLI to invoke the registered model (@Champion by default).
• sop.yaml: Example SOP with planner config, UC tools, guardrails, failure policy, and judging rubric.
• sql/uc_functions.sql: Example UC SQL FUNCTIONS for lookup/recall/validation.

Quickstart

1) Environment

• Python 3.11+
• uv for environment management (brew install uv or see https://astral.sh/uv)
• Databricks workspace with UC and Model Registry
• SQL Warehouse credentials and a serving endpoint for model tools

2) Install

uv sync --python 3.11
# activate
source .venv/bin/activate  # zsh/bash
# or
source .venv/bin/activate.fish  # fish

3) Configure

Set these environment variables:

• MLFLOW_REGISTRY_URI=databricks-uc (default in registrar)
• DB_HOST — workspace hostname
• DB_WAREHOUSE — SQL warehouse HTTP path
• DB_TOKEN — Personal Access Token/Service Principal for SQL
• DATABRICKS_TOKEN — Token used by the planner and model-serving calls

4) Register a model version (bundles the SOP)

python -m register.register_agent_langchain_uc \
  # defaults: model_name=procure.platform.agent, alias omitted

Edit register/register_agent_langchain_uc.py to set sop_path and (optionally) alias (e.g., Champion). The registrar logs a LangChain model, pins/bundles the SOP, and (if set) assigns an alias.

5) Invoke

# invoke Champion by default
python invoke/invoke_champion.py --company-name "ACME Ltd" --amount 750 --pretty

# or provide the model URI and JSON payload
python invoke/invoke_champion.py \
  --model-uri "models:/procure.platform.agent@Challenger" \
  --input-json '{"company_name":"ACME Ltd","amount":750}' --pretty

SOP anatomy (YAML)

The SOP controls the planner and the available tools and policies:

• planner: model/base_url/temperature/max_steps and the system prompt
• tools: allow-listed UC tools with JSON Schemas for arguments
  • kind: sql_function → uc_fn: "catalog.schema.fn"
  • kind: model_function → uc_model + uc_endpoint (Serving)
• guardrails: UC FUNCTIONS for structural checks + CEL gates for policy
• failure: max retries/backoff and retryability per tool
• judging: pass threshold and rubric for MLflow judges

See sop.yaml for a working example.

UC tools

• SQL reads: implement as UC SQL FUNCTIONs (see sql/uc_functions.sql).
• Actions: publish an MLflow pyfunc and expose via Databricks Model Serving; reference it with uc_model and uc_endpoint in the SOP.

Assurance Gate

Executed after the agent’s tool run(s):

1. Validate structure/PII with UC FUNCTIONS (uc_validate_payload)
2. CEL policy decisions (allow/fail_and_retry/require_hitl)
3. MLflow judges compute groundedness, relevance, safety → score + verdict

If policy fails or judges fail, the chain returns REQUIRES_HITL; otherwise COMPLETED with a confidence score.

Observability

• Use MLflow Tracing and export to OpenTelemetry (see SOP observability fields)
• SOP is bundled with each model version → clean lineage and reproducibility

Troubleshooting

• Import errors: run commands from the repo root or execute modules, e.g. python -m register.register_agent_langchain_uc.
• Authentication: ensure DB_TOKEN and DATABRICKS_TOKEN are set and valid.
• Model URI: confirm the model is registered in UC and the alias (e.g., Champion) exists.

Further reading

• Agents & Process Automation 101 — the why: LinkedIn
• Levels of Autonomy (LoA) for Enterprise Automation — the dial: LinkedIn

License

Licensed under the Apache License, Version 2.0. See the LICENSE file for details.