AgentOps Gateway

A secure execution platform for AI tool calls. An AI agent can ask to deploy a service, roll back a release, pull pod logs, or open an incident — but it never touches your infrastructure directly. Every tool call goes through this gateway, which authenticates the caller, authorizes the action against policy, validates and rate-limits it, logs and traces it, and only then executes the approved action.

The point: letting an LLM run kubectl apply against production directly is a bad idea. A hallucinated argument or a poisoned prompt shouldn't be one network hop away from deleting a Deployment. The gateway puts a policy engine and a human-in-the-loop approval gate between the model and the cluster.

---

The pipeline

Every tool call passes through the same stages. Safe reads sail through; risky writes stop for sign-off.

AI agent ──▶ [ authenticate ] ──▶ [ authorize ] ──▶ [ validate ] ──▶ [ rate-limit ] ──▶ [ approve? ] ──▶ [ execute ] ──▶ result
                                       │                                                     │                 │
                                       └──────────────── audit log + OpenTelemetry trace at every stage ───────┘

Stage	What it does	Status
authenticate	Identify the calling agent / user	planned (caller identity is currently trusted from the request body)
authorize	RBAC policy check — can this caller use this tool?	done (internal/policy)
validate	Schema-check tool parameters before execution	planned
rate-limit	Throttle per-caller call volume	planned
approve	Human sign-off for state-changing actions	done (internal/approval)
execute	Run the action against the cluster	done (internal/deploy, via kubectl)
observe	Audit log (JSON) + distributed trace (OTel)	done (internal/audit, internal/tracing)

---

Tools

The agent-facing tool surface. Read tools run immediately; write tools require approval.

Tool	Risk	Approval	Backed by (today)
check_health	read	no	kubectl get / kubectl describe
get_pod_logs	read	no	kubectl logs
query_metrics	read	no	planned (Prometheus query)
deploy_service	write	yes	kubectl apply / kubectl rollout
rollback_release	emergency	no (fast path)	kubectl rollout undo
create_incident	write	yes	planned (incident API)

Naming note: the verbs above are the target API. The running code still uses kubectl-flavored tool ids (kubectl-get, kubectl-apply, kubectl-rollout, kubectl-rollback, kubectl-logs) in internal/mcp/types.go, configs/policy.yaml, and the executor in internal/api/handlers.go. Renaming those to the verbs above is a follow-up. The runnable examples below use the current ids so they actually work.

---

How it works

An AI agent sends a tool call as a JSON POST:

{
  "id": "call-001",
  "tool": "kubectl-apply",
  "caller_id": "agent-bob",
  "caller_group": "sre",
  "parameters": {
    "namespace": "prod",
    "manifest": "apiVersion: apps/v1\nkind: Deployment..."
  }
}

The gateway then:

1. Authorizes — RBAC rules in configs/policy.yaml decide if the caller can use this tool, and whether it needs approval first.
2. Gates on approval — if the matched rule sets require_approval: true, the call is queued and returns 202 Accepted with an approval id. An approver must POST /v1/approvals/{id}/approve before it runs.
3. Executes — once cleared, it shells out to kubectl and returns the output.

Every step emits a JSON audit line and an OpenTelemetry span.

POST /v1/tools/invoke               submit a tool call
GET  /v1/approvals?status=pending   list pending approvals
POST /v1/approvals/{id}/approve     approve a queued call
POST /v1/approvals/{id}/reject      reject it with a reason
GET  /v1/health                     liveness check

---

Quickstart (local)

# needs Go 1.22+ (see scripts/setup.sh)
go mod tidy

# run with the example policy
go run ./cmd/gateway
# or: make run

# in another terminal — a read (allowed, no approval)
curl -s -X POST localhost:8080/v1/tools/invoke \
  -H 'Content-Type: application/json' \
  -d '{"id":"1","tool":"kubectl-get","caller_id":"alice","caller_group":"developers","parameters":{"resource":"pods","namespace":"default"}}' | jq

# a write (queued for approval)
curl -s -X POST localhost:8080/v1/tools/invoke \
  -H 'Content-Type: application/json' \
  -d '{"id":"2","tool":"kubectl-apply","caller_id":"bob","caller_group":"sre","parameters":{"namespace":"default","manifest":"..."}}' | jq

# check the pending queue
curl -s "localhost:8080/v1/approvals?status=pending" | jq

# approve it (note: approval id is "apr-" + the call id)
curl -s -X POST localhost:8080/v1/approvals/apr-2/approve \
  -H 'Content-Type: application/json' \
  -d '{"approver_id":"group:senior-sre"}' | jq

This assumes kubectl is on PATH and pointed at a cluster. Without a cluster the policy/approval flow still works end-to-end — only the final kubectl execution fails.

---

Project layout

cmd/gateway/         main — wires everything together
internal/
  mcp/               ToolCall / ToolResult types (MCP-style) + tool registry
  policy/            RBAC engine — loads policy.yaml, evaluates (caller, tool)
  approval/          state machine — pending → approved/rejected/expired → executed
  audit/             structured JSON audit log (one line per event)
  deploy/            kubectl wrapper — apply, rollback, rollout, get, describe, logs
  tracing/           OpenTelemetry setup — OTLP HTTP export to a collector
  api/               HTTP handlers — invocation, approval, health
configs/             policy.yaml (RBAC rules) and config.yaml (reference defaults)
k8s/
  gateway/           Deployment, Service, ConfigMap
  rbac/              ServiceAccount, ClusterRole, ClusterRoleBinding
  monitoring/        OTel Collector + Grafana
dashboards/          Grafana dashboard JSON (gateway-overview)
.github/workflows/   CI (test + Docker build) and deploy (tag-triggered rollout)
docs/                approach-guide.txt — read this first if you're new to Go/K8s
scripts/             setup.sh — installs Go and kubectl
legacy-streamview/   an unrelated older project, archived out of the way (see below)

---

Policy rules

Rules live in configs/policy.yaml, evaluated top-to-bottom — last match wins. Put a default deny-all at the top and layer exceptions below it.

policies:
  - name: deny-all              # default deny
    subjects: ["*"]
    tools: ["*"]
    allow: false

  - name: dev-read              # developers read, no approval
    subjects: ["group:developers"]
    tools: [kubectl-get, kubectl-describe, kubectl-logs]
    allow: true

  - name: sre-apply             # SREs apply, but need senior-sre sign-off
    subjects: ["group:sre"]
    tools: [kubectl-apply, kubectl-rollout]
    allow: true
    require_approval: true
    approvers: ["group:senior-sre"]

  - name: deny-delete           # nobody deletes, ever
    subjects: ["*"]
    tools: ["kubectl-delete"]
    allow: false

Subjects match exactly (user:alice, group:sre) or via *.

---

Observability

Set OTEL_EXPORTER_OTLP_ENDPOINT to your collector address and traces start flowing. The dashboard in dashboards/gateway-overview.json assumes Grafana Tempo as the trace backend.

The audit log is newline-delimited JSON on stdout — separate from traces, meant for security/compliance (who did what, when, approved by whom):

{"time":"2026-05-22T14:01:00Z","event":"tool_request","caller_id":"bob","tool":"kubectl-apply","call_id":"2","trace_id":"abc123"}
{"time":"2026-05-22T14:01:00Z","event":"policy_check","caller_id":"bob","tool":"kubectl-apply","call_id":"2","decision":"pending_review"}
{"time":"2026-05-22T14:03:11Z","event":"approved","approver_id":"carol","call_id":"2","approval_id":"apr-2"}
{"time":"2026-05-22T14:03:12Z","event":"tool_executed","caller_id":"bob","tool":"kubectl-apply","call_id":"2","duration_ms":430}

---

Deploying to Kubernetes

make deploy       # apply RBAC + gateway manifests, wait for rollout
make monitoring   # install OTel Collector + Grafana
make rollback     # roll the gateway back one revision

CI (ci.yaml) runs go test -race and builds a Docker image on every push to main. Deploy (deploy.yaml) triggers on v* tags, updates the image, waits for the rollout, and auto-rolls back on failure.

---

Status / roadmap

Implemented: RBAC policy, approval gate, audit log, OTel tracing, kubectl execution, k8s manifests, CI/CD.

Not done yet (rough edges):

• Authentication — caller identity is taken on trust from the request body. Real deployments need mTLS or signed tokens.
• Parameter validation and rate-limiting — stages exist in the design, not yet in code.
• Tool rename — kubectl-* ids → the verb API (deploy_service, rollback_release, …).
• query_metrics / create_incident — designed, not wired to backends.
• Approval store is in-memory; a restart loses pending approvals (back it with Redis/DB).
• No approval notifications — approvers poll /v1/approvals (add a Slack webhook).
• kubectl-exec is registered but not implemented; no mTLS between gateway and callers.

---