SATE AI

Fault Injection and Reliability Framework for On-Device AI across Flutter, CLI, Android, Web, and more

🔍 Looking for runtime observability? Pair SATE AI with EdgePulse: the observability companion that traces memory, thermal state, battery draw, and latency during inference, and works directly alongside SATE AI's fault injection.

📄 Published research: Three papers published on Zenodo cover SATE AI and on-device AI reliability:

1. SATE AI Framework Paper: DOI: 10.5281/zenodo.23250418
2. EdgePulse Telemetry Paper: DOI: 10.5281/zenodo.23248718
3. Joint Fault Injection & Tracing Study: DOI: 10.5281/zenodo.23262270 (280 real-device traces across TFLite, ONNX, and GGUF)

Platform Support

Platform	Language	Package / Target	Status
Flutter (Mobile, Desktop)	Dart	packages/sate_ai	✅ Available
CLI (Linux, macOS, Windows)	Dart	packages/sate_cli	✅ Available
Android Native	Kotlin	packages/sate_android	✅ Available
Web / Node.js	TypeScript / npm	packages/sate_web	✅ Available
iOS / macOS Native	Swift	packages/sate_ios	🔭 Planned (Roadmap)
Python Evaluation Runner	Python	packages/sate_python	🔭 Planned (Roadmap)

Architecture Overview

SATE AI is designed with clean decoupling between business logic, runtime execution, and target platforms:

                      +-----------------------------+
                      |       Standard Report       |
                      |   (JSON / Markdown / HTML)  |
                      +-----------------------------+
                                     ^
                                     |
               +---------------------+---------------------+
               |                     |                     |
               v                     v                     v
      +-----------------+   +-----------------+   +-----------------+
      |    sate_core    |   |  sate_android   |   |    sate_web     |
      |   (Pure Dart)   |   | (Kotlin / JVM)  |   |  (TypeScript)   |
      +-----------------+   +-----------------+   +-----------------+
         |           |               |                     |
         v           v               v                     v
   +----------+ +----------+   +-----------+       +---------------+
   | sate_ai  | | sate_cli |   |  Android  |       |  Web / Node   |
   | (Flutter)| |  (CLI)   |   |  TFLite   |       | TFJS / ONNX   |
   +----------+ +----------+   +-----------+       +---------------+

• sate_core: The pure Dart foundational engine containing fault injector interfaces, report structures, golden baselines, i18n, webhooks, and scheduler. Contains zero Flutter or platform dependencies.
• sate_cli: Standalone command-line executable distributed as an independent Dart application. Directly depends on sate_core without requiring Flutter SDK.
• sate_ai: Flutter package providing UI integrations, DevTools extensions, and Flutter-specific native adapters (ONNX, TFLite, Fllama, MediaPipe, Core ML, Google ML Kit).
• sate_android: Idiomatic Kotlin/Android SDK providing fault injection directly to Android applications using TensorFlow Lite without Flutter.
• sate_web: TypeScript/npm SDK with support for TensorFlow.js and ONNX Runtime Web in browser or Node.js runtimes.

Repository structure

This repository is organized as a monorepo containing the following packages:

Package	Type	Description
packages/sate_core	Pure Dart	Framework-agnostic core: injectors, adapters interface, reports, baseline, templates, webhooks, i18n
packages/sate_ai	Flutter	Flutter package wrapping sate_core, plus native adapters (ONNX, TFLite, Fllama, MediaPipe, Core ML, Google ML Kit)
packages/sate_cli	Pure Dart	CLI binary: installable without Flutter via dart pub global activate sate_cli
packages/sate_android	Kotlin	Android native SDK for fault injection on TFLite models
packages/sate_web	TypeScript/npm	TypeScript/npm SDK for browser and Node.js (TensorFlow.js, ONNX Runtime Web)
packages/sate_ai_devtools	Flutter	Flutter DevTools extension for real-time fault injection monitoring

Packages are orchestrated with Melos.

Development

# Install Melos
dart pub global activate melos

# Bootstrap all packages
melos bootstrap

# Run everything
melos run ci

Individual package commands:

melos run test:core   # pure Dart core tests
melos run test:cli    # CLI tests
melos run test:ai     # Flutter package tests

Overview

SATE AI is a fault injection and resilience testing framework for on-device AI models across Flutter, mobile (Android/iOS), web, Node.js, and command-line environments. It simulates real-world failure scenarios: memory pressure, malformed inputs, quantization drift, thermal throttling, latency spikes, model corruption, network drops, GPU VRAM pressure, and confidence degradation: so developers can validate model reliability before shipping to production.

Features

• 11 fault injectors covering memory, I/O, thermal, and network failure modes
• Adaptive Chaos Injection modulated by live device CPU, RAM, and thermal telemetry
• Stress test cost and resource estimation for duration, memory, battery, and cloud API spend
• Model Quantization Analyzer for ONNX and TFLite models
• 8 model adapters: MockAdapter, OnnxAdapter, TFLiteAdapter, FllamaAdapter, MediaPipeAdapter, CoreMLAdapter, GoogleMLKitAdapter, and custom adapters
• CLI tool with subcommands for stress, benchmark, batch, schedule, serve, health-check, badge, template output, and quantization analysis
• HTML report export with Chart.js charts
• Real-time SSE monitoring dashboard
• Golden baseline regression detection
• Stress test retries and flaky test detection
• SQLite storage for historical report tracking
• CI/CD status badges (SVG)
• Custom report templates (JSON/YAML)
• Interactive guided onboarding tutorial in the Flutter example app
• Project showcase gallery featuring real-world deployments and edge AI benchmarks
• Stress scheduler (cron)
• VS Code extension

Installation

CLI (no Flutter required)

The CLI can be installed with only the Dart SDK:

dart pub global activate sate_cli
sate_ai --model model.gguf --injectors memoryPressure

If sate_ai is not found after install, add the pub cache to your PATH:

export PATH="$PATH:$HOME/.pub-cache/bin"

Flutter package

Add to your pubspec.yaml:

dependencies:
  sate_ai: ^0.11.0

flutter pub get

Web builds are not supported when the TFLite adapter is active. The tflite_flutter package relies on native FFI bindings that cannot be transpiled to JavaScript. Web projects should use MockAdapter or a web-compatible adapter.

Flutter developers who want both

If you already have Flutter installed and want the CLI alongside the package:

flutter pub global activate sate_ai

CI/CD Integration

Add SATE AI to any pipeline with the Dart SDK alone - no Flutter needed:

- name: Install SATE AI CLI
  run: |
    dart pub global activate sate_cli
    echo "$HOME/.pub-cache/bin" >> $GITHUB_PATH

- name: Run stress tests
  run: |
    sate_ai --model models/model.gguf \
            --injectors memoryPressure,malformedInput \
            --output report.json

- name: Upload report
  uses: actions/upload-artifact@v4
  with:
    name: sate-ai-report
    path: report.json

Quick Start

import 'package:sate_ai/sate_ai.dart';

Future<void> main() async {
  final model = MockAdapter(modelId: 'my-model');

  final report = await SateAI.stress(
    model: model,
    injectors: [
      MemoryPressureInjector(limitMb: 100),
      MalformedInputInjector(),
    ],
    retryCount: 3,
    flakyThreshold: 2,
  );

  if (report.passed) {
    print('Model passed all stress tests.');
  } else {
    print('Model failed: ${report.failureCount} failures.');
    print(report.toMarkdown());
  }
}

Fault Injectors

Injector	Fault Type
MemoryPressureInjector	memoryPressure
MalformedInputInjector	malformedInput
QuantizationDriftInjector	quantizationDrift
ThermalThrottleInjector	thermalThrottle
LatencyInjector	latency
ModelSwapInjector	modelSwap
NetworkLatencyDropInjector	networkFailure
ConfidenceThresholdInjector	confidenceValidation
GpuMemoryPressureInjector	gpuMemoryPressure
DataCorruptionInjector	dataCorruption
ModelVersionMismatchInjector	modelVersionMismatch

Model Adapters

Adapter	Runtime
MockAdapter	Pure Dart (testing)
OnnxAdapter	ONNX Runtime
TFLiteAdapter	TensorFlow Lite
FllamaAdapter	llama.cpp (Llama, Phi, Gemma)
MediaPipeAdapter	Google MediaPipe (vision)
CoreMLAdapter	Apple Core ML (iOS)
GoogleMLKitAdapter	Google ML Kit

CLI

# Basic stress test
sate_ai --model model.gguf --injectors memoryPressure,malformedInput

# With retry and flaky detection
sate_ai --model model.gguf --injectors memoryPressure --retry 3 --flaky-threshold 2

# Benchmark mode
sate_ai --model model.gguf --benchmark --benchmark-runs 20

# Batch mode with auto-detection
sate_ai --models model1.onnx,model2.tflite,model3.gguf --auto-detect

# Health check
sate_ai --model model.gguf --health-check

# HTML report
sate_ai --model model.gguf --injectors memoryPressure --html --output report.html

# Custom template
sate_ai --model model.gguf --injectors memoryPressure --template templates/slack.yaml

# Generate badges
sate_ai --model model.gguf --injectors memoryPressure --badge-output docs/sate_ai

# Save to SQLite
sate_ai --model model.gguf --injectors memoryPressure --db history.db

# List history
sate_ai --db history.db --db-history

# Golden baseline
sate_ai --model model.gguf --baseline
sate_ai --model model.gguf --compare --tolerance 5.0

# Real-time monitoring dashboard
sate_ai serve --port 8080

# Scheduling
sate_ai --model model.gguf --schedule "0 2 * * *"

# Compare two reports
sate_ai --compare-reports report1.json,report2.json --diff-html --diff-output diff.html

# Profiling
sate_ai --model model.gguf --profile --profile-runs 5

# Multi-language localized reports (en, es, fr, de, pt)
sate_ai --model model.gguf --markdown --language es --output report_es.md
sate_ai --model model.gguf --html --language fr --output report_fr.html

# Webhook notifications
sate_ai --model model.gguf --injectors memoryPressure \
  --webhook-url https://hooks.slack.com/services/XXX/YYY/ZZZ \
  --webhook-type slack

# Generate a custom injector
sate_ai create injector MyCustomInjector

# Pre-flight stress test cost estimation
sate_ai --estimate --injectors memoryPressure,thermalThrottle --estimate-profile edge

Showcase

Discover how teams, researchers, and developers use SATE AI to harden on-device models:

• Project Showcase Gallery: Interactive web gallery with filterable categories, testimonials, and reliability metrics.
• SHOWCASE.md: Submission guidelines, project details, and official "Tested with SATE AI" badge snippets for your README.

Documentation

• Website
• Project Showcase Gallery
• API Reference
• Research Paper
• Cost and Resource Estimation Cookbook
• Adaptive Chaos Testing Cookbook
• Contributing Guide

Citation

If you use SATE AI or EdgePulse in your research, please cite the corresponding works:

@software{ullah2026sateai,
  author       = {Muhammad Assad Ullah},
  title        = {SATE AI: A Fault Injection and Reliability Engineering Framework for On-Device AI Models in Mobile Applications},
  year         = {2026},
  publisher    = {Zenodo},
  version      = {0.1.0},
  doi          = {10.5281/zenodo.23250418},
  url          = {https://doi.org/10.5281/zenodo.23250418}
}

@software{ullah2026edgepulse,
  author       = {Muhammad Assad Ullah},
  title        = {EdgePulse: A Runtime Observability Framework for Quantized Large Language Models on Consumer Edge Devices},
  year         = {2026},
  publisher    = {Zenodo},
  version      = {1.0.0},
  doi          = {10.5281/zenodo.23248718},
  url          = {https://doi.org/10.5281/zenodo.23248718}
}

@software{ullah2026joint,
  author       = {Muhammad Assad Ullah},
  title        = {Characterising On-Device AI Failure Modes Under Joint Fault Injection and Runtime Observability},
  year         = {2026},
  publisher    = {Zenodo},
  version      = {1.0.0},
  doi          = {10.5281/zenodo.23262270},
  url          = {https://doi.org/10.5281/zenodo.23262270}
}

Contributing

Contributions are welcome. See CONTRIBUTING.md.

License

MIT: see LICENSE.