[Feat] Add agent skill and Claude Code plugin

.claude-plugin/marketplace.json

@@ -0,0 +1,15 @@
+{
+  "name": "ml-energy-data",
+  "owner": {
+    "name": "Jae-Won Chung",
+    "email": "jwnchung@umich.edu"
+  },
+  "description": "ML.ENERGY Data Toolkit plugins for Claude Code.",
+  "plugins": [
+    {
+      "name": "skill",
+      "source": "./",
+      "description": "Analyze The ML.ENERGY Benchmark data with the mlenergy-data Python toolkit."
+    }
+  ]
+}

.claude-plugin/plugin.json

@@ -0,0 +1,11 @@
+{
+  "name": "skill",
+  "description": "Analyze The ML.ENERGY Benchmark data (LLM and diffusion inference energy/latency/throughput on H100 and B200) using the mlenergy-data Python toolkit.",
+  "author": {
+    "name": "Jae-Won Chung",
+    "email": "jwnchung@umich.edu"
+  },
+  "homepage": "https://ml.energy/data",
+  "repository": "https://github.com/ml-energy/data",
+  "license": "Apache-2.0"
+}

README.md

@@ -62,6 +62,28 @@ for arch, group in runs.task("gpqa").gpu_model("B200").group_by("architecture").
 - [**OpenG2G**](https://gpu2grid.io/openg2g/): Datacenter-grid coordination simulation framework; loads benchmark data and fits models.
 - [**The ML.ENERGY blog**](https://ml.energy/blog): Analysis scripts for blog posts.
 
+## Agent skill
+
+[`skills/mlenergy-data/`](skills/mlenergy-data/SKILL.md) is a portable [Agent Skill](https://agentskills.io/) that teaches AI coding agents (Claude Code, Cursor, Goose, OpenCode, Gemini CLI, GitHub Copilot, etc.) to answer natural-language questions about the benchmark using this toolkit.
+
+For Claude Code, install via the bundled plugin marketplace:
+
+```
+/plugin marketplace add ml-energy/data
+/plugin install skill@ml-energy-data
+```
+
+For other tools, drop the skill directly into that tool's skills directory:
+
+```bash
+# or ~/.config/goose/skills, ~/.gemini/skills, ~/.claude/skills, ...
+mkdir -p ~/.cursor/skills && \
+  curl -fsSL https://github.com/ml-energy/data/archive/refs/heads/master.tar.gz | \
+  tar -xz -C ~/.cursor/skills --strip-components=2 'data-master/skills/mlenergy-data'
+```
+
+Gemini CLI also supports `gemini skills install https://github.com/ml-energy/data --consent --scope user` directly.
+
 ## Documentation
 
 See the full [documentation site](https://ml.energy/data/) for:

skills/mlenergy-data/references/DATA_CATALOG.md

@@ -0,0 +1,130 @@
+# Data Catalog
+
+Concrete enumeration of what's in `ml-energy/benchmark-v3` as of the current release. Use this as a lookup so you don't have to re-download the parquet to check which combinations exist. To verify any of these are still current, run `LLMRuns.from_hf()` / `DiffusionRuns.from_hf()` and group as needed.
+
+## Counts
+
+- LLM runs (stable): 838 (1138 total; ~300 marked unstable)
+- Diffusion runs: 1020
+
+## Tasks
+
+| Task | Domain | Description |
+|---|---|---|
+| `gpqa` | LLM | GPQA Diamond — long-chain reasoning |
+| `lm-arena-chat` | LLM | LM Arena chat-style requests |
+| `sourcegraph-fim` | LLM | Code fill-in-the-middle (Sourcegraph) |
+| `image-chat` | LLM (multimodal) | Vision-language chat with image inputs |
+| `video-chat` | LLM (multimodal) | Vision-language chat with video inputs |
+| `text-to-image` | Diffusion | Text-to-image generation |
+| `text-to-video` | Diffusion | Text-to-video generation |
+
+Display names used in the leaderboard JSON (only for human output, not for filtering):
+GPQA → "GPQA Diamond", lm-arena-chat → "LLM Chat (LM Arena)", sourcegraph-fim → "Fill-in-the-Middle (Sourcegraph)", image-chat → "Image Chat", video-chat → "Video Chat", text-to-image → "Text to Image", text-to-video → "Text to Video".
+
+## GPUs and parallelism
+
+- `gpu_model`: `"H100"`, `"B200"`.
+- `num_gpus`: 1, 2, 4, 8.
+- LLM parallelism fields: `tensor_parallel`, `expert_parallel`, `data_parallel` (DP for attention + EP for MLP experts is the typical MoE setup).
+- Diffusion parallelism: `ulysses_degree`, `ring_degree` (sequence parallelism).
+
+## LLM batch sizes (`max_num_seqs`)
+
+Observed values (powers of 2 plus a few midpoints): 8, 16, 32, 64, 96, 128, 192, 256, 384, 512, 768, 1024, 1536, 2048, 3072, 4096. Not every (model, GPU, num_gpus) cell covers the full range.
+
+## Diffusion batch sizes
+
+1, 2, 4, 8, 16, 32.
+
+## Precisions
+
+- LLM `weight_precision`: `"bfloat16"`, `"fp8"`, `"mxfp4"` (mxfp4 is GPT OSS only).
+- Diffusion `weight_precision`: `"bfloat16"` only in this release.
+
+## Architectures (LLM)
+
+Three literal strings — match exactly:
+
+| Value | Examples |
+|---|---|
+| `"Dense Transformer"` | Llama 3.1 8B/70B/405B Instruct, Qwen 3 8B/14B/32B, Gemma 3 12B/27B, Qwen 3 VL 8B/32B Instruct |
+| `"MoE"` | Qwen 3 30B/235B (and Coder 30B/480B), GPT OSS 20B/120B, DeepSeek R1/V3.1, Llama 4 Maverick/Scout, Qwen 3 Omni 30B A3B, Qwen 3 VL 30B/235B Instruct |
+| `"Mamba-Transformer Hybrid"` | NVIDIA Nemotron Nano 9B V2, 12B V2, 12B V2 VL |
+
+## LLM models (per-model coverage)
+
+Format: `nickname` — `model_id` — `task list` — `(GPU, num_gpus)` cells. Total params (B) and active params (B) listed for MoE/Hybrid where they differ from total.
+
+### Dense Transformer
+
+- **Gemma 3 12B** — google/gemma-3-12b-it — image-chat, lm-arena-chat — (B200,1), (H100,1)
+- **Gemma 3 27B** — google/gemma-3-27b-it — image-chat, lm-arena-chat — (B200,1), (H100,1), (H100,2)
+- **Llama 3.1 8B Instruct** — meta-llama/Llama-3.1-8B-Instruct — lm-arena-chat — (B200,1), (H100,1)
+- **Llama 3.1 70B Instruct** — meta-llama/Llama-3.1-70B-Instruct — lm-arena-chat — (B200,2), (H100,4)
+- **Llama 3.3 70B Instruct** — meta-llama/Llama-3.3-70B-Instruct — lm-arena-chat — (B200,2), (H100,4)
+- **Llama 3.1 405B Instruct** (BF16, 405B) — meta-llama/Llama-3.1-405B-Instruct — lm-arena-chat — (B200,8)
+- **Llama 3.1 405B Instruct FP8** — meta-llama/Llama-3.1-405B-Instruct-FP8 — lm-arena-chat — (B200,4), (H100,8)
+- **Qwen 3 8B** — Qwen/Qwen3-8B — gpqa, lm-arena-chat — (B200,1), (H100,1)
+- **Qwen 3 14B** — Qwen/Qwen3-14B — gpqa, lm-arena-chat — (B200,1), (H100,1)
+- **Qwen 3 32B** — Qwen/Qwen3-32B — gpqa, lm-arena-chat — (B200,1), (H100,1), (H100,2)
+- **Qwen 3 VL 8B Instruct** — Qwen/Qwen3-VL-8B-Instruct — image-chat, video-chat — (B200,1), (H100,1)
+- **Qwen 3 VL 32B Instruct** — Qwen/Qwen3-VL-32B-Instruct — image-chat, video-chat — (B200,1), (H100,2)
+
+### MoE (total / active)
+
+- **Qwen 3 30B A3B Instruct** — Qwen/Qwen3-30B-A3B-Instruct-2507 — lm-arena-chat — (30B/3B) — (B200,1), (H100,1), (H100,2)
+- **Qwen 3 30B A3B Thinking** — Qwen/Qwen3-30B-A3B-Thinking-2507 — gpqa — (30B/3B) — (B200,1), (H100,1), (H100,2)
+- **Qwen 3 Coder 30B A3B** — Qwen/Qwen3-Coder-30B-A3B-Instruct — sourcegraph-fim — (30B/3B) — (B200,1), (H100,1), (H100,2)
+- **Qwen 3 Omni 30B A3B Instruct** — Qwen/Qwen3-Omni-30B-A3B-Instruct — image-chat, video-chat — (30B/3B) — (B200,1), (H100,1), (H100,2)
+- **Qwen 3 VL 30B A3B Instruct** — Qwen/Qwen3-VL-30B-A3B-Instruct — image-chat, video-chat — (30B/3B) — (B200,1), (H100,1), (H100,2)
+- **Qwen 3 235B A22B Instruct** — Qwen/Qwen3-235B-A22B-Instruct-2507 — lm-arena-chat — (235B/22B) — (B200,4), (B200,8), (H100,8)
+- **Qwen 3 235B A22B Instruct FP8** — Qwen/Qwen3-235B-A22B-Instruct-2507-FP8 — lm-arena-chat — (235B/22B) — (B200,2), (H100,4), (H100,8)
+- **Qwen 3 235B A22B Thinking** — Qwen/Qwen3-235B-A22B-Thinking-2507 — gpqa — (235B/22B) — (B200,4), (B200,8), (H100,8)
+- **Qwen 3 235B A22B Thinking FP8** — Qwen/Qwen3-235B-A22B-Thinking-2507-FP8 — gpqa — (235B/22B) — (B200,2), (B200,4), (H100,4), (H100,8)
+- **Qwen 3 VL 235B A22B Instruct** — Qwen/Qwen3-VL-235B-A22B-Instruct — image-chat, video-chat — (235B/22B) — (B200,4), (B200,8), (H100,8)
+- **Qwen 3 VL 235B A22B Instruct FP8** — Qwen/Qwen3-VL-235B-A22B-Instruct-FP8 — image-chat, video-chat — (235B/22B) — (B200,2), (B200,4), (H100,4), (H100,8)
+- **Qwen 3 Coder 480B A35B** — Qwen/Qwen3-Coder-480B-A35B-Instruct — sourcegraph-fim — (480B/35B) — (B200,8)
+- **Qwen 3 Coder 480B A35B FP8** — Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8 — sourcegraph-fim — (480B/35B) — (B200,4), (B200,8)
+- **GPT OSS 20B** — openai/gpt-oss-20b — gpqa — (21B/4B), MXFP4 capable — (B200,1), (H100,1)
+- **GPT OSS 120B** — openai/gpt-oss-120b — gpqa — (117B/5B), MXFP4 capable — (B200,1), (B200,2), (H100,1), (H100,2)
+- **DeepSeek R1** — deepseek-ai/DeepSeek-R1-0528 — gpqa — (671B/37B), **natively FP8** (no `" FP8"` suffix) — (B200,8)
+- **DeepSeek V3.1** — deepseek-ai/DeepSeek-V3.1 — gpqa, lm-arena-chat — (671B/37B), **natively FP8** — (B200,8)
+- **Llama 4 Maverick 17B 128E Instruct FP8** — meta-llama/Llama-4-Maverick-17B-128E-Instruct-FP8 — image-chat, lm-arena-chat — (400B/17B) — (H100,8)
+- **Llama 4 Scout 17B 16E Instruct** — meta-llama/Llama-4-Scout-17B-16E-Instruct — image-chat, lm-arena-chat — (109B/17B) — (H100,4), (H100,8)
+
+### Mamba-Transformer Hybrid
+
+- **NVIDIA Nemotron Nano 9B V2** — nvidia/NVIDIA-Nemotron-Nano-9B-v2 — gpqa, lm-arena-chat — (B200,1), (H100,1)
+- **NVIDIA Nemotron Nano 12B V2** — nvidia/NVIDIA-Nemotron-Nano-12B-v2 — gpqa, lm-arena-chat — (B200,1), (H100,1)
+- **NVIDIA Nemotron Nano 12B V2 VL** — nvidia/NVIDIA-Nemotron-Nano-12B-v2-VL — image-chat — (H100,1)
+
+## Diffusion models
+
+| Nickname | model_id | Task | Output size | inference_steps | Notes |
+|---|---|---|---|---|---|
+| FLUX.1 Dev | black-forest-labs/FLUX.1-dev | text-to-image | 1024×1024 | 50 | |
+| Hunyuan-DiT 1.2 | Tencent-Hunyuan/HunyuanDiT-v1.2 | text-to-image | 1024×1024 | 50 | |
+| PixArt-Σ | PixArt-alpha/PixArt-Sigma-XL-2-1024-MS | text-to-image | 1024×1024 | 20 | |
+| SANA 1.5 1.6B | Efficient-Large-Model/SANA1.5_1.6B_1024px_diffusers | text-to-image | 1024×1024 | 20 | |
+| SANA 1.5 4.8B | Efficient-Large-Model/SANA1.5_4.8B_1024px_diffusers | text-to-image | 1024×1024 | 20 | |
+| Stable Diffusion 3.5 Medium | stabilityai/stable-diffusion-3.5-medium | text-to-image | 1024×1024 | 28 | |
+| Stable Diffusion 3.5 Large | stabilityai/stable-diffusion-3.5-large | text-to-image | 1024×1024 | 28 | |
+| CogVideoX 2B | THUDM/CogVideoX-2b | text-to-video | 480×720, 49 frames @ 8 fps | 50 | |
+| CogVideoX 1.5 5B | THUDM/CogVideoX1.5-5B | text-to-video | 768×1360, 81 frames @ 8 fps | 50 | |
+| Wan 2.1 1.3B | Wan-AI/Wan2.1-T2V-1.3B-Diffusers | text-to-video | 480×832, 81 frames @ 15 fps | 50 | |
+| Wan 2.1 14B | Wan-AI/Wan2.1-T2V-14B-Diffusers | text-to-video | 480×832, 81 frames @ 15 fps | 50 | |
+| HunyuanVideo | tencent/HunyuanVideo | text-to-video | 720×1280, 129 frames @ 15 fps | 50 | |
+
+All diffusion runs in this release are `bfloat16`, `use_torch_compile=False`. Diffusion has run-count coverage on (B200, 1/2/4/8) and (H100, 1/2/4/8) for most models — the multi-GPU variants test sequence-parallel (Ulysses + Ring).
+
+## Stability (LLM only)
+
+A run is marked unstable if any of:
+
+- `steady_state_duration < 20s`
+- `energy_per_token_joules` missing or non-positive
+- `avg_batch_size / max_num_seqs < 0.85` ("low_batch_utilization")
+- A smaller batch in the same (model, task, GPU, num_gpus) group was unstable ("cascade_from_unstable_batch:N")
+
+`stable_only=True` (the default for `LLMRuns.from_hf()`) excludes them. Use `LLMRuns.from_hf(stable_only=False)` then `.unstable()` to inspect.

skills/mlenergy-data/references/PITFALLS.md

@@ -0,0 +1,109 @@
+# Extended Pitfalls
+
+The most critical pitfalls — the ones that cause silent wrong answers — are inlined in [`SKILL.md`](../SKILL.md) ("Critical gotchas" and "Units"). This file holds the longer-tail items: API edge cases, raw-JSON details, and reproducibility notes.
+
+## API edge cases
+
+### `num_gpus` / `max_num_seqs` / `batch_size` filters cannot mix exact + range
+
+```python
+runs.num_gpus(1, 2)            # exact OR
+runs.num_gpus(min=2, max=8)    # inclusive range
+runs.num_gpus(1, min=2)        # raises ValueError
+runs.num_gpus()                # raises TypeError (no args)
+```
+
+### `unstable()` requires a non-stable-only collection
+
+```python
+LLMRuns.from_hf().unstable()                      # raises ValueError
+LLMRuns.from_hf(stable_only=False).unstable()     # works
+```
+
+### `group_by` key shape depends on arg count
+
+- One field: keys are scalars. `group_by("task")` → `{"gpqa": LLMRuns(...), ...}`.
+- Multiple fields: keys are tuples. `group_by("model_id", "task")` → `{("Qwen/...", "gpqa"): LLMRuns(...), ...}`.
+
+### `to_dataframe()` strips private fields
+
+Path fields (`_results_path`, `_prometheus_path`) and HF metadata (`_hf_repo_id`, etc.) are excluded. Use `r.read_results_json()` or `r.read_prometheus_json()` to access raw payloads — don't try to read paths from a `to_dataframe()` row.
+
+### `download_raw_files()` returns the same collection
+
+It's an in-place pre-warm — chain it for ergonomics; the return value is the same `LLMRuns`/`DiffusionRuns` object. The optional `file=` arg can be `"results"` or `"prometheus"` for LLM; default is both. For diffusion, only `results.json` exists.
+
+### Cache lives in `HF_HOME`
+
+Repeated calls after the first download are instant. To force a re-download, clear the HF cache or pass `revision=` to `from_hf()`.
+
+## Raw JSON details
+
+### Inter-token latencies in raw `results.json` are not smoothed
+
+For chunked-prefill engines (vLLM), some ITL entries are zero — the engine batched multiple decode tokens into one step. The toolkit's `inter_token_latencies()` and the `*_itl_ms` percentile fields **already handle this** by spreading the accumulated latency across the covered tokens (`_smooth_chunked_itl` in `runs.py`).
+
+Only an issue if you read raw `results.json` directly via `run.read_results_json()` — then you'll see a long zero spike at the front. Use the toolkit's accessors unless you specifically want raw values.
+
+### Per-device timeline columns
+
+`run.timelines()` exposes only the GPU-summed `value`. Per-device samples are present in raw `results.json` under `timeline.power.device_instant.<gpu_id>` if needed. The internal helper `mlenergy_data.records.timelines.extract_device_timeline` returns wide-form per-device columns plus the total — use it directly if you're doing per-GPU analysis.
+
+### Stability cascade rule
+
+A run is unstable if any of: `steady_state_duration < 20s`, `energy_per_token_joules <= 0`, `avg_batch_size / max_num_seqs < 0.85` ("low_batch_utilization"), or a smaller batch in the same `(model, task, GPU, num_gpus)` group was unstable ("cascade_from_unstable_batch:N"). The cascade reflects the assumption that smaller batches are the safe baseline; if the smallest batch is unstable, larger batches don't get a free pass.
+
+To diagnose:
+
+```python
+all_llm = LLMRuns.from_hf(stable_only=False)
+{r.unstable_reason.split(":")[0] for r in all_llm.unstable()}
+```
+
+In the current release: ~95% of unstable runs are `low_batch_utilization`, the remainder are `cascade_from_unstable_batch` and `short_steady_state`.
+
+## Coverage
+
+### (model, GPU, num_gpus, task) cells are sparse
+
+The catalog ([`DATA_CATALOG.md`](DATA_CATALOG.md)) is the authoritative list. When a subset comes up empty for a comparison, prefer to skip and tell the user, rather than fabricate by widening filters.
+
+### Across-the-board comparison: intersect available tasks first
+
+```python
+candidates = {
+    nick for nick, group in llm.group_by("nickname").items()
+    if {"gpqa", "lm-arena-chat"} <= {r.task for r in group}
+}
+```
+
+## Field semantics
+
+### Diffusion's single `energy_per_generation_joules`
+
+The toolkit uses one generic field for both image and video runs. Check `r.is_text_to_image` / `r.is_text_to_video` (or `r.task`) to decide labels and which size fields are populated. `num_frames` and `fps` are `None` for image runs. The leaderboard-build script splits this into `energy_per_image_joules` / `energy_per_video_joules` in its JSON output, but that's a downstream rename, not a toolkit field.
+
+## `mlenergy_data.modeling` is not recommended for general use
+
+The toolkit exports `LogisticModel` (4-parameter sigmoid for batch sweeps) and `ITLMixtureModel` (two-component lognormal for inter-token latency). Both classes work, but their fit quality is unreliable enough that the skill does not surface them. Concretely, on the v3 dataset:
+
+- **`LogisticModel`** on `energy_per_token_joules`, `avg_power_watts`, `median_itl_ms`, `output_throughput_tokens_per_sec` vs `log2(batch_size)`: median R² ≥ 0.98 across (model, task, gpu, num_gpus) groups, but **22%–38% of groups have max relative residuals above 15%, with worst cases over 100%**. There's no in-band signal that tells you whether your fit is in the good majority or the bad tail.
+- **`ITLMixtureModel`**: median empirical-mean recovery is 2%, but **p90 is 172% and the worst case is 1430%** (model predicts ~15× the true mean). Vision-language models on `image-chat` and `video-chat` are the consistent failure mode — the lognormal "stall" component fits a heavy tail that blows up the analytical mean.
+
+If a user explicitly asks for these fits, use them — but always validate by:
+- For `LogisticModel`: compute residuals against the actual data points (`fit.eval(batch=b)` vs the measured value at each `b` in your sweep), and reject the fit if `max(|residual / y|) > 0.15` or any held-out point disagrees by more than ~10%.
+- For `ITLMixtureModel`: compare `fit.mean_var()` to `np.mean(samples_s), np.std(samples_s)` and reject if either differs by more than ~10–20%. KS-test the model's `sample_one` draws against the empirical distribution.
+
+For most analyses, prefer interpolating directly between measured batch sizes (the dataset is dense) over fitting a parametric model.
+
+## Reproducibility
+
+The `seed` and `num_request_repeats` fields are populated for LLM runs. The toolkit doesn't aggregate across seeds — multiple seeds appear as multiple `LLMRun` rows sharing `(task, model_id, gpu_model, num_gpus, max_num_seqs)`. To measure variability across seeds, group by everything except `seed` and look at the spread of the metric of interest:
+
+```python
+keys = ("task", "model_id", "gpu_model", "num_gpus", "max_num_seqs", "weight_precision")
+for k, group in llm.group_by(*keys).items():
+    if len(group) > 1:
+        es = [r.energy_per_token_joules for r in group]
+        print(f"{k}: n={len(es)} energy spread {min(es):.4f}..{max(es):.4f}")
+```