diff --git a/.claude-plugin/marketplace.json b/.claude-plugin/marketplace.json new file mode 100644 index 0000000..ea71065 --- /dev/null +++ b/.claude-plugin/marketplace.json @@ -0,0 +1,15 @@ +{ + "name": "ml-energy-data", + "owner": { + "name": "Jae-Won Chung", + "email": "jwnchung@umich.edu" + }, + "description": "ML.ENERGY Data Toolkit plugins for Claude Code.", + "plugins": [ + { + "name": "skill", + "source": "./", + "description": "Analyze The ML.ENERGY Benchmark data with the mlenergy-data Python toolkit." + } + ] +} diff --git a/.claude-plugin/plugin.json b/.claude-plugin/plugin.json new file mode 100644 index 0000000..e885abb --- /dev/null +++ b/.claude-plugin/plugin.json @@ -0,0 +1,11 @@ +{ + "name": "skill", + "description": "Analyze The ML.ENERGY Benchmark data (LLM and diffusion inference energy/latency/throughput on H100 and B200) using the mlenergy-data Python toolkit.", + "author": { + "name": "Jae-Won Chung", + "email": "jwnchung@umich.edu" + }, + "homepage": "https://ml.energy/data", + "repository": "https://github.com/ml-energy/data", + "license": "Apache-2.0" +} diff --git a/README.md b/README.md index f331f94..8bbf9a8 100644 --- a/README.md +++ b/README.md @@ -62,6 +62,28 @@ for arch, group in runs.task("gpqa").gpu_model("B200").group_by("architecture"). - [**OpenG2G**](https://gpu2grid.io/openg2g/): Datacenter-grid coordination simulation framework; loads benchmark data and fits models. - [**The ML.ENERGY blog**](https://ml.energy/blog): Analysis scripts for blog posts. +## Agent skill + +[`skills/mlenergy-data/`](skills/mlenergy-data/SKILL.md) is a portable [Agent Skill](https://agentskills.io/) that teaches AI coding agents (Claude Code, Cursor, Goose, OpenCode, Gemini CLI, GitHub Copilot, etc.) to answer natural-language questions about the benchmark using this toolkit. + +For Claude Code, install via the bundled plugin marketplace: + +``` +/plugin marketplace add ml-energy/data +/plugin install skill@ml-energy-data +``` + +For other tools, drop the skill directly into that tool's skills directory: + +```bash +# or ~/.config/goose/skills, ~/.gemini/skills, ~/.claude/skills, ... +mkdir -p ~/.cursor/skills && \ + curl -fsSL https://github.com/ml-energy/data/archive/refs/heads/master.tar.gz | \ + tar -xz -C ~/.cursor/skills --strip-components=2 'data-master/skills/mlenergy-data' +``` + +Gemini CLI also supports `gemini skills install https://github.com/ml-energy/data --consent --scope user` directly. + ## Documentation See the full [documentation site](https://ml.energy/data/) for: diff --git a/skills/mlenergy-data/SKILL.md b/skills/mlenergy-data/SKILL.md new file mode 100644 index 0000000..2b60fd9 --- /dev/null +++ b/skills/mlenergy-data/SKILL.md @@ -0,0 +1,300 @@ +--- +name: mlenergy-data +description: Answer natural-language questions about The ML.ENERGY Benchmark (LLM and diffusion inference energy/latency/throughput measurement data on GPUs) using the `mlenergy-data` Python toolkit. Use when the user asks about energy per token, energy per image/video, throughput, ITL latency, power timelines, GPU comparisons, batch-size sweeps, MoE/Dense/Hybrid architecture comparisons, or FP8/BF16/MXFP4 precision comparisons. +license: Apache-2.0 +compatibility: Requires Python 3.10+, `pip install mlenergy-data`, and a Hugging Face access token in `HF_TOKEN` (the dataset is gated). +--- + +# ML.ENERGY Data Toolkit + +The toolkit (`mlenergy_data`) loads and analyzes [The ML.ENERGY Benchmark v3.0](https://huggingface.co/datasets/ml-energy/benchmark-v3) — LLM and diffusion inference runs on NVIDIA H100 and B200 GPUs, published as parquet summaries plus per-run raw JSONs on HF Hub. + +Use this skill whenever the user wants to query, compare, aggregate, plot, or model the benchmark data. + +## What the data covers (orient before coding) + +- **Two domains**, each with its own collection class: + - LLM (text + multimodal LLM): `LLMRuns` / `LLMRun`. Tasks: `gpqa`, `lm-arena-chat`, `sourcegraph-fim`, `image-chat`, `video-chat`. Image-chat and video-chat are LLMs that ingest images/videos — they are NOT diffusion. + - Diffusion (text-to-image, text-to-video): `DiffusionRuns` / `DiffusionRun`. Tasks: `text-to-image`, `text-to-video`. +- GPUs: `H100`, `B200`. Multi-GPU configs at `num_gpus ∈ {1, 2, 4, 8}`. +- LLM batch sizes (`max_num_seqs`): from 8 up to 4096, mostly powers of 2 (8, 16, 32, 64, 96, 128, 192, 256, 384, 512, 768, 1024, 1536, 2048, 3072, 4096). +- Diffusion batch sizes: 1, 2, 4, 8, 16, 32. +- LLM precisions: `bfloat16`, `fp8`, `mxfp4` (mxfp4 is GPT OSS only). Diffusion precisions: only `bfloat16` in the current release. +- LLM architectures: `"Dense Transformer"`, `"MoE"`, `"Mamba-Transformer Hybrid"` (the literal strings — not "Dense"/"Hybrid"). +- A complete catalog of models, tasks-per-model, GPU coverage, and image/video sizes is in [`references/DATA_CATALOG.md`](references/DATA_CATALOG.md). Read it before guessing model names or whether a (model, GPU, num_gpus) cell exists. + +## Setup + +**The dataset is gated on Hugging Face Hub.** The user must have requested access *and* set an `HF_TOKEN` for any `from_hf()` call to succeed. + +If running inside this repo, `source .venv/bin/activate` first. Otherwise `pip install mlenergy-data` (or `uv pip install mlenergy-data`). + +```python +from mlenergy_data.records import LLMRuns, DiffusionRuns + +llm = LLMRuns.from_hf() # default: stable_only=True, ~few MB parquet only +diff = DiffusionRuns.from_hf() +``` + +Both classes also have `from_directory(root)` (compiled local copy) and `from_raw_results(*roots)` (re-parse raw `results.json` files). Default to `from_hf()` unless the user specifies a local directory. + +`from_hf()` only downloads the parquet summary. Raw files (per-request output lengths, per-token ITL, power timelines) auto-download on demand when you call `output_lengths()`, `inter_token_latencies()`, `timelines()`, or `read_results_json()`. **Always filter the collection before calling those methods** — the full raw dataset is ~100 GB. + +### If the user hits a gated-dataset error + +Recognize these signals from a `from_hf()` call (or any raw-data method that triggers a download): + +- A `RuntimeError` containing the phrase `"This is a gated dataset"` (the toolkit wraps HF errors with a friendly message). +- A `huggingface_hub.errors.GatedRepoError` or `RepositoryNotFoundError`. +- HTTP 401 / 403 in the traceback, or a message about an expired/missing token. +- The user reports they can't load anything but the toolkit is installed correctly. + +Don't just dump the traceback. Walk the user through the fix in plain language: + +1. **Request access to the dataset.** Open https://huggingface.co/datasets/ml-energy/benchmark-v3 in a browser and click "Request access". Approval is automatic and usually instant — the page should show "You have been granted access" within seconds. If the user isn't logged into Hugging Face, they'll need a free account first (https://huggingface.co/join). +2. **Create an access token.** Go to https://huggingface.co/settings/tokens and create a new token. A "Read" token is sufficient — no write permissions needed. Copy the token (it starts with `hf_…`). +3. **Export it in the shell.** `export HF_TOKEN=hf_…` for the current session, or add it to `~/.zshrc` / `~/.bashrc` / `.env` for persistence. The user must do this in the *same shell* that runs Python — exporting in one terminal does not affect another. +4. **Retry.** Re-run the `from_hf()` call. If it still fails, ask the user to (a) confirm the access page now shows "granted", (b) verify `echo $HF_TOKEN` prints a non-empty `hf_…` string in the same shell. + +If the user can't get access for any reason, stop and tell them — don't fabricate results. + +## The collection model + +`LLMRuns` and `DiffusionRuns` are **immutable, chainable, lazy filtered collections**. Every filter returns a new collection; you iterate to get individual frozen-dataclass records. + +```python +gpqa_b200 = llm.task("gpqa").gpu_model("B200") # chain freely +len(gpqa_b200) # count +gpqa_b200[0] # index +list(gpqa_b200) # materialize +gpqa_b200.to_dataframe() # one row per run +``` + +Iteration yields typed records — the IDE/autocomplete works. Use `min`/`max`/`sorted` directly with a `key=` lambda; do not write helper functions. + +## Filter cheat sheet + +| Method | LLMRuns | DiffusionRuns | Notes | +|---|---|---|---| +| `task(*names)` | yes | yes | OR within args | +| `gpu_model(*names)` | yes | yes | e.g. `"H100"`, `"B200"` | +| `num_gpus(*counts)` or `num_gpus(min=, max=)` | yes | yes | exact OR range, not both | +| `max_num_seqs(*sizes)` or `max_num_seqs(min=, max=)` | yes | — | LLM batch size | +| `batch_size(*sizes)` or `batch_size(min=, max=)` | — | yes | diffusion batch size | +| `precision(*values)` | yes | yes | `"bfloat16"`, `"fp8"`, `"mxfp4"` | +| `architecture(*values)` | yes | — | `"Dense Transformer"`, `"MoE"`, `"Mamba-Transformer Hybrid"` | +| `nickname(*names)` | yes | yes | human-friendly display name | +| `model_id(*ids)` | yes | yes | `"org/model"` HF identifier | +| `where(predicate)` | yes | yes | arbitrary lambda | +| `stable()` / `unstable()` | yes | — | LLM-only (see Stability) | +| `group_by(*fields)` | yes | yes | returns `{value: subcollection}` | +| `+` (concat) | yes | yes | union of two collections | + +Filters are AND across calls (`.task("gpqa").gpu_model("B200")` ⇒ both). Within a call, multiple positional args are OR (`.task("gpqa", "lm-arena-chat")`). Keyword `min=`/`max=` are inclusive bounds. + +## Field cheat sheet + +The full set of fields lives in the `LLMRun` and `DiffusionRun` docstrings (`mlenergy_data/records/runs.py`) and in `to_dataframe()`. The most common ones for analysis: + +**LLMRun (35 public fields, see [`references/DATA_CATALOG.md`](references/DATA_CATALOG.md) for the full list):** +- Identity: `task`, `model_id`, `nickname`, `architecture`, `weight_precision`, `gpu_model`, `num_gpus`, `max_num_seqs` +- Size: `total_params_billions`, `activated_params_billions` (equals total for dense; smaller for MoE) +- Energy/throughput: `energy_per_token_joules`, `energy_per_request_joules`, `output_throughput_tokens_per_sec`, `request_throughput_req_per_sec`, `avg_power_watts` +- Workload: `total_output_tokens`, `completed_requests`, `avg_output_len`, `avg_batch_size` (actual concurrent sequences observed in steady state — usually < `max_num_seqs`) +- Latency: `mean_itl_ms`, `median_itl_ms`, `p50_itl_ms`, `p90_itl_ms`, `p95_itl_ms`, `p99_itl_ms` (ITL = inter-token latency) +- Parallelism: `tensor_parallel`, `expert_parallel`, `data_parallel` +- Stability: `is_stable`, `unstable_reason` + +**DiffusionRun (22 public fields):** +- Identity: `task`, `model_id`, `nickname`, `weight_precision`, `gpu_model`, `num_gpus`, `batch_size` +- Generation params: `inference_steps`, `height`, `width`, `num_frames` (video only), `fps` (video only) +- Parallelism: `ulysses_degree`, `ring_degree`, `use_torch_compile` +- Energy: `energy_per_generation_joules` (generic — image OR video), `avg_power_watts`, `batch_latency_s`, `throughput_generations_per_sec` +- `is_text_to_image`, `is_text_to_video` are convenience properties. + +DiffusionRun has no `architecture` field and no stability metadata. There is no `energy_per_image_joules` / `energy_per_video_joules` — use `energy_per_generation_joules` and check `r.task` if you need a label. + +## Recipes for common questions + +These map natural-language asks to short, idiomatic code. For longer or trickier patterns, [`references/RECIPES.md`](references/RECIPES.md) covers: + +- **FP8 vs BF16 pairing of the same model** — strip the `" FP8"` nickname suffix, group on `(base, gpu, num_gpus, batch)`, compare energy/latency ratios. +- **Iso-latency comparison (B200 vs H100)** — for each (model, precision), find the lowest-energy config under a median-ITL deadline on each GPU, compute the gain. +- **Multi-GPU scaling: same batch, more GPUs** — count, across a task suite, how often adding GPUs at the same batch trades energy for latency. +- **Power timeline: median-aggregate across runs** — resample per-run timelines onto a common grid and take the median (the OpenG2G simulation pattern). +- **Output-length distribution per (model, task)** — leaderboard-style histograms with consistent bins. +- **Best-of-K with multiple constraints** — find the lowest-energy config that simultaneously meets median-ITL, p95-ITL, and throughput floors. +- **When a run isn't there** — handling empty subsets and the sparse `(model, gpu, num_gpus, task)` grid. + +Read RECIPES.md only if the user's question matches one of the patterns above. + +### "Which model has the lowest energy per token on GPQA?" +```python +best = min(llm.task("gpqa"), key=lambda r: r.energy_per_token_joules) +print(f"{best.nickname} on {best.num_gpus}x{best.gpu_model} batch={best.max_num_seqs}: " + f"{best.energy_per_token_joules:.3f} J/tok") +``` + +### "Best energy efficiency per GPU type, on GPQA" +```python +for gpu, group in llm.task("gpqa").group_by("gpu_model").items(): + best = min(group, key=lambda r: r.energy_per_token_joules) + print(f"{gpu}: {best.nickname} {best.num_gpus}x batch={best.max_num_seqs} " + f"-> {best.energy_per_token_joules:.4f} J/tok") +``` + +### "Compare best efficiency between B200 and H100 on a task" +```python +chat = llm.task("lm-arena-chat") +b = min(chat.gpu_model("B200"), key=lambda r: r.energy_per_token_joules) +h = min(chat.gpu_model("H100"), key=lambda r: r.energy_per_token_joules) +delta = (h.energy_per_token_joules - b.energy_per_token_joules) / h.energy_per_token_joules * 100 +print(f"B200: {b.nickname} {b.num_gpus}x batch={b.max_num_seqs} -> {b.energy_per_token_joules:.4f} J/tok") +print(f"H100: {h.nickname} {h.num_gpus}x batch={h.max_num_seqs} -> {h.energy_per_token_joules:.4f} J/tok") +print(f"B200 vs H100: {delta:+.1f}% (positive = B200 wins)") +``` + +### "Best per model, restricted to B200" +```python +for nick, group in llm.task("gpqa").gpu_model("B200").group_by("nickname").items(): + best = min(group, key=lambda r: r.energy_per_token_joules) + print(f"{nick}: {best.energy_per_token_joules:.4f} J/tok @ batch={best.max_num_seqs}") +``` + +### "Energy per token vs batch size for one model" +```python +runs = sorted( + llm.task("gpqa").nickname("DeepSeek R1").gpu_model("B200").num_gpus(8), + key=lambda r: r.max_num_seqs, +) +batches = [r.max_num_seqs for r in runs] +energies = [r.energy_per_token_joules for r in runs] +``` + +### "MoE vs dense at similar active parameters" +```python +gpqa = llm.task("gpqa").gpu_model("B200").precision("bfloat16") +for arch, group in gpqa.group_by("architecture").items(): + best = min(group, key=lambda r: r.energy_per_token_joules) + print(f"{arch}: {best.nickname} ({best.activated_params_billions:.0f}B active) " + f"@ {best.energy_per_token_joules:.3f} J/tok") +``` + +### "Find configs that meet a latency SLA, then minimize energy" +```python +median_itl_deadline_ms = 100 +candidates = llm.task("lm-arena-chat").nickname("Qwen 3 235B A22B Instruct FP8").gpu_model("B200") +ok = candidates.where(lambda r: r.median_itl_ms <= median_itl_deadline_ms) +best = min(ok, key=lambda r: r.energy_per_token_joules) if ok else None +``` + +### "Lowest energy per image (text-to-image) on B200" +```python +t2i = diff.task("text-to-image").gpu_model("B200") +best = min(t2i, key=lambda r: r.energy_per_generation_joules) +print(f"{best.nickname} {best.height}x{best.width} batch={best.batch_size}: " + f"{best.energy_per_generation_joules:.0f} J/image") +``` + +### "Average power timeline of a run" +```python +run = llm.task("gpqa").nickname("Qwen 3 8B").gpu_model("B200").num_gpus(1).max_num_seqs(64)[0] +tl = run.timelines(metric="power.device_instant") # auto-downloads raw on first call +# columns: timestamp, relative_time_s, value, metric (long-form, summed across GPUs) +``` + +`timelines()` accepts `metric ∈ {"power.device_instant", "power.device_average", "temperature"}`. For LLM runs, the timeline is restricted to the steady-state window. For diffusion runs, it returns the full run. + +### "Distribution of output lengths for a model" +```python +sub = llm.task("gpqa").nickname("DeepSeek R1").gpu_model("B200").num_gpus(8).max_num_seqs(128) +df = sub.output_lengths() # columns: task, model_id, num_gpus, max_num_seqs, output_len +df["output_len"].describe() +``` + +### "Per-token inter-token latency samples for distributional analysis" +```python +sub = llm.task("lm-arena-chat").nickname("Qwen 3 8B").gpu_model("B200").num_gpus(1).max_num_seqs(64) +df = sub.inter_token_latencies() # columns: ..., itl_s (seconds) +``` + +## Bulk raw data — when and how + +By default, only the parquet summary is downloaded (~few MB). The collection is enough to answer any aggregate-stat question. Reach for raw files only when: + +- **Power/temperature timeline** of a run → `run.timelines(metric=...)` or `runs.timelines(metric=...)`. +- **Per-request output lengths** → `run.output_lengths()` or `runs.output_lengths()`. +- **Per-token inter-token latencies** (for distributional analysis or percentiles beyond what's pre-computed) → `run.inter_token_latencies()` or `runs.inter_token_latencies()`. +- **Anything not exposed as a field** → `run.read_results_json()` returns the parsed `results.json` payload; `run.read_prometheus_json()` returns vLLM/Prometheus stats (LLM only). + +Auto-download is per-record and lazy — the first `timelines()` or `output_lengths()` call on a run hits the Hub once, then HF caches it locally (`HF_HOME`). To pre-warm in parallel: + +```python +sub = llm.task("gpqa").gpu_model("B200").download_raw_files() # both results + prometheus +sub = llm.task("gpqa").gpu_model("B200").download_raw_files(file="results") # results only +``` + +Always filter before `download_raw_files()` (or before bulk methods) — the unfiltered dataset is ~100 GB. + +## Units + +Mixing these up silently produces wrong answers — there's no type system to catch it. + +- **Energy**: joules (`*_joules`). +- **Power**: watts (`avg_power_watts`). +- **Duration**: seconds (`steady_state_duration_seconds`, `batch_latency_s`). +- **ITL percentile fields** (`mean_itl_ms`, `median_itl_ms`, `p50/p90/p95/p99_itl_ms`): **milliseconds**. +- **Raw ITL samples** from `run.inter_token_latencies()` and `runs.inter_token_latencies()`: **seconds**. +- **Throughput**: tokens/sec (LLM `output_throughput_tokens_per_sec`), requests/sec (LLM `request_throughput_req_per_sec`), generations/sec (diffusion `throughput_generations_per_sec`). + +When comparing a percentile against a deadline, the deadline must be in ms. + +## Critical gotchas + +[`references/PITFALLS.md`](references/PITFALLS.md) has the full list (API edge cases, ITL smoothing semantics for raw JSON, reproducibility notes). The ones that cause silent wrong answers — read these every time: + +1. **FP8 nickname suffix is inconsistent.** Most FP8 variants have `" FP8"` appended to the BF16 nickname (e.g. `"Qwen 3 235B A22B Instruct"` ↔ `"Qwen 3 235B A22B Instruct FP8"`). But `DeepSeek R1` and `DeepSeek V3.1` are **natively FP8** — the nickname has no suffix, and `weight_precision == "fp8"`. To enumerate FP8 vs BF16 pairs of the *same architecture*, key on the BF16 nickname stripped of `" FP8"` (see `_analyze_fp8_vs_bf16` in the blog script). + +2. **Architecture strings are literal.** Use `"Dense Transformer"` (not `"Dense"`), `"MoE"`, and `"Mamba-Transformer Hybrid"`. README/docs occasionally show `"Dense"` for brevity — that string does not match. + +3. **Stability is LLM-only.** `LLMRuns.from_hf()` defaults to `stable_only=True` and excludes runs that failed quality checks (low batch utilization, short steady state, cascade-from-unstable-batch). To inspect rejected runs, call `LLMRuns.from_hf(stable_only=False)` and use `.unstable()`. `DiffusionRuns` has no stability filtering — every parquet row is returned. + +4. **`max_num_seqs` is the configured cap, `avg_batch_size` is what was actually achieved** during steady state. They differ when client request load couldn't fill the configured batch. + +5. **Image-chat and video-chat are LLM tasks**, not diffusion. They use multimodal LLMs (`Qwen 3 VL`, `Llama 4 Scout`, etc.) and live in `LLMRuns`. + +6. **Diffusion has only `bfloat16`** in the current release. Filtering `diff.precision("fp8")` returns an empty collection. Diffusion has no `is_stable`, no `architecture`, and no `use_torch_compile=True` runs. + +7. **Bulk methods auto-download**, so an unfiltered `runs.timelines()` or `runs.download_raw_files()` will download ~100 GB. Always filter first. + +8. **Tasks are not interchangeable.** Each model is benchmarked on a specific subset of tasks (see [`references/DATA_CATALOG.md`](references/DATA_CATALOG.md)). DeepSeek R1 has only `gpqa`; Qwen 3 Coder has only `sourcegraph-fim`; etc. Always check coverage before requesting a comparison. + +9. **`energy_per_request_joules` and `request_throughput_req_per_sec` can be `None`** when the run had no completed requests or `avg_output_len` is missing. The identity `avg_output_len × energy_per_token_joules ≈ energy_per_request_joules` holds only when the right-hand side is non-null. Don't multiply blindly. + +10. **`group_by` keys are tuples for multi-field groupings.** `runs.group_by("model_id", "task")` returns `{(model_id, task): subcollection}`. + +11. **Power timeline `value` is summed across GPUs and windowed differently per domain.** `LLMRun.timelines()` slices to the steady-state window (post-warm-up); `DiffusionRun.timelines()` returns the full run. The `value` column is total power across all GPUs — divide by `r.num_gpus` for per-GPU, and don't compare LLM and diffusion timelines without accounting for the different windowing. + +## Working code-first + +When the user asks an analysis question, default to: + +1. Filter the collection (one chain — no intermediate variables unless they're reused). +2. Iterate / `min` / `max` / `sorted` with a `key=` lambda for the answer. +3. Print the answer in a single line that names the model, GPU, batch, and the metric value with units. +4. Reach for `to_dataframe()` only if the user wants a table or a multi-column summary; reach for raw bulk methods only if the answer truly requires per-request or per-token data. + +If the user asks for a plot, follow project plotting conventions (zero-anchored axes, deterministic SVG when applicable). The blog script ([`analysis/ml-energy-leaderboard-v3.0.py`](https://github.com/ml-energy/blog/blob/master/analysis/ml-energy-leaderboard-v3.0.py)) is the reference for production-quality plots. + +## Where to read more + +- **API reference** (auto-generated from docstrings): https://ml.energy/data/api/records/ and https://ml.energy/data/api/modeling/ +- **Guide** (progressive walkthrough): https://ml.energy/data/guide/ +- **Real-world examples** (study these for production patterns): + - Leaderboard JSON build: https://github.com/ml-energy/leaderboard/blob/master/scripts/build_data.py + - Blog figures (FP8 pairing, iso-latency, multi-GPU): https://github.com/ml-energy/blog/blob/master/analysis/ml-energy-leaderboard-v3.0.py + - OpenG2G simulation traces and ITL fits: https://github.com/gpu2grid/openg2g/blob/master/openg2g/datacenter/workloads/inference.py +- **Bundled references in this skill**: + - [`references/DATA_CATALOG.md`](references/DATA_CATALOG.md) — exhaustive enumeration of models, tasks, GPU coverage, sizes + - [`references/RECIPES.md`](references/RECIPES.md) — longer analysis recipes + - [`references/PITFALLS.md`](references/PITFALLS.md) — extended gotchas list diff --git a/skills/mlenergy-data/references/DATA_CATALOG.md b/skills/mlenergy-data/references/DATA_CATALOG.md new file mode 100644 index 0000000..972b8a8 --- /dev/null +++ b/skills/mlenergy-data/references/DATA_CATALOG.md @@ -0,0 +1,130 @@ +# Data Catalog + +Concrete enumeration of what's in `ml-energy/benchmark-v3` as of the current release. Use this as a lookup so you don't have to re-download the parquet to check which combinations exist. To verify any of these are still current, run `LLMRuns.from_hf()` / `DiffusionRuns.from_hf()` and group as needed. + +## Counts + +- LLM runs (stable): 838 (1138 total; ~300 marked unstable) +- Diffusion runs: 1020 + +## Tasks + +| Task | Domain | Description | +|---|---|---| +| `gpqa` | LLM | GPQA Diamond — long-chain reasoning | +| `lm-arena-chat` | LLM | LM Arena chat-style requests | +| `sourcegraph-fim` | LLM | Code fill-in-the-middle (Sourcegraph) | +| `image-chat` | LLM (multimodal) | Vision-language chat with image inputs | +| `video-chat` | LLM (multimodal) | Vision-language chat with video inputs | +| `text-to-image` | Diffusion | Text-to-image generation | +| `text-to-video` | Diffusion | Text-to-video generation | + +Display names used in the leaderboard JSON (only for human output, not for filtering): +GPQA → "GPQA Diamond", lm-arena-chat → "LLM Chat (LM Arena)", sourcegraph-fim → "Fill-in-the-Middle (Sourcegraph)", image-chat → "Image Chat", video-chat → "Video Chat", text-to-image → "Text to Image", text-to-video → "Text to Video". + +## GPUs and parallelism + +- `gpu_model`: `"H100"`, `"B200"`. +- `num_gpus`: 1, 2, 4, 8. +- LLM parallelism fields: `tensor_parallel`, `expert_parallel`, `data_parallel` (DP for attention + EP for MLP experts is the typical MoE setup). +- Diffusion parallelism: `ulysses_degree`, `ring_degree` (sequence parallelism). + +## LLM batch sizes (`max_num_seqs`) + +Observed values (powers of 2 plus a few midpoints): 8, 16, 32, 64, 96, 128, 192, 256, 384, 512, 768, 1024, 1536, 2048, 3072, 4096. Not every (model, GPU, num_gpus) cell covers the full range. + +## Diffusion batch sizes + +1, 2, 4, 8, 16, 32. + +## Precisions + +- LLM `weight_precision`: `"bfloat16"`, `"fp8"`, `"mxfp4"` (mxfp4 is GPT OSS only). +- Diffusion `weight_precision`: `"bfloat16"` only in this release. + +## Architectures (LLM) + +Three literal strings — match exactly: + +| Value | Examples | +|---|---| +| `"Dense Transformer"` | Llama 3.1 8B/70B/405B Instruct, Qwen 3 8B/14B/32B, Gemma 3 12B/27B, Qwen 3 VL 8B/32B Instruct | +| `"MoE"` | Qwen 3 30B/235B (and Coder 30B/480B), GPT OSS 20B/120B, DeepSeek R1/V3.1, Llama 4 Maverick/Scout, Qwen 3 Omni 30B A3B, Qwen 3 VL 30B/235B Instruct | +| `"Mamba-Transformer Hybrid"` | NVIDIA Nemotron Nano 9B V2, 12B V2, 12B V2 VL | + +## LLM models (per-model coverage) + +Format: `nickname` — `model_id` — `task list` — `(GPU, num_gpus)` cells. Total params (B) and active params (B) listed for MoE/Hybrid where they differ from total. + +### Dense Transformer + +- **Gemma 3 12B** — google/gemma-3-12b-it — image-chat, lm-arena-chat — (B200,1), (H100,1) +- **Gemma 3 27B** — google/gemma-3-27b-it — image-chat, lm-arena-chat — (B200,1), (H100,1), (H100,2) +- **Llama 3.1 8B Instruct** — meta-llama/Llama-3.1-8B-Instruct — lm-arena-chat — (B200,1), (H100,1) +- **Llama 3.1 70B Instruct** — meta-llama/Llama-3.1-70B-Instruct — lm-arena-chat — (B200,2), (H100,4) +- **Llama 3.3 70B Instruct** — meta-llama/Llama-3.3-70B-Instruct — lm-arena-chat — (B200,2), (H100,4) +- **Llama 3.1 405B Instruct** (BF16, 405B) — meta-llama/Llama-3.1-405B-Instruct — lm-arena-chat — (B200,8) +- **Llama 3.1 405B Instruct FP8** — meta-llama/Llama-3.1-405B-Instruct-FP8 — lm-arena-chat — (B200,4), (H100,8) +- **Qwen 3 8B** — Qwen/Qwen3-8B — gpqa, lm-arena-chat — (B200,1), (H100,1) +- **Qwen 3 14B** — Qwen/Qwen3-14B — gpqa, lm-arena-chat — (B200,1), (H100,1) +- **Qwen 3 32B** — Qwen/Qwen3-32B — gpqa, lm-arena-chat — (B200,1), (H100,1), (H100,2) +- **Qwen 3 VL 8B Instruct** — Qwen/Qwen3-VL-8B-Instruct — image-chat, video-chat — (B200,1), (H100,1) +- **Qwen 3 VL 32B Instruct** — Qwen/Qwen3-VL-32B-Instruct — image-chat, video-chat — (B200,1), (H100,2) + +### MoE (total / active) + +- **Qwen 3 30B A3B Instruct** — Qwen/Qwen3-30B-A3B-Instruct-2507 — lm-arena-chat — (30B/3B) — (B200,1), (H100,1), (H100,2) +- **Qwen 3 30B A3B Thinking** — Qwen/Qwen3-30B-A3B-Thinking-2507 — gpqa — (30B/3B) — (B200,1), (H100,1), (H100,2) +- **Qwen 3 Coder 30B A3B** — Qwen/Qwen3-Coder-30B-A3B-Instruct — sourcegraph-fim — (30B/3B) — (B200,1), (H100,1), (H100,2) +- **Qwen 3 Omni 30B A3B Instruct** — Qwen/Qwen3-Omni-30B-A3B-Instruct — image-chat, video-chat — (30B/3B) — (B200,1), (H100,1), (H100,2) +- **Qwen 3 VL 30B A3B Instruct** — Qwen/Qwen3-VL-30B-A3B-Instruct — image-chat, video-chat — (30B/3B) — (B200,1), (H100,1), (H100,2) +- **Qwen 3 235B A22B Instruct** — Qwen/Qwen3-235B-A22B-Instruct-2507 — lm-arena-chat — (235B/22B) — (B200,4), (B200,8), (H100,8) +- **Qwen 3 235B A22B Instruct FP8** — Qwen/Qwen3-235B-A22B-Instruct-2507-FP8 — lm-arena-chat — (235B/22B) — (B200,2), (H100,4), (H100,8) +- **Qwen 3 235B A22B Thinking** — Qwen/Qwen3-235B-A22B-Thinking-2507 — gpqa — (235B/22B) — (B200,4), (B200,8), (H100,8) +- **Qwen 3 235B A22B Thinking FP8** — Qwen/Qwen3-235B-A22B-Thinking-2507-FP8 — gpqa — (235B/22B) — (B200,2), (B200,4), (H100,4), (H100,8) +- **Qwen 3 VL 235B A22B Instruct** — Qwen/Qwen3-VL-235B-A22B-Instruct — image-chat, video-chat — (235B/22B) — (B200,4), (B200,8), (H100,8) +- **Qwen 3 VL 235B A22B Instruct FP8** — Qwen/Qwen3-VL-235B-A22B-Instruct-FP8 — image-chat, video-chat — (235B/22B) — (B200,2), (B200,4), (H100,4), (H100,8) +- **Qwen 3 Coder 480B A35B** — Qwen/Qwen3-Coder-480B-A35B-Instruct — sourcegraph-fim — (480B/35B) — (B200,8) +- **Qwen 3 Coder 480B A35B FP8** — Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8 — sourcegraph-fim — (480B/35B) — (B200,4), (B200,8) +- **GPT OSS 20B** — openai/gpt-oss-20b — gpqa — (21B/4B), MXFP4 capable — (B200,1), (H100,1) +- **GPT OSS 120B** — openai/gpt-oss-120b — gpqa — (117B/5B), MXFP4 capable — (B200,1), (B200,2), (H100,1), (H100,2) +- **DeepSeek R1** — deepseek-ai/DeepSeek-R1-0528 — gpqa — (671B/37B), **natively FP8** (no `" FP8"` suffix) — (B200,8) +- **DeepSeek V3.1** — deepseek-ai/DeepSeek-V3.1 — gpqa, lm-arena-chat — (671B/37B), **natively FP8** — (B200,8) +- **Llama 4 Maverick 17B 128E Instruct FP8** — meta-llama/Llama-4-Maverick-17B-128E-Instruct-FP8 — image-chat, lm-arena-chat — (400B/17B) — (H100,8) +- **Llama 4 Scout 17B 16E Instruct** — meta-llama/Llama-4-Scout-17B-16E-Instruct — image-chat, lm-arena-chat — (109B/17B) — (H100,4), (H100,8) + +### Mamba-Transformer Hybrid + +- **NVIDIA Nemotron Nano 9B V2** — nvidia/NVIDIA-Nemotron-Nano-9B-v2 — gpqa, lm-arena-chat — (B200,1), (H100,1) +- **NVIDIA Nemotron Nano 12B V2** — nvidia/NVIDIA-Nemotron-Nano-12B-v2 — gpqa, lm-arena-chat — (B200,1), (H100,1) +- **NVIDIA Nemotron Nano 12B V2 VL** — nvidia/NVIDIA-Nemotron-Nano-12B-v2-VL — image-chat — (H100,1) + +## Diffusion models + +| Nickname | model_id | Task | Output size | inference_steps | Notes | +|---|---|---|---|---|---| +| FLUX.1 Dev | black-forest-labs/FLUX.1-dev | text-to-image | 1024×1024 | 50 | | +| Hunyuan-DiT 1.2 | Tencent-Hunyuan/HunyuanDiT-v1.2 | text-to-image | 1024×1024 | 50 | | +| PixArt-Σ | PixArt-alpha/PixArt-Sigma-XL-2-1024-MS | text-to-image | 1024×1024 | 20 | | +| SANA 1.5 1.6B | Efficient-Large-Model/SANA1.5_1.6B_1024px_diffusers | text-to-image | 1024×1024 | 20 | | +| SANA 1.5 4.8B | Efficient-Large-Model/SANA1.5_4.8B_1024px_diffusers | text-to-image | 1024×1024 | 20 | | +| Stable Diffusion 3.5 Medium | stabilityai/stable-diffusion-3.5-medium | text-to-image | 1024×1024 | 28 | | +| Stable Diffusion 3.5 Large | stabilityai/stable-diffusion-3.5-large | text-to-image | 1024×1024 | 28 | | +| CogVideoX 2B | THUDM/CogVideoX-2b | text-to-video | 480×720, 49 frames @ 8 fps | 50 | | +| CogVideoX 1.5 5B | THUDM/CogVideoX1.5-5B | text-to-video | 768×1360, 81 frames @ 8 fps | 50 | | +| Wan 2.1 1.3B | Wan-AI/Wan2.1-T2V-1.3B-Diffusers | text-to-video | 480×832, 81 frames @ 15 fps | 50 | | +| Wan 2.1 14B | Wan-AI/Wan2.1-T2V-14B-Diffusers | text-to-video | 480×832, 81 frames @ 15 fps | 50 | | +| HunyuanVideo | tencent/HunyuanVideo | text-to-video | 720×1280, 129 frames @ 15 fps | 50 | | + +All diffusion runs in this release are `bfloat16`, `use_torch_compile=False`. Diffusion has run-count coverage on (B200, 1/2/4/8) and (H100, 1/2/4/8) for most models — the multi-GPU variants test sequence-parallel (Ulysses + Ring). + +## Stability (LLM only) + +A run is marked unstable if any of: + +- `steady_state_duration < 20s` +- `energy_per_token_joules` missing or non-positive +- `avg_batch_size / max_num_seqs < 0.85` ("low_batch_utilization") +- A smaller batch in the same (model, task, GPU, num_gpus) group was unstable ("cascade_from_unstable_batch:N") + +`stable_only=True` (the default for `LLMRuns.from_hf()`) excludes them. Use `LLMRuns.from_hf(stable_only=False)` then `.unstable()` to inspect. diff --git a/skills/mlenergy-data/references/PITFALLS.md b/skills/mlenergy-data/references/PITFALLS.md new file mode 100644 index 0000000..75ddf1e --- /dev/null +++ b/skills/mlenergy-data/references/PITFALLS.md @@ -0,0 +1,109 @@ +# Extended Pitfalls + +The most critical pitfalls — the ones that cause silent wrong answers — are inlined in [`SKILL.md`](../SKILL.md) ("Critical gotchas" and "Units"). This file holds the longer-tail items: API edge cases, raw-JSON details, and reproducibility notes. + +## API edge cases + +### `num_gpus` / `max_num_seqs` / `batch_size` filters cannot mix exact + range + +```python +runs.num_gpus(1, 2) # exact OR +runs.num_gpus(min=2, max=8) # inclusive range +runs.num_gpus(1, min=2) # raises ValueError +runs.num_gpus() # raises TypeError (no args) +``` + +### `unstable()` requires a non-stable-only collection + +```python +LLMRuns.from_hf().unstable() # raises ValueError +LLMRuns.from_hf(stable_only=False).unstable() # works +``` + +### `group_by` key shape depends on arg count + +- One field: keys are scalars. `group_by("task")` → `{"gpqa": LLMRuns(...), ...}`. +- Multiple fields: keys are tuples. `group_by("model_id", "task")` → `{("Qwen/...", "gpqa"): LLMRuns(...), ...}`. + +### `to_dataframe()` strips private fields + +Path fields (`_results_path`, `_prometheus_path`) and HF metadata (`_hf_repo_id`, etc.) are excluded. Use `r.read_results_json()` or `r.read_prometheus_json()` to access raw payloads — don't try to read paths from a `to_dataframe()` row. + +### `download_raw_files()` returns the same collection + +It's an in-place pre-warm — chain it for ergonomics; the return value is the same `LLMRuns`/`DiffusionRuns` object. The optional `file=` arg can be `"results"` or `"prometheus"` for LLM; default is both. For diffusion, only `results.json` exists. + +### Cache lives in `HF_HOME` + +Repeated calls after the first download are instant. To force a re-download, clear the HF cache or pass `revision=` to `from_hf()`. + +## Raw JSON details + +### Inter-token latencies in raw `results.json` are not smoothed + +For chunked-prefill engines (vLLM), some ITL entries are zero — the engine batched multiple decode tokens into one step. The toolkit's `inter_token_latencies()` and the `*_itl_ms` percentile fields **already handle this** by spreading the accumulated latency across the covered tokens (`_smooth_chunked_itl` in `runs.py`). + +Only an issue if you read raw `results.json` directly via `run.read_results_json()` — then you'll see a long zero spike at the front. Use the toolkit's accessors unless you specifically want raw values. + +### Per-device timeline columns + +`run.timelines()` exposes only the GPU-summed `value`. Per-device samples are present in raw `results.json` under `timeline.power.device_instant.` if needed. The internal helper `mlenergy_data.records.timelines.extract_device_timeline` returns wide-form per-device columns plus the total — use it directly if you're doing per-GPU analysis. + +### Stability cascade rule + +A run is unstable if any of: `steady_state_duration < 20s`, `energy_per_token_joules <= 0`, `avg_batch_size / max_num_seqs < 0.85` ("low_batch_utilization"), or a smaller batch in the same `(model, task, GPU, num_gpus)` group was unstable ("cascade_from_unstable_batch:N"). The cascade reflects the assumption that smaller batches are the safe baseline; if the smallest batch is unstable, larger batches don't get a free pass. + +To diagnose: + +```python +all_llm = LLMRuns.from_hf(stable_only=False) +{r.unstable_reason.split(":")[0] for r in all_llm.unstable()} +``` + +In the current release: ~95% of unstable runs are `low_batch_utilization`, the remainder are `cascade_from_unstable_batch` and `short_steady_state`. + +## Coverage + +### (model, GPU, num_gpus, task) cells are sparse + +The catalog ([`DATA_CATALOG.md`](DATA_CATALOG.md)) is the authoritative list. When a subset comes up empty for a comparison, prefer to skip and tell the user, rather than fabricate by widening filters. + +### Across-the-board comparison: intersect available tasks first + +```python +candidates = { + nick for nick, group in llm.group_by("nickname").items() + if {"gpqa", "lm-arena-chat"} <= {r.task for r in group} +} +``` + +## Field semantics + +### Diffusion's single `energy_per_generation_joules` + +The toolkit uses one generic field for both image and video runs. Check `r.is_text_to_image` / `r.is_text_to_video` (or `r.task`) to decide labels and which size fields are populated. `num_frames` and `fps` are `None` for image runs. The leaderboard-build script splits this into `energy_per_image_joules` / `energy_per_video_joules` in its JSON output, but that's a downstream rename, not a toolkit field. + +## `mlenergy_data.modeling` is not recommended for general use + +The toolkit exports `LogisticModel` (4-parameter sigmoid for batch sweeps) and `ITLMixtureModel` (two-component lognormal for inter-token latency). Both classes work, but their fit quality is unreliable enough that the skill does not surface them. Concretely, on the v3 dataset: + +- **`LogisticModel`** on `energy_per_token_joules`, `avg_power_watts`, `median_itl_ms`, `output_throughput_tokens_per_sec` vs `log2(batch_size)`: median R² ≥ 0.98 across (model, task, gpu, num_gpus) groups, but **22%–38% of groups have max relative residuals above 15%, with worst cases over 100%**. There's no in-band signal that tells you whether your fit is in the good majority or the bad tail. +- **`ITLMixtureModel`**: median empirical-mean recovery is 2%, but **p90 is 172% and the worst case is 1430%** (model predicts ~15× the true mean). Vision-language models on `image-chat` and `video-chat` are the consistent failure mode — the lognormal "stall" component fits a heavy tail that blows up the analytical mean. + +If a user explicitly asks for these fits, use them — but always validate by: +- For `LogisticModel`: compute residuals against the actual data points (`fit.eval(batch=b)` vs the measured value at each `b` in your sweep), and reject the fit if `max(|residual / y|) > 0.15` or any held-out point disagrees by more than ~10%. +- For `ITLMixtureModel`: compare `fit.mean_var()` to `np.mean(samples_s), np.std(samples_s)` and reject if either differs by more than ~10–20%. KS-test the model's `sample_one` draws against the empirical distribution. + +For most analyses, prefer interpolating directly between measured batch sizes (the dataset is dense) over fitting a parametric model. + +## Reproducibility + +The `seed` and `num_request_repeats` fields are populated for LLM runs. The toolkit doesn't aggregate across seeds — multiple seeds appear as multiple `LLMRun` rows sharing `(task, model_id, gpu_model, num_gpus, max_num_seqs)`. To measure variability across seeds, group by everything except `seed` and look at the spread of the metric of interest: + +```python +keys = ("task", "model_id", "gpu_model", "num_gpus", "max_num_seqs", "weight_precision") +for k, group in llm.group_by(*keys).items(): + if len(group) > 1: + es = [r.energy_per_token_joules for r in group] + print(f"{k}: n={len(es)} energy spread {min(es):.4f}..{max(es):.4f}") +``` diff --git a/skills/mlenergy-data/references/RECIPES.md b/skills/mlenergy-data/references/RECIPES.md new file mode 100644 index 0000000..e6dc694 --- /dev/null +++ b/skills/mlenergy-data/references/RECIPES.md @@ -0,0 +1,163 @@ +# Recipes + +Longer, idiomatic patterns drawn from real production code (the leaderboard, blog, and OpenG2G use cases). Read these when the user asks for an analysis that goes beyond a single filter+min. + +## FP8 vs BF16 pairing of the same model + +Most FP8 variants append `" FP8"` to the BF16 nickname. Pair them by stripping the suffix; key on the (base nickname, gpu, num_gpus, batch). Skip DeepSeek R1/V3.1 — they are natively FP8 and have no BF16 sibling. + +```python +from collections import defaultdict +from mlenergy_data.records import LLMRuns + +llm = LLMRuns.from_hf() + +groups: dict[tuple, dict[str, "LLMRun"]] = defaultdict(dict) +for r in llm.task("lm-arena-chat"): + base = r.nickname.removesuffix(" FP8") + key = (base, r.gpu_model, r.num_gpus, r.max_num_seqs) + groups[key][r.weight_precision] = r + +for (base, gpu, n_gpus, batch), prec_runs in groups.items(): + if "fp8" in prec_runs and "bfloat16" in prec_runs: + fp8, bf16 = prec_runs["fp8"], prec_runs["bfloat16"] + ratio = fp8.energy_per_token_joules / bf16.energy_per_token_joules + print(f"{base} on {n_gpus}x{gpu} batch={batch}: FP8 / BF16 = {ratio:.2f}") +``` + +## Iso-latency comparison (B200 vs H100) + +For each (model, precision) that has runs on both GPUs, find the lowest-energy config that meets a median-ITL deadline. + +```python +def best_for_deadline(runs, model, gpu, precision, deadline_ms, latency_attr, energy_attr): + matching = [ + r for r in runs + if r.nickname == model and r.gpu_model == gpu + and r.weight_precision == precision + and getattr(r, latency_attr) <= deadline_ms + ] + return min(matching, key=lambda r: getattr(r, energy_attr)) if matching else None + +llm = LLMRuns.from_hf() +gpqa = llm.task("gpqa") + +deadline_ms = 100 +for model, prec in sorted({(r.nickname, r.weight_precision) for r in gpqa}): + h = best_for_deadline(gpqa, model, "H100", prec, deadline_ms, "median_itl_ms", "energy_per_token_joules") + b = best_for_deadline(gpqa, model, "B200", prec, deadline_ms, "median_itl_ms", "energy_per_token_joules") + if h and b: + gain = (h.energy_per_token_joules - b.energy_per_token_joules) / h.energy_per_token_joules * 100 + print(f"{model:40s} {prec:8s} H100={h.energy_per_token_joules:.3f} B200={b.energy_per_token_joules:.3f} gain={gain:+.0f}%") +``` + +For diffusion, swap to `r.batch_latency_s` (latency) and `r.energy_per_generation_joules` (energy). + +## Multi-GPU scaling: same batch, more GPUs + +Across all (model, gpu_type) groups, count how often adding GPUs *at the same batch size* trades energy for latency. + +```python +from collections import defaultdict + +llm = LLMRuns.from_hf().task("gpqa", "lm-arena-chat", "sourcegraph-fim") +groups: dict[tuple[str, str], dict[int, list]] = defaultdict(lambda: defaultdict(list)) +for r in llm: + groups[(r.nickname, r.gpu_model)][r.num_gpus].append(r) + +for gpu in ("B200", "H100"): + e_up = lat_down = total = 0 + for (nick, g), per_count in groups.items(): + if g != gpu: + continue + counts = sorted(per_count.keys()) + for i, n in enumerate(counts): + for m in counts[i + 1:]: + a = {r.max_num_seqs: r for r in per_count[n]} + b = {r.max_num_seqs: r for r in per_count[m]} + for batch in set(a) & set(b): + total += 1 + if b[batch].energy_per_token_joules >= a[batch].energy_per_token_joules: + e_up += 1 + if b[batch].median_itl_ms < a[batch].median_itl_ms: + lat_down += 1 + print(f"{gpu}: more GPUs at same batch -> energy up {e_up}/{total}, latency down {lat_down}/{total}") +``` + +## Power timeline: median-aggregate across runs + +Used by OpenG2G to build a representative per-GPU power trace per (model, batch_size, num_gpus). Resample each run to a common grid, then take the median. + +```python +import numpy as np +import pandas as pd + +sub = ( + llm.task("lm-arena-chat") + .nickname("Llama 3.1 8B Instruct") + .gpu_model("H100") + .num_gpus(1) + .max_num_seqs(64) + .download_raw_files(file="results") +) + +# Concatenate timelines from each run +frames = [] +for i, run in enumerate(sub): + tl = run.timelines(metric="power.device_instant") + tl["run_index"] = i + frames.append(tl) +all_tl = pd.concat(frames, ignore_index=True) + +dt_s = 0.1 +t_end = float(np.median([all_tl[all_tl.run_index == i].relative_time_s.max() for i in range(len(sub))])) +grid = np.arange(0.0, t_end + 1e-12, dt_s) +mat = np.vstack([ + np.interp(grid, g.relative_time_s.to_numpy(), g.value.to_numpy()) + for _, g in all_tl.groupby("run_index", sort=True) +]) +median_trace_w = np.median(mat, axis=0) # per-timestep median power across runs (total across GPUs) +per_gpu_w = median_trace_w / sub[0].num_gpus +``` + +The full version with multi-batch grouping lives at `_build_trace_store_from_timelines` in OpenG2G's `inference.py`. + +## Output-length distribution per (model, task) + +For leaderboard-style histograms with consistent bins across configurations: + +```python +import numpy as np + +sub = llm.task("gpqa").nickname("DeepSeek R1").download_raw_files(file="results") +all_lengths = [] +per_run = [] +for r in sub: + lens = np.array(r.output_lengths(), dtype=int) + per_run.append(lens) + all_lengths.extend(lens.tolist()) + +_, bin_edges = np.histogram(all_lengths, bins=50) +agg_counts, _ = np.histogram(all_lengths, bins=bin_edges) +per_run_counts = [np.histogram(lens, bins=bin_edges)[0] for lens in per_run] +``` + +## Best-of-K with multiple constraints + +Find the lowest energy-per-token config that simultaneously meets: +- median ITL deadline +- p95 ITL deadline +- minimum throughput floor + +```python +ok = llm.task("lm-arena-chat").nickname("Qwen 3 32B").gpu_model("B200").where( + lambda r: r.median_itl_ms <= 80 + and r.p95_itl_ms <= 250 + and r.output_throughput_tokens_per_sec >= 800 +) +best = min(ok, key=lambda r: r.energy_per_token_joules) if ok else None +``` + +## When a run isn't there + +If `min(empty_collection, key=...)` raises `ValueError`, that's a sign the (model, GPU, num_gpus, task) cell isn't populated. Check `len(sub)` before reducing, or wrap with `if sub:`. The catalog ([`DATA_CATALOG.md`](DATA_CATALOG.md)) lists what cells exist.