[Leaderboard] Permute Core - Claude Opus 5 - 83.30% Pass@1

[ericmillsio]

Permute Core - Leaderboard Submission

Agent name: Permute Core
Team: Permute (permute.ai)
Contributors: @cnav01 and @ericmillsio
Backbone LLM: Claude Opus 5 (max reasoning effort)
Hints: Yes
Tuned prompt: Yes
Trials: 5 per query (270 total)
Stratified Pass@1: 83.30%

Architecture

Permute Core is a single-agent analytical executor operating over the datasets provided for each task.

The system normalizes each sanctioned data source into Parquet-backed tables and provides the agent with a consistent DuckDB query interface. Access is read-only and restricted to the stores sanctioned for each DAB dataset.

We’ve created an evidence layer that pre-processes each dataset using ML and algorithmic techniques looking to identify relevant tables, columns, relationships, predicates, and analytical operations. Relevant evidence is filtered at runtime for each query using deterministic lexical search and evidence-graph scoring.

• duckdb_query: queries the task data with read-only SQL.
• python_analysis: provides restricted Python analysis over query results, with external I/O disabled.
• semantic_probe: provides deterministic data-profiling and normalization routines for common analytical ambiguities.
• select_result_candidate: enforces result provenance, conflict resolution, and completion requirements (agent still produces final response).

We use a stable system prompt across all queries. For each query, our system automatically retrieves and filters the relevant evidence at runtime using the same evidence-selection pipeline; the payload is not manually curated per question. Because this query-specific evidence becomes part of the agent’s context, we conservatively marked Tuned Prompt: Yes.

Results

Dataset	Pass@1
bookreview	1.00
googlelocal	1.00
music_brainz_20k	1.00
PATENTS	1.00
stockindex	1.00
yelp	1.00
crmarenapro	0.97
stockmarket	0.96
PANCANCER_ATLAS	0.67
DEPS_DEV_V1	0.50
GITHUB_REPOS	0.50
agnews	0.40
Stratified Pass@1	0.8330

Notes

• 270 entries: 54 queries × 5 trials.
• All tool calls restricted to sanctioned data sources only
• External I/O is blocked by the SQL and Python tool policies. Web and external-fetch tools are disabled.
• All answers were produced from the sanctioned task databases.
• Each trial uses a fresh agent session, and validator results are produced only after completion and are never exposed to the agent.
• The task-specific evidence configuration was reviewed and does not provide benchmark ground-truth answers or validator output.
• Results were evaluated against the official per-query validators at benchmark commit 9a55e34271a2500a41cbc009e239edd94599e374.
• Submission answers match their corresponding final trace responses in all 270/270 trials.
• Trace audit found 0 attempts to read validate.py, ground-truth files, benchmark answer artifacts, prior-run results, or other submissions.
• Trace audit found 0 external dataset or web-fetch attempts.
• Submission file: leaderboard_submissions/permute_core.json
• Execution traces: permute_core_traces.zip