An evidence-first daily radar for newly released AI benchmarks, evaluation methods, datasets, leaderboards, and data-quality work.
Every day, GitHub Actions queries primary or structured sources, deduplicates records, classifies them with a transparent taxonomy, ranks them using explainable signals, and publishes a GitHub Issue. It is inspired by agents-radar, with sources and scoring redesigned for benchmark and AI-data research.
- New AI/LLM benchmarks and challenge sets
- Evaluation frameworks, judge models, safety/capability evals, and leaderboards
- Public AI datasets, preference data, synthetic data, and data releases
- Data contamination, leakage, provenance, deduplication, and annotation-quality work
Default sources:
| Source | Required secret | Role |
|---|---|---|
| arXiv | No | Primary paper discovery |
| Hugging Face Hub | No | Dataset repository discovery |
| GitHub | No in Actions | Code and artifact discovery |
| OpenAlex | OPENALEX_API_KEY |
Scholarly metadata enrichment |
| Brave Search | BRAVE_API_KEY |
Web and lab-blog discovery |
The report remains useful without optional secrets. Missing optional sources are shown as warnings in the source-health table instead of being silently ignored.
Each item receives four visible scores:
- Relevance: matches against benchmark, evaluation, dataset, and data-quality taxonomy
- Evidence: primary/structured source, authorship, and cross-source artifact evidence
- Recency: time since publication or material update
- Adoption: logarithmically scaled stars, downloads, likes, or citations
The default priority is:
0.40 relevance + 0.25 evidence + 0.20 recency + 0.15 adoption
This is triage, not scientific quality adjudication or endorsement.
python3 -m venv .venv
source .venv/bin/activate
python -m pip install -e '.[dev]'
benchmark-radarOutputs:
out/report.md: the exact GitHub Issue bodyout/items.json: machine-readable evidence and source-health snapshot
Run checks:
ruff check .
pytest -qEdit config.yml to change the lookback, threshold, queries, taxonomy, and
report size. Copy .env.example to .env only for local use; never commit credentials.
Optional repository secrets:
OPENALEX_API_KEY
BRAVE_API_KEY
The built-in GITHUB_TOKEN is used automatically in Actions.
.github/workflows/daily-radar.yml runs at 12:15 UTC and can also be started manually.
It:
- collects and renders with read-only repository permission;
- uploads the Markdown and JSON evidence for 30 days;
- creates or updates the issue for that UTC date;
- prevents duplicate daily issues by exact-title lookup.
The workflow needs repository Issues enabled. The labels daily-radar and automated
must exist; they are created during initial repository setup.
- Every entry links to its discovered primary or structured record.
- Optional-source failures are visible.
- Reports can contain false positives; always inspect the source.
- A repository update is not necessarily a new release.
- Publication dates differ across preprints, code, datasets, and formal publications.
- This system does not automatically create ANX-Bench events or research claims.
MIT