Tiansheng HuN, Yichuan WangB, James V. TalwarR, Nicholas TomlinT, Chen ZhaoNY, Sewon MinB
NNYU Shanghai BUniversity of California, Berkeley RRational Dynamics TToyota Technological Institute at Chicago NYNew York University
What it is · How it works · Evaluate · Data · Training · Citation
git clone https://github.com/StarTrail-org/DRPixel && cd DRPixel/eval
pip install -e '.[client]'
export MODEL_NAME=gpt-4o MODEL_BASE_URL=https://api.openai.com/v1 MODEL_API_KEY=sk-...
export SEARCH_URL=... # the search endpoint, see eval/README.md
export JUDGE_API_KEY=sk-... # for the gpt-5.4-mini judge
python -m drbench.doctor # about 20 s: checks the model, the search endpoint and image input
python -m drbench.eval --config Page # runs and grades all 500 questionsDeep research agents usually work on documents only after OCR has turned them into text, and that conversion loses or corrupts much of what matters in a scientific paper: tables, figures and equations. DR Pixel is a page-native deep research agent. It renders each paper page by page, retrieves page images with a vision-language embedding model, and reasons over the images directly, so the text and visual content of a page are retrieved and read together.
This repository contains the code for the paper:
eval/ |
the evaluation harness for DRPixelBench, 500 questions over 1.06M pages from 99,055 scientific papers. Point it at any OpenAI-compatible model and one command runs and grades it. |
train/ |
data preparation and training for counterfactual credit reweighting, the SFT recipe behind the trained DR Pixel model. |
The agent runs a ReAct loop. At each turn it writes a thought and then either searches or gives its final answer. Search results stay in context, and each question allows at most 20 searches. The settings differ only in what the search tool returns. DR Pixel searches 1,062,153 page images, rendered at 150 dpi and embedded with Qwen3-VL-Embedding-8B, and gets back the top three pages. The OCR-based baselines in the paper search the same papers after parsing them into text:
--config |
what the agent searches | needs vision |
|---|---|---|
Page |
whole-page images, top 3 (DR Pixel) | yes |
Text |
256-word OCR chunks, top 5, embedded with Qwen3-Embedding-8B | no |
Text_Fig |
the same chunks, plus a second tool that searches cropped figures | yes |
To train an open-weight model, we fine-tune Qwen3.6-27B on GPT-5.4 trajectories. Standard SFT imitates every teacher step, including repeated or low-value searches, and the student learns to search too much. Counterfactual credit reweighting instead scores each reasoning and search step by how much it changes the frozen base model's likelihood of the gold answer, turns these scores into per-step loss weights, and trains on the weighted data with ordinary SFT.
Any endpoint that speaks the OpenAI chat-completions API works, hosted or local. For any other API,
subclass Client and override complete() in eval/drbench/model.py,
which is the only code that talks to the model.
eval/README.md covers setup, the search server, grading, the output files and
reproducibility. Keep two conventions in mind when comparing numbers:
- Every question is grounded in exactly one gold page. PgHit is the share of questions whose gold
page was retrieved; under
Textit is not measurable, because a text chunk carries no page number. - Acc|Pg is accuracy on the questions whose gold page was retrieved, and Tool/q counts searches only, not the final answer.
Everything is in one HuggingFace dataset,
HughieHu/drpixel-bench:
benchmark/ |
the 500 questions |
corpus/ |
the list of all 99,055 papers, their OCR text, and all 1,062,153 page images (473 GB), rendered exactly as the agent sees them |
index/ |
the page and passage embedding indices (8.7 GB and 36 GB) |
The corpus combines papers we collected from arXiv and open-access conference proceedings with the
papers of AirQA, IRPapers and M3SciQA, so that all four benchmarks share one corpus. The 13,946
papers that come from those benchmarks remain under their original releases. We do not
redistribute PDFs; corpus/papers.jsonl gives an arXiv id for 76.3% of the corpus.
The trained model is released as
HughieHu/DR-Pixel-Trained, and its training set,
2,496 GPT-5.4 trajectories with their per-step weights and raw credits, in the train/ directory of
HughieHu/drpixel-bench.
train/README.md shows how to train on it and describes the data-preparation
steps, from scoring each step's credit to writing the training file, the small ms-swift plugin that
applies the weights, and the training recipe.
@misc{hu2026drpixel,
title = {DR Pixel: Page-Native Deep Research with Vision-Language Models},
author = {Hu, Tiansheng and Wang, Yichuan and Talwar, James V. and Tomlin, Nicholas and Zhao, Chen and Min, Sewon},
year = {2026}
}The code is released under Apache-2.0. For the data, see the dataset cards on HuggingFace; papers from the external benchmarks remain under the terms of their original releases.