Elicit paper search evaluation on BioASQ

This repository reproduces the paper-search recall evaluation reported in Elicit's Search API launch materials. It contains the frozen BioASQ inputs, the Elicit outputs used for the reported result, a deterministic scorer, and a small runner for supported search APIs.

The checked-in result reproduces Elicit's macro recall of 0.390, 0.483, 0.603, 0.670, and 0.708 at 10, 20, 50, 100, and 200 results.

Reproduce the reported Elicit score

uv is the only tool you need. The project is pinned to Python 3.11 and has no runtime dependencies.

git clone https://github.com/elicit/elicit-paper-search-eval.git
cd elicit-paper-search-eval
uv run score.py --strict-complete

Expected output:

Metric	@10	@20	@50	@100	@200
Macro recall	0.390	0.483	0.603	0.670	0.708
Micro recall	0.204	0.306	0.466	0.571	0.632

The scorer should also report:

num_scored_questions: 5486
total_gold: 56168
match_tier_counts: doi=40, pmid=35060, title=413

data/manifest.json records the full-precision expected values and SHA-256 hashes for the frozen artifacts.

Reported provider comparison

The table below reports macro recall for each provider on its completed BioASQ question set. A dash means the provider did not return results at that depth.

System	@10	@20	@50	@100	@200
Elicit	0.390	0.483	0.603	0.670	0.708
Consensus	0.369	0.434	—	—	—
Semantic Scholar	0.254	0.311	0.397	0.465	0.535
OpenAlex (keyword)	0.286	0.364	0.474	0.557	0.630
OpenAlex (semantic)	0.183	0.258	0.299	—	—
Google Scholar	0.287	0.333	0.406	0.458	0.502

Consensus returned about 20 results per question and OpenAlex semantic returned at most 50. Frozen competitor responses are not included in this repository. Live reruns may differ as provider APIs, indexes, and ranking models change.

Web search APIs

General-purpose web search APIs, each restricted to scientific sources so that a recall benchmark does not penalise them for returning news or blogs. Exa was restricted to its publication category; the other three were restricted to the pubmed.ncbi.nlm.nih.gov and pmc.ncbi.nlm.nih.gov domains. All four received the original natural-language question, so none of them uses a generated keyword query.

System	@10	@20	@50	@100	@200
Elicit	0.391	0.483	0.603	0.670	0.708
Perplexity (PubMed + PMC)	0.357	0.423	—	—	—
Brave (PubMed + PMC)	0.319	0.363	0.379	0.379	—
Parallel (PubMed + PMC)	0.292	0.301	—	—	—
Exa (publications)	0.281	0.340	0.396	0.447	—

Perplexity and Parallel returned at most 20 results, and Exa and Brave at most 100.

Two adjustments apply to this table and not to the academic comparison above, because these APIs need them and the academic APIs did not:

• Publication-year cutoff. Only Exa can filter on a paper's publication date, via endPublishedDate. For Brave, Parallel and Perplexity the cutoff is applied after retrieval by resolving every result's publication year from PubMed. Removing results leaves gaps in a provider's ranking, so the surviving results are renumbered; otherwise a request for the first 10 results returns fewer than 10 papers and the measurement reflects how much of a provider's index postdates the question rather than how well it searches.
• One credit per paper. A provider returning both a paper's PubMed page and its PMC page is credited once.

Elicit needs neither adjustment: it filters by publication year at request time, and none of its result lists has a gap.

Exa's figures were corrected on 2026-08-05. Earlier materials reported 0.268 / 0.325 / 0.382 / 0.433. Our DOI extraction accepted a DOI only when the result URL's host was doi.org, which discarded the DOI from publisher URLs such as onlinelibrary.wiley.com/doi/10.1111/.... That affected 59,618 of Exa's 433,012 results, leaving them with no identifier and matchable only by exact title. Recovering those DOIs raises Exa by 1.3 to 1.5 points. The same defect was fixed in the Brave and Parallel adapters, where it changes nothing measurable: every one of their results already carries a PMID. Exa reported the discrepancy and its diagnosis was correct.

Brave's later pages repeat earlier ones heavily, though not completely: the fifth page still added 388 URLs unseen on earlier pages, across 327 questions. After the cutoff its lists run to a median of 63 results, and to a median of 23 once repeated papers are collapsed; no question reaches 100. Its figures at 50 and 100 are its ceiling repeated rather than deeper measurements. Exa's lists run to a median of 79.

Run a search API

Copy the environment template and add the key for the provider you want to run:

cp .env.example .env

Run a small pilot first. Each completed question is appended immediately, so an interrupted run resumes without repeating successful requests.

uv run search.py elicit --limit 100 --concurrency 2
uv run score.py --results runs/elicit.jsonl --failures runs/elicit.failures.jsonl

After checking the pilot, run the full input:

uv run search.py elicit --concurrency 2 --confirm-large-run

The same interface supports the ten provider modes included in the runner:

uv run search.py consensus --limit 100
uv run search.py semantic_scholar --limit 100
uv run search.py openalex_keyword --limit 100
uv run search.py openalex_semantic --limit 100
uv run search.py google_scholar --limit 100
uv run search.py exa --limit 100
uv run search.py brave --limit 100
uv run search.py parallel --limit 100
uv run search.py perplexity --limit 100

The four web providers return page URLs rather than paper records, so the runner records a PMID or DOI parsed from the URL and, for PMC pages, the PMCID. BioASQ gold is keyed on PMIDs, so PMC pages need one more step before they can be scored:

uv run search.py brave --limit 100
uv run resolve_pmcids.py --results runs/brave.jsonl --email you@example.com
uv run score.py --results runs/brave.resolved.jsonl

resolve_pmcids.py maps PMCIDs to PMIDs through NCBI's ID Converter, which is free and needs no key. Skipping it will understate the three PubMed/PMC providers badly — 59% of Brave's results sit on a PMC host and would go unmatched.

Provider responses and failure sidecars are written under runs/, which is gitignored.

Calls and provider limits

The full benchmark contains 5,486 questions. At the evaluated depths, a complete run can make up to:

Provider	Query	Maximum depth	Calls for 5,486 questions
Elicit	Original natural-language question	200	5,486
Consensus	Original natural-language question	about 20	5,486
Semantic Scholar	Provider-specific keyword query	200	10,972
OpenAlex keyword	Provider-specific keyword query	200	10,972
OpenAlex semantic	Original natural-language question	50	5,486
Google Scholar via SerpAPI	Provider-specific keyword query	200	54,860
Exa	Original natural-language question	100	5,486
Brave	Original natural-language question	100	27,430
Parallel	Original natural-language question	20	5,486
Perplexity	Original natural-language question	20	5,486

Brave is paginated in pages of 20, so a depth-100 run makes up to five paid requests per question. Google Scholar uses up to ten paid SerpAPI searches per question. Check your SerpAPI plan and current pricing before running it. Quotas and prices can change.

Method

Benchmark

The input is a pooled set of 5,486 BioASQ Task B questions from multiple challenge years. Questions repeated in different years remain separate when their gold paper set or publication cutoff differs. Each API request excludes papers published after that question's BioASQ challenge year.

data/bioasq.jsonl has one self-contained record per question:

{
  "question_id": "golden_enriched:...",
  "question": "Is Tuberous Sclerosis a genetic disease?",
  "max_year": 2014,
  "gold": [{"pmid": "...", "doi": "...", "title": "..."}],
  "queries": {
    "semantic_scholar": {"query": "...", "query_version_id": "..."},
    "openalex_keyword": {"query": "...", "query_version_id": "..."},
    "google_scholar": {"query": "...", "query_version_id": "..."}
  }
}

Every keyword-query record also includes the model, source, exact prompt, prompt hash, query version, and exact request string sent after provider-specific normalization. This runner consumes those frozen queries; it does not call an LLM or regenerate them.

Keyword-generation prompts and final queries

The repository reports one provenance row for every keyword query used in the evaluation: 5,486 questions across Semantic Scholar, OpenAlex keyword search, and Google Scholar, for 16,458 rows total. Each row contains:

• the natural-language BioASQ question and search provider;
• the model, source, exact prompt, prompt hash, and prompt provenance;
• generated_query, the query parsed from the model response; and
• request_query_used, the exact string sent to the provider.

Export the complete report as JSONL:

uv run keyword_queries.py --output runs/keyword_queries.jsonl

Inspect the generation breakdown:

uv run keyword_queries.py --summary

Claude Opus 4.8 generated 16,353 queries. GPT-5.6 Sol generated 105 replacements for Opus refusals and later repaired provider-rejected syntax in 83 of the Opus-generated queries. Repair prompts include the rejected query and validation error. For 164 Semantic Scholar queries, request_query_used differs from generated_query because unsupported hyphens were replaced with spaces before the API request.

To inspect all three keyword queries, prompts, and versions for one question:

uv run keyword_queries.py \
  --question-id golden_enriched:52b06a68f828ad283c000005

Search configuration

Semantic providers receive the original BioASQ question. Keyword providers receive the final provider-specific query in the input file. Elicit uses:

POST https://elicit.com/api/v2/search/papers
Authorization: Bearer $ELICIT_API_KEY
Content-Type: application/json

{
  "query": "<BioASQ question>",
  "searchMode": "semantic",
  "corpus": "elicit",
  "filters": {"maxYear": 2014},
  "maxResults": 200
}

The frozen Elicit output came from the evaluated 300-candidate reranking configuration and contains exactly 200 returned papers per question.

Matching

Candidates are examined in provider rank order. A paper is credited at most once, through the first available matching tier:

1. exact normalized PMID;
2. exact normalized DOI; then
3. exact normalized title after lowercasing, Unicode normalization, punctuation removal, and whitespace collapse.

The matcher commits to the first tier that points to a gold paper. If that gold paper was already claimed by an earlier result, the candidate receives no additional credit and does not fall through to a weaker tier. This prevents duplicate results from claiming multiple gold papers.

Macro recall is computed per question and then averaged, giving each question equal weight. Micro recall pools the matched and gold paper counts across questions.

Failures

Successful questions are scored; questions with recorded retrieval failures are reported and excluded. A successful response containing no candidates remains a zero-recall result. Google Scholar's explicit no-results response is recorded as a terminal failure to preserve the convention used in the reported comparison.

Transient network errors, HTTP 429s, and HTTP 5xx responses are retried with exponential backoff, jitter, and Retry-After support. Exhausted transient errors remain retryable on the next invocation. Terminal errors are skipped on resume unless --retry-terminal is supplied.

Files

data/bioasq.jsonl              Frozen questions, gold papers, queries, and prompts
data/elicit_results.jsonl.gz   Frozen Elicit top-200 outputs
data/manifest.json             Counts, expected metrics, provenance, and checksums
uv.lock                        Locked Python and development-tool environment
keyword_queries.py             Exact prompts and final keyword-query report
search.py                      Resumable API runner for all six provider modes
score.py                       Deterministic recall scorer
tests/test_keyword_queries.py  Keyword provenance report regression test
tests/test_score.py            Matching and normalization regression tests
tests/test_search.py           Provider request and parser regression tests

Data attribution and terms

The benchmark records are derived from BioASQ Task B and PubMed metadata. Courtesy of the U.S. National Library of Medicine. NLM does not endorse this repository or its results.

The snapshot does not represent the most current or complete NLM data. BioASQ and NLM data remain subject to their respective terms, including the NLM data terms. The MIT license in this repository applies to the code, not to third-party benchmark records or API response content.

Limitations

BioASQ's gold articles are sufficient evidence selected for its question-answering benchmark, not an exhaustive set of every relevant paper. The benchmark is biomedical and PubMed-oriented. Recall therefore measures recovery of the frozen BioASQ gold set; it does not measure precision, search quality in every discipline, or the relevance of unlabelled results.

The checked-in outputs reproduce the reported score, but they do not make the original acquisition metadata fully self-contained: the final one-question resume overwrote the Elicit run's first-pass metadata. The validated artifact still contains all 5,486 questions and 1,097,200 uniquely ranked candidates; data/manifest.json records this provenance limitation explicitly.

License

The source code is available under the MIT License.