Panelist

An extensible, configurable multi-perspective research framework, inspired by the STORM paper (Shao et al., 2024, Stanford NLP). Unlike the original auto-discovery approach, Panelist gives full control over expert perspectives while preserving the auto-generate mode as an optional strategy.

Panelist runs a panel of configured "expert" perspectives against a topic — each researching it independently via web search and/or local documents — then synthesizes their findings into an outline and a final artifact: an article, a structured risk report, a decision brief that preserves disagreement, or a numeric risk score. What experts exist, how they research, and what shape the output takes are all config, not code.

uv run panelist run configs/example_marketing_strategy.yaml

Table of contents

• How this differs from the original STORM
• Architecture
• Setup
• Configuration
• Output formats
• Real use cases
• Testing & CI
• Citation

How this differs from the original STORM

STORM (Shao et al., NAACL 2024) has an LLM discover article perspectives by surveying similar existing articles, then simulates conversations between an AI writer and each perspective to gather information before drafting a Wikipedia-style article. Panelist keeps the "multiple perspectives research independently, then get synthesized" core idea, but changes three things:

1. Experts are configured, not discovered, by default. experts.mode: guided means you specify exactly who's on the panel and what they focus on. classic (full LLM auto-discovery, closer to the original paper) and hybrid (your fixed experts + 1-2 LLM-suggested additions) are both available as an ExpertDiscoveryStrategy — a Strategy-pattern choice, not a hardcoded default.
2. Output isn't always an article. writing.output_format can be article, structured_report, decision_brief, or risk_score — see Output formats.
3. Research is a swappable dependency, down to a human. ExpertResearcher has an LLM-backed implementation and a HumanResearcher that asks a person for answers instead — same interface, same pipeline.

This is not affiliated with, endorsed by, or an official extension of the original STORM project or Stanford NLP/OVAL.

Architecture

interfaces/cli.py            Typer CLI — the only entry point, talks to RunStormPipeline
application/                 Use cases: BudgetTracker, OutlineBuilder, OutputWriter,
                              HumanApprovalGate, the LangGraph assembly, RunStormPipeline
researchers/                 ExpertResearcher implementations (LLM-based, human)
domain/                      Pure models + ExpertDiscoveryStrategy — zero infra imports
infrastructure/               Concrete adapters: Anthropic/OpenAI/Fake LLM
                              providers, Tavily/Chroma/Fake retrievers
config/                       Pydantic schema + YAML/.env loader

Full write-up, including why each layer is shaped this way, in docs/architecture.md and docs/design-decisions.md.

Setup

Requires uv.

git clone <this-repo>
cd panelist
uv sync

cp .env.example .env
# fill in whichever of these your configs use:
#   ANTHROPIC_API_KEY / OPENAI_API_KEY / TAVILY_API_KEY

uv run panelist validate configs/example_marketing_strategy.yaml   # sanity-check a config
uv run panelist run configs/example_marketing_strategy.yaml        # run it
uv run panelist diff old/output.json new/output.json              # what changed between two runs

Output lands in project.output_dir (output.json, a research_log.json grounding trail, and a trace.jsonl event stream always; output.md too for article output format). For real end-to-end runs — actual Anthropic + Tavily calls, real sourced findings, real spend each — see examples/lead_scoring/ (a risk_score) and examples/competitive_monitoring/ (a structured_report).

API keys don't have to live in .env: every api_key field also accepts a literal value or a ${VAR_NAME} placeholder directly in the YAML. See Configuration for the full precedence order.

Running the test suite

uv run pytest      # all tests use fakes — no network calls, no API keys needed
uv run ruff check .
uv run mypy src/panelist --ignore-missing-imports

Configuration

Full reference: configs/schema.md. Six ready-to-edit examples in configs/, one per use case below plus the original marketing-strategy sample. Minimal shape:

project:
  topic: "Marketing strategy for Company X entering Market Y"
  output_dir: "./output"

experts:
  mode: guided # classic | guided | hybrid
  definitions:
    - id: brand_strategist
      role: "Brand Strategist"
      focus: "Positioning and perception"

research:
  max_rounds_per_expert: 4
  stop_condition: llm_judge # llm_judge | fixed_rounds | manual
  sources:
    web_search:
      enabled: true
      provider: tavily
      api_key: "${TAVILY_API_KEY}"  # or a literal key, or omit to use the env var directly
    local_rag:
      enabled: false               # Chroma over local docs (txt/md/pdf/docx/xlsx/html) when enabled

outline:
  max_sections: 6
  require_human_approval: false

writing:
  output_format: article # article | structured_report | decision_brief | risk_score

llm:
  provider: anthropic # anthropic | openai
  model: claude-sonnet-5
  api_key: "${ANTHROPIC_API_KEY}"
  max_tokens_budget: 100000 # hard cap for the whole run — cost control
  max_cost_usd: 5.00 # optional: also stop the run if estimated spend crosses this

The run prints its estimated dollar cost (est_cost=$…) on completion; per-model rates live in pricing.py and are editable estimates, not billing-accurate quotes.

A malformed config raises ConfigError immediately, before any LLM call is made.

Researching local files

Panelist isn't limited to web search — experts can research your own documents (contracts, financials, internal wikis, transcripts) via a local Chroma vector index, with no extra API key required (it uses Chroma's bundled local embedding model). Text and Markdown work out of the box; PDF, DOCX, XLSX, and HTML need the optional documents extra (uv sync --extra documents). Enable it under research.sources.local_rag:

research:
  sources:
    local_rag:
      enabled: true
      documents_path: "./data/company_docs" # recursive; txt/md/pdf/docx/xlsx/html
      chunk_size: 800 # target chunk size for the structure-aware recursive chunker
      chunk_overlap: 100
      persist_directory: "./data/.chroma" # optional — omit to re-index in memory each run

Documents are chunked with a recursive, structure-aware splitter (paragraph → line → sentence → word boundaries) rather than fixed-width slicing, so clauses and paragraphs stay intact. A persisted index is fingerprinted and rebuilt whenever the corpus or chunking params change.

local_rag and web_search can be enabled together (experts' queries fan out to both); at least one source must be enabled. This is what powers the due-diligence, content-audit, and decision-support use cases below, where the source material is internal documents rather than the public web.

Grounded, auditable citations

Research answers aren't free text with a citation stapled on. Each finding is broken into individual claims, and a claim may only cite sources that were actually retrieved for that question — any source id the model invents is dropped, and a claim with no support is surfaced as [UNCITED] rather than hidden. Retrieved content is also fed to the model as clearly delimited, untrusted data (a first-line defense against prompt injection from a scraped page or a target's own documents). This is deliberately aimed at the high-stakes use cases below, where "where did this come from?" has to have an answer.

Output formats: not just articles

The same discover → research → outline → write pipeline produces four different shapes, selected purely by writing.output_format:

Format	Shape	Fits
article	Markdown article, one written section per outline section	blog posts, analytical reports
structured_report	title, executive summary, per-expert findings, red_flags list	due diligence, monitoring digests
decision_brief	per-expert positions, unresolved_conflicts, one recommendation	management decision support — disagreement is preserved, not smoothed over
risk_score	numeric overall_score + level, per-expert factors, one verdict	lead scoring, pre-publish compliance checks

Real use cases

Six scenarios this framework maps onto directly, each with a ready-to-run config in configs/ and a saved example run in examples/. Ordered from the simplest operational setup to the most complex (more sources, more human checkpoints, richer output shapes):

1. Lead / partner qualification before a sales cycle — config · real run

A B2B company gets an inbound lead and wants a fast read on whether it's worth sales time. Experts = "Financial Stability Analyst," "Product Fit Analyst," "Reputation Risk Analyst," researching the lead company via web search. Output is risk_score — a structured verdict a script can act on, not free text. This directly automates work an SDR currently does by hand (googling the company, reading the site, forming an opinion); faster qualification means more leads processed per rep-hour.

2. Competitive monitoring for product/marketing teams — config · real run

Teams need a recurring read (e.g. monthly) on what 3-5 key competitors are doing on product, pricing, and hiring. Experts = "Product Analyst," "Pricing Analyst," "Hiring & Growth Signals Analyst," each researching one competitor via web search (site changes, reviews, press, job postings). Designed to run on a schedule (cron); panelist diff <last>/output.json <this>/output.json reports exactly what changed month over month (score/level moves, new or cleared red flags, per-finding changes), and enabling the cache means a re-run only pays for what actually changed.

3. Pre-publish risk audit for company communications — config

Before publishing an important statement — an earnings release, an annual report, a crisis response — legal, PR, and compliance need to independently check it. Experts = "Legal Risk Reviewer," "Reputational Risk Reviewer," "Brand Guidelines Reviewer," "Financial Compliance Reviewer," each reviewing the same draft (loaded via local_rag) and giving a verdict on specific problem passages, not a rewrite. Output is risk_score, meant as a pre-publish gate in the workflow. The value here is risk mitigation — avoiding a costly legal or reputational error — not content generation, and that's how it should be pitched internally.

4. Go-to-market / marketing strategy — config

A company entering a new market wants a strategy write-up covering positioning, acquisition channels, and retention risk — the kind of deliverable a marketing consultancy produces. Experts = "Brand Strategist," "Performance Marketer," "Customer Experience Specialist," with hybrid mode letting the LLM add one or two perspectives the requester didn't think to name (a partnerships angle, a content/SEO angle). Output is a full article: one written section per outline section, not a structured verdict — a larger and more open-ended generation surface than the first three use cases.

5. M&A / investment due diligence — config

Before an investment or acquisition, an investor needs the target checked from a dozen angles — financials, legal risk, reputation, tech debt. Experts here are "Financial Analyst," "M&A Legal Counsel," "Technical Due Diligence Lead," "Reputation & PR Risk Analyst," each researching via local_rag over uploaded company documents (financials, contracts, litigation) plus web search for news and public mentions — the first use case to combine both retrieval sources, plus a human sign-off gate on the outline before writing begins. Output is a structured_report with explicit red_flags per area — not prose, a risk report.

6. Decision support synthesizing internal data — config

Leadership needs to make a call (enter a new market, change pricing, sunset a product tier) with information scattered across CRM, financial dashboards, support tickets, and internal discussions. Experts map to internal stakeholder perspectives ("Sales Perspective," "Support/CX Perspective," "Finance Perspective"), each researching the same internal corpus via local_rag. hybrid mode lets the LLM add perspectives the requester didn't think to include. Output is a decision_brief: explicit for/against positions per department plus unresolved_conflicts — disagreement is surfaced, not averaged away, because that's the useful signal. This is the most complex example: dynamic expert discovery, a single internal corpus shared across all experts, and the richest output shape of the four. It's a decision-support system, a category companies pay directly for (the same information-gathering McKinsey/BCG do over weeks, here in minutes).

The common thread across all six: the framework's job is a structured decision output — risk score, verdict, red flags, a brief with conflicting positions, or a full article — not a single hardcoded shape. That's why OutputFormat is a config field on the domain model rather than a hardcoded assumption: the same pipeline serves all six without touching code.

Testing & CI

Every test uses FakeLLMProvider / FakeRetriever — no network calls, no API keys required to run the suite. tests/unit/ covers each layer in isolation (config validation, budget tracking, discovery strategies, the outline builder, structured-output retry/backoff, researchers). tests/integration/test_pipeline_end_to_end.py runs the full RunStormPipeline — LangGraph included — once per OutputFormat. CI (.github/workflows/ci.yml) runs ruff, mypy, and pytest on every push/PR via uv.

Citation

Panelist is inspired by, but is not an implementation of or affiliated with:

Yijia Shao, Yucheng Jiang, Theodore Kanell, Peter Xu, Omar Khattab, Monica Lam. "Assisting in Writing Wikipedia-like Articles From Scratch with Large Language Models." NAACL 2024. arXiv:2402.14207

License

MIT — see LICENSE. (Update the copyright line there if you fork this for your own use.)