Task emails must contain real, sourced data — eliminate hallucinated metrics #1833

Follow-up to chat#1829 (task-email delivery, resolved 2026-07-01). Delivery and the empty-body guard now work — but task reports fabricate metrics when their data source is missing or errors. Today's focus: accurate tool calling so task emails carry real, sourced data with zero hallucinations.

Motivating case (2026-07-01, verified): the "Apache YouTube CPM Weekly Report" to stephanie.guerrero@onerpm.com was built after its only metrics call — /api/artists/{id}/socials — returned "Artist not found", with no YouTube/analytics/CPM source ever called. The agent then shipped invented CPM/CTR/revenue figures as if measured. Full tool-call trace: see the pinned comment.

Goal

Every metric in a task email is backed by a successful data-source call. When the source is missing or errors, the agent omits or labels the metric (or aborts the send) — it never fabricates. Target: 0% of delivered task emails contain hallucinated data (baseline below). Surfaces: app/lib/workflows/runAgentStep.ts (agent loop), the recoup-platform-api-access skill, and the per-task data connectors.

PRs (updated 2026-07-01)

All 15 matrix PRs are merged (last: api#740, 2026-07-02). Scrape persistence now covers all 7 supported platforms. Remaining open items: the runAgentStep headless honest-send (constructive-fallback follow-up) and the task→data-source mapping.

Ship order by impact: the two grounding rules (runAgentStep first, then the skill) go first — they cap hallucinated data at ~0 for all tasks, even ones with no data source. Then the enablers (default-skill install, /api/artists id fix) restore report value by getting real data flowing again.

Done

• api#740 — persist scrape results for the remaining 5 platforms.
  ✅ Merged 2026-07-02 (squash to main, commit 7b8c4b05). TikTok/X/YouTube/Threads/Facebook now attach getApifyWebhooks() on start and persist via real-shape handlers + registry entries (actor ids resolved from the Apify API). Round-trip hazards handled: X URLs lowercased (actor echoes display casing; naive upsert would duplicate), YouTube keyed on inputChannelUrl (actor returns /channel/UC…, stored rows are @handle).
  Verified on preview, live on real rows — writes made by the webhooks: TikTok @apache_207 918,100→917,500; X theasf 66,112→66,211; YouTube @apachelasminas 0→1,010,000 (+avatars/bios) — with zero duplicate rows (1 theasf row, 0 new /channel/… rows). Threads/Facebook covered by unit tests from real runs (@zuck 5.64M, facebook.com/facebook 154M). Results.
  All 7 supported platforms now persist scrape results — the stale-socials gap is closed.

• api#738 — registry webhook dispatch + LinkedIn persistence end-to-end.
  ✅ Merged 2026-07-01 (squash to main, commit bd1958ca). getApifyResultHandler (actorId → handler registry, replacing the IG-only switch) + the first non-IG platform persisted: startLinkedinProfileScraping attaches getApifyWebhooks(), and handleLinkedinProfileScraperResults upserts socials (username/avatar/bio/followerCount/region, keyed profile_url) — mapping from a real harvestapi run, 404-wrapper skipped.
  Root cause confirmed: webhooks are per-run and only IG's start modules attached them — other platforms never called /api/apify at all.
  Verified end-to-end on preview (223752f6): before/after DB diff written by the webhook itself — followerCount 0 → 13,203, bio/avatar/region null → real profile data, via run rgJsSe0By69c7tShB → Apify POST → registry → handler. Results.

• api#737 — LinkedIn profile scraping (harvestapi).
  ✅ Merged 2026-07-01 (squash to main). startLinkedinProfileScraping → harvestapi/linkedin-profile-scraper; linkedin.com dispatch in scrapeProfileUrl; LINKEDIN in getSocialPlatformByLink. Preview verification caught + fixed 3 real bugs pre-merge: the actor takes urls not profiles; getUsernameFromProfileUrl returned the /in/ path prefix as the username; and (caught live — a run scraped linkedin.com/in/in → 404) the module now loudly rejects prefix pseudo-handles from legacy rows (KISS: shared dispatcher untouched per review).
  Verified end-to-end on preview (b65c1d69): prod 400 Unsupported → preview 200 + run TNwgXpQmhvoCHsKg8 → SUCCEEDED with a real profile (sweetmaneth, 14,242 connections). Results. Persistence to socials still pends api#738.

• chat#1835 — delete the in-app /docs subroute.
  ✅ Merged 2026-07-01 (squash to main). Removed app/docs/ (4 pages + constants) and components/docs/ (5 support components used only by those pages) — 13 files. The hand-maintained in-app docs duplicated and drifted from the official docs codebase (the exact drift cubic flagged on chat#1834); documentation now lives only in recoupable/docs (reconciled to the live API in docs#257).
  Verified pre-merge: nothing outside app/docs/ imported the deleted code; no Link/href to the /docs route anywhere; tsc clean of docs-related dangling imports.

• docs#257 — reconcile /api/artists + /socials docs ↔ API.
  ✅ Merged 2026-07-01 (to main, commit 767caa06). Artist schema: +id(=account_id, documented as the sub-resource alias), +instruction/knowledges/label, socials→account_socials; SocialProfile: −id (api#735 removed it).
  Verified against live prod (api.recoupable.dev running Veerbalsingh369/myc 2252 youtube videos success pending #734+Added youtube thumbnail result UI #735, Apache fixture): both schemas match the live responses field-for-field, zero drift in either direction (results comment). Docs ↔ API ↔ prod now tell the same story.

• api#735 — drop the dead account_socials.id from /socials (keep social_id).
  ✅ Merged 2026-07-01 (squash to main, commit 1bbefabb). flattenAccountSocials no longer returns id (the account_socials.id); social_id is the single identity.
  Verified on preview (contrast vs main, Apache socials): /socials returns no id (only social_id) vs main's id+social_id; POST /api/socials/{social_id}/scrape → 200, while the old id (04b864b8) → 404 (the exact scrape footgun). Companion chat#1834 (React keys social.id → social_id) ✅ merged 2026-07-01 — the chat UI now keys on social_id, matching the dropped id. Docs follow-up (drop id from the /socials schema) folded into the reconciliation item below.

• api#734 — /api/artists id is now account_id (drop the leaked account_info.id).
  ✅ Merged 2026-07-01 (squash to main, commit 756252ac). getFormattedArtist no longer spreads the raw account_info row — id = account_id, and the leaked job_title/role_type/company_name/organization are gone.
  Verified on preview (same-token contrast vs main, Apache fixture): id === account_id (ebae4bb9) vs main's 33adb9fe (account_info.id); GET /api/artists/{id}/socials using the list id → 200 on preview vs 404 on main (the exact OneRPM footgun). The docs↔API reconciliation (document id/instruction/knowledges/label, fix socials↔account_socials) is the small follow-up tracked above.

• api#733 — install all 6 default sandbox skills (build-os ref + resilient install).
  ✅ Merged 2026-07-01 (squash to main, commit b3bac3cd). Corrected DEFAULT_GLOBAL_SKILL_REFS recoup-platform-build-workspace → recoup-platform-build-os, and made installGlobalSkills best-effort (log + continue instead of throw) so one bad ref can't abort the batch.
  Verified on preview (chat edc3d328): a fresh sandbox ls ~/.agents/skills/ lists all 6 default skills (was 2); recoup-roster-list-artists loads success:true (it returned "Skill not found" on the original OneRPM run, chat c38d62cd). Task sandboxes now get their full toolkit → agents can resolve/verify the roster instead of guessing.

• api#739 — withhold ask_user_question from headless runs.
  ✅ Merged 2026-07-01 (squash to main, commit db0e2b2f). The tool (no server execute — only the chat UI fulfills it) is now gated behind an interactive flag on buildAgentTools, threaded RunAgentWorkflowInput → runAgentStep; handleStartChatRun (headless) passes interactive:false, interactive chat keeps it.
  Verified end-to-end on preview (chat 9ccfabbb, Sonnet 5, Apache): with no ask-the-user escape, the agent scraped real data (@Apachelasminas — 1.01M subs, 404.6M views, 275 videos), recognized CPM isn't sourceable (0 fabrication), and delivered an honest report (Resend 235b7356) — vs. the original prod run (chat c38d62cd) which fabricated CPM/revenue and emailed it. Converts the OneRPM benchmark from "paused question, nothing delivered" → "honest, real-data report, delivered."

• tasks#154 — default the scheduled-task model to claude-sonnet-5.
  ✅ Merged 2026-07-01 (squash to main, commit 9e9c6da). customer-prompt-task now defaults to anthropic/claude-sonnet-5 (DEFAULT_TASK_MODEL) when the task config omits a model, instead of falling back to haiku-4.5. Directly cuts the live fabrication risk: on the OneRPM/Apache benchmark, haiku-4.5 fabricated CPM/revenue and sent it twice; sonnet-5 refused and grounded on real scraped data (1.54M subs, 1.11B views).
  Verified: RED→GREEN test (defaults to sonnet-5 when unset; explicit model still forwarded); full tasks suite green (357).

• skills#70 — harden recoup-platform-api-access (grounding + get-the-data fallback).
  ✅ Merged 2026-07-01 (squash to main, commit 006ff76). Mirrors the runtime grounding rule into the always-loaded skill — stop rule "never invent metrics/data"; use account_id (not the list id) for /artists/{id}/*; read the embedded account_socials before calling /socials. Plus the constructive fallback: when a source isn't connected, scrape public data (/socials/{id}/scrape), check GET /api/connectors isConnected, and mint + surface a Composio connect link (POST /api/connectors → redirectUrl) instead of omitting. Kept caller-agnostic per review — the headless "send an honest email instead of asking" behavior stays in the task layer (constructive-fallback item, still open below).
  Verified: CI green (check_resolvable, run_resolver_eval, validate_manifests).

• api#736 — runAgentStep data-grounding rule (the PRIMARY lever).
  ✅ Merged 2026-07-01 (squash to main, commit 57846cb). buildAgentSystemPrompt now always emits a DATA_GROUNDING_SECTION: the agent may state only figures retrieved from a successful tool call this run; on missing/failed/empty data it says so and omits/stops — never estimate/"industry average"/sample.
  Verified on preview (built from 58685e3c): replayed the exact OneRPM "Weekly YouTube CPM Report" prompt (scheduled_actions 70956e7e; recipient rewritten to a staff address). vs. the original prod run (chat c38d62cd): 0 CPM/revenue numbers invented, 0 report files written, 0 emails sent — the agent recognized "CPM … requires YouTube Analytics access" and asked for the connection instead of fabricating. Results comment.
  The same test surfaced the constructive-fallback gap (refused to lie, but also didn't get the data) — tracked below.

• skills#69 — recoup-internal-task-email-audit (the detection tool).
  ✅ Merged 2026-07-01 (squash to main, commit c501770). The standing audit that enumerates Trigger.dev customer-prompt-task runs, correlates them to email_send_log sends, and extracts each run's tool-call trace + flags has_hallucinated_data / expected_data_source. This is the tool that produced this issue's baseline and root-caused the Apache→OneRPM fabrication (§ evidence comment).
  Verified: CI green (check_resolvable, run_resolver_eval, validate_manifests); ran live on the 2026-07-01 prod window (per-run table + baseline produced).

Open — API contract: list endpoints return an id their own sub-resources reject (systemic)

This has now bitten us three times — /api/artists (id=account_info.id), /api/artists/{id}/socials (id=account_socials.id), and the social-scrape sub-call — each returns an id that its own {id} sub-resource rejects, and the load-bearing id is a different field. Worth a systemic pass: the id a list returns must be the id its sub-resources accept.

• Reconcile /api/artists + /socials docs ↔ API — ✅ shipped docs#257, see Done.

• Delete the chat in-app /docs subroute — ✅ shipped chat#1835, see Done.

• /api/artists id-consistent — ✅ shipped api#734, see Done. (docs reconciliation follow-up above)

• /api/artists/{id}/socials id-consistent — ✅ shipped api#735 + chat#1834 (React keys), see Done.

• Add LinkedIn to social scraping — ✅ shipped api#737, see Done. (persistence → api#738)

• Persist the remaining platforms' scrape results — ✅ shipped api#740, see Done. All 7 platforms persist.

Open — data grounding

• runAgentStep no-fabrication grounding rule — ✅ shipped api#736, see Done.

• Harden recoup-platform-api-access (grounding + embedded-socials + get-the-data fallback) — ✅ shipped skills#70, see Done.

• Constructive fallback when data is missing — scrape + connect link, not a dead-end. (skill part ✅ shipped skills#70; runAgentStep headless honest-send follow-up open)

  • Why: the api#736 preview replay proved the grounding rule stops fabrication — but the agent then dead-ended on ask_user_question (3×) and never tried to get the data: 0 calls to /socials/{id}/scrape, 0 to POST /api/connectors (the Composio connect link), 0 to /socials. In a headless scheduled task no one answers, so the run delivers nothing — barely better than the fabrication it replaced. api#736 results.
  • Fix: when a metric's source isn't connected: (1) POST /api/socials/{social_id}/scrape (or /artist/socials/scrape) to pull the public data available (YouTube/TikTok/IG/X/Threads/FB); (2) GET /api/connectors → read isConnected; (3) POST /api/connectors {"connector":"youtube"} → { redirectUrl } and embed that Composio OAuth link in the email so the customer self-connects; (4) send an honest report (real public data + connect link) — never ask_user_question in a headless task. Added to the always-loaded skill in skills#70; a runAgentStep/headless nudge to send the honest "data unavailable + connect here" email instead of asking is the remaining follow-up.
  • Done when: replaying the OneRPM CPM prompt on preview yields an email with the real public YouTube data + a working redirectUrl connect link — not a fabricated report, not an unanswered question.

• Default the scheduled-task model to anthropic/claude-sonnet-5 — ✅ shipped tasks#154, see Done.

• Remove ask_user_question from headless runs — ✅ shipped api#739, see Done.

• Forbid scheduled-task creation in headless task runs. (no PR yet — observed on the 2026-07-02 benchmark re-test)

  • Why: scheduled-task prompts embed their own schedule ("Run every day at 9:00 AM America/New_York…"). A headless agent reading that instruction creates a new scheduled task via POST /api/tasks — the LV8R re-test run created "LV8R Daily Trends Digest (9AM ET)" (b0495f6e, enabled=true) on the invoking account (deleted via DELETE /api/tasks after the test). In production this means every task run can spawn duplicate self-perpetuating schedules.
  • Fix: headless runs (customer-prompt-task / /api/chat/runs) should tell the agent it IS the scheduled execution — "you are already running on the schedule; never create/modify scheduled tasks" (in the runAgentStep prompt or the task wrapper). Optionally withhold task-creation capability from headless runs the way ask_user_question was (api#739 pattern).
  • Done when: re-running a "run every day…" prompt headless creates no new scheduled_actions row.

• Map each recurring task type → its required data source; flag tasks with no available source.

  • Why: some tasks (e.g. "YouTube CPM") have no data source wired at all — YouTube Analytics/revenue needs a connector Recoup doesn't have, which is why the agent invents. Others (trend reports) need web_fetch/search but don't call it.
  • Fix: per the baseline's expected_data_source column, either wire the connector or have the task decline honestly.
  • Done when: every recurring email task maps to a real, callable source (or is explicitly marked "no source → decline").

Source references

• Parent: chat#1829 (delivery — resolved; its "data-grounding for research tasks" follow-up is this issue).
• Detection tool: skills#69 (recoup-internal-task-email-audit).
• Evidence exhibit: Apache→OneRPM run, chat c38d62cd (full tool-call trace in the comment below).