docs: expand query hubs and add crawler entrypoints

[johnxie]

Summary

Expands the generated Query Hub from 6 broad surfaces to 15 high-intent surfaces and adds lightweight crawler entrypoints.

What changed

• Added new generated query hubs for:
  • multi-agent frameworks and orchestration
  • AI code audit, review, and governance
  • local-first/self-hosted AI stacks
  • agent memory and knowledge systems
  • agent context, memory, and skills
  • browser automation and web agents
  • LLM observability, evals, and guardrails
  • workflow automation and internal AI tools
  • AI data platforms, analytics, and storage
• Added a concise README goal row for coding-agent context, memory, and skills.
• Regenerated discoverability/query-hub.md and discoverability/query-coverage.json from the generator.
• Added root robots.txt and sitemap.xml as lightweight crawl hints for the primary repository surfaces.

Validation

• python3 scripts/docs_health.py --root .
• python3 scripts/check_index_format_v2.py --root .
• python3 scripts/staleness_audit.py --max-age-days 10 --fail-on-stale
• Generated discoverability assets to /tmp and diffed all CI-validated outputs.