Improve AgentSea search and AI discovery

[Cahl-Dee]

Summary

• publish robots.txt with sitemap and explicit search/AI crawler access
• publish AgentSea llms.txt with canonical product and priority guide URLs
• add self-canonicals to the homepage and static pages
• add crawlable homepage links to the OpenClaw and Hermes guides
• make invalid agent/cloud routes return 404 instead of 500
• stop emitting synthetic deployment-time lastmod values and replace the stale spawn.thegrid.ai fallback

Verification

• UI typecheck passes
• UI build passes (44 static pages generated)
• UI lint passes with no warnings or errors
• local production server returns correct canonicals, robots.txt, llms.txt, and 404s for invalid routes
• full repository suite: 941 pass, 5 skip, 3 pre-existing TTY/OAuth fallback failures in this non-interactive environment

[vercel]

@Cahl-Dee is attempting to deploy a commit to the Spectral Developers Team on Vercel.

A member of the Team first needs to authorize it.

[Cahl-Dee]

Follow-up: the sitemap is now limited to agent/cloud combinations that are actually exposed in the homepage UX (Local and DigitalOcean for the five public agents). CLI-only/hidden providers such as Sprite, AWS, GCP, Hetzner, and Daytona remain routable but are no longer advertised for indexing. This reduces the sitemap from 39 URLs to 14 and avoids presenting unverified hidden deployment paths as public landing pages.