Claude Scans Your Site Live: Experiment Tests AI Legibility in Real Time

Claude Opus 4.6’s live test exposed a fundamental mismatch between modern JavaScript‑heavy sites and the way Anthropic’s agents crawl the web. According to Johnbrennan’s “Can Claude Read Your Website?” case study, all three React single‑page applications initially returned empty HTML shells, leaving Claude with no article content, navigation links, or discoverable URLs. The agent’s fetch tools do not execute client‑side JavaScript, so any content rendered after the initial load is invisible to the model—a problem that turns today’s SPA‑driven sites into “dark” pages for AI readers.

The experiment’s turning point was the addition of a plain‑text sitemap ( sitemap.txt ). Johnbrennan reports that a single file listing one URL per line transformed each opaque site into a fully navigable corpus, allowing Claude to autonomously discover and read every piece of content. This finding aligns with broader industry observations that AI legibility hinges on static, server‑generated signals rather than dynamic rendering. In parallel, the study confirmed that server‑side HTML injection can restore visibility, but edge‑caching layers may mask those injections, as a stale cache kept a site dark for more than an hour despite a correctly injected payload.

Beyond sitemaps, the report highlights the value of structured Markdown endpoints ( .md  files). Because Markdown carries clean front‑matter metadata and a hierarchical layout, Claude can parse, cite, and reason about the content with “zero friction,” according to the author. Conversely, novel file extensions such as .toon — used for the Unified TOON Meta‑Index (UTMI) — were initially unreadable when served with the default binary MIME type (application/octet‑stream). Once the MIME type was corrected to a plain‑text designation, Claude could immediately parse the consolidated index, which bundles robots.txt rules, sitemap data, llms.txt references, and API tool registrations into a single token‑optimized manifest.

The study also underscores the critical role of a discoverable homepage. Johnbrennan notes that even when every endpoint on a site is correctly exposed, an AI agent stalls if the root page returns nothing navigable. The homepage serves as the entry point for Claude’s crawling logic; without it, the agent cannot bootstrap its traversal, effectively rendering the rest of the site invisible. This insight dovetails with recent coverage of Anthropic’s broader tooling, such as the Claude Code Security suite that identified 500+ vulnerabilities, emphasizing that the same attention to surface‑level accessibility must extend to AI‑focused content delivery (VentureBeat).

Collectively, these findings suggest that developers seeking AI‑readable sites must adopt a “server‑first” mindset: provide static sitemaps, expose content in plain‑text or Markdown formats, ensure correct MIME types for custom files, and guarantee a functional homepage. As Claude and competing agents become integral to search, summarization, and downstream applications, the gap between SPA convenience and AI legibility could become a competitive differentiator. Websites that fail to adapt may find their content excluded from emerging AI‑driven ecosystems, while those that implement the outlined artifacts will likely enjoy broader visibility and new revenue channels.