AI crawlers still do not run JavaScript

They still don't. The one measurement everyone cites is Vercel's, published December 2024 and never re-run. searchVIU tested five AI systems again in December 2025 and only Gemini rendered on a live fetch. If your body text exists only after JavaScript runs, the crawler leaves with an empty page and content quality never comes into it.

Rendering here means executing the JavaScript on a page and reading the DOM that comes out the other end. Googlebot does it on a second pass. A crawler that skips that pass sees the HTML from the first response, which on a client-rendered React app is a near-empty shell with a script tag in it.

Desses sells an AEO audit that opens with crawlability and rendering checks. Telling you your rendering is broken is how we get paid. There's a one-line command below that answers the question without me.

What Vercel measured, and why the date matters

Vercel monitored its own network and nextjs.org and published the results in December 2024. ChatGPT's crawler pulled JavaScript files as 11.50% of its total requests. Claude's pulled them as 23.84%. Neither executed a single one. Applebot rendered, and Gemini rendered because it runs on Googlebot's infrastructure. Everything else fetched the file, stored it, and moved on.

Now the part most articles quoting this study skip. December 2024. Nobody has re-run it, including the people citing it every week, including this page. A 2024 measurement describing 2026 crawlers is old evidence about software that ships new versions constantly, and it remains the canonical citation because it's the only one at that scale. Being the only one is not the same as being current.

searchVIU checked again a year later

The independent confirmation came on 2 December 2025. searchVIU built one test page with eight product prices placed five different ways, then queried five AI systems against it.

Gemini pulled 50% of the prices and was the only system that rendered JavaScript on a direct live fetch. ChatGPT managed 37.5%, from visible HTML only. Google AI Mode got 25%, Perplexity 12.5%, and Claude extracted nothing.

AI Mode and Perplexity both picked up the JavaScript-rendered price after indexing, so their crawlers do execute JavaScript somewhere in the pipeline. Neither runs it at the moment a user asks a question. If a page has to be fetched fresh to be useful, it has to be readable fresh.

One page, one run, so treat it as directional. Two methods, a year apart, landing on the same answer is better evidence than either one alone.

How much of your traffic this is

Cloudflare's own published line is that more than 50% of traffic on the Internet is now non-human. In its 2025 Radar Year in Review, published 15 December 2025, AI crawlers were 20% of verified bot traffic, and AI crawlers were the most frequently fully disallowed user agents found in robots.txt files. Sharper decimals circulate widely and most are readings taken off a live dashboard on one afternoon, so check what a bot-traffic percentage measures before you quote it.

Training bots and retrieval bots are not interchangeable

A training bot collects text to build a model with. A retrieval bot fetches your page to answer a question a person is asking right now, and it decides whether you appear in the answer. Both arrive over HTTPS, both read robots.txt, and most robots.txt files treat them as one thing.

Blocking GPTBot costs you nothing in AI search. Blocking OAI-SearchBot removes you from ChatGPT.

Anthropic publishes its crawler IP ranges as a machine-readable file, and you should verify by IP rather than by user agent string. The file is IP prefixes only, no bot names, so it tells you a request came from Anthropic and not which of its three agents sent it. A user agent is a line of text anyone can type.

Two errors this whole category repeats

Blocking Google-Extended does not opt you out of AI Overviews

Google's own documentation says Google-Extended "does not impact a site's inclusion in Google Search nor is it used as a ranking signal". It's a control token for training and grounding, and it doesn't crawl anything. AI Overviews are built on Googlebot's index. Add Google-Extended to robots.txt and your pages keep appearing in AI Overviews exactly as before.

The only user agent that removes you from AI Overviews is Googlebot, and blocking Googlebot removes you from Google. Applebot-Extended works the same way: Apple's documentation says outright that it "does not crawl webpages". This one survives because the name has "Google" in it and because blocking something feels like doing something.

robots.txt no longer governs user-directed agents

Perplexity's own documentation says Perplexity-User generally ignores robots.txt rules. OpenAI runs ChatGPT-User, Anthropic runs Claude-User, and Meta ships the same class of fetcher. The published reasoning is consistent across all of them: a person pasted a URL and asked for the page, and every provider treats that fetch as the person's, not the crawler's.

You can agree or disagree with that reading. It's now how the industry operates, and a robots.txt written on the assumption that it covers everything arriving at your server is a robots.txt describing 2022.

When client-side rendering is the right call

Server rendering costs money. A pre-rendered page is a file on a CDN. A server-rendered one is a running process, a cache layer, cold starts and a per-request bill, plus somebody who owns it at 3am. That's a real trade.

A product surface behind a login should stay client-rendered. Dashboards, editors, admin panels, anything user-specific: no crawler should see it, no crawler will see it, and rebuilding it for AI visibility is spending money to make an unindexed page load slower. We turn down rebuilds like that. It's also worth knowing how small this channel still is before anyone rearchitects anything for it.

The split that solves this for most teams is boring and it works. Marketing and content pre-rendered. Application client-rendered. Separate route groups or separate subdomains, one build each.

What we'd do

Start with the check. Ten seconds, and it settles the argument:

curl -s -A "GPTBot" https://yoursite.com/your-page | grep -c "a distinctive phrase from your body copy"

Zero means your content isn't in the HTML. Run it on a blog post, a product page and your homepage, since a lot of sites are fine on two out of three.

On Framer this comes back clean by default. Framer's documentation states that every page is pre-rendered to HTML on its servers before it's served, and that agents which don't execute JavaScript still receive the full page text. Framer also serves a Markdown version of any optimised page through an Accept: text/markdown request header or a ?md suffix, which is the feature almost nobody knows about.

On Next.js the answer is architectural. Anything that has to be readable gets static generation or server components, and the client boundary starts below the content. That's the bulk of our AI and custom development work: deciding which parts of an app were never supposed to be interactive.

Then robots.txt, written from the table rather than from a template. Allow Googlebot, Bingbot, OAI-SearchBot, Claude-SearchBot and PerplexityBot without exception. Decide on GPTBot and ClaudeBot as a training question, on its own merits, knowing the decision has no effect on whether you get cited.

Last, read your server logs. Log analysis is the only method that shows whether retrieval bots reach your content at all. The third-party tools sample prompts and infer from the answers, which tells you about the model and not your server.

Our own log study, unfinished

We're running one. Server logs across client Framer and Next.js sites, looking at which AI user agents arrive, what they request, what they get back, and whether any of them behave like they're executing anything. The point is to put a 2026 number next to Vercel's 2024 one, with the sample size and the window printed beside it.

It isn't finished and I won't preview a result we don't have. When it's done we'll publish it, including if the answer is that nothing has changed. A study you only publish when it's interesting isn't a study.

Send me your URL and I'll run it as GPTBot, then send back what came out of the HTML and what didn't. Takes about five minutes and you'll know where you stand.