What AI crawler logs really say: Markdown, llms.txt and fake bots

Shared analyses of AI crawler traffic point to three lessons: serve clean text, don't over-rely on llms.txt, and never trust a user-agent alone.

Several practitioners recently shared aggregated server logs covering AI crawler traffic across many websites. The figures are theirs and we have not verified them independently, but the patterns match what we observe when reading logs for stores, and they are useful to anyone who wants products discovered, cited and recommended by AI.

1. AI crawlers like clean text

When a site offered a Markdown version of its pages, a large share of GPTBot and ClaudeBot requests reportedly went to that version rather than the HTML. That is logical: a model wants facts, not navigation, scripts and cookie banners.

For a store, the practical takeaway is not to rebuild everything in Markdown. It is to make sure product facts (name, brand, price, currency, availability, key attributes) exist in server-rendered HTML and in structured data, and stay consistent between both.

2. llms.txt is a map, not a ranking lever

The same analyses report that llms.txt files are fetched rarely by general AI crawlers. We still publish one for every store because it costs nothing and helps disambiguate the entity, but it never replaces a complete Product schema and dense product pages.

3. A user-agent proves nothing

A meaningful share of requests claiming to be GPTBot or ClaudeBot reportedly came from ordinary scrapers borrowing the name. On small sites the ratio can be far worse, because real AI crawlers visit rarely while scrapers never stop.

4. Read your own site, not the average

Pooled numbers hide huge variation between sites. What matters is your catalog: which pages real crawlers fetch, how often, and whether those pages carry the facts an answer engine needs.

How MagicGEO applies this

MagicGEO focuses on the product page itself: consistent facts, complete JSON-LD, a per-product GEO score explaining what is missing, and AI visibility measured repeatedly rather than from a single check. llms.txt is generated as a complement, and crawler visits are logged so trends can be read per store. Our own site also answers Markdown requests on its home page with a curated text summary.