What Is llms.txt? A Guide to Managing AI Crawlers

Q: Where on my site should I put the llms.txt file?

In the root directory, so it can be reached at `https://yoursite.com/llms.txt`. Putting it in your static file folder (`public/`, `static/` or the server's root folder) is enough. An llms.txt placed in a subdirectory is not looked for by crawlers.

Q: What is the difference between llms.txt and llms-full.txt?

`llms.txt` is a table of contents: headings, short descriptions and links. `llms-full.txt` collects the plain-text version of all the content in a single file. The second usually makes sense for documentation sites; for a company site or a blog, llms.txt alone is enough and far easier to maintain.

llms.txt is a summary file in your site's root directory that tells AI models, in plain Markdown, what your site is, which pages matter and where to look.

Jeremy Howard of Answer.AI published it as a proposal in September 2024. The starting point was a simple annoyance: a language model visiting your site meets navigation menus, cookie notices, JavaScript-loaded components and ad boxes. With a limited context window, digging "what does this site do, who does it serve, which page is useful to me" out of that noise is expensive.

llms.txt makes that cheaper, but alone it is not enough. Before a crawler can read the file it has to get into your site, and that is where most ground is lost: even with a correct robots.txt, the CDN layer in front can turn the bot away silently. This article covers how the file is written and how those invisible barriers are found, using our own live configuration.

What problem does llms.txt solve?

A site's HTML is designed for people. On a product page the real information might be 200 words while the source is 300 KB. Processing that 300 KB either burns a large share of the model's context or forces it to read the page truncated — both work against you.

llms.txt puts a clean, hand-written table of contents next to that clutter. It contains:

The site's identity in a single sentence (who, what they do, for whom)
A list of services or products with short descriptions
Full URLs of the important pages and what they contain
Blog or documentation titles, each with a one-line summary
A way to get in touch

The return is the chance of being quoted directly. Asked for "an agency doing GEO for clinics in Istanbul", ChatGPT or Perplexity prefers the clearest, least ambiguous source. Write your identity in your own sentence and the model does not have to guess it.

What is the difference between llms.txt and robots.txt?

The two get confused often, but their jobs are opposites.

robots.txt is a permission file. It answers "may this crawler enter this folder?" It is a decades-old, agreed-upon protocol and crawlers generally obey it.

llms.txt is a content file. It blocks nothing and releases nothing. It says "now that you're in, here is the summary." It is not an official standard either; it is a proposal, and no provider is guaranteed to read it.

So the order matters: first you open the door with robots.txt, then you put directions inside with llms.txt. Writing directions while the door is locked is pointless.

There is a third layer too: structured data. JSON-LD schema repeats what llms.txt says in machine-verifiable form and signals that this is a marked-up entity; we covered that side in our article on Schema Markup and E-E-A-T.

What does a real llms.txt look like?

Below is a shortened version of the live llms.txt on nextgeoagency.com:

# Next GEO Agency

> Turkiye merkezli GEO (Generative Engine Optimization) ve SEO ajansi.
> Yerel hizmet isletmelerinin ve profesyonel danismanlarin ChatGPT, Gemini,
> Perplexity ve Google gibi arama motorlarinda bulunmasini saglayarak musteri
> kazanmasina calisir. Hizmet verdigi baslica alanlar: klinikler,
> hukuk burolari, emlak ofisleri, guzellik salonlari ve KOBI'ler.

Site: https://nextgeoagency.com
Diller: Turkce (varsayilan), Ingilizce
Guncelleme: 2026-08-24

## Hizmetler
- GEO (Generative Engine Optimization): ChatGPT, Gemini, Claude, Perplexity ve
  Google SGE gibi yapay zeka arama motorlarinda marka gorunurlugu
- SEO: teknik SEO, anahtar kelime arastirmasi, icerik optimizasyonu, yerel SEO
- Kurumsal Web Sitesi Tasarimi: hizli, mobil uyumlu, SEO ve GEO uyumlu siteler

## Sayfalar
- [Ana Sayfa](https://nextgeoagency.com/)
- [Cozumler](https://nextgeoagency.com/cozumler)
- [Blog](https://nextgeoagency.com/blog)
- [Iletisim](https://nextgeoagency.com/iletisim)

## Makaleler
- [Schema Markup ve E-E-A-T](https://nextgeoagency.com/blog/108) — Schema.org
  yapılandırılmış verisi ve E-E-A-T sinyalleri nasıl kurulur?
- [GEO Stratejisi Nasıl Kurulur?](https://nextgeoagency.com/blog/112) — Sıfırdan
  GEO stratejisi için adım adım pratik yol haritası.

## Iletisim
https://nextgeoagency.com/iletisim

Three things to notice. First, the file opens with # Heading, then a > block quote — the specification requires only the H1, but the summary quote is the first thing the model reads and should never be empty. Second, every link is an absolute URL; the model may read the file cut off from its context, so a relative path is useless. Third, the description after the dash on each line looks optional but carries the real value: the model can read that line and use the page without visiting it.

How do you allow AI bots on the robots.txt side?

On most sites the problem is not a ban but an omission: Allow: / sits under User-agent: *, yet the AI crawlers are not named one by one. Some read the wildcard rule correctly, others look for a block in their own name; spelling it out is safest.

The relevant section of our own robots.txt:

# Icerik sinyalleri: arama, AI cevaplarinda kaynak gosterme ve
# model egitimi icin izin verilmistir.
User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=yes
Allow: /

# --- Yapay zeka arama motorlari ve asistanlar (GEO icin kritik) ---
User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: CCBot
Allow: /

Sitemap: https://nextgeoagency.com/sitemap.xml

The Content-Signal line comes from Cloudflare's content signals policy and declares three uses separately: indexing in search (search), being an input to an AI answer (ai-input) and use in model training (ai-train). It is a statement of preference, not an enforcement — but it beats declaring nothing.

Do not skip the Sitemap directive on the last line; many AI crawlers begin discovery from robots.txt.

Which bot belongs to whom, and what does it crawl for?

Look less at who owns the bot and more at what it does. The same company's search bot and training bot produce different outcomes: block the search bot and you vanish from AI answers; block the training bot and you only stay out of future models' memory.

Bot	Owner	What it crawls for	Should it be allowed?
`GPTBot`	OpenAI	General crawling for model training	Yes if visibility is the goal; debatable if content licensing is a concern
`OAI-SearchBot`	OpenAI	Showing the source or link in ChatGPT search results	Absolutely yes — this is the route to appearing in ChatGPT
`ChatGPT-User`	OpenAI	Fetching a page on the spot when a user supplies a link	Yes; blocking it means refusing a direct user request
`ClaudeBot`	Anthropic	General crawling and training	Yes
`Claude-SearchBot`	Anthropic	Citing sources in Claude's search answers	Absolutely yes
`PerplexityBot`	Perplexity	Building a search index, citing in answers	Absolutely yes
`Google-Extended`	Google	Not a separate crawler; the permission switch deciding whether your content is used in Gemini training and grounding	Yes — blocking it does not affect Google search but stops you being a source in Gemini
`Applebot-Extended`	Apple	Content-use permission on the Apple Intelligence side	Yes
`CCBot`	Common Crawl	Produces an open data set; many models' training data comes from here	Yes, it gives broad indirect reach
`Bytespider`	ByteDance	General crawling and training	Optional; if there are complaints of aggressive crawling, a rate limit can be considered

The most critical row is Google-Extended. Plenty of sites mistake it for a crawler and block it; it does not touch Googlebot's normal crawling at all, it only shuts your content out of Gemini answers. You lose on the GEO side without gaining a thing on classic SEO.

The nastiest trap: your CDN layer can override robots.txt

The most expensive information in this article is here. Your file may be written perfectly; if your site sits behind a CDN or proxy (Cloudflare, Fastly and the like), that layer can turn the request away without it ever reaching your server.

For Cloudflare there are two settings to watch:

"Block AI bots" / AI Crawl Control. Switched on with one click, it stops every AI crawler it recognises at the edge server. While it is on your robots.txt has no force whatsoever — the bot cannot even come to read the file. Cloudflare splits the control into three categories, Search, Agent and Training, and on newly added domains some can arrive blocked by default. On a site with a GEO goal all three must be Allow.
Managed robots.txt. If the CDN starts serving its own robots.txt, the file in your public/ folder is never seen. After a change, check from outside with curl, not from your browser — is the content you see really the content you wrote?

A third problem in the same family is almost never discussed: Email Obfuscation. This Cloudflare feature wraps page email addresses in JavaScript to protect them from spam bots; in the HTML source the address becomes a placeholder like [email protected], resolved only when JS runs in the browser. A large share of AI crawlers do not run JavaScript. The result: the model reads your contact page, cannot see the address and says "no contact information was found for this business." For local service businesses that is GEO taken straight through the heart. The fix: switch the feature off for the contact page, or give the address as plain text inside the JSON-LD schema as well.

We walked through testing whether this barrier really exists, with the bot user-agent and the commands, in are AI bots getting into your site.

How do you verify the setup with curl?

The page opening in your browser proves nothing; you are not a bot. The only way to verify is to send the request with a bot identity:

# 1) Are robots.txt and llms.txt really live, and is the content the one you wrote?
curl -s https://nextgeoagency.com/robots.txt | head -20
curl -s https://nextgeoagency.com/llms.txt  | head -15

# 2) Does the home page return 200 for a bot user-agent?
curl -A "GPTBot/1.1" -sI https://nextgeoagency.com/ | head -1
curl -A "OAI-SearchBot/1.0" -sI https://nextgeoagency.com/ | head -1
curl -A "PerplexityBot/1.0" -sI https://nextgeoagency.com/ | head -1
curl -A "ClaudeBot/1.0" -sI https://nextgeoagency.com/blog/112 | head -1

# 3) Is the HTML the bot receives full, or an empty <div id="root">?
curl -A "GPTBot/1.1" -s https://nextgeoagency.com/blog/112 | grep -o "<h1[^>]*>[^<]*" | head -3

# 4) Is the email address present as plain text in the source HTML?
curl -A "GPTBot/1.1" -s https://nextgeoagency.com/iletisim | grep -c "email-protection"

Expected results: every line in the second group should return HTTP/2 200. A 403 means a security layer has come in between. The third command should print a real heading — if it prints nothing, your site renders on the client and the bot sees an empty page; you then need server-side rendering or static generation. The fourth should return 0; 1 or more means email obfuscation is on.

Repeat the checks after every deploy. When a CDN setting is switched on by accident nothing visible changes on the site — you simply go silently invisible on the AI side.

Why you should not edit these files by hand

A hand-written llms.txt is accurate on day one and lying by month three. New articles get added and never reach the file; the model learns your site as it was six months ago.

On our site llms.txt and sitemap.xml are never opened by hand. scripts/generate-seo-files.ts reads the article list before every build and writes both from scratch:

// public/llms.txt — the article section is generated from the post list
const llms = `# Next GEO Agency

> ${ozet}

Guncelleme: ${bugun}

## Sayfalar
${sayfalar.map((s) => `- [${s.ad}](${SITE}${s.yol})`).join('\n')}

## Makaleler
${posts.map((p) => `- [${esc(p.title)}](${SITE}/blog/${p.id}) — ${esc(p.excerpt)}`).join('\n')}
`;
writeFileSync(resolve(root, 'public/llms.txt'), llms, 'utf8');

The script hangs off the prebuild step in package.json, so it fires every time npm run build runs. Publishing a new article means adding one record to the content file; sitemap, llms.txt and the update date correct themselves. No step is left for a human to forget.

The same logic works on any stack: a plugin or scheduled task on WordPress, a route handler in Next.js, a build script on a static site. What matters is that the file is generated, not written. The wider frame is in our guide to building a GEO strategy.

Frequently Asked Questions

Is llms.txt mandatory, and will my site be invisible in AI searches without it?

It is not mandatory and not an official standard. AI crawlers can crawl your site and use your content even without the file. llms.txt is a convenience layer that raises the odds of being understood correctly; what actually decides the outcome is your site being open to crawlers and your content being rendered on the server.

Where on my site should I put the llms.txt file?

In the root directory, so it can be reached at https://yoursite.com/llms.txt. Putting it in your static file folder (public/, static/ or the server's root folder) is enough. An llms.txt placed in a subdirectory is not looked for by crawlers.

If I allow GPTBot, will my content be used in model training?

Yes, that is GPTBot's declared purpose. If you do not want to be used in training but do want to appear as a source in ChatGPT answers, set GPTBot to Disallow and allow OAI-SearchBot and ChatGPT-User. Those two are for search and on-the-spot fetching, not training.

I fixed my robots.txt but bots still cannot get in, what could be the reason?

The most common cause is a CDN or security layer. With a proxy such as Cloudflare, the "Block AI bots" feature, bot management rules or a WAF rule can stop the request before it reaches your server. If curl -A "GPTBot/1.1" -sI https://yoursite.com/ returns 403, the problem is not in robots.txt but in the layer in front of it.

What is the difference between llms.txt and llms-full.txt?

llms.txt is a table of contents: headings, short descriptions and links. llms-full.txt collects the plain-text version of all the content in a single file. The second usually makes sense for documentation sites; for a company site or a blog, llms.txt alone is enough and far easier to maintain.

Running the curl commands above against your own domain takes ten minutes and often turns up a surprise. For a wider check, look at Next GEO Agency's technical infrastructure solutions or ask for a free technical audit from the contact page; we review robots.txt, llms.txt, CDN settings and render behaviour together and produce a concrete list of fixes.

llms.txt Nedir? AI Tarayıcı Yönetimi Rehberi