llms.txt Generator: Create llms.txt and AI Crawler Rules
Builds an llms.txt file that follows the llmstxt.org specification, plus robots.txt rules for GPTBot, ClaudeBot, Google-Extended and other AI crawlers, with training and search handled separately. Runs entirely in your browser. Nothing is sent or stored.
For sites that want to point AI systems to their key pages, or block specific AI crawlers. Not a Google ranking tool: Google Search does not read llms.txt.
Generator
Your llms.txt follows the llmstxt.org specification (version 2).
Created in your browser. Your input is not sent anywhere and not stored. Reloading the page clears it.
llms.txt
# tiny web GmbH
> tiny web GmbH is a Munich-based consultancy run by Hans Jung. It builds free SEO tools and AI workflows for marketing teams in small and mid-sized companies.
Consulting services are offered in German for clients in Germany, Austria and Switzerland. All tools are free and need no sign-up.
## Tools
- [llms.txt Generator](https://tinyweb.com/en/llms-txt-generator/): llms.txt and robots.txt rules for AI crawlers
- [Common Crawl Checker](https://tinyweb.com/en/common-crawl-checker/): check whether a domain is in the Common Crawl index
- [HTTP Header Canonical Generator for PDF](https://tinyweb.com/en/pdf-header-canonical-generator/): canonical tags for PDFs on Apache and Nginx
- [Ultimate 410 Gone WordPress Plugin](https://tinyweb.com/en/ultimate-410-status-code-wp-plugin/): serve 410 instead of 404 for deleted content
## Company
- [About tiny web and Hans Jung](https://tinyweb.com/en/about-us/): background, talks and way of working
## Optional
- [Contact](https://tinyweb.com/en/contact/): book an introductory call
robots.txt
# AI crawler rules, created with the llms.txt Generator by tiny web
# Bot list as of October 2026, template: Visible in search and training
# Training
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
Allow: /
# AI search
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /
# User fetch (robots.txt only partly applies)
User-agent: ChatGPT-User
User-agent: Claude-User
User-agent: Perplexity-User
Allow: /
Sitemap: https://tinyweb.com/sitemap.xml
Created in your browser. Nothing is sent or stored.
| Key | English text |
|---|---|
| err.site_name | Please enter a name with up to 80 characters. |
| err.site_url | Please enter your domain with https, for example https://your-domain.com. |
| err.summary | Please describe your site in 20 to 400 characters. |
| err.notes | Notes cannot contain headings. Remove the # at the start of the line. |
| err.section | Please give this section a name. |
| err.section_optional | Use the checkbox below for an “Optional” section. |
| err.link_title | Please add a title for this link. |
| err.link_url | Please enter a full address or a path starting with /. |
| err.link_note | The note can have up to 200 characters. |
| err.limit | An llms.txt should be a selection. Up to 200 links are allowed. |
| warn.other_host | One link points to another domain. That is allowed, but check that you meant it. |
| warn.http | One link uses http. Use https where you can. |
| warn.duplicate | One link appears twice. We kept it once. |
| warn.few_links | Your llms.txt has fewer than three links. Add your most important pages. |
| warn.md | Check that the .md copies of your pages exist. Otherwise these links lead nowhere. |
| warn.block_all | Your robots.txt blocks all crawlers. Google cannot read your pages either. |
| warn.googlebot | Your robots.txt blocks Googlebot. Your pages will be missing from Google Search and AI Overviews. |
| warn.size | Your robots.txt is larger than 500 KiB. Crawlers may not read all of it. |
| warn.line | Line {n} could not be read and was left unchanged. |
| status.ok | Your llms.txt follows the llmstxt.org specification (version 2). |
| status.warn | Your files are ready. Please review {n} note(s) below. |
| status.error | Please fix the highlighted fields. |
| copy.fallback | Please copy with Ctrl+C or Cmd+C. |
| download.llms | Upload the file to the root of your site so it is available at {site_url}/llms.txt. |
| download.robots | Upload the file to the root of your site so it is available at {site_url}/robots.txt. |
| comment.head | # AI crawler rules, created with the llms.txt Generator by tiny web |
| comment.list | # Bot list as of October 2026, template: {template} |
| comment.training / search / user | # Training · # AI search · # User fetch (robots.txt only partly applies) |
Your result
What your llms.txt and robots.txt rules actually do
Short answer: Your llms.txt gives language models and AI agents a short map of your key pages. Your robots.txt rules decide which AI crawlers may read your site. Only robots.txt is binding, and only for crawlers that follow the standard.
The llms.txt file
It opens with your name and a one-paragraph summary, followed by sections of links with short notes. An AI system that reads it learns in a few lines who you are and which pages matter. Google Search does not use it. Ahrefs found that 97 % of llms.txt files received no request at all in May 2026. Treat it as a small extra, not a ranking lever.
The robots.txt rules
Each block names one or more crawlers and says
AlloworDisallow. Once a crawler has its own block, your generalUser-agent: *group no longer applies to it. That is why the generator copies your blocked paths into every allowed block. For user-triggered fetches, robots.txt only partly helps: OpenAI and Perplexity say these fetches do not always follow it.Which template to pick
Ask one question first: does your business lose money if models learn from your content? For most company sites the answer is no, and a model that knows your brand is more likely to mention it. Publishers and paid databases often block training and keep search open.
How to
How to upload and test your llms.txt and robots.txt
Short answer: Both files belong in the root of your domain. Open them in a browser to confirm they load. Expect effects over days to weeks.
- 01
Upload the files
Put llms.txt and robots.txt in your site root, for example via FTP, your CMS or an SEO plugin with a robots.txt editor. Only replace an existing robots.txt if you pasted it into the generator first.
- 02
Check they load
Open your-domain.com/llms.txt and your-domain.com/robots.txt. Both should load as plain text without a redirect. Chrome Lighthouse flags a server error on llms.txt.
- 03
Watch the effect
Crawlers re-read robots.txt; RFC 9309 limits caching to 24 hours. OpenAI and Perplexity name about 24 hours for their search bots.
AI crawlers in detail
Eleven AI crawlers: purpose, group and what blocking does
Short answer: Providers split training, search and user fetches into separate bots. You can stay citable in AI answers while keeping your content out of training.
| Token | Provider | Group | If you block it | Source, date |
|---|---|---|---|---|
| GPTBot | OpenAI | Training | no training on new content; ChatGPT search still possible | OpenAI crawler docs, retrieved 21 Oct 2026 |
| ClaudeBot | Anthropic | Training | future content excluded from training | Anthropic, 7 Apr 2026 |
| CCBot | Common Crawl | Training (archive) | missing from future archives | Common Crawl, retrieved 21 Oct 2026 |
| Google-Extended | Gemini training and grounding | no effect on Google Search or AI Overviews | Google, 15 Oct 2026 | |
| Applebot-Extended | Apple | Training | no effect on Siri, Spotlight or Safari | Apple, 4 Sep 2026 |
| OAI-SearchBot | OpenAI | AI search | no longer a source in ChatGPT search answers | OpenAI crawler docs, retrieved 21 Oct 2026 |
| Claude-SearchBot | Anthropic | AI search | not in Claude’s search index | Anthropic, 7 Apr 2026 |
| PerplexityBot | Perplexity | AI search | missing from Perplexity answers | Perplexity, retrieved 21 Oct 2026 |
| ChatGPT-User | OpenAI | User fetch | robots.txt may not apply | OpenAI crawler docs, retrieved 21 Oct 2026 |
| Claude-User | Anthropic | User fetch | Claude cannot read your page on request | Anthropic, 7 Apr 2026 |
| Perplexity-User | Perplexity | User fetch | robots.txt generally ignored | Perplexity, retrieved 21 Oct 2026 |
Footnote: Google-Extended and Applebot-Extended are control tokens, not crawlers. Some crawlers fake the names CCBot or GPTBot; providers publish IP lists for verification (for example openai.com/gptbot.json and index.commoncrawl.org/ccbot.json).
Limits
What this llms.txt generator does not do
Short answer: It writes files. It does not read your site and cannot promise that any AI assistant will cite you.
- It does not crawl your site. You pick the pages, because an llms.txt is meant to be a selection, not a sitemap.
- It does not check whether linked pages load.
- It does not protect data. robots.txt is a request to crawlers, not access control.
- It cannot reliably block user-triggered fetches. That needs server rules, for example based on the providers’ IP lists.
- It does not improve Google rankings. Indexing, content and links do that.
When this is not enough: if your site has several sections with their own documentation, add one llms.txt per section, such as /docs/llms.txt. The spec says the most specific file applies.
Method and sources behind the llms.txt generator
Short answer: The llms.txt follows llmstxt.org version 2, dated 10 August 2026. The robots.txt rules follow RFC 9309. The bot list comes from the documentation of OpenAI, Anthropic, Google, Perplexity, Apple and Common Crawl as of October 2026.
We review the bot list once a year and whenever a provider changes its documentation. Sources: llmstxt.org specification v2 (Jeremy Howard, Answer.AI), 10 Aug 2026 · RFC 9309 Robots Exclusion Protocol, September 2022 · OpenAI, Overview of OpenAI Crawlers, retrieved 21 Oct 2026 · Anthropic Help Center, 7 Apr 2026 · Google, Google’s common crawlers, 15 Oct 2026, and Guide to optimizing for generative AI features, 10 Jul 2026 · Perplexity Crawlers, retrieved 21 Oct 2026 · Apple, About Applebot, 4 Sep 2026 · Common Crawl, CCBot and FAQ, retrieved 21 Oct 2026 · Ahrefs llms.txt study of 137,210 domains, 15 Jun 2026.
“llms.txt Generator by tiny web, https://tinyweb.com/en/llms-txt-generator/, October 2026.”
llms.txt generator FAQ
Where do I put my llms.txt file?
Do I have to replace my current robots.txt?
Does the tool store what I enter?
Does llms.txt help with Google?
Can I block ChatGPT from reading my site?
Next step
Want a second pair of eyes on your AI visibility?
tiny web is Hans Jung’s consultancy in Munich. Consulting runs in German, for companies in Germany, Austria and Switzerland. If that fits, see our SEO consulting for small and mid-sized companies (in German). Otherwise, book an introductory call.