Common Crawl Checker
Free, no sign-up: see whether a domain appears in the three latest Common Crawl archives and what its robots.txt allows AI crawlers such as GPTBot, ClaudeBot and CCBot to do. Common Crawl is an open web archive that many language models were trained on. Treat the result as a signal, not as proof.
Check a domain
tinyweb.com is in Common Crawl.
Example for tinyweb.com- GPTBot
- allowed
- ClaudeBot
- allowed
- CCBot
- allowed
- Google-Extended
- allowed
- Applebot-Extended
- allowed
- OAI-SearchBot
- allowed
- Claude-SearchBot
- allowed
- PerplexityBot
- allowed
- September 2026 (CC-MAIN-2026-39)
- 22 records
- August 2026 (CC-MAIN-2026-34)
- 14 records
- Juli 2026 (CC-MAIN-2026-30)
- 19 records
create an llms.txt and AI crawler rules
Query date: 9 September 2026. Eight bots for training and AI search are checked.
Your result
What your Common Crawl Checker result means
Short answer: If your domain is in the latest archive and CCBot is allowed, your pages can end up in future training data. If it is missing, the usual reasons are few links pointing to your site, a young domain or a block.
| Result | What it means | What to do |
|---|---|---|
| In the latest archive, CCBot allowed | Pages from your domain are in the newest open archive. Model builders who use Common Crawl can use them. | Nothing to change. Check that the archived URLs are your key pages. |
| Only in older archives | Common Crawl samples different pages each month. A month without hits is normal. | Check again next month. If you miss three in a row, see the next row. |
| In none of the three archives | Few external links, a new domain, or a block in the past. | Review robots.txt, earn links from known sites, keep your sitemap current. |
| CCBot blocked | New content stays out of future archives. Past archives remain. | Decide on purpose: training yes or no. |
| Search bots blocked (OAI-SearchBot, Claude-SearchBot, PerplexityBot) | AI search tools stop citing your pages as sources. | Usually a mistake. Fix it with the generator. |
| robots.txt not reachable | Crawlers treat a server error as a block. | Have your server or host checked. |
Use cases
Three ways SEO and PR teams use a Common Crawl checker
Short answer: Check your own domain, vet media outlets before outreach, and find old URLs that survive in archives after a relaunch.
Check your own site
Is your domain in the archives that language models learn from, and does your robots.txt let in the AI crawlers you want?
Vet outlets before you pitch
A guest article only helps in AI answers if AI systems know the outlet. Check target publications before you invest in outreach.
Spot old URLs after a relaunch
Archives keep old addresses. The list of archived URLs shows which redirects your relaunch needs.
Example
Common Crawl Checker example: tinyweb.com
Short answer: tinyweb.com appears in all three latest archives, with 14 to 22 records per month. The September 2026 archive still holds two URLs that our relaunch redirects.
| Archive | Capture window | Records |
|---|---|---|
| CC-MAIN-2026-39 | 4 to 17 Sep 2026 | 22 (21 pages plus robots.txt), captured on 9 Sep 2026 |
| CC-MAIN-2026-34 | 7 to 20 Aug 2026 | 14 |
| CC-MAIN-2026-30 | 10 to 23 Jul 2026 | 19 |
What we take from it: the September archive still lists two old German service URLs that now redirect. Models trained on it know the old structure. Clean redirects make sure crawlers and AI search find the new address.
Limits
What a Common Crawl result does not tell you
Short answer: A hit in the archive does not mean a given model learned your page. A miss does not mean no model knows you.
- Archive is not model. Providers filter Common Crawl before training and add other sources. Most do not disclose which archives went into which model.
- Time lag. A model learns from data up to a cutoff date. New pages only show up in later models.
- Sample, not full copy. Common Crawl archives part of a domain’s pages, and most of its content is in English.
- Blocks only work forward. Blocking CCBot removes nothing from past archives.
- Home page as test path. Bot values check the path
/. Rules for single folders show as “partly blocked”. - Not a Google crawl test. Whether Google crawls and indexes your pages is shown in Google Search Console.
Method and sources behind the Common Crawl Checker
Short answer: The checker queries Common Crawl’s public index API for the three newest archives and reads your robots.txt following RFC 9309. Requests run one after another, because Common Crawl rate-limits the API heavily.
- Archives: the three newest entries in index.commoncrawl.org/collinfo.json. On 21 October 2026 these were CC-MAIN-2026-39 (September), CC-MAIN-2026-34 (August) and CC-MAIN-2026-30 (July). In 2026 Common Crawl published one archive per month.
- Query: all records for the domain and its subdomains (
matchType=domain), up to 500 per archive. - robots.txt: evaluated per bot for the path
/under RFC 9309. Bot list from the documentation of OpenAI, Anthropic, Google, Perplexity, Apple and Common Crawl, as of October 2026. - Context: Mozilla Foundation, “Training Data for the Price of a Sandwich” (February 2024): at least 64 % of 47 language models published between 2019 and October 2023 used Common Crawl.
- No storage: results stay in memory for up to one hour to respect the rate limit, then they are gone.
Common Crawl Checker FAQ
What is Common Crawl?
If my site is in Common Crawl, does ChatGPT know it?
How do I test whether AI crawlers can access my site?
How often is a new Common Crawl archive released?
How do I remove my site from Common Crawl?
User-agent: CCBot and Disallow: /. It applies to future archives. Archives already published stay as they are.Next step
Turn the result into rules for AI crawlers
If the check shows bots you did not mean to block, adjust your AI crawler rules with the generator. tiny web is Hans Jung’s consultancy in Munich; consulting runs in German for companies in Germany, Austria and Switzerland. See our SEO consulting for small and mid-sized companies (in German), or book an introductory call.