llms.txt / ai.txt Inspector
Check a domain’s AI crawler files — llms.txt, ai.txt, and AI policy — and cross-check them against robots.txt.
What is the llms.txt / ai.txt Inspector?
The llms.txt / ai.txt Inspector checks how a domain communicates with AI crawlers and assistants. It looks for the emerging set of AI guidance files — llms.txt and llms-full.txt (curated content maps for language models), ai.txt (robots-style rules for AI crawlers), and a .well-known AI policy file — then parses and validates each one.
Because these files live at fixed paths on the registered domain, this is a domain-level check: the path of the URL you enter does not matter, only the domain that hosts the files. The inspector also reads robots.txt and cross-checks well-known AI crawlers so you can catch conflicts between what your files invite and what they block. It returns an A–F grade and a clear list of fixes.
How to Use the Inspector
- Enter a domain name or URL and click Inspect llms.txt.
- Read the Summary tab to see which AI guidance files the domain publishes.
- Open the llms.txt tab to browse the parsed title, summary, sections, and links.
- Use the ai.txt tab to review each crawler’s Allow and Disallow directives.
- Check the Robots Cross-Check tab for AI-bot conflicts, then read Findings and export from Raw.
How the Inspector Works
When you submit a domain or URL, we extract the registered domain and request the well-known AI files over HTTPS: /llms.txt, /llms-full.txt, /ai.txt, and /.well-known/ai-policy.json. Missing files are treated as simply absent rather than errors, so a domain that publishes only some of them still returns a clean report.
For llms.txt we parse the Markdown structure — the H1 title, the blockquote summary, each ## section, and the curated links with their descriptions — and present it as a browsable outline. For ai.txt we parse each User-Agent group and its Disallow and Allow directives, and we capture the AI policy JSON as-is.
We then fetch robots.txt and evaluate how it treats a list of well-known AI crawlers, comparing that against what your ai.txt and policy files declare. Contradictions, invalid llms.txt formatting, and missing files each produce a finding, and the combined result maps to an A–F grade so you can see how coherent and complete your AI guidance is.
Key Features
- Full file discovery — llms.txt, llms-full.txt, ai.txt, and the .well-known AI policy in one pass.
- llms.txt outline parser — Title, summary, sections, and links rendered as a readable tree.
- ai.txt directive parsing — Every User-Agent group with its Allow and Disallow rules.
- Robots cross-check — Well-known AI crawlers compared against robots.txt to surface conflicts.
- Graded findings — An A–F score with contradictions and format issues called out.
- Raw JSON — The complete response for scripting and monitoring.
Common Use Cases
- Content and SEO teams — Confirm your best pages are exposed to AI assistants via llms.txt.
- Site owners — Decide and clearly declare how AI crawlers may use your content.
- Developers — Validate a newly published llms.txt or ai.txt for structure and conflicts.
- Compliance reviewers — Check that AI policy files and robots.txt tell a single, consistent story.
Frequently Asked Questions
What is llms.txt?
llms.txt is a proposed standard file, served at the root of a domain as /llms.txt, that gives large language models a curated, Markdown map of your most important content. It typically starts with an H1 title, a short blockquote summary of the site, and then ## sections containing lists of links — each a Markdown link with an optional description — that point to the pages you most want an AI assistant to read. Think of it as a sitemap written for language models rather than search engine crawlers.
How is llms.txt different from ai.txt and robots.txt?
They solve different problems. robots.txt controls which crawlers may access which paths. ai.txt uses the same robots-style User-Agent / Disallow / Allow syntax but is aimed specifically at AI and training crawlers, letting you state whether your content may be used for AI. llms.txt is not about permission at all — it is a positive, curated guide that helps a model find and understand your best content. Many sites publish all three, so this tool checks for each and shows how they interact.
What does this inspector check?
It looks for /llms.txt, /llms-full.txt, /ai.txt, and /.well-known/ai-policy.json on the domain you enter. For llms.txt it parses the title, summary, sections, and links so you can see the outline a model would read. For ai.txt it parses each User-Agent group and its Disallow and Allow directives. It also reads robots.txt and cross-checks well-known AI crawlers, so you can spot contradictions such as inviting a bot in one file while blocking it in another.
Do I need an llms.txt file?
It is optional and still an emerging convention, but publishing one is low-cost and increasingly worthwhile as AI assistants become a real source of traffic and citations. A good llms.txt makes it far easier for a model to find your canonical documentation, product pages, or articles rather than guessing from raw HTML. If you already maintain a sitemap, writing a focused llms.txt that highlights your best pages is a natural next step.
What is a policy contradiction?
A contradiction happens when your files disagree about a crawler. For example, your ai.txt might Allow a specific AI bot while your robots.txt Disallows that same bot, or your AI policy file might claim content is off-limits while llms.txt actively lists it for models to read. Crawlers resolve these conflicts inconsistently, so the inspector flags contradictions explicitly and lowers the grade until your files present one coherent intent.