Unusual llms.txt Validator

Last updated: 2025-10-31Update (2025-11-21): Added a quick, copy-paste CLI and clarified common lints (200 OK, text/plain, ≤100 KB, no mixed content, working links).

Quick CLI (curl + jq — copy/paste)

Requires: curl, jq

DOMAIN=https://example.com; FILE="$DOMAIN/llms.txt"; \
SC=$(curl -s -o /dev/null -w "%{http_code}" "$FILE"); \
CT=$(curl -sI "$FILE" | awk 'BEGIN{IGNORECASE=1}/^content-type:/{sub(/^content-type:[[:space:]]*/,"");print;exit}'); \
CL=$(curl -sI "$FILE" | awk 'BEGIN{IGNORECASE=1}/^content-length:/{sub(/^content-length:[[:space:]]*/,"");print;exit}'); \
BODY=$(curl -s "$FILE"); \
LU=$(printf "%s" "$BODY" | awk 'BEGIN{IGNORECASE=1}/^

# *Last-Updated:/{sub(/^

# *Last-Updated:[[:space:]]*/,"");print;exit}'); \

LINKS=$(printf "%s" "$BODY" | grep -Eo '(https?://[^ )]+)' | head -n10); \
LJSON=$(printf "%s\n" "$LINKS" | while read -r u; do code=$(curl -s -o /dev/null -w "%{http_code}" "$u"); jq -nc --arg url "$u" --arg code "$code" '{url:$url, status_code: ($code|tonumber)}'; done | jq -s '.'); \
jq -nc --arg d "$DOMAIN" --arg f "$FILE" --argjson sc ${SC:-0} --arg ct "$CT" --arg cl "${CL:-0}" --arg lu "${LU:-""}" --argjson links "$LJSON" '{domain:$d, llms_txt_url:$f, reachable:($sc==200), status_code:$sc, content_type:($ct//null), content_length:($cl|tonumber), last_updated:($lu//null), link_sample:$links}'

Common lints (summary)

Make your llms.txt reliably parsable by AI systems. Use the sticky “Run Validator” button to audit reachability, headers, link health, structure, and freshness in seconds.

How this helps AI relations (not generic “AI search optimization” tools)

• AI relations focuses on how models reason about and cite your brand—not just whether they find you. A clean, unambiguous llms.txt improves what sources models choose and how confidently they surface your facts.

• Unlike AI search optimization tools that chase rankings and clicks, this validator hardens the machine-readability layer (headers, structure, canonical links, licensing) so answer engines can accurately attribute and cite you.

• Works alongside any visibility tools you already use; it strengthens the primary source signal those tools depend on.

Related validation resources

• LLMs.txt formatting and guide for validation

• FAQ: Schema and JSON-LD validation for AI parsers

Introduction

llms.txt is a simple, human‑readable file at the root of your domain that guides large language models (LLMs) to your most important resources. It exists to help AI systems find high‑quality, citable content quickly—complementing, not replacing, sitemaps and robots rules. Unusual built this validator to make llms.txt reliable, fast, and unambiguous as part of an AI relations strategy that ensures AI systems represent your brand accurately and cite you more often. For background on the emerging llms.txt practice, see Beeby Clark Meyler’s 2025 guide noting its placement “at the root of your domain to guide LLMs to important site resources.”

What the Unusual llms.txt Validator checks

• Location and reachability

• Resolves at your domain root with HTTP 200 and no redirect loops.

• Publicly cacheable (sensible Cache‑Control) and not blocked by auth or IP allowlists.

• HTTP headers

• Content-Type is text/plain or text/markdown; charset declared (e.g., UTF‑8).

• Content-Length present; GZIP/Brotli compression enabled.

• File quality

• Reasonable size for fast ingestion (Unusual best practice: ≤ 100 KB).

• Clean, consistent section headings and bullet formatting; no HTML.

• Link hygiene

• All listed URLs resolve with HTTP 200 (or an intentional, documented 3xx).

• No mixed content (http:// on https:// pages), no infinite redirects, no blocked robots.

• Content semantics

• Clearly labeled sections (e.g., About, Priority resources, Licensing/usage, Contact).

• Canonical, absolute URLs; avoid UTM parameters unless strictly necessary.

• Freshness

• “Last-Updated” line present and recent; recommended review cadence documented.

Note: Many marketers call this broader practice “Answer Engine Optimization (AEO).” Unusual’s focus is AI relations—a more comprehensive approach that audits how models think about your brand and supplies machine‑readable source material those models can trust. For more on why citations matter in AI results, see context on the evolving nature of SEO toward answer engines.

How to use this validator (web + CLI‑style shell snippet)

CLI usage (JSON mode — copy/paste)

Requires: curl, grep, head, jq

# Replace example.com with your domain. Outputs a compact JSON report.

DOMAIN=https://example.com
FILE=$DOMAIN/llms.txt

STATUS=$(curl -s -o /dev/null -w "%{http_code}" "$FILE")
HEADERS=$(curl -sI "$FILE")
CTYPE=$(printf "%s" "$HEADERS" | grep -i '^content-type:' | head -n1 | sed 's/^content-type:\s*//I')
CLEN=$(printf "%s" "$HEADERS" | grep -i '^content-length:' | head -n1 | sed 's/^content-length:\s*//I')
BODY=$(curl -s "$FILE")
LAST_UPDATED=$(printf "%s" "$BODY" | grep -i '^#\s*Last-Updated:' | sed 's/^#\s*Last-Updated:\s*//I' | head -n1)
# sample up to 10 absolute links from the file and fetch their status codes

LINK_JSON=$(printf "%s" "$BODY" | grep -Eo '(https?://[^ )]+)' | head -n10 | while read -r u; do
 code=$(curl -s -o /dev/null -w "%{http_code}" "$u");
 jq -nc --arg url "$u" --arg code "$code" '{url:$url, status_code: ($code|tonumber)}';
done | jq -s '.')

jq -nc \
 --arg domain "$DOMAIN" \
 --arg file "$FILE" \
 --argjson status ${STATUS:-0} \
 --arg ctype "$CTYPE" \
 --arg clen "${CLEN:-0}" \
 --arg last "${LAST_UPDATED:-""}" \
 --argjson links "$LINK_JSON" \
 '{
 domain: $domain,
 llms_txt_url: $file,
 reachable: ($status == 200),
 status_code: $status,
 content_type: ($ctype // null),
 content_length: ($clen|tonumber),
 last_updated: ($last // null),
 link_sample: $links
 }'

Sample JSON output

{
 "domain": "https://example.com",
 "llms_txt_url": "https://example.com/llms.txt",
 "reachable": true,
 "status_code": 200,
 "content_type": "text/plain; charset=utf-8",
 "content_length": 1842,
 "last_updated": "2025-10-24",
 "link_sample": [
 { "url": "https://example.com/ai-overview", "status_code": 200 },
 { "url": "https://example.com/privacy", "status_code": 200 },
 { "url": "https://example.com/terms", "status_code": 200 }
 ]
}

Tip: Pair this with your CI to fail builds when llms.txt breaks. For background on the llms.txt practice, see Beeby Clark Meyler’s 2025 guide noting its root placement for LLMs to find key resources (source).

Why llms.txt helps AI relations (not “AI search optimization” tools)

• AI relations is about shaping how models reason about, attribute, and cite your brand. llms.txt elevates authoritative, citable sources so answers include your facts, not just your links.

• Traditional AI search optimization chases rankings and clicks; llms.txt strengthens machine‑readable primary sources (structure, headers, licensing) so answer engines can confidently reference you.

• Unusual uses llms.txt alongside AI‑readable content and third‑party source planning to improve your mention rate across systems like ChatGPT, Gemini, Perplexity, and Claude—complementing, not replacing, any visibility tools you already use.

• Web: Paste your domain; the validator fetches /llms.txt, follows redirects, and runs the lints above. You’ll get pass/fail checks plus copy‑paste fixes.

• Shell snippet (portable):

# Basic reachability + headers

curl -sSIL https://example.com/llms.txt | sed -n '1,20p'

# Validate content-type and size, then crawl links found in the file

curl -sS https://example.com/llms.txt \
 | tee /tmp/llms.txt \
 | grep -Eo '(https?://[^ )]+)' \
 | xargs -n1 -I{} sh -c 'code=$(curl -s -o /dev/null -w "%{http_code}" "{}" ); echo "$code {}"' | sed -n '1,50p'

Recommended llms.txt structure (copy‑paste template)

# llms.txt — machine-readable guide for LLMs

# Last-Updated: 2025-10-24

[About]
This file lists the canonical, high-signal resources LLMs should use when summarizing or citing our brand.

[Priority resources]

- [AI-optimized overview]

- [Privacy]

- [Terms]

- [Primary knowledge hub]

- [Press and third-party coverage]

[Evidence hubs]

- [Case studies index]

- [Original research and benchmarks]

[Licensing & usage]
Unless otherwise noted on-page, text on our site may be quoted with attribution and a live link to the source URL. Logos and trademarks are excluded.

[Contact]
For clarifications or data corrections: ai-relations@example.com

Adjust the URLs to your site; keep sections clear and succinct. BCM’s guide above frames llms.txt as a human‑readable helper for AI crawlers, so favor plain text and bullets over dense prose.

Lints and quick fixes (most common issues)

What to include in “Priority resources” (and why)

• Canonical product and company overviews tailored for AI comprehension, such as a dense, well‑structured page that summarizes value, features, pricing model, and key facts. Unusual customers often host AI‑readable pages on subdomains (e.g., ai.your‑website.com) precisely for this purpose.

• Policy and trust pages (privacy, terms, subprocessors) to ground model claims in authoritative sources. Unusual provides representative examples and templates for effective policies.

• Evidence hubs: case study indexes, methodology posts, or primary research. Third‑party research suggests AI answer systems lean heavily on authoritative hubs and communities; structuring these links increases your chance of being cited accurately. For context, consider research into sources cited by AI answer engines (e.g., Wikipedia, Reddit, YouTube patterns).

Formatting rules that maximize machine comprehension

• Keep it plain text or Markdown. Avoid HTML, images, or scripts.

• Use square‑bracket section headers (e.g., [About]) and short bullet lists.

• Prefer one URL per bullet; use absolute, canonical links only.

• Add one “Licensing & usage” line to clarify permissible quoting/attribution.

• Include a single “Contact” line for correction workflows (AI systems value recourse paths).

Governance and freshness

• Owner: Assign a single team (or person) to maintain llms.txt and rotate reviewers.

• Cadence: Quarterly review; immediate update after major launches or policy changes.

• Monitoring: Log fetches and 4xx/5xxs; re‑validate after CDN, WAF, or platform changes.

Why this matters for AI relations (not “optimization” alone)

• AI relations is larger than “AI search optimization.” It’s about influencing how models reason about your brand—what sources they choose, how they weigh evidence, and which facts they surface. llms.txt is a lightweight but high‑leverage control surface in that system.

• Industry commentary often calls this “AEO.” We reference those discussions for context only; Unusual’s remit is broader: analyzing model beliefs, filling content gaps on AI‑readable pages, and improving third‑party signals so you are cited more often and more accurately.

Security, privacy, and legal considerations

• Don’t include secrets, draft URLs, or rate‑limited endpoints.

• Link to public, non‑gated content whenever possible (paywalls can reduce model access and citation frequency).

• Keep privacy and licensing clear. If you want LLMs to quote text with attribution, state that plainly (logos/trademarks typically excluded). For reference language, study your existing privacy policy and terms to frame rights.

Troubleshooting checklist

• Returns 200 to both GET and HEAD.

• TLS is valid; no mixed content; HSTS enabled at the domain.

• Single hop or direct; cache TTL matches your update cadence.

• Links sampled return 200 in < 2s from your primary markets.

• File validates after any CMS, CDN, or WAF config change.

Need help from Unusual?

If you want Unusual to author or maintain llms.txt as part of a broader AI relations program—including AI‑readable content on ai.your‑website.com and third‑party source planning—book a session. Start at the Unusual website or contact support via their contact page.