Server-render your key pages
Client-side-only rendering is the single most common reason brands are invisible to LLMs. Retrievers execute limited or no JavaScript. Server-render or statically generate your homepage, category pages, product/service pages, and long-form content.
Use semantic HTML
Prefer <article>, <section>, <nav>, <header>, <footer>, <main>, and proper heading hierarchy over generic <div> soup. Semantic tags help retrievers segment content and identify the primary answer block.
Add JSON-LD structured data
At minimum, ship Organization on every page, FAQPage on FAQ pages, Article on long-form content, and Product/SoftwareApplication on product pages. Validate with Google's Rich Results Test.
Publish llms.txt
Add a /llms.txt file at your root - a markdown index of your most important URLs with short descriptions. It doesn't replace a sitemap but signals machine-readability intent and helps selective retrieval.
Configure robots.txt for AI crawlers
Explicitly allow the crawlers you want to be visible to: GPTBot, PerplexityBot, ClaudeBot, Google-Extended, CCBot, and Applebot. Blocking them removes you from those engines entirely.
- GPTBot - OpenAI / ChatGPT
- PerplexityBot - Perplexity
- ClaudeBot - Anthropic / Claude
- Google-Extended - Google AI (Gemini, AI Overviews training)
- CCBot - Common Crawl (used by many models)
- Applebot - Apple Intelligence
Canonical URLs and clean architecture
Set canonical tags on every page. Avoid duplicate content across parameterized URLs. Keep your URL structure shallow, descriptive, and consistent.
Performance and stability
Fast TTFB, no soft 404s, and stable URLs matter. Retrievers deprioritize slow or flaky origins.
The complete checklist
- Server-render or SSG all key pages
- Semantic HTML with clean heading hierarchy
- JSON-LD on every template
- /llms.txt at the root
- AI crawlers allowed in robots.txt
- Canonical URLs everywhere
- Fast TTFB, no soft 404s
- Visible last-updated dates and author bylines