AI answer engines don't crawl the web the way Google does, and they don't reward the same things. If you want to be the source they cite, you have to be readable, attributable, and unambiguous. This is the working checklist behind the Website Verdict AI-discoverability score — every item is something we measure on real scans. Work through it top to bottom; the sections are ordered by impact.
1. Let AI crawlers in
The most common failure we find is also the dumbest: sites that block AI crawlers in robots.txt (often via a copy-pasted 'block everything weird' template) and then wonder why they never appear in AI answers. Decide deliberately which crawlers you allow. These are the user-agents that matter most:
| Crawler | Operator | Feeds |
|---|---|---|
| GPTBot / OAI-SearchBot | OpenAI | ChatGPT browsing and search citations |
| ClaudeBot | Anthropic | Claude's web answers |
| PerplexityBot | Perplexity | Perplexity answers and source slots |
| Google-Extended | Gemini grounding (Googlebot itself feeds AI Overviews) | |
| Bingbot | Microsoft | Copilot answers and Bing citations |
Also check that your CDN or WAF isn't silently challenging these bots with JavaScript interstitials — a 403 or a challenge page reads as an empty site. Run a scan on the homepage and the Website Verdict crawler-access check will tell you exactly who gets in.
2. Publish llms.txt and keep it honest
llms.txt is a plain-markdown index at your site root that tells AI systems what your site is, what it offers, and which pages matter. It is cheap to add and increasingly read. Ours lists every guide on this blog. Two rules: keep it in sync with your actual sitemap (a stale llms.txt is worse than none), and write the summary line as a factual sentence — it is frequently quoted verbatim in AI answers. You can generate one automatically with our free llms.txt generator.
3. Make your identity machine-readable
- Add Organization and WebSite structured data with a stable name, logo, and URL on every page.
- Keep the same brand name, one-line description, and domain across your site, schema, social profiles, and directories — entity resolution is string-matching at heart, and inconsistency splits your identity.
- Maintain a crawlable about page that states in plain text who you are, what you do, and who you serve.
- Use one canonical domain (www or apex, http never) and absolute canonical URLs on every page.
4. Structure content so LLMs can lift answers
- Lead with the answer, then explain — inverted-pyramid style. The first two sentences under a heading are what gets extracted.
- Use descriptive, question-style headings that map to real queries ('How much does X cost?', not 'Pricing philosophy').
- Add FAQ sections with direct, factual answers and matching FAQPage schema on key pages.
- Keep one idea per paragraph and keep paragraphs short; extraction happens at paragraph granularity.
- Use tables for comparisons and specs — engines parse them cleanly and quote them often.
- Server-render everything that matters. Most AI crawlers read initial HTML only and never execute your JavaScript.
5. Earn trust signals
Citations, author bylines, dates, and corroborating links all increase the odds an AI engine treats your page as authoritative. This is where legitimate backlinks pull double duty — they help classic SEO and reinforce trust for AI. Answer engines also cross-check claims across pages, so consistency matters more than volume: one accurate, dated, attributed page beats five thin variants. Our ethical link-building guide covers earning those signals without schemes.
6. Measure it
- Score your site: run a free scan here and read the AI-discoverability category — it checks crawler access, llms.txt, schema, and answerability in one pass, or use the standalone AI visibility checker.
- Track citations: ask ChatGPT, Perplexity, and Gemini the questions your customers ask and record whether you're cited; re-check monthly. See our Perplexity rank-tracking guide for a systematic approach.
- Watch referral traffic from chatgpt.com, perplexity.ai, and gemini.google.com in your analytics — it's small but it converts well, and it trends with citation share.
The one-screen version
- 1Allow the AI crawlers you want citing you (and verify your WAF isn't blocking them).
- 2Publish an accurate llms.txt and keep it current.
- 3Ship Organization + WebSite schema and a plain-language about page.
- 4Lead with answers, use question headings, add FAQPage schema.
- 5Server-render key content; keep it in initial HTML.
- 6Build consistency and legitimate links; measure citations monthly.