Question 1

What is crawlability?

Accepted Answer

Crawlability is whether a search engine or AI bot is allowed and able to fetch a page. It's set by your robots.txt rules, meta robots directives, and any firewall rule blocking bots. A page that isn't crawlable can't be indexed, ranked or cited.

Question 2

How do I check if my page is crawlable?

Accepted Answer

To check if your page is crawlable, paste its URL into the test above. It reads your robots.txt, meta robots directives and llms.txt, then returns a verdict per AI crawler with the exact rule behind any block. No account needed.

Question 3

What's the difference between crawlability and indexability?

Accepted Answer

Crawlability is permission to fetch a page. Indexability is permission to store and serve it. A “noindex” tag leaves a page crawlable but unindexable. Crawlability comes first: if a bot can't fetch the page, it never reads the directive inside it.

Question 4

What are AI crawlers?

Accepted Answer

AI crawlers are bots run by AI companies to fetch web pages. Some collect training data, some build the indexes behind AI answers, and some fetch a page live when a user asks. Each has its own user-agent and obeys robots.txt separately.

Question 5

Can AI crawlers execute JavaScript?

Accepted Answer

Most AI crawlers can't execute JavaScript. They read the raw HTML your server returns, with no second rendering pass like Googlebot's. Content, links or meta tags that appear only after hydration are invisible to them. Put anything you want cited in the server response.

Question 6

Do I need an llms.txt file?

Accepted Answer

You don't need an llms.txt file, and no crawler is obliged to honour one. It's a Markdown file at your root pointing AI systems to your key pages. Cheap to publish, and a missing one blocks nothing. A broken one is more common.

Question 7

Why is my site crawlable but still absent from AI answers?

Accepted Answer

Being crawlable only makes you eligible to appear. Visibility then depends on whether your content answers the prompts people type, whether other sources corroborate it, and how well your pages match retrieval queries. That's a content problem rather than a technical one.

Can AI bots read your site?Free. Instant. Any URL.

What this crawlability test checks

robots.txt, per crawler

Meta robots & headers

llms.txt

How to test website crawlability

Crawlability issues we find most often

You can be crawlable and still be invisible

FAQs