Methodology

How the check works

We check 13 crawlers two ways, score every page we can find, and never dock you for opting out of model training. Here is exactly what runs when you enter a domain.

The scan, step by step

Read what the site declares

We fetch robots.txt, llms.txt and sitemap.xml from the domain you entered. A bare Disallow: means nothing is disallowed, which is the opposite of Disallow: /. Getting that wrong marks an open site as fully blocked, so we parse it properly.

Verify it from the outside

Declared rules are only half the story. Bot protection at the CDN can return 403 to a crawler your robots.txt explicitly welcomes, and nothing in the file tells you. We send a live request using each crawler's real user-agent string and compare. A crawler counts as allowed only when the file permits it and the server actually answers.

Find pages worth checking

We read the sitemaps your robots.txt names plus /sitemap.xml, follow a sitemap index one level deep, and add the pages your homepage links to, so a site without a sitemap still gets more than one page checked. The selection takes one page per section before a second from any, so you get /docs, /blog and /pricing instead of ten sibling doc pages, and it picks the same pages every time an unchanged site is scanned, so two scans compare like for like. Up to 10 pages on the domain itself, plus up to 2subdomains found in the homepage's own links. Each subdomain gets its own robots.txt, its own probes and its own score.

Analyze each page

For every page we fetch the raw HTML and measure what an answer engine would actually get: semantic structure, whether content exists before JavaScript runs, whether paragraphs survive being retrieved alone, which schema types are present, and when the page was last updated.

Score it

Three weighted categories plus a structured-data bonus, computed per page and for the site as a whole. The headline score describes the domain you entered. Subdomains appear in the matrix and the averages.

What the score is made of

AI Visibility

50%

Can the 6 search and answer engines reach this page? Resolved per page, because robots.txt rules are path-scoped.

·robots.txt rules evaluated per crawler, per path
·Live HTTP request using each crawler's real user-agent string
·A crawler counts as allowed only when both signals agree

Content Structure

25%

Once a crawler is in, can it extract anything usable? Measured on each page's own markup.

·Semantic HTML5 elements present (main, header, nav, article, section)
·Meaningful text in the raw HTML, before any JavaScript runs
·Paragraphs long enough to stand on their own when retrieved out of context

Technical Infrastructure

25%

Host-level foundations. Pages on the same host share this score.

·robots.txt present
·sitemap.xml present, including sitemap indexes
·llms.txt present, with a fallback to the legacy llm.txt
·HTTPS enforced, checked by requesting the site over plain http
·Homepage response time under a second

Structured Data

+10 bonus

A bonus rather than a category, so a perfectly accessible site is never dragged below a B for skipping schema. Three parts: the markup parses, it uses a type an answer engine can actually quote (FAQPage, HowTo, Product, Article and similar, not Organization or BreadcrumbList), and the page says when it was last updated.

Training crawlers are reported, not scored

This is where we differ from every other visibility score, and it is deliberate. Blocking GPTBot or ClaudeBot is a licensing decision. Plenty of publishers make it on purpose. A tool that counts those blocks and tells you to reverse them is giving advice about a business decision it never asked about.

So we split the two. The 6 search and answer engines drive your score, because those decide whether you can be found and cited today. The 7 training crawlers are reported as a consent posture, open, selective or opted out, worth zero points either way.

Scored · search and answers

GooglebotGoogle Search
BingbotBing / Copilot
ChatGPT-UserChatGPT live browsing
OAI-SearchBotOpenAI search crawler
Claude-UserClaude live browsing
PerplexityBotPerplexity crawler

Reported only · training

Google-ExtendedGoogle Gemini & AI Overviews training
GPTBotOpenAI GPTBot (training)
ClaudeBotAnthropic ClaudeBot (training)
CCBotCommon Crawl (used to train many LLMs)
BytespiderByteDance / TikTok AI
Applebot-ExtendedApple Intelligence training
meta-externalagentMeta AI training

What we do not do

→We do not ask a language model to judge your site. Every check is deterministic, so the same site scores the same twice.
→We do not render JavaScript. That is the point: we measure what a crawler that does not run a browser can see.
→We do not crawl your whole site. Page fetches are capped and we identify ourselves with a normal browser user agent.
→We do not keep a log of every domain checked. Results are cached in memory for about an hour and cleared on redeploy.

Run it yourself

The same check is available as a JSON API and as an MCP server, so an AI assistant can run it for you.

curl -X POST https://isvisible.ai/api/is-visible \
  -H "Content-Type: application/json" \
  -d '{"url": "example.com"}'

Check your site