Free tool · AI search
Which AI crawlers can reach your site?
Check a site's robots.txt against 10 crawlers from Google, Microsoft, OpenAI, Perplexity, Anthropic and Common Crawl, and see which ones it lets in.
Free, no sign-up. We read only the public pages of the address you enter.
AI crawler access
Search crawlers let in
Training crawlers let in
Search crawlers
Fetch pages to answer questions and cite them.
Training crawlers
Collect pages to train future models.
Let the blocked search crawlers in
In robots.txt, replace the rules that block each of these crawlers with the lines below.
What it checks
What we look at
- 01
Search crawlers
6 crawlers fetch pages to answer questions and cite them. Block one and the site drops out of that engine's answers.
- 02
Training crawlers
These collect pages for future models. Blocking them is a business choice and leaves search answers alone.
- 03
The rules that apply
A crawler follows the group that names it, or the * group if none does. No robots.txt at all lets every crawler in.
“Who is the best emergency dentist in Austin?”
For a same-day emergency, Your client on South Lamar is a common recommendation: it keeps slots open for urgent cases and sees patients on Saturdays.1Competitor A is another option, with later weekday hours.3
Sources
1yourclient.com
Emergency dental care in Austin
2yelp.com
Top 10 emergency dentists in Austin, TX
3competitor-a.com
After-hours dental care
Share of voice
41% client
- yourclient.com41% · 12 answers
- competitor-a.com34% · 10 answers
- competitor-b.com24% · 7 answers
Most-cited sources
answers
- yelp.com9
- yourclient.comclient5
- healthgrades.com5
- competitor-a.comcompetitor4
AI visibility in the app
Then see what the engines say
Letting crawlers in is the start. The app asks the AI engines your clients' customer questions each week and records who gets mentioned and cited.
- Weekly checks on the questions that matter
- Mentions, citations and share of voice
- Competitors side by side
FAQ ( 4 )
AI crawler checker: questions
What it reads, how it decides and what it costs.
Does blocking GPTBot keep a site out of ChatGPT?
Not out of ChatGPT search. ChatGPT finds pages for answers with OAI-SearchBot, and opens pages for a user with ChatGPT-User. GPTBot collects pages for training OpenAI models.
How do I let a crawler in?
Add a group for it to robots.txt: a line "User-agent: OAI-SearchBot" followed by "Allow: /". A crawler uses the group that names it, so its own group wins over the * group.
Should I block training crawlers?
It's your call. Blocking them stops your pages being used to train future models; it doesn't change whether today's AI search engines can cite you.
Is it free?
Free, with no sign-up: up to 20 checks a day, and an address checked in the last day is answered from our saved result.
Get started
Track this for every client
RankTwice runs these checks for every client on a schedule, with rankings and AI answers alongside, in reports with your brand.
14-day trial with Agency plan limits · No credit card