Start with the live form
Enter a domain, URL, IP, email, or record and get a focused result without setup.
All tools
Web Data
Domain Intelligence
DNS & Infrastructure
Security & Trust
Identity & Verification
Monitoring & Automation
Robots.txt checker
Enter a website to read its robots.txt, test whether a crawler may fetch a path, and see how GPTBot, ClaudeBot, PerplexityBot and other AI crawlers are treated.
Results
Interactive intelligence tool
Run the check, then ship the workflow
Use the browser tool for a fast answer, then move the same logic into scripts, monitoring, or product flows when it becomes repeatable.
Review decision-ready signals
Outputs highlight statuses, risks, records, and next actions instead of raw provider noise.
Automate the workflow that works
Use the request and response examples to turn a one-off check into an API call or recipe.
Used by people at these companies
What this tool helps you decide
Each page is shaped around a practical operational question, not just a raw lookup.
See the current DNS, registration, security, pricing, or reputation evidence.
Compare the result with related checks so the next move is easier to trust.
Copy examples, open linked tools, or move into API documentation when you need scale.
Trust signals before you integrate
Transparent docs, authenticated requests, and visible reliability details make it easier to evaluate DomScan before you ship.
Service status
API artifacts
OpenAPI, Swagger, Postman, CLI, SDK, and MCP links are one click away.
API keys
Protected access
Authenticated endpoints use API keys with clear credit costs before you call them.
Free allowance
Sign up for free
Start with 10,000 monthly credits and upgrade only when usage grows.
Active
Example request
Start from the curl and HTTP samples, then map the parameters into your application code.
Key Features
RFC 9309 parsing
Groups, allow and disallow rules, wildcards and end anchors are matched the way the standard describes, with the longest rule winning.
AI crawler matrix
Root access for 14 AI crawlers, including training, search and user-initiated agents from OpenAI, Anthropic, Google, Perplexity and others.
Honest unknown states
If the file cannot be fetched because of a block, a timeout or a server error, the result is unknown, never a silent allow.
Example request
GET /v1/robots?url=example.comGET /v1/robots?url=example.com&user_agent=GPTBot&path=/blog/Built on the same API
The browser experience previews DomScan's structured endpoints, so teams can validate a use case before writing code.
Frequently asked questions
What happens when a site has no robots.txt?
A 404 or 410 response means there are no crawl restrictions, so every path is allowed. The checker reports this as not found.
How do I block AI crawlers?
Add a user-agent group for each crawler, for example GPTBot or ClaudeBot, with a Disallow rule. The AI crawler table shows which bots your current rules already cover.
Does robots.txt stop all bots?
No. Robots.txt is a request that well-behaved crawlers follow. It does not enforce access control, so private content needs authentication.