Start with the live form
Enter a domain, URL, IP, email, or record and get a focused result without setup.
All tools
Web Data
Domain Intelligence
DNS & Infrastructure
Security & Trust
Identity & Verification
Monitoring & Automation
Robots.txt Tester
Check whether crawlers may visit your URLs. Test up to 100 URLs against several user agents at once, using the live robots.txt of a site or a draft you paste before publishing it.
Results
Interactive intelligence tool
Run the check, then ship the workflow
Use the browser tool for a fast answer, then move the same logic into scripts, monitoring, or product flows when it becomes repeatable.
Review decision-ready signals
Outputs highlight statuses, risks, records, and next actions instead of raw provider noise.
Automate the workflow that works
Use the request and response examples to turn a one-off check into an API call or recipe.
Used by people at these companies
What this tool helps you decide
Each page is shaped around a practical operational question, not just a raw lookup.
See the current DNS, registration, security, pricing, or reputation evidence.
Compare the result with related checks so the next move is easier to trust.
Copy examples, open linked tools, or move into API documentation when you need scale.
Trust signals before you integrate
Transparent docs, authenticated requests, and visible reliability details make it easier to evaluate DomScan before you ship.
Service status
API artifacts
OpenAPI, Swagger, Postman, CLI, SDK, and MCP links are one click away.
API keys
Protected access
Authenticated endpoints use API keys with clear credit costs before you call them.
Free allowance
Sign up for free
Start with 10,000 monthly credits and upgrade only when usage grows.
Active
Example request
Start from the curl and HTTP samples, then map the parameters into your application code.
Key Features
Test a draft before publishing
Paste a robots.txt and see how it treats your URLs without changing anything on your site.
Many URLs, many crawlers
Up to 100 URLs and 10 user agents in one check, with a verdict for every combination.
The rule that decides
Each verdict shows the matching Allow or Disallow rule and its line number, following the official robots.txt standard.
Example request
{"url":"example.com","urls":["/blog/","/admin/"],"user_agents":["Googlebot","GPTBot"]}Built on the same API
The browser experience previews DomScan's structured endpoints, so teams can validate a use case before writing code.
Frequently asked questions
Can I test a robots.txt before it goes live?
Yes. Paste the content in the robots.txt box and nothing is fetched from your site. Leave it empty to test the file that is live now.
Can I paste a full user agent string?
Yes. A full string such as the one Googlebot sends is reduced to its crawler name, which is what robots.txt rules match.
What does unknown mean?
The live robots.txt could not be read, for example because the site blocked the request or timed out. Unknown is never treated as allowed, and the check is refunded.