Live
Service status
Check live service health and documented failure responses before you integrate.
Web Data
Robots.txt Tester API
Send a list of URLs and user agents and get a verdict for every combination. Test the live robots.txt of a site or a draft you paste, see the Allow or Disallow rule that decides and its line, and catch syntax mistakes before crawlers do.
Category
Web Data
Endpoints
1
Key Features
4
Frequently asked questions
4
Used by people at these companies
Trust signals before you integrate
Transparent docs, authenticated requests, and visible reliability details make it easier to evaluate DomScan before you ship.
OpenAPI
API artifacts
OpenAPI, Swagger, Postman, CLI, SDK, and MCP links are one click away.
API keys
Protected access
Authenticated endpoints use API keys with clear credit costs before you call them.
10,000
Free allowance
Start with 10,000 monthly credits and upgrade only when usage grows.
What this API helps you ship
Use this page as a production brief: endpoints, examples, response shape, and the workflow pieces needed to plug DomScan into your own product.
Embed domain checks, DNS intelligence, risk signals, or enrichment into onboarding, search, and internal tools.
Replace repeated manual lookups with scheduled jobs, alerting, and reproducible investigation steps.
Use predictable fields, documented status codes, and credit costs instead of scraping provider pages.
Feed agents, dashboards, SOAR playbooks, and CRMs through OpenAPI, SDK, Postman, or MCP.
Integration workflow
A simple path from first request to repeatable production usage.
Authenticate once
Send your API key with the documented header and keep requests consistent across services.
Query with examples
Start from the curl and HTTP samples, then map the parameters into your application code.
Operate and monitor
Use status codes, credit costs, and response fields to build retries, logs, and alerts.
Developer kit
Jump from this page into machine-readable docs, request collections, SDKs, or agent tooling.
OpenAPI schema
Generate clients or inspect every request and response shape.
Postman collectionImport ready-made requests for manual testing and team handoff.
SDKs and CLIUse maintained packages and command-line workflows instead of writing boilerplate.
MCP integrationExpose domain intelligence to AI agents and internal assistant workflows.
Parameters and response map
Scan the inputs, output fields, and status codes before wiring the endpoint into your client.
Parameter
robots_txturlurlsuser_agents
Example response
sourcestatussummarysummary.testssummary.allowedsummary.disallowedsummary.unknownresultsissues
HTTP status codes
200400401402409422429
Endpoints
POST
/v1/robots/test
Trust signals before you integrate
Transparent docs, authenticated requests, and visible reliability details make it easier to evaluate DomScan before you ship.
Service status
API artifacts
OpenAPI, Swagger, Postman, CLI, SDK, and MCP links are one click away.
API keys
Protected access
Authenticated endpoints use API keys with clear credit costs before you call them.
Free allowance
Sign up for free
Start with 10,000 monthly credits and upgrade only when usage grows.
Active
Example request
Start from the curl and HTTP samples, then map the parameters into your application code.
Key Features
Pasted or live file
Send robots_txt to test a draft without fetching anything, or send url to test the file that is live on the site.
Bulk verdicts
Up to 100 URLs and 10 user agents per request, with totals per crawler and a row for every combination.
Standard matching
Rules are matched as the official robots.txt standard (RFC 9309) describes: longest match wins, Allow wins ties, and * and $ wildcards are supported.
Syntax checks
Typos, rules outside a group, invalid lines and other issues are listed with their line numbers.
Example request
curl -X POST -H "X-API-Key: $DOMSCAN_API_KEY" -H "Content-Type: application/json" \
-d '{"url":"example.com","urls":["/blog/","/admin/"],"user_agents":["Googlebot","GPTBot"]}' \
"https://domscan.net/v1/robots/test"Example response
{
"source": "fetched",
"status": "found",
"summary": { "tests": 4, "allowed": 1, "disallowed": 3, "unknown": 0 },
"results": [
{
"url": "/blog/",
"user_agent": "Googlebot",
"allowed": true,
"reason": "no_matching_rule"
},
{
"url": "/admin/",
"user_agent": "Googlebot",
"allowed": false,
"reason": "matched_disallow",
"matched_rule": { "type": "disallow", "path": "/admin/", "line": 5 }
}
],
"issues": [{ "code": "directive_typo", "count": 1, "lines": [9] }]
}Frequently asked questions
How is this different from the Robots.txt API?
The Robots.txt API reads a site's live file and tests one crawler and path. The tester checks many URLs and crawlers in one request and can also test a robots.txt you paste.
Which user agent is matched?
The crawler name, without letter case. A full browser-style string such as Googlebot's is reduced to its crawler name first, and the response shows the name that was used.
What happens with URLs on another host?
When the live file is used, a URL on a different host than the site is reported as other_host without a verdict, because each host has its own robots.txt.
How much does it cost?
1 credit per request, for up to 1,000 verdicts. When the live robots.txt cannot be read, the verdicts are unknown and the credit is refunded.
Related tools and resources
HTTP status codes
These are the HTTP status codes to handle, so you can tell apart successful responses, auth issues, credits, rate limits, missing data, and upstream failures.
OK
200
Request successful
Bad Request
400
Invalid parameters
Unauthorized
401
Missing or invalid API key/session.
Payment Required
402
Not enough credits to run this request.
HTTP 409
409
These are the HTTP status codes to handle, so you can tell apart successful responses, auth issues, credits, rate limits, missing data, and upstream failures.
HTTP 422
422
These are the HTTP status codes to handle, so you can tell apart successful responses, auth issues, credits, rate limits, missing data, and upstream failures.
Too Many Requests
429
Rate limit exceeded
Test your robots.txt