Web Data

Robots.txt API

Fetch and parse any website robots.txt with one request. Get crawler groups, rules, declared sitemaps, an allow or disallow verdict for a user agent and path, and root access for the main AI crawlers.

View documentation Try the free tool Explore all APIs

Endpoints Active

GET /v1/robots

Example request

curl -H "X-API-Key: $DOMSCAN_API_KEY" "https://domscan.net/v1/robots?url=example.com&user_agent=GPTBot&path=/blog/"

Example response 200 OK

{
  "status": "found",
  "sitemaps": ["https://example.com/sitemap.xml"],
  "test": {
    "user_agent": "GPTBot",
    "path": "/blog/",
    "allowed": false,
    "matched_rule": { "type": "disallow", "path": "/", "line": 2 }
  },
  "ai_crawlers": [
    { "bot": "GPTBot", "owner": "openai", "root_access": "disallowed", "group": "specific" },
    { "bot": "Cla
...

Example request GET /v1/robots

Key Features 4

Frequently asked questions 4

HTTP status codes 200 / 400 / 401 / 402 / 429 / 502

Category Web Data

Endpoints 1

Key Features 4

Frequently asked questions 4

Used by people at these companies

Trust signals before you integrate

Transparent docs, authenticated requests, and visible reliability details make it easier to evaluate DomScan before you ship.

Live Service status

Check live service health and documented failure responses before you integrate.

OpenAPI API artifacts

OpenAPI, Swagger, Postman, CLI, SDK, and MCP links are one click away.

API keys Protected access

Authenticated endpoints use API keys with clear credit costs before you call them.

10,000 Free allowance

Start with 10,000 monthly credits and upgrade only when usage grows.

What this API helps you ship

Use this page as a production brief: endpoints, examples, response shape, and the workflow pieces needed to plug DomScan into your own product.

Product workflows

Embed domain checks, DNS intelligence, risk signals, or enrichment into onboarding, search, and internal tools.

Analyst automation

Replace repeated manual lookups with scheduled jobs, alerting, and reproducible investigation steps.

Clean JSON data

Use predictable fields, documented status codes, and credit costs instead of scraping provider pages.

AI and ops tooling

Feed agents, dashboards, SOAR playbooks, and CRMs through OpenAPI, SDK, Postman, or MCP.

Integration workflow

A simple path from first request to repeatable production usage.

Authenticate once

Send your API key with the documented header and keep requests consistent across services.

Query with examples

Start from the curl and HTTP samples, then map the parameters into your application code.

Operate and monitor

Use status codes, credit costs, and response fields to build retries, logs, and alerts.

Developer kit

Jump from this page into machine-readable docs, request collections, SDKs, or agent tooling.

OpenAPI schema

Generate clients or inspect every request and response shape.

Postman collection

Import ready-made requests for manual testing and team handoff.

SDKs and CLI

Use maintained packages and command-line workflows instead of writing boilerplate.

MCP integration

Expose domain intelligence to AI agents and internal assistant workflows.

Parameters and response map

Scan the inputs, output fields, and status codes before wiring the endpoint into your client.

Request parameters

Parameter

urldomainuser_agentpathskip_cache

Response fields

Example response

statussitemapstesttest.user_agenttest.pathtest.allowedtest.matched_ruletest.matched_rule.typetest.matched_rule.pathtest.matched_rule.lineai_crawlers

Status coverage

HTTP status codes

200400401402429502

Endpoints

GET /v1/robots

Credits: 1Authentication: Authentication required

Trust signals before you integrate

Transparent docs, authenticated requests, and visible reliability details make it easier to evaluate DomScan before you ship.

Service status API artifacts

OpenAPI, Swagger, Postman, CLI, SDK, and MCP links are one click away.

API keys Protected access

Authenticated endpoints use API keys with clear credit costs before you call them.

Free allowance Sign up for free

Start with 10,000 monthly credits and upgrade only when usage grows.

Active Example request

Start from the curl and HTTP samples, then map the parameters into your application code.

Key Features

Standard matching

Rules are matched under RFC 9309: the most specific user-agent group, the longest path match, allow on ties, and support for * and $.

AI crawler coverage

Root access for GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Google-Extended, PerplexityBot, CCBot and more, with owner and purpose.

Path testing

Pass a user agent and a path to get the verdict, the rule that decided it and the group it came from.

Clear failure states

Blocked, rate-limited and failed fetches return unknown and are refunded, so a missing answer is never reported as allowed.

Example request

GET /v1/robots?url=example.com&user_agent=GPTBot&path=/blog/ bash

curl -H "X-API-Key: $DOMSCAN_API_KEY" "https://domscan.net/v1/robots?url=example.com&user_agent=GPTBot&path=/blog/"

Example response

200 OK json

{
  "status": "found",
  "sitemaps": ["https://example.com/sitemap.xml"],
  "test": {
    "user_agent": "GPTBot",
    "path": "/blog/",
    "allowed": false,
    "matched_rule": { "type": "disallow", "path": "/", "line": 2 }
  },
  "ai_crawlers": [
    { "bot": "GPTBot", "owner": "openai", "root_access": "disallowed", "group": "specific" },
    { "bot": "ClaudeBot", "owner": "anthropic", "root_access": "allowed", "group": "wildcard" }
  ]
}

Frequently asked questions

How is a missing robots.txt reported?

A 404 or 410 response returns not_found, which under RFC 9309 means crawlers may access everything. 401, 403, 429 and 5xx responses return unknown.

Which AI crawlers are included?

14 agents from OpenAI, Anthropic, Google, Perplexity, Common Crawl, ByteDance, Apple, Meta and Amazon, grouped as training, search or user-initiated fetches.

Is the result cached?

Definitive results are cached for 15 minutes. Add skip_cache=1 to fetch the file again. The path test is always computed for your request.

How many credits does it cost?

One credit per request. Requests that end in an unknown state are refunded.

Related tools and resources

HTTP status codes

These are the HTTP status codes to handle, so you can tell apart successful responses, auth issues, credits, rate limits, missing data, and upstream failures.

OK 200

Request successful

Bad Request 400

Invalid parameters

Unauthorized 401

Missing or invalid API key/session.

Payment Required 402

Not enough credits to run this request.

Too Many Requests 429

Rate limit exceeded

Bad Gateway 502

Upstream RDAP error

Check a robots.txt

Start free View documentation Try the free tool