Web Data

Collect public web content with the context to use it

Move from a URL to readable content, its final destination, visible technologies, category, and URL-level signals.

For developers, researchers, and agent builders

Use focused APIs when a product or workflow needs current public page evidence in a structured response.

Start with the output you need

Collect content for analysis, trace a destination before following it, or enrich a known page with technology and category context.

Inputs

URLDomain

Outcomes

CollectAssessDiscoverEnrichVerifyCompare

Choose the narrowest product that answers the question

Use Scrape for page content. Add Redirects, Technology Detection, Website Categorization, or URL Intelligence only when that context changes the decision.

Web Scraping

Fetch public web pages and documents as HTML, Markdown, LLM-ready text or plain text, one page at a time, in batches, or as a whole-site crawl.

Choose this when: Choose this when you need clean HTML, text or Markdown from public pages, PDFs or spreadsheets, from one page up to a whole site, with rendering or asynchronous collection only when supported and necessary.

Inputs
URL
Outcomes
Collect
Credits
0-20
SynchronousAsynchronousRESTMCPSDK

Redirect Chain

Trace the full redirect chain for a URL.

Choose this when: Choose this when the destination and redirect chain matter more than the page body.

Inputs
URL, Domain
Outcomes
Collect, Assess
Credits
2
SynchronousRESTMCPSDKInteractive tool

Tech Stack Detection

Detect the technology stack used by a website including CMS, frameworks, and services.

Choose this when: Choose this when you need evidence of exposed frameworks, platforms, analytics, or infrastructure technologies.

Inputs
URL, Domain
Outcomes
Discover, Enrich, Assess
Credits
0-9
SynchronousBulkAsynchronousRESTMCPSDKInteractive tool

Robots.txt API

Parse robots.txt, test any crawler and path, and see which AI crawlers are blocked.

Choose this when: Choose this to read the robots.txt of a site, test whether a crawler may fetch a path, and see which AI crawlers are allowed or blocked.

Inputs
URL, Domain
Outcomes
Verify, Assess
Credits
1
SynchronousRESTMCPSDKInteractive tool

XML sitemap audit API

Find and audit XML sitemaps with URL counts, lastmod freshness and protocol issues.

Choose this when: Choose this to find the XML sitemaps of a site and check URL counts, lastmod freshness, and protocol issues within a fixed file budget.

Inputs
URL, Domain
Outcomes
Discover, Verify, Assess
Credits
2
SynchronousRESTMCPSDKInteractive tool

On-page SEO audit API

Audit titles, meta tags, headings, links, images and structured data with an SEO score.

Choose this when: Choose this to audit one page for on-page SEO, from meta tags and headings to links, images and structured data, with or without JavaScript rendering.

Inputs
URL, Domain
Outcomes
Verify, Assess
Credits
2-6
SynchronousRESTMCPSDKInteractive tool

JavaScript SEO rendering check API

Compare raw and rendered HTML to see which SEO signals depend on JavaScript.

Choose this when: Choose this to see which SEO signals of a page only exist after JavaScript runs, by comparing its raw and rendered HTML.

Inputs
URL, Domain
Outcomes
Compare, Assess
Credits
7
SynchronousRESTMCPSDKInteractive tool

Structured data validation API

Extract and validate JSON-LD, Microdata and RDFa schema.org markup.

Choose this when: Choose this to extract JSON-LD, Microdata and RDFa from a page and check it against schema.org rules for common rich result types.

Inputs
URL, Domain
Outcomes
Verify, Assess
Credits
2-6
SynchronousRESTMCPSDKInteractive tool

Keyword density API

Find the most frequent keywords and phrases on a page with density percentages.

Choose this when: Choose this to measure word and phrase frequency on a page and check where target keywords appear.

Inputs
URL, Domain
Outcomes
Assess
Credits
2-6
SynchronousRESTMCPSDKInteractive tool

Page links and broken link checker API

List page links and find broken ones with bounded status checks.

Choose this when: Choose this to list the links on a page and find broken ones, with blocked checks kept as unknown.

Inputs
URL, Domain
Outcomes
Discover, Verify
Credits
2-6
SynchronousRESTMCPSDKInteractive tool

Hreflang validation API

Validate hreflang codes, x-default, self references and return links.

Choose this when: Choose this to validate the hreflang tags of a page and confirm that alternate language pages link back.

Inputs
URL, Domain
Outcomes
Verify, Assess
Credits
3
SynchronousRESTMCPSDKInteractive tool

Robots.txt Tester API

Test many URLs and crawlers against a pasted or live robots.txt.

Choose this when: Choose this to test many URLs and crawlers at once against a pasted or live robots.txt.

Inputs
URL, Domain
Outcomes
Verify, Assess
Credits
1
SynchronousRESTMCPSDKInteractive tool

ads.txt and sellers.json validation API

Validate ads.txt and app-ads.txt records and confirm sellers against sellers.json.

Choose this when: Choose this to validate a site's ads.txt or app-ads.txt and confirm authorized sellers against sellers.json.

Inputs
URL, Domain
Outcomes
Verify, Assess
Credits
2-5
SynchronousRESTMCPSDKInteractive tool

Duplicate content checker API

Compare pages for duplicate content, titles and meta descriptions.

Choose this when: Choose this to compare up to 10 pages for duplicate content and repeated titles, meta descriptions and H1s.

Inputs
URL, Domain
Outcomes
Compare, Assess
Credits
5
SynchronousRESTMCPSDKInteractive tool

Site audit and SEO crawler API

Crawl up to 50 pages and find broken links, duplicates and on-page issues.

Choose this when: Choose this to crawl up to 50 pages of a site and find broken links, duplicates and on-page SEO issues in one report.

Inputs
URL, Domain
Outcomes
Discover, Assess
Credits
10-35
SynchronousRESTMCPSDKInteractive tool

Website screenshot API

Capture any page as an image or PDF on desktop or mobile.

Choose this when: Choose this to capture a page as PNG, JPEG, WebP or PDF on desktop or mobile.

Inputs
URL, Domain
Outcomes
Discover, Verify
Credits
10
SynchronousRESTMCPSDKInteractive tool

Accessibility audit API

Run an automated WCAG audit with violations, affected elements and a score.

Choose this when: Choose this to run an automated WCAG audit of a page with violations by impact and affected elements.

Inputs
URL, Domain
Outcomes
Verify, Assess
Credits
8
SynchronousRESTMCPSDKInteractive tool

Cookie and tracker scanner API

List cookies, trackers and consent manager seen before consent.

Choose this when: Choose this to see the cookies, trackers and consent manager a page loads before any consent.

Inputs
URL, Domain
Outcomes
Discover, Assess
Credits
8
SynchronousRESTMCPSDKInteractive tool

Page weight and resource API

Measure page bytes, requests, third parties and load timings.

Choose this when: Choose this to measure a page's bytes, requests, third parties, render-blocking resources and timings.

Inputs
URL, Domain
Outcomes
Assess
Credits
6
SynchronousRESTMCPSDKInteractive tool

Website Categorization

Categorize a website by industry, content type, and topics.

Choose this when: Choose this when you need a broad content or business category for routing, research, or review.

Inputs
URL, Domain
Outcomes
Enrich, Assess
Credits
0-3
SynchronousBulkRESTMCPSDKInteractive tool

Unified URL Intelligence

Extract normalized page metadata, preview fields, structured data, and public identity signals from a URL.

Choose this when: Choose this when you need normalized URL structure and risk-oriented URL signals before deeper collection.

Inputs
URL
Outcomes
Collect, Enrich
Credits
2
SynchronousBulkRESTMCPSDK

Workflows

Inspect the URL, collect the page as markdown, then add category or technology evidence required by the agent task.

Page to agent-ready context

Inspect the URL, collect the page as markdown, then add category or technology evidence required by the agent task.

Website research snapshot

Collect a representative page, resolve its final destination, and combine content with website category and exposed technology signals.

Collect one public page as markdown

Submit a public URL to POST /v1/scrape with markdown output. Use standard mode first, then select rendering only when the page requires JavaScript.

Example request
curl -X POST "https://domscan.net/v1/scrape" -H "X-API-Key: $DOMSCAN_API_KEY" -H "Content-Type: application/json" -d '{"url":"https://example.com","output":"markdown"}'
Example response
{
  "data": {
    "url": "https://example.com",
    "outcome": "success",
    "content": "# Example Domain",
    "truncated": false
  },
  "billing": { "credits_charged": 1, "credits_refunded": 0 }
}

Synchronous first, jobs when the workload requires them

Scrape one page synchronously or submit 1 to 100 public URLs to a paid asynchronous scrape job. Technology detection also supports product-specific bulk and job paths.

SynchronousBulkAsynchronousRESTMCPSDKInteractive tool

Bounded public-page collection

Collection uses bounded redirects, response sizes, concurrency, and processing modes. It does not promise access to every page.

Start with one page, then add only the context you need

Review the five Web Data products or make a first Scrape request from the documentation.