For developers, researchers, and agent builders
Use focused APIs when a product or workflow needs current public page evidence in a structured response.
Try a product, outcome, endpoint path, or API topic.
Web Data
Move from a URL to readable content, its final destination, visible technologies, category, and URL-level signals.
Use focused APIs when a product or workflow needs current public page evidence in a structured response.
Collect content for analysis, trace a destination before following it, or enrich a known page with technology and category context.
Use Scrape for page content. Add Redirects, Technology Detection, Website Categorization, or URL Intelligence only when that context changes the decision.
Fetch public web pages and documents as HTML, Markdown, LLM-ready text or plain text, one page at a time, in batches, or as a whole-site crawl.
Choose this when: Choose this when you need clean HTML, text or Markdown from public pages, PDFs or spreadsheets, from one page up to a whole site, with rendering or asynchronous collection only when supported and necessary.
Trace the full redirect chain for a URL.
Choose this when: Choose this when the destination and redirect chain matter more than the page body.
Detect the technology stack used by a website including CMS, frameworks, and services.
Choose this when: Choose this when you need evidence of exposed frameworks, platforms, analytics, or infrastructure technologies.
Parse robots.txt, test any crawler and path, and see which AI crawlers are blocked.
Choose this when: Choose this to read the robots.txt of a site, test whether a crawler may fetch a path, and see which AI crawlers are allowed or blocked.
Find and audit XML sitemaps with URL counts, lastmod freshness and protocol issues.
Choose this when: Choose this to find the XML sitemaps of a site and check URL counts, lastmod freshness, and protocol issues within a fixed file budget.
Audit titles, meta tags, headings, links, images and structured data with an SEO score.
Choose this when: Choose this to audit one page for on-page SEO, from meta tags and headings to links, images and structured data, with or without JavaScript rendering.
Compare raw and rendered HTML to see which SEO signals depend on JavaScript.
Choose this when: Choose this to see which SEO signals of a page only exist after JavaScript runs, by comparing its raw and rendered HTML.
Extract and validate JSON-LD, Microdata and RDFa schema.org markup.
Choose this when: Choose this to extract JSON-LD, Microdata and RDFa from a page and check it against schema.org rules for common rich result types.
Find the most frequent keywords and phrases on a page with density percentages.
Choose this when: Choose this to measure word and phrase frequency on a page and check where target keywords appear.
List page links and find broken ones with bounded status checks.
Choose this when: Choose this to list the links on a page and find broken ones, with blocked checks kept as unknown.
Validate hreflang codes, x-default, self references and return links.
Choose this when: Choose this to validate the hreflang tags of a page and confirm that alternate language pages link back.
Test many URLs and crawlers against a pasted or live robots.txt.
Choose this when: Choose this to test many URLs and crawlers at once against a pasted or live robots.txt.
Validate ads.txt and app-ads.txt records and confirm sellers against sellers.json.
Choose this when: Choose this to validate a site's ads.txt or app-ads.txt and confirm authorized sellers against sellers.json.
Compare pages for duplicate content, titles and meta descriptions.
Choose this when: Choose this to compare up to 10 pages for duplicate content and repeated titles, meta descriptions and H1s.
Crawl up to 50 pages and find broken links, duplicates and on-page issues.
Choose this when: Choose this to crawl up to 50 pages of a site and find broken links, duplicates and on-page SEO issues in one report.
Capture any page as an image or PDF on desktop or mobile.
Choose this when: Choose this to capture a page as PNG, JPEG, WebP or PDF on desktop or mobile.
Run an automated WCAG audit with violations, affected elements and a score.
Choose this when: Choose this to run an automated WCAG audit of a page with violations by impact and affected elements.
List cookies, trackers and consent manager seen before consent.
Choose this when: Choose this to see the cookies, trackers and consent manager a page loads before any consent.
Measure page bytes, requests, third parties and load timings.
Choose this when: Choose this to measure a page's bytes, requests, third parties, render-blocking resources and timings.
Categorize a website by industry, content type, and topics.
Choose this when: Choose this when you need a broad content or business category for routing, research, or review.
Extract normalized page metadata, preview fields, structured data, and public identity signals from a URL.
Choose this when: Choose this when you need normalized URL structure and risk-oriented URL signals before deeper collection.
Inspect the URL, collect the page as markdown, then add category or technology evidence required by the agent task.
Inspect the URL, collect the page as markdown, then add category or technology evidence required by the agent task.
Collect a representative page, resolve its final destination, and combine content with website category and exposed technology signals.
Submit a public URL to POST /v1/scrape with markdown output. Use standard mode first, then select rendering only when the page requires JavaScript.
curl -X POST "https://domscan.net/v1/scrape" -H "X-API-Key: $DOMSCAN_API_KEY" -H "Content-Type: application/json" -d '{"url":"https://example.com","output":"markdown"}'
{
"data": {
"url": "https://example.com",
"outcome": "success",
"content": "# Example Domain",
"truncated": false
},
"billing": { "credits_charged": 1, "credits_refunded": 0 }
}
Scrape one page synchronously or submit 1 to 100 public URLs to a paid asynchronous scrape job. Technology detection also supports product-specific bulk and job paths.
Collection uses bounded redirects, response sizes, concurrency, and processing modes. It does not promise access to every page.
Choose the lowest sufficient collection mode, retain destination evidence, and enrich the result only when the workflow needs it.
Learn moreLet an agent collect a page, understand the domain behind it, and request deeper evidence only when the task requires it.
Learn moreCollect public pages and connect them with domain, company, technology, DNS, and infrastructure context.
Learn moreReview the five Web Data products or make a first Scrape request from the documentation.