Subdomain Finder: Passive Discovery, DNS Verification, and Coverage Limits

Subdomain discovery is an evidence-gathering exercise, not a database query with a guaranteed complete answer. A public hostname might appear in a certificate, a passive DNS dataset, an archived URL, a search index, or nowhere that an outside observer can query. A private hostname may not be visible from the public internet at all. Even the authoritative DNS zone does not normally offer a standard operation that lists every owner name.

That distinction matters when you use a subdomain finder for inventory or security work. A returned name is a lead supported by one source. A missing name is not proof that it does not exist. Good workflows record where each lead came from, test current state separately, and compare repeated snapshots without turning uncertainty into a false promise of complete coverage.

What DomScan's Subdomain Finder does

DomScan's `GET /v1/subdomains` endpoint performs passive discovery. The public `sources` parameter currently accepts only `ct`, and defaults to that value. This is a logical compatibility selector for the implemented discovery pipeline. It is not a list of provider names, and callers cannot request `crtsh`, `wayback`, or another underlying provider directly.

The pipeline starts with crt.sh. If crt.sh returns at least one concrete hostname, DomScan returns that result without querying later fallbacks. If crt.sh is empty, unavailable, or temporarily in cooldown, the service tries HackerTarget Host Search, then ThreatMiner's subdomain query, then the Internet Archive Wayback CDX index. Cert Spotter is last because anonymous full-domain certificate searches have a small request budget. The pipeline stops when a source produces concrete names. It is a resilience chain, not a union of every source on every request.

The response reports the providers actually contacted successfully in `summary.sources_used` and `intelligence_summary.data_sources`. Depending on availability and empty results, those values can include `crtsh`, `hackertarget`, `threatminer`, `wayback`, or `certspotter`. They are more informative than the request's logical `sources=ct` selector when you need to understand provenance.

DomScan normalizes names to lowercase, rejects values outside the requested domain, removes duplicates, and sorts the concrete hostname list alphabetically. The service does not generate candidate labels, mutate discovered names, brute force a wordlist, try AXFR, walk DNSSEC denial records, scan ports, fetch web pages, or vary HTTP Host headers. Those are separate techniques and have different authorization and operational risks.

Why Certificate Transparency helps, and where it stops

Certificate Transparency provides append-only logs of public TLS certificate chains so certificate authority activity can be audited. RFC 9162 describes CT for certificates issued by public certificate authorities and notes that certificate chains can be submitted by certificate authorities, certificate holders, or third parties. DNS names in a certificate's Subject Alternative Name extension therefore create useful hostname evidence.

CT is not a DNS inventory. A hostname can exist without ever receiving a publicly logged certificate. A wildcard certificate such as `*.example.com` does not reveal each concrete label that may exist below it. A logged certificate can also contain a name that no longer resolves, never served traffic, was entered by mistake, or appeared in a precertificate before final issuance. Treat CT as certificate evidence, not proof of present DNS state or service ownership.

For rows sourced from crt.sh or Cert Spotter, DomScan maps the earliest certificate `not_before` value it retained to `first_seen`. That field is a certificate validity timestamp. It is not the date DomScan first discovered the hostname, the date the DNS name was created, or the date a service first became reachable. HackerTarget, ThreatMiner, and Wayback fallback rows do not provide an equivalent value in this pipeline, so their `first_seen` is `null`.

Request parameters

• `domain` is required. DomScan normalizes and validates the domain before discovery.
• `sources` defaults to `ct`. It is the only accepted selector today, even though the fallback chain can return evidence from non-CT passive providers.
• `verify` defaults to `false`. Accepted true values are `1`, `true`, and `yes`; accepted false values are `0`, `false`, and `no`.
• `include_wildcards` defaults to `false` and uses the same accepted Boolean forms.
• `limit` defaults to 500 and must be an integer from 1 through 2000. DomScan applies it before optional verification.
• `prefer_cache` defaults to `false`. When true, DomScan serves only a fresh or stale cached result and never waits for an upstream discovery request.

curl 'https://domscan.net/v1/subdomains?domain=example.com&sources=ct&verify=true&include_wildcards=true&limit=200' \
  -H 'Authorization: Bearer YOUR_API_KEY'

The route is authenticated and the endpoint registry currently prices it at three credits. Consult the live OpenAPI document and account response for the current contract and billing behavior rather than copying an old example into permanent automation.

Reading a successful response

{
  "domain": "example.com",
  "subdomains": [
    {
      "name": "www.example.com",
      "source": "crtsh",
      "first_seen": "2024-01-15T00:00:00Z",
      "verified": true,
      "dns_records": { "A": ["192.0.2.10"] }
    }
  ],
  "wildcards": [
    {
      "pattern": "*.example.com",
      "source": "crtsh",
      "first_seen": "2024-01-15T00:00:00Z"
    }
  ],
  "summary": {
    "total_found": 1,
    "returned": 1,
    "verified_count": 1,
    "unverified_count": 0,
    "sources_used": ["crtsh"],
    "apex_included": false,
    "wildcard_suppressed_count": 1,
    "wildcard_returned_count": 1
  },
  "intelligence_summary": {
    "data_sources": ["crtsh"],
    "source_count": 1,
    "cache_status": "live",
    "returned_count": 1,
    "total_found": 1,
    "truncated": false,
    "limit": 200,
    "verification_requested": true,
    "include_wildcards": true,
    "verified_count": 1,
    "verified_ratio": 1,
    "live_dns_record_count": 1,
    "apex_included": false,
    "wildcard_suppressed_count": 1,
    "wildcard_returned_count": 1,
    "first_seen_oldest": "2024-01-15T00:00:00Z",
    "first_seen_newest": "2024-01-15T00:00:00Z",
    "warning_count": 0
  },
  "meta": {
    "query_time_ms": 187,
    "cached": false
  }
}

This is an illustrative shape, not a result for `example.com`. In a real response, `summary.total_found` counts the full normalized list before the requested limit, while `summary.returned` counts the slice returned to the caller. `intelligence_summary.truncated` is true when the total is larger than the returned count. Because sorting happens before limiting, a limited response is the alphabetically first slice, not a risk-ranked or randomly sampled set.

The `source` on each row identifies the provider that supplied that hostname. `warnings` appears when the discovery service has a warning to expose. `meta.cached`, optional `meta.stale`, and `intelligence_summary.cache_status` tell you whether the response was assembled live, read from a fresh cache, or served from the longer-lived stale cache. Always store these fields with your evidence.

What verify=true means

When `verify=true`, DomScan takes the already limited list and queries each name for DNS type A through DNS-over-HTTPS. It reads A and CNAME records present in that answer. A row is marked `verified: true` only when at least one A address or CNAME was collected. The verifier works in batches with an overall time budget, so an unfinished or failed lookup remains false rather than delaying the whole request indefinitely.

This is deliberately narrower than the phrase live host sometimes implies. It does not issue a separate AAAA query, inspect MX or TXT records, connect to a port, send HTTP, validate a TLS certificate, or establish control by the expected organization. An IPv6-only hostname, a name with only another record type, a temporary resolver failure, or a timed-out batch can be reported as unverified. Conversely, an A or CNAME answer does not prove that an application is healthy or safe.

Cached and live data also answer different questions. Check `cache_status` before interpreting verification as a fresh observation. If current DNS matters to your decision, query the hostname through the DNS Lookup Tool or an authoritative resolver at decision time, and retain the response and timestamp.

Wildcard evidence stays separate

A certificate SAN such as `*.example.com` is a pattern, not a concrete hostname. DomScan suppresses wildcard SANs from the concrete `subdomains` array. When `include_wildcards=true`, it returns the patterns in a separate `wildcards` array and reports how many were suppressed and returned. DomScan does not expand a pattern into guessed labels.

Certificate wildcards and DNS wildcards are also different evidence. A wildcard certificate says which hostnames a certificate could match under TLS rules. A wildcard DNS record synthesizes answers under the rules clarified by RFC 4592. Finding one does not prove the other exists. During authorized brute force, test for wildcard DNS behavior first or a synthesized answer can make thousands of nonexistent guesses look valid.

Cache, pending, and upstream behavior

Subdomain providers can be slow, empty, rate limited, or temporarily unavailable. DomScan keeps separate cache entries for the domain, source selector, verification choice, and wildcard choice. A non-empty unverified result is fresh for up to 24 hours, an empty result for up to six hours, and a result requested with verification for up to 30 minutes. A stale copy can remain available for up to seven days as an outage fallback.

Normal requests check the fresh cache before contacting providers. If upstream discovery fails and no live result is available, DomScan can serve the stale copy. When every upstream is unavailable and no stale result exists, the route returns HTTP 503 rather than presenting a charged empty success. The billing layer refunds the charged credits for that failure path, and the response advises a later retry.

`prefer_cache=true` is useful for polling without waiting on providers. If neither fresh nor stale data exists, DomScan returns HTTP 202 with code `CACHE_MISS_REFRESH_QUEUED`, `retry_after: 30`, zero credits charged, and `billing_status: not_charged`. It queues a background refresh. A client should honor `Retry-After`, repeat the cache-only request, and handle another 202 without treating it as an empty discovery result.

Why no method finds every subdomain

Each passive source observes a different event. CT sees names placed in public certificate chains. HackerTarget and ThreatMiner expose their own collected host data. The Wayback CDX index exposes hostnames present in archived URLs. A name absent from one source may appear in another, but combining sources still cannot reveal a label that never reached any collected dataset. Provider retention, quotas, indexing delays, parsing rules, and outages add more gaps.

Active DNS guessing explores a different space. A tester can resolve common labels, generate permutations based on known naming conventions, recurse below discovered labels, or examine NSEC responses where appropriate. Zone-transfer testing asks an authoritative server for AXFR. Virtual-host discovery varies an HTTP Host header against approved infrastructure. None of these techniques guarantees completeness either, and none is performed by DomScan's Subdomain Finder.

Avoid vague tool comparisons. ProjectDiscovery describes Subfinder as a passive enumeration tool. Other products may add active resolution, permutations, crawling, or vulnerability scanning depending on their mode and configuration. Compare the exact version, flags, enabled providers, credentials, rate limits, and time of collection. A raw count without those controls says little about quality.

A responsible discovery workflow

1. Define the approved roots, excluded subsidiaries, third-party services, allowed techniques, rate limits, schedule, contacts, and stop conditions.
2. Run passive discovery and retain every hostname with its provider, evidence timestamp, cache status, warnings, and query time.
3. Preserve wildcard patterns separately from concrete names. Do not generate hosts from a wildcard certificate alone.
4. Resolve candidates with current DNS queries. Record A, AAAA, CNAME, and other relevant types rather than treating DomScan's narrow verified flag as a complete DNS assessment.
5. Within scope, probe only approved services and ports. Separate DNS existence, network reachability, application response, ownership, and security posture into distinct findings.
6. Assign each confirmed asset to an owner and lifecycle state. Unknown ownership is an inventory problem that deserves follow-up, not automatic evidence of compromise.
7. Store snapshots and compare them on a schedule. Review additions, removals, CNAME changes, certificate evidence, and unexplained changes with the responsible team.

Subdomain takeover requires more than a dangling-looking record

A stale CNAME can create takeover risk when its target refers to a third-party resource that has been deleted and can be claimed by another account. NXDOMAIN at the target, a provider-specific unconfigured-resource response, or a released address can justify investigation. None is universal proof. Provider behavior changes, some platforms reserve old identifiers, and control requirements differ by product.

Validate suspected takeover conditions against the provider's current official documentation and your own account records. Do not claim a third-party resource as a test unless the rules of engagement explicitly permit it and the provider permits that activity. The safest remediation is usually to remove the obsolete DNS record or reattach it to an organization-controlled resource, then verify DNS and application behavior after propagation.

Monitoring without an automatic-discovery claim

DomScan's Domain Monitor tracks domains and subdomains that a user explicitly adds. It checks their configured availability, DNS, certificate, and status signals over time. The current monitor does not automatically rerun root-domain subdomain discovery and add every newly found hostname. Adding a root domain therefore does not create a continuous subdomain inventory.

To monitor discovery, schedule your own authorized calls to `/v1/subdomains`, preserve each complete response, and diff normalized hostname sets. Treat source and cache changes carefully: a name can disappear because a fallback provider changed or a limited slice moved, not because DNS was removed. Use an appropriate limit, inspect `total_found` and `truncated`, and confirm additions or removals with current DNS before alerting. Known high-value hostnames can be added individually to Domain Monitor for ongoing checks.

The practical conclusion

DomScan's Subdomain Finder is a bounded passive discovery endpoint with a clear fallback strategy. It starts at crt.sh, uses HackerTarget, ThreatMiner, Wayback, and Cert Spotter as sequential fallbacks, can run a narrow A/CNAME verification pass over the limited result set, separates wildcard certificate patterns, and exposes cache and truncation metadata. Those details make the output useful because they show what the evidence can and cannot support.

Use it as one collection step in a documented inventory process. Preserve provenance, verify current state independently, add authorized active methods when the assessment calls for them, and repeat the process with consistent settings. The honest goal is not to claim that you found everything. It is to know how each hostname entered the inventory, what was checked afterward, and which coverage gaps remain.