API reference

Website Crawler

Fetch raw HTML (request mode, no rendering) with status, headers and redirect chain.

GEThttp://localhost:4000/v1/website/crawler

Credit cost: 1 per successful request
Async: Supported (is_async)
404 billed: Yes
Plans: All plans

Authenticate with the x-api-key header. See authentication and credits & billing.

Parameters

Basics

Parameter Description

Parameter	Description
`query` stringrequired	A website URL with the http or https protocol. Example: `https://example.com`
`allow_redirects` boolean	Set to false to not follow redirects. Default: `true`
`return_page_source` boolean	Return the raw HTML source as text/html instead of JSON.

query

stringrequired

A website URL with the http or https protocol.

Example: https://example.com

allow_redirects

boolean

Set to false to not follow redirects.

Default: true

return_page_source

boolean

Return the raw HTML source as text/html instead of JSON.

Global parameters

Accepted by every billable endpoint. Details in Global parameters.

Parameter	Description
`soft_fail` boolean	Return HTTP 200 even when an error occurs, with the message in `error`. Default: `false`
`is_async` boolean	Process asynchronously; returns a `job_id` to poll on /v1/jobs/{id}. Default: `false`
`fields` string	Comma-separated list of response fields to keep (dot notation for nested).
`webhook_url` string	With is_async=true, POST the finished job to this URL (signed, paid plans).

Code examples

curl "http://localhost:4000/v1/website/crawler?query=https%3A%2F%2Fexample.com" \
  -H "x-api-key: $SCREENSHOTDUDE_API_KEY"

Responses

200 — application/json

{
  "success": true,
  "data": {
    "url": "http://acme.example",
    "final_url": "https://acme.example/",
    "status": 200,
    "headers": {
      "content-type": "text/html; charset=utf-8",
      "server": "nginx",
      "cache-control": "max-age=600"
    },
    "redirects": [
      {
        "url": "http://acme.example",
        "status": 301,
        "location": "https://acme.example/"
      }
    ],
    "content_type": "text/html; charset=utf-8",
    "html": "<!doctype html><html lang=\"en\"><head><title>Acme</title>..."
  }
}

200 — with return_page_source=true (text/html)

<!doctype html>
<html lang="en">
  <head><title>Acme</title></head>
  …
</html>

How it differs from the Content API

The website crawler performs a plain HTTP request — no browser, no JavaScript. It's faster and returns exactly what the server sent: status code, headers, redirect chain and source. Use the Content API when you need the rendered page.

Response fields

Field	Type	Description
`data.url`	string	The requested URL (`query`).
`data.final_url`	string	URL after redirects.
`data.status`	number	HTTP status of the final response.
`data.headers`	object	Response headers (lower-cased names).
`data.redirects`	object[]	Each hop: `{ url, status, location }`. Empty when `allow_redirects=false` or none occurred.
`data.content_type`	string	`Content-Type` of the final response.
`data.html`	string	Response body as text.

With return_page_source=true the body is returned directly as text/html instead of JSON.

Billing of 404

Unlike other endpoints, a 404 from the target site is billed: the crawl succeeded and you received the target's response. Errors from the API itself (invalid query, auth, credits) remain free.

Errors

Errors return JSON { "error": "…" }. Failed requests are never billed (except a 404 from the target site on this endpoint). Full list in Status codes.

Status	Meaning
400	Missing or invalid parameter (e.g. invalid URL, unsupported format, private network target)
401	Invalid, inactive or expired API key; plan rate limit (`Rate limit exceeded: …`) or key quota (`…: quota daily`)
403	`IP temporarily blocked` after repeated authentication failures; endpoint not included in your plan
402	`Payment required`, `You have open invoices: …` or `Insufficient credits: X required, Y available`
404	Target returned 404 — billed for this endpoint
413	Result too large (e.g. extremely tall full-page capture)
500	Rendering failed or timed out (60 s) — not billed; retry or use `is_async=true`