Feeding LLMs clean web content with Markdown extraction

If you've ever pasted raw HTML into a prompt, you know the problem: navigation menus, cookie notices, inline scripts and tracking pixels eat your context window and confuse the model. For retrieval-augmented generation (RAG) and agents, clean Markdown is the format you want.

Why Markdown?

• Token-efficient. Markdown typically needs a fraction of the tokens of the equivalent HTML.
• Structure survives. Headings, lists, tables and links keep their meaning, which helps chunking and citations.
• Models are fluent in it. LLMs read and write Markdown natively.

Rendered, not just fetched

Many pages build their content in the browser. A plain HTTP request returns an empty shell. The Content API renders the page in Chromium first, then converts the final DOM to Markdown:

curl -G "https://api.screenshotdude.com/v1/content" \
  --data-urlencode "url=https://docs.example/guide" \
  -d include_markdown=true \
  -H "x-api-key: $SCREENSHOTDUDE_API_KEY"

Keep only the main content

Even rendered pages contain boilerplate. Add use_defuddle=true to keep only the main article — removing navigation, sidebars, footers and comments:

curl -G "https://api.screenshotdude.com/v1/content" \
  --data-urlencode "url=https://blog.example/post" \
  -d include_markdown=true -d use_defuddle=true \
  -d fields=data.markdown,data.metadata.title,data.metadata.canonical \
  -H "x-api-key: $SCREENSHOTDUDE_API_KEY"

The fields parameter trims the response to what you'll store, using dot notation.

A minimal ingestion pipeline

import { ScreenshotDudeClient } from "@screenshotdude/sdk";

const client = new ScreenshotDudeClient({ apiKey: process.env.SCREENSHOTDUDE_API_KEY });

const results = await client.batch(urls, (url) =>
  client.content({ url, include_markdown: true, use_defuddle: true }),
  { concurrency: 5 },
);

for (const r of results) {
  if (r.status !== "fulfilled") continue;
  // chunk r.value.data.markdown by headings, embed, upsert into your vector store
}

Discover pages with the sitemap

Need a whole site? Call once with include_sitemap=true to get its URLs, then extract each page. For hundreds of pages, use is_async=true with a webhook_url so nothing times out.

Give your agent the tools directly

If you use Claude, Cursor or another MCP client, connect the ScreenshotDude MCP server and the model can fetch Markdown on its own when it needs fresh web content.

Next: the Markdown playbook has complete examples in cURL, TypeScript and Python.