Skip to content
ScreenshotDude

Playbooks

Playbook: extract Markdown

Turn any page into clean, main-content Markdown for LLMs and search — in cURL, TypeScript and Python.

This playbook extracts the main content of an article as Markdown, with title and canonical URL — the ideal input for RAG pipelines and prompts.

The request

Parameter Why
include_markdown=true Return Markdown.
use_defuddle=true Keep only the main content (no nav, sidebars, footers, comments).
remove_cookie_banners=true Avoid consent text leaking into content.
fields=markdown,metadata.title,metadata.canonical Return only what you'll store.

Code

curl -G "http://localhost:4000/v1/content" \
  --data-urlencode "url=https://example.com/blog/post" \
  -d include_markdown=true -d use_defuddle=true \
  -d remove_cookie_banners=true \
  -d fields=markdown,metadata.title,metadata.canonical \
  -H "x-api-key: $SNAPLENS_API_KEY"
{
  "success": true,
  "data": {
    "metadata": { "title": "How we scaled our search", "canonical": "https://example.com/blog/post" },
    "markdown": "# How we scaled our search\n\nLast year our index grew 10×…"
  }
}

Crawling a whole site

  1. Call once with include_sitemap=true&fields=sitemap to get every URL.
  2. Extract each page with is_async=true and a webhook_url, or the SDK's batch() helper with a concurrency within your rate limit.
  3. Chunk Markdown by headings and embed.

Tips

  • Without use_defuddle, Markdown includes navigation and footers — useful for link analysis, noisy for LLMs.
  • Add stealth=true for sites that block automated browsers (paid plans).
  • block_resources=image,media,font speeds up extraction when you only need text.