What "LLM-ready markdown" actually means

Every scraping API claims to return "LLM-ready" output. Most of the time that phrase hides four separate engineering problems, and whether they're solved determines whether your RAG pipeline eats good data or garbage. Here's what to actually check.

1. Boilerplate stripping

A product page rendered to markdown should contain the product — not the nav bar, cookie banner, footer links, and newsletter popup. Good pipelines strip chrome before conversion; bad ones hand your LLM 3,000 tokens of menu. The test: scrape a news article and check whether the first token is the headline or "Sign in".

2. JavaScript rendering

Roughly half the modern web renders content client-side. A plain HTTP fetch returns an empty shell with a root div and a script tag. Real rendering means a browser loads, executes, and settles before extraction — SPAs, infinite scroll, and hydration included. If your scraper can't read React apps, it can't read the web your users browse.

3. Token cost

Clean output isn't just about quality — it's about cost. Every token of boilerplate is a token you pay for twice: once to fetch, once in every LLM call that processes it. Well-stripped markdown routinely cuts input tokens by 20–40% versus raw text dumps. On a pipeline doing thousands of reads a day, the metering model of your scraper (flat per page vs per output token) changes your unit economics.

4. Structure preservation

Headings, lists, links, and tables carry meaning. Flattened output forces the model to guess hierarchy; structured markdown lets retrieval chunk by section. Check that headings survive as headings, links keep their hrefs, and tables don't collapse into comma soup.

The quick evaluation recipe

# 1. A JS-heavy SPA (should NOT be an empty shell)
curl -X POST https://fastcrawl.net/api/v1/scrape/ \
  -H "Authorization: Bearer $KEY" \
  -d '{"url":"https://todomvc.com/examples/react/dist/"}'

• If you get real todo items back → rendering works.
• If the first line is a headline → boilerplate stripping works.
• If headings are marked up → structure survives.
• Divide output tokens by the page's visible word count → your true cost ratio.