HTTP-only vs browser rendering: when you don't need a browser

Every scraping API ships with a browser. Most of the time that's the wrong tool. A full headless Chrome render takes 2 to 5 seconds and bills browser-hours. A plain HTTP fetch of the same page takes 50 to 150 milliseconds. The two outputs are often identical for the majority of the public web — documentation sites, blogs, static product catalogs, Wikipedia, marketing pages, government portals. If you're spending 30× the time and money to get the same markdown, you're doing scraping wrong.

The hard part is knowing which sites need the browser and which don't. The answer is a single setting — fetch_mode in Fastcrawl, with three values: auto, http, and browser. The rest of this post is the rule for picking one.

The cost gap is bigger than people think

Browser rendering is the most expensive thing a scraping API does. It rents a remote Chrome instance, opens a tab, navigates, waits for the page to settle, and snapshots the DOM. The work itself is fast — but the round trip to a browser fleet, the warm-up of a session, the wait for the JavaScript to execute and the network to go idle, all add up. End-to-end: 2 to 5 seconds per page. At a million pages a month, that's a thousand browser-hours.

An HTTP fetch is a single GET request. No JavaScript executes. No remote browser is involved. The result is the raw HTML the server sent — which for the majority of sites on the public web already contains the full content, because the site was built to be indexed by Googlebot, which doesn't run JavaScript either. Latency: 50 to 150 milliseconds. Cost: a fraction of a credit. The two endpoints do identical work for static or server-rendered sites; the HTTP path is just 20 to 50 times faster.

How to know which mode a site needs

The signal is whether the meaningful content is in the raw HTML response. Three quick checks, in order:

1. Fetch it with curl and look. If the content you want appears in the first 200KB of HTML before any JavaScript runs, you don't need a browser. Most blogs, docs sites, news articles, Wikipedia, government pages, and server-rendered product pages pass this test.

2. Search for SPA markers. Open the page source and look for shell patterns: a single empty <div id="root"> or <div id="app">, __NEXT_DATA__ JSON blobs, Angular ng-version attributes, Vue data-server-rendered. If the document is mostly an empty shell waiting for JavaScript, the HTTP path returns nothing useful — you need a browser.

3. Check for client-side data fetching. If the page shows a spinner or "Loading…" in the raw HTML and the actual data appears only after a fetch to an internal API, the content is loaded dynamically. A browser is required.

The first check is the one that matters in practice. Step 2 and 3 are the fallback for cases where the initial HTML looks suspicious.

The three fetch modes

Most scraping APIs force one path. Fastcrawl exposes three, on the same /api/v1/scrape endpoint:

fetch_mode: "browser" (the default before the upgrade) — always render in a real browser. Safe, slow, expensive. Use this when you know the site is a heavy SPA.

fetch_mode: "http" — fetch the raw HTML with a Googlebot user-agent and convert to markdown. Skips the browser entirely. Use this when you know the site is static or server-rendered and you want the fastest possible scrape. Wikipedia, documentation, and most blogs fall here.

fetch_mode: "auto" (the new default) — try HTTP first, and if the result is suspiciously short (less than 200 characters of markdown, which catches empty shells, cookie walls, CAPTCHAs, and SPA loaders), fall back to a browser render. You get the speed of HTTP for the easy cases and the reliability of browser for the hard ones, with no code change.

# Force HTTP-only — known static site
curl -X POST https://fastcrawl.net/api/v1/scrape \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer ***" \
  -d '{
    "url": "https://en.wikipedia.org/wiki/Web_scraping",
    "fetch_mode": "http"
  }'

# Auto — let the engine decide
curl -X POST https://fastcrawl.net/api/v1/scrape \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer ***" \
  -d '{
    "url": "https://example.com",
    "fetch_mode": "auto"
  }'

The auto path is the one most callers want. It gives existing users the speedup with zero code change, and the fallback handles the long tail of sites that need rendering. We measured Wikipedia in the auto path at 14 milliseconds — versus 2 to 3 seconds for a forced browser render. The same page, the same markdown, the same API, 100× faster.

When the browser still wins

There are real cases where fetch_mode: "browser" is the right call. If the page is a Single-Page Application that hydrates on the client — React apps that fetch data from internal APIs, Angular dashboards, anything that shows a loading spinner before content — the HTTP path returns the empty shell and you're done. You need the browser to wait for JavaScript to execute and the data to render.

The harder case is sites that gate content behind anti-bot. Cloudflare challenges, DataDome, PerimeterX — these serve a JavaScript challenge that must be solved in-browser before the real content loads. The HTTP path sees the challenge page; the browser path solves it and gets the content. This is also why the auto fallback exists: if the HTTP path returns the challenge page, the markdown is short, and the engine retries in a real browser.

For sites that need a longer wait — a multi-step SPA hydration, a chart that renders over two seconds, a page that lazy-loads images on scroll — pair the browser path with a wait_for_timeout value. That tells the engine to keep the browser open and wait for the dynamic content to settle before capturing.

A simple decision rule

For the vast majority of scraping workloads — monitoring competitor pages, tracking prices, collecting articles, building training data, aggregating documentation — the right answer is fetch_mode: "auto". It gets the speed of HTTP when speed applies, and the reliability of a browser when it doesn't. You stop thinking about render modes entirely.

Use http explicitly when you've tested the site and confirmed the content is in the raw HTML and you want to lock in the speedup. Use browser when you've tested the site and confirmed the content only appears after JavaScript runs. The third case — the one where the site is a mix of static and dynamic — is the one auto was designed for.

Browser rendering will always be the right answer for some fraction of the web. But it's not the right answer for your workload, on your sites, most of the time. The fastest, cheapest scrape is the one that doesn't open a browser.

This is the other half of the speed story. The JS-heavy scraping guide covers the three-layer problem (render, extract, evade) — this post is the counterweight: when the browser is overkill, the HTTP path is the right call.