How to scrape JavaScript-heavy websites without getting blocked

Fetch a static page with curl and you get content. Fetch a React or Next.js app the same way and you get a shell — an empty HTML skeleton with a script tag and a prayer. JavaScript-heavy sites don't render until a real browser executes their code, which means traditional HTTP scrapers hit a wall on a growing share of the modern web. Meanwhile, those same sites are investing heavily in bot detection, so just spinning up Puppeteer isn't enough either.

This post covers the three-layer problem (render, extract, evade) and the practical tools that solve each layer — whether you use Fastcrawl's managed rendering or roll your own stack.

Why most HTTP scrapers fail on modern sites

Single-page applications (SPAs), React hydration, lazy-loaded components, and server-side rendering with client-side hydration all share the same property: the meaningful content isn't in the initial HTML. It's injected by JavaScript after the DOM loads. When you fetch the page with a plain HTTP client, you get the loader — not the data.

This isn't limited to flashy frontends. Increasingly, SaaS dashboards, e-commerce product pages, government portals, and even news sites use client-side rendering for critical content. The gap between "site works in a browser" and "content is in the raw HTML" is widening every year.

Layer 1: render the page properly

You need a real browser engine — something that parses HTML, executes JavaScript, waits for dynamic content to settle, and then hands you the fully rendered DOM. The options are:

• Headless Chromium (Puppeteer, Playwright, Chrome DevTools Protocol) — the standard approach. You control a real Chrome instance, wait for network idle or specific selectors, then scrape the DOM. Works, but requires infrastructure, error handling, and tuning per site.
• Rendering APIs (Fastcrawl scrape, Browserless, etc.) — you send a URL, they handle the browser pool, rendering, and cleanup. You get clean HTML or markdown back without managing infrastructure.
• Server-side rendering detection — some frameworks (Next.js, Nuxt) render on the server too. Check if the raw HTML already has your content before paying the cost of a full browser render.

The right choice depends on your scale. If you're scraping fewer than 100 pages per day, a managed API saves you weeks of browser maintenance. At thousands of pages, a self-hosted rendering farm gives you control over cost and latency.

Layer 2: extract structured data from rendered content

Once you have the rendered DOM, you still need to pull out the data you actually want. The raw HTML is verbose — it includes navigation, footers, scripts, and layout noise you don't care about.

This is where clean markdown conversion earns its keep. Fastcrawl's scrape endpoint returns LLM-ready markdown, not raw HTML — boilerplate stripped, content structured, tokens minimized. If you're feeding this into an AI pipeline, the extraction quality directly affects your token costs and output accuracy.

# Scrape a JS-heavy page with Fastcrawl
curl -X POST https://fastcrawl.net/api/v1/scrape \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{
    "url": "https://example.com/products/widget-123",
    "format": "markdown",
    "render_js": true
  }'

The render_js: true flag triggers full browser rendering. The response comes back as clean markdown with the product content extracted — no navigation, no cookie banners, no tracking pixels. For bulk extraction, the crawl endpoint handles breadth-first traversal with the same rendering pipeline.

Layer 3: don't get blocked

Rendering the page is table stakes. The harder problem is that sites actively detect and block scrapers. Here's what they look for and how to handle it:

Browser fingerprinting: Sites check for headless indicators — missing WebGL, navigator.webdriver set to true, absent browser plugins, missing media codecs. Modern headless browsers have closed most of these gaps, but you need to verify your fingerprint is clean. Tools like puppeteer-extra-plugin-stealth patch the common leaks.

Request patterns: A sudden burst of requests from a single IP, all hitting different pages with no referrer, no cookies, and uniform user-agent strings screams bot. Spread requests over time, rotate IP addresses via residential proxies, and maintain a cookie jar that simulates a real browsing session. The simplest anti-detection is behaving like a human: add delays between requests, vary your access patterns, and respect robots.txt even when you technically can ignore it.

JavaScript challenges: Some sites (Cloudflare, Datadome, PerimeterX) serve a JavaScript challenge that must be solved in-browser before the real content loads. These get more sophisticated every quarter. Managed scraping APIs like Fastcrawl handle these automatically — the challenge is solved inside their browser pool before the content reaches you.

When to use an API vs. self-host

The decision matrix is straightforward:

• Use a managed API when you want zero infrastructure overhead, automatic anti-bot handling, and you're scraping fewer than ~50K pages per month. Fastcrawl's free tier covers experimentation; paid plans scale with credits.
• Self-host when you need sub-second latency, have sensitive data that can't leave your infrastructure, or are scraping at a scale where per-page API costs don't make sense.
• Hybrid is often the right answer: use an API for the hard-to-scrape sites (those with bot detection) and a self-hosted stack for the simpler targets.

A practical workflow

Here's a concrete pipeline for scraping a JavaScript-heavy product catalog:

1. Discover URLs — use Fastcrawl's map endpoint or a sitemap parser to get the list of product pages.
2. Render and extract — scrape each URL with JS rendering enabled, returning structured markdown or JSON.
3. Normalize — strip timestamps, normalize prices to a common currency, deduplicate variants.
4. Store — write to your database or data warehouse.
5. Monitor — set up a Fastcrawl monitor on the catalog index page to detect new products or price changes.

This five-step pattern works whether you're building a price comparison engine, a market research tool, or feeding clean data into an LLM pipeline. The key insight is that the rendering and extraction layers are separable from the monitoring layer — get the data right first, then watch for changes.