Crawling vs scraping: which one does your project actually need?

The two terms get used interchangeably, but they're different operations with different costs, failure modes, and etiquette rules. Picking the wrong one is the most common mistake teams make when wiring agents to the web.

Scraping is one page. Crawling is a graph.

Scraping fetches a single URL and converts it to structured data. You already know the address — a product page, a doc page, an article. One request in, one result out.

Crawling starts at one URL and follows links to discover and fetch many pages under the same site — bounded by depth, count, and domain rules. You don't know all the addresses yet; the crawl finds them.

The decision rule

• Know the exact URL? → scrape it.
• Need "everything about X on this site"? → crawl, then scrape or extract from the results.
• Just need the list of URLs? → map (sitemap + link discovery) without fetching full pages.

Why the difference matters in practice

Crawl cost scales with pages: a 500-page site at one credit/page is 500 credits whether you needed them or not — so good crawlers enforce same-domain boundaries, skip assets (images, PDFs, archives), and dedupe URLs by canonical form. Scraping fails differently: single pages get blocked, move, or render thin — so retry logic and SPA detection matter there.

A concrete example

# Scrape: you know the page
POST /api/v1/scrape  {"url": "https://example.com/pricing"}

# Map: you want the URL list first
POST /api/v1/map     {"url": "https://example.com"}

# Crawl: you want everything
POST /api/v1/crawl   {"url": "https://example.com", "max_pages": 100}

Etiquette still applies

Crawl politely: respect robots.txt, throttle concurrency, prefer off-peak schedules for recurring jobs, and never re-crawl what hasn't changed — that's what monitors with change detection are for. Your future self (and your IP reputation) will thank you.