How to handle pagination in web scraping: offset, cursor, and load-more

Almost every list worth scraping is paginated, and almost every scraper that fails in production failed on pagination rather than on parsing. The listing page itself is easy. The hard part is knowing where the list continues, how to prove you have seen all of it, and how to stop without either looping forever or leaving 60% of the rows behind.

This is the working version: how each pagination style behaves, how to find the endpoint that actually returns rows instead of the HTML shell around them, and the termination and dedupe rules that keep a run bounded and correct.

The four pagination shapes you will meet

Offset is ?page=3 or ?offset=40&limit=20. The URL encodes position, so you can jump anywhere and parallelize freely. It is also the only style that silently returns duplicates: if rows are inserted while you walk the list, an item shifts from page 2 to page 1 and you see it twice, or miss it entirely.

Cursor is ?after=eyJpZCI6MTIzfQ — an opaque token pointing at the last row you saw. It is stable under inserts and always resumable, but strictly sequential: you cannot jump to page 7, and tokens expire. Load-more / infinite scroll is the same thing behind a button, and the button usually calls a JSON endpoint instead of rendering HTML, which is good news for you. Sitemaps and index pages are the fourth shape: sometimes the cleanest enumeration of a paginated site is not the list UI at all but /sitemap.xml or a category index that links every detail page.

The mistake is writing one loop for all four. Offset wants concurrency and dedupe; cursor wants a sequential loop with a stored token; load-more wants endpoint discovery; index pages want link filtering and no page math at all.

Find the endpoint that returns rows, not the page around them

Paginated HTML pages are usually a rendering of a feed that already exists as JSON. Before you build a loop over ?page=N, check whether the page itself gives you the URL set — themap endpoint returns every link reachable from a start URL, which is enough to enumerate most index pages without guessing page numbers at all:

# enumerate the real listing URLs instead of guessing ?page=N
curl -s https://fastcrawl.net/api/v1/map/ \
  -H "Authorization: Bearer ***" -H "Content-Type: application/json" \
  -d '{"url":"https://example.com/jobs","max_urls":2000}' \
  | jq -r '.links[]' | grep -E '/job/[0-9]+' > urls.txt
wc -l urls.txt

If the site has a stable item-URL pattern, this beats pagination entirely: you get a flat list, dedupe is a sort, and you can hand the whole thing to batch/scrape later. Where the map is thin because the list lives behind JavaScript, ask for the render:

curl -s https://fastcrawl.net/api/v1/scrape/ \
  -H "Authorization: Bearer ***" -H "Content-Type: application/json" \
  -d '{"url":"https://example.com/jobs?page=2","fetchMode":"browser","formats":["links"],"maxAge":0}' \
  | jq -r '.links[]' | grep -E '/job/[0-9]+'

fetchMode matters here. On an SPA, http fetches the same empty shell for every ?page=N value, and you get a loop that terminates instantly with zero rows while looking perfectly healthy. browser costs 2-5 seconds per page; auto starts HTTP and escalates when content is sparse. The trade-off is the same one covered inHTTP-only vs browser rendering.

Terminate on evidence, never on a page counter

A fixed loop — "fetch pages 1 through 50" — is wrong in both directions. It overruns when the list is short and it truncates when the list grows. Terminate on a signal from the data instead. The four signals that are safe to trust:

No new item IDs, compared against the set you have already collected. An explicit empty result or the site's own end marker. A repeated content hash, which happens when a site clamps an out-of-range page to the last page instead of returning nothing. Or the site's declared total, reached exactly.

import hashlib, json, os, requests

API = "https://fastcrawl.net/api/v1/scrape/"
H = {"Authorization": f"Bearer {os.environ['FASTCRAWL_KEY']}"}

def fetch(url):
    r = requests.post(API, headers=H, timeout=90, json={
        "url": url, "fetchMode": "auto", "formats": ["links"], "maxAge": 0,
    })
    return r.json()

seen_ids, page, prev_hash = set(), 1, None
while page <= 200:                     # hard cap is a guardrail, not the exit
    data = fetch(f"https://example.com/jobs?page={page}")
    h = hashlib.sha256(json.dumps(data.get("links", []), sort_keys=True).encode()).hexdigest()
    ids = {l.rsplit("/", 1)[-1] for l in data.get("links", []) if "/job/" in l}
    fresh = ids - seen_ids
    if not fresh or h == prev_hash:    # nothing new, or site clamped to last page
        break
    seen_ids |= fresh
    prev_hash, page = h, page + 1

print(len(seen_ids), "unique items from", page - 1, "pages")

Note what makes this safe: it stops on no new IDs, not on a page number, and it also stops on a repeated payload, which is what a clamped page looks like. The 200-page cap exists only so a bug cannot burn a night of credits. On a cursor site the same loop carries after forward from the previous response instead of incrementing page.

Dedupe on the item's identity, in its own table

Offset pagination guarantees duplicates on a live list, so dedupe is not optional. Use a stable key derived from the item itself — the numeric ID in the detail URL, a SKU, a requisition number — and never the title. Titles are near-duplicates all day ("Senior Engineer" at two companies, one job reposted with a dash swapped in).

The lazy version is a UNIQUE constraint on that key and an upsert: the boundary duplicates disappear in the database rather than in your loop, and re-runs are idempotent for free. Keep the natural key and the first-seen timestamp, and you also get a cheap "posted since" filter.

For sites you are going to walk repeatedly, keep the last page of a link-shaped list and stop when page 1 no longer contains anything new — the list is newest-first nearly everywhere, so a small overlap check beats re-walking the archive. And if the list is stable enough that only the top of it moves, stop paginating on a schedule altogether and put a monitor on the first page: you pay for one fetch a day and get a webhook when the set of IDs at the top changes. Themonitors vs polling post covers where that line sits.

Batch, crawl, and pay attention to the bill

Once you have a flat list of item URLs — from a map, from a paginated walk, from a sitemap — collecting them with batch/scrape is the right shape: up to 50 URLs per request, fetched 10-wide, with the same formats, maxAge and fetchMode options as a single scrape.

jq -n --argjson u "$(head -50 urls.txt | jq -R . | jq -s .)" \
  '{urls:$u, formats:["markdown"], fetchMode:"auto"}' \
| curl -s https://fastcrawl.net/api/v1/batch/scrape/ \
    -H "Authorization: Bearer ***" -H "Content-Type: application/json" -d @- \
| jq -r '.results[] | select(.success) | "\(.url)\t\(.markdown | length)"'

Two cost facts worth internalizing. Batching cuts wall-clock latency and request count, not credits — it is one credit per page either way, and failed pages are never charged. And the paginated walk itself is the line item people forget: a 60-page list is 60 credits every run, so schedule the walk on the cadence the data actually changes, not on your cron's convenience.

If the list is link-driven rather than parameterized, POST /api/v1/crawl does the walking for you: it starts at a URL, follows same-domain links in parallel up to max_depth(default 3) and max_pages (default 50, max 1000), and returns a job id you poll atGET /api/v1/crawl/{id}/. It canonicalizes each URL first — lowercase host, no fragment, default ports collapsed, query strings kept distinct — so ?page=2 and?page=2#top count once, not twice.

Pitfalls that turn a paginated run into a wrong answer

Duplicate spares. "Load more" lists commonly repeat the last row of the previous batch as an anchor. Dedupe or you will double-count the boundary item on every page.

The clamped last page. Many sites serve page 1 (or the final page) for any out-of-range value with a 200 status. A loop that only checks HTTP status never stops. Hash the payload.

Page-1-only selectors. Ad slots, featured rows and CMS-injected cards appear on page 1 but not page 5. If your extractor keys off container position, page 1 yields garbage. Key off the item's own link pattern instead.

Cache hiding fresh rows. Scrapes are cached for one hour by default, so a re-walk inside the hour returns the previous traversal. Pass "maxAge": 0 when freshness is the point, and leave the default on for archive passes where it saves credits.

Being a bad citizen. A 60-page walk is 60 requests to one host in a minute. Keep concurrency modest on small sites, identify your client, and read robots.txt; a board that blocks you turns a five-minute job into a rewrite. Full detail on that inscraping job boards at scale, where pagination is the whole game.