Why your scraping API bill is unpredictable

Every scraping vendor publishes a rate: $1.50 per 1,000 records, $0.60 per 1,000 pages, "from $0.30 per 1K". Those numbers are usually honest. The problem is that they are rates on a unit you don't control yet. Your invoice is the rate multiplied by a unit count that depends on how long the page was, which endpoint you hit, whether a browser was needed, whether the request succeeded, and which add-ons were switched on at runtime.

That is why the same workload can come in at $40 one month and $190 the next without anyone changing the code. This post walks through the four mechanics that make it happen, and what a bill looks like when they aren't in play.

Four ways a per-request price becomes a variable bill

Almost all unpredictability reduces to one of these: token metering (the unit is output size), per-endpoint multipliers (the unit is which call, or how many seconds the browser ran), add-on stacking (a proxy tier, an unlocker tier, an extraction pass, a PDF render, each priced separately), and billing failures(a blocked page or a retry still costs you).

None of these is dishonest on its own. They are all attempts to price something expensive to provide. But each one adds a factor to your bill that you can't read off the rate card, and the factors multiply. A token-metered endpoint that also needs a premium proxy tier and a structured-extraction pass isn't 1x the headline rate. It's somewhere between 3x and 10x, and the exact number varies by page.

Token metering: the same URL costs different amounts

Token-metered pricing makes the unit of work the length of the output. Jina Reader works this way: a long article costs more than a short one for the identical operation, and its ReaderLM-v2 mode costs roughly 3x the tokens for the same fetch. Your cost per URL is not a number, it's a distribution, and the number you budget with is the distribution's tail, not its mean.

The practical consequence is that forecasting becomes a modelling exercise. If you scrape 10,000 pages a month and page length varies by 3x across your target set, your bill estimate carries the same error bar unless you sample page lengths first. Nobody samples page lengths before they start. They find out in month two.

There's a second-order effect that bites harder. Token metering couples your scraping cost to your LLM cost, because both are measured in the same currency. That sounds tidy until a target site adds a 400-token nav block to every page, and your scraping spend moves without a single line of your code changing. The site you scrape controls your invoice.

Multipliers and add-ons

Other vendors price per endpoint and per capability. Search costs 2 credits, browser-interact costs 2 per minute of session time, crawl is per page plus an extraction pass, residential proxies are a higher tier than datacenter, "premium" unlocker endpoints are higher again. Each line is defensible in isolation. Stacked, a single page can cost many times the headline rate.

The reason this is hard to forecast isn't the arithmetic, it's that the vendor decides the tier per request, at runtime, based on signals you can't see. Which proxy pool your target resolved to, whether the anti-bot layer engaged, whether the render fell back to a browser. You are billed for decisions made inside someone else's scheduler. Two URLs on the same domain can land on different tiers on the same day.

Managed-dataset pricing has a related shape. Bright Data's dataset products run around $1.50 per 1,000 records, which is a clean unit right up until you notice that record count is not page count and varies by how much structured data the page happens to expose. The useful question to ask any vendor is not "what is your rate" but "what will this URL cost at this tier, and who decides the tier". If the answer is "we decide, per request", you've found the unpredictability.

You pay for failures and retries

On several platforms a blocked request still consumes credits. That inverts the incentive: your worst day, the one where a WAF turned on or a rate limit tripped, is also your most expensive day. Add an automatic retry loop and the failure is billed twice, or five times, with the retry logic itself becoming the cost driver.

Failures-as-results is the better contract, and it's worth checking for explicitly rather than assuming. A scrape that returns a 422 with a target_blocked error code and aretryable: false flag should cost nothing, because you got no data. Fastcrawl charges zero for failed requests, and the same policy covers the free tier, which is the only way a "try it and see" evaluation is meaningful. Cache hits are free too: re-scraping a URL inside its freshness window with maxAge returns the stored copy and bills nothing.

This is the single highest-leverage line item to audit in your own usage data. Pull last month's requests, count the non-2xx responses, and ask what they cost you. If the answer is "something", that's a vendor line you can eliminate by switching, not by optimising.

What a forecastable bill looks like

The alternative is boring, which is the point. One unit per call, identical across every endpoint and every format: scrape, crawl page, map, search, extract, screenshot, PDF, parse. No token counting, no per-endpoint multiplier, no browser-minute surcharge, and no tier decision made behind your back. HTTP fetch and browser render cost the same credit, so the choice between them is a latency decision rather than a budget one.

When the unit is flat, the forecast is arithmetic. 20,000 pages a month is 20,000 credits. The only variable left is how many calls you make, which is entirely under your control, and the two levers that reduce it are caching and change detection. That's the subject ofplanning your scraping budget, and ofscheduled monitors vs polling — a monitor re-fetches and hashes each run and spends a credit per run, but it only fires a webhook when the page actually moved, so you pay for the checks and get told about the changes.

The last piece is a hard ceiling. Fastcrawl's Go plan is $5 for 5,000 credits, with overage metered at $1.00 per 1,000 extra and switchable off entirely in the dashboard if you want a stop rather than a surprise. A bill that can be capped is a bill you can put in a budget line without a margin for mystery.

Measure your own cost per page

Don't take a vendor's rate card as your unit cost. Run 20 representative URLs, read your credit balance before and after, and divide. If the answer is 20, your page length doesn't matter and your forecast is page count times runs. If it's 31, you now know the variance you are budgeting against, and you know it before you commit to a plan size.

# credits before
curl -s https://fastcrawl.net/api/v1/usage/ -H "Authorization: Bearer YOUR_KEY"

# 50 URLs through the same pipeline as /scrape, 10-wide parallel
curl -s -X POST https://fastcrawl.net/api/v1/batch/scrape/ \
  -H "Authorization: Bearer YOUR_KEY" -H "Content-Type: application/json" \
  -d '{"urls":["https://example.com/pricing","https://example.com/docs"],
       "formats":["markdown"],"timeout":30,"fetchMode":"auto"}' \
  | python3 -c 'import json,sys; d=json.load(sys.stdin); print(d["succeeded"],"of",d["total"],"ok")'

# credits after — the delta divided by the URL count is your real cost per page

Run that once a quarter. The number that should move your bill is how many pages you scrape, not how long they happen to be. If a page-shape change on a target site shows up as a cost change on your side, you're on a metered unit. If it doesn't, you can stop budgeting for it.

If you want the shorter version of this comparison, thevendor comparisons break down each competitor's unit, floor price and what they charge for failures — including theJina Reader head-to-head on token metering. TheAPI docs list every endpoint and confirm the flat-credit rule per call.