API rate limits and credit budgets: plan your scraping capacity

Every scraping API gives you two ceilings, and people plan for the wrong one. The rate limit is the short-term ceiling: how many requests you can push per second or per minute. The credit budget is the long-term ceiling: how many pages you can fetch this month before the meter starts running. Hitting the first one gets you a 429. Hitting the second one gets you a bill.

Both are predictable, which is the part most teams skip. You can compute your monthly credit burn before you write a line of code, and you can handle rate-limit responses in about ten lines of retry logic. This post is the math and the retry pattern, plus the guardrails that stop a runaway crawler from draining an account overnight.

Two ceilings: requests per second and credits per month

A rate limit protects the provider's infrastructure, and occasionally the target site's. It is measured in requests per minute, or in concurrent requests. Exceed it and the API returns 429 with a hint about when to retry. It resets constantly: wait a minute and you have a full budget again. A rate limit almost never costs money directly, it costs time.

A credit budget is measured in pages, and it does not reset until the month does. Each successful request consumes credits — typically one per page per format. This is the ceiling that actually determines which plan you need, because it maps directly onto the size of your job. The failure mode is asymmetric too: blow through a rate limit and you get throttled; blow through a credit budget and either your pipeline dies mid-month or, worse, overage metering quietly starts charging per thousand pages.

Practical consequence: pick your plan by monthly page count, then check whether the plan's rate limit is even in the same universe as your peak concurrency. If your monthly math says 4,000 pages and the plan allows 20 requests per minute, you are fine — 4,000 pages is a few hours of crawling at that rate.

Do the crawl math before you pick a plan

The formula is boring and it works:

credits/month = pages × frequency × formats.

A product monitor over 500 pages, refreshed daily, one format: 500 × 30 = 15,000 credits. A one-off crawl of 10,000 URLs: 10,000 credits, once. An agent that fetches three pages per conversation and runs 200 conversations a day: 3 × 200 × 30 = 18,000 credits. Run the numbers for your actual job and the plan choice makes itself.

Two adjustments matter. First, fetch mode: an HTTP fetch and a full browser render usually cost the same on credit-based APIs, but they cost very differently in time, and time caps your daily throughput. Auto mode that tries HTTP first and falls back to a browser only when the page needs it keeps your average latency low without a second code path — the same tiering logic as theHTTP-only vs browser rendering decision. Second, failures: good APIs don't charge for failed requests. If yours does, timeouts and bot blocks become line items, and your real burn is higher than the math above.

Polling frequency is where budgets die

The crawl math is rarely what wrecks a budget. Frequency is. A dashboard poll every two minutes sounds cheap — 720 polls a day. Over a month that is 21,600 credits for one page, most of them fetching identical content. Multiply by a dozen watched pages and a $5 plan is gone before the first real data point arrives.

Before raising any frequency, ask what the data actually does. Rankings move daily. Prices move hourly at best. A changelog moves weekly. Match the interval to the fastest real change, not to how fast the poller could run. And where the API offers it, push the scheduling to the provider: a scheduled monitor with change detectionspends credits only when content actually differs, and it replaces the cron loop that would otherwise burn the identical-content polls.

The general rule: crawl depth and page count are engineering decisions, but polling interval is the single highest-leverage cost decision in most scraping projects. Halve the frequency and you halve the bill, with no loss of data anyone would notice.

Handle 429s and Retry-After properly

When you do hit a rate limit, the response usually tells you when to come back. The correct behavior is to honor it, add jitter so a thousand parallel workers don't stampede the same second, and cap retries so a permanently-blocked URL can't wedge your pipeline. Ten lines:

import random, time, requests

def fetch(url, key, max_retries=5):
    for attempt in range(max_retries):
        r = requests.post(
            "https://fastcrawl.net/api/v1/scrape/",
            headers={"Authorization": f"Bearer {key}"},
            json={"url": url, "formats": ["markdown"], "fetchMode": "auto"},
            timeout=60,
        )
        if r.status_code == 429:
            wait = float(r.headers.get("Retry-After", 2 ** attempt))
            time.sleep(wait + random.uniform(0, 1))   # honor + jitter
            continue
        r.raise_for_status()
        return r.json()
    raise RuntimeError(f"gave up on {url} after {max_retries} retries")

Two details people get wrong. Exponential backoff without the Retry-After header wastes minutes retrying a limit that resets in seconds. And retrying 4xx errors other than 429 is pointless — a 400 will fail identically forever, so fail fast and log it instead. On Fastcrawl, failed requests are never charged, so a retry loop costs you latency, not credits.

Guardrails: hard caps, usage checks, and batch discipline

Backoff handles the short ceiling. For the monthly budget, you want hard guardrails, because a retry loop with an off-by-one can turn 500 pages into 50,000 before anyone looks at a dashboard. Three that are worth wiring in on day one:

• A hard cap, not just a plan. If your provider meters overage past the plan allowance, decide deliberately whether a runaway crawl may spend it. Fastcrawl's Go plan meters overage at $0.25 per 1,000 pages, and you can disable overage in the dashboard to hard-cap at 5,000 credits — requests past that fail instead of billing. For unattended agents, a cap that fails loudly beats a meter that bills silently.
• Check the meter before big batches. A usage endpoint (GET /api/v1/usage) turns "I think we have credits left" into a fact. Gate long-running jobs on it: read the balance, subtract a safety margin, and refuse to start a 3,000-URL crawl with 500 credits in the tank.
• Batch at the API, not in a thread pool. Where a batch endpoint exists, use it — on Fastcrawl, POST /api/v1/batch/scrape takes up to 10 URLs in one call. Fewer client-side concurrency knobs means fewer ways to accidentally burst past a rate limit.

# Budget gate: refuse to start a big job below a safety floor
BALANCE=$(curl -s https://fastcrawl.net/api/v1/usage/ \
  -H "Authorization: Bearer ***" | jq -r '.credits_remaining')

if [ "$BALANCE" -lt 3000 ]; then
  echo "only $BALANCE credits left - skipping batch job"; exit 1
fi

The pattern generalizes: any unattended scraping job should have a budget check between "idea" and "first request." It is the scraping equivalent of a circuit breaker, and it costs one curl call.

A worked budget

Put it together for a realistic job: an AI agent that reads five pages per conversation, plus a daily price check over 300 product pages. The agent does 150 conversations a day: 5 × 150 × 30 = 22,500 pages. The price check is 300 × 30 = 9,000 — or far less if a monitor only charges on change. Total: roughly 30,000 pages a month.

At that volume the free tier (2,000 credits) is out, and the honest comparison is between a plan with metered overage and one that hard-caps. The polling discipline matters more than the plan: drop the price check from hourly to twice daily and the job shrinks by thousands of pages with no observable difference in the data. That is the whole exercise — the crawl math tells you the plan, and the frequency decisions tell you whether you needed the bigger plan at all.