Scheduled monitors vs polling: change detection patterns that don't waste credits

"Watch these pages and tell me when something changes" is one of the most common things people build on a scraping API, and one of the easiest to build badly. The obvious implementation — a cron job that re-fetches every URL on a fixed interval and compares the result — works fine at ten pages and becomes a budget problem at a hundred. Most of what it spends is spent discovering that nothing happened.

The alternative is to move the schedule, the hashing and the diffing into the scraping layer, and have it call you only when there is something to say. This is the comparison between the two patterns, the cost arithmetic behind it, and the cases where polling is still the right call.

What polling actually costs

Take a realistic setup: 100 pages you care about, checked hourly. That is 2,400 fetches a day, roughly 72,000 a month. On a flat per-page credit model that is 72,000 credits — which on theFastcrawl Go plan means tens of dollars of overage every month, and on a per-1K-record competitor it is worse. Now consider what those 72,000 fetches bought you: if the pages are competitor pricing pages, changelogs or job listings, the number of runs that actually produced a change is likely somewhere between 20 and 200. Everything else was spent confirming the status quo.

The cost is not only credits. Hourly polling of 100 URLs means your scraper is issuing requests around the clock, which is exactly the traffic pattern that gets a range of IPs throttled or blocked. It also means your worker is awake at 3am doing nothing useful, and your storage is filling with 72,000 near-identical snapshots you will never query. Being a good citizen on someone else's site argues for the same conclusion: fetch at the cadence the data actually changes at, not the cadence your cron syntax made convenient.

What a scheduled monitor does instead

A monitor inverts the relationship. You register a URL, a schedule and an optional webhook; the scraping service runs the fetch on time, normalizes and hashes the content, compares it against the last successful run, and fires a webhook only when the hash moved. You pay one credit per run rather than per poll, and — more importantly — you stop writing the diffing code, the retry logic and the failure-state bookkeeping yourself.

curl -X POST https://fastcrawl.net/api/v1/monitors/ \\
  -H "Authorization: Bearer ***" \\
  -H "Content-Type: application/json" \\
  -d '{
    "url": "https://competitor.com/pricing",
    "schedule": "daily",
    "hour": 9,
    "timezone": "Asia/Hong_Kong",
    "webhook_url": "https://your.app/hooks/monitor"
  }'

The response carries the monitor id. From there the schedule is the service's problem, not yours: no cron container to keep alive, no clock drift, no "did the job run" dashboard. You can force a run out of band with POST /api/v1/monitors/{id}/run when you want a fresh read right now, and GET /api/v1/monitors/ lists every monitor with its next scheduled time.

The webhook payload is small and complete: event, monitor_id,url, changed: true, the new content_hash, and the markdown truncated to 50,000 characters so your handler can act on the diff immediately without a second fetch. If nothing changed, nothing is sent — that single property is where the savings come from.

Picking a cadence that matches the data

The biggest lever is not the technology, it is the interval. Match the schedule to how often the source genuinely changes, not to how eager you are. Stock and ticketing pages justify hourly. Competitor pricing, job boards and changelogs are almost always fine at daily. Documentation, terms of service and regulatory pages are weekly or monthly. A daily monitor on ten pages costs about 300 credits a month; the same ten pages polled hourly cost 7,200.

Cadence also decides when you find out. A daily monitor at 09:00 in your timezone means a price change posted at 10:00 is not visible to you until the following morning. That is usually an acceptable trade, and when it is not — flash sales, limited inventory, compliance deadlines — it is a valid reason to poll a small set of URLs aggressively while everything else stays on a schedule.

The hard part is the diff, not the fetch

Change detection fails in one of two directions: it misses real changes, or it reports changes that are not real. The second is more common and more corrosive, because a monitor that cries wolf gets muted and then deleted. Raw HTML is useless as a comparison key. Ad slots rotate, CSRF tokens regenerate, session identifiers appear in markup, A/B tests reorder a hero section, and a "last updated 2 minutes ago" ticker fires on every single request. Diff that and every run looks like a change.

The fix is to hash normalized content rather than the response body. Fastcrawl hashes the cleaned markdown with blank lines collapsed and each line trimmed, so whitespace churn and pagination artifacts stop registering as changes while genuine copy, price or structure edits still do. The comparison is against the last successful hash, so a transient failure — a 503, a bot wall, a timeout — never wipes your baseline and never produces a false positive on the next run. That baseline behaviour is the part hand-rolled monitor scripts get wrong most often: they compare against the last run, the last run stored an empty string, and suddenly every page in your fleet "changed" at once.

When polling is still right

Polling wins when you need the data regardless of whether it changed. If you are building a time series — price history, inventory levels, review counts — you want every sample, and a monitor that only notifies on change gives you a sparse series with gaps you cannot reconstruct. Poll those, and consider a cheaper fetch mode for them: most of these pages do not need a browser, and the HTTP-only vs browser rendering split cuts both latency and cost on high-frequency jobs.

Polling also wins when you need sub-minute freshness, or when the "change" you care about is not in the rendered text at all — a stock counter flipping to zero, a status badge changing colour, an element appearing for twelve seconds. Those need a fast loop you control, and a monitor on a daily or hourly schedule is the wrong tool. In practice most teams end up hybrid: a scheduled monitor on the long tail of pages that change rarely, and a tight polling loop on the handful that change constantly.

What to do with a change once you have one

Detection is the boring half; interpretation is where the value is. A webhook that says "this page changed" still requires a human to open it. Point an agent at the diff instead and ask it to classify — price increase, new plan tier, deprecated endpoint, new job posting — and route the result to the right channel. Because the payload already carries clean markdown rather than raw HTML, that classification step costs a fraction of what it would on uncleaned markup, which is the same argument as cutting your LLM token bill: the cheapest token is the one you never send.

The end state looks like this: monitors on a sensible cadence handle detection and stay quiet, a webhook wakes a small agent only on real change, and that agent decides whether a human needs to know. We covered the agent side of this inmonitoring websites with AI agents; the scheduling and cost side is what this post adds. Build the quiet part first — a monitor fleet that never false-positives is worth more than a clever classifier on top of a noisy one.