Monitoring websites with AI agents: change detection that doesn't wake you at 3am

"Watch this page and tell me when it changes" sounds trivial until you build it. Pages change hourly for meaningless reasons — ads rotate, timestamps tick, A/B tests shuffle a hero image. Naive diffing floods you with noise; polling burns credits; cron scripts rot. The pattern below is how to do it properly.

The pattern: schedule → hash → diff → notify

1. Schedule the re-scrape at a human cadence (daily for competitor pages, weekly for docs).
2. Hash the normalized content — not raw HTML, which changes on every asset rotation.
3. Diff against the last successful run, not the last run — so a transient failure doesn't erase history.
4. Notify only on real change, via webhook, into whatever your team actually reads.

Normalization is where most attempts fail

Before hashing, strip everything that isn't content: whitespace collapses, timestamps removed, ad containers dropped. Otherwise you'll get pings for phantom changes. This is why monitoring belongs in the scraping layer — the same pass that strips boilerplate for LLMs also stabilizes the hash.

A concrete setup

POST /api/v1/monitors
{
  "url": "https://competitor.com/pricing",
  "schedule": "daily",
  "webhook_url": "https://your.app/hooks/monitor"
}

Each run re-renders the page in a real browser, hashes the clean markdown, compares to the previous success, and POSTs a JSON payload to your webhook only when something actually changed — with a summary of what moved. One credit per run; a daily monitor on ten pages costs less than a coffee per month.

What agents add on top

Once the firehose is quiet, point an agent at the diffs: summarize what changed, classify it (price move? new feature? wording tweak?), and route accordingly. That's the actual payoff — not knowing that a page changed, but knowing what it means without reading it yourself.

Use cases worth wiring up today

• Competitor pricing pages — catch moves within a day
• Changelogs and release notes — feed your roadmap reviews
• Job boards and career pages — signal of team growth
• Docs of APIs you depend on — deprecation warnings before they break you
• Regulatory/compliance pages in your industry