Web scraping for SEO monitoring: rankings, on-page, competitors

SEO tools are a scraping business with a dashboard on top. A rank check is a search plus a position lookup. An on-page audit is a fetch plus a field extraction. A competitor alert is a diff. Once you see that, the question stops being "which suite should I subscribe to" and starts being "can I run these four jobs cheaper with an API and a cron."

Usually yes, if your tracked set is under a few thousand URLs. This post is the working version: the three jobs, the endpoint for each, the cache trap that makes rank data lie to you, and the credit math that tells you whether to keep the subscription.

The four jobs any SEO monitor performs

Strip the product pages and an SEO monitor does four things on a schedule. It finds out where you rank for a query. It reads your own pages and checks whether the technical layer still holds (title, canonical, headings, links, alt text). It watches competitors for price, packaging, and content moves. And it tells a human only when one of those three actually changed.

Three of the four need a fetch. Only the last needs judgement, and that is the part an agent or a webhook handler can do. So the whole system reduces to: search, scrape, compare, notify. Nothing in that chain requires a $129/month seat unless you are tracking tens of thousands of keywords across multiple countries and devices.

One caveat before the code: position data is never one number. Google personalizes by account, history, location and device, and SERPs differ by city. A rank check returns a distribution, not a scalar. Teams that forget this build dashboards nobody trusts.

Rank checks: search, then locate your URL

The simplest version returns ranked results for a query and you find your position in the list. Fastcrawl's POST /api/v1/search takes a query and a limit (max 20) and returnstitle, url, snippet per result:

# where do we show up for a query?
curl -s https://fastcrawl.net/api/v1/search/ \
  -H "Authorization: Bearer ***" \
  -H "Content-Type: application/json" \
  -d '{"query":"headless cms for technical documentation","limit":20}' \
  | jq -r '.results | to_entries[] | "\(.key + 1)\t\(.value.url)"' \
  | grep example.com

Two things to get right. First, force fresh content: the scrape cache defaults to one hour (maxAge: 3600), and cached results make yesterday's rank look like today's. Pass"maxAge": 0 on anything where freshness is the measurement. Second, record the parameters with the result: query, engine, limit, timestamp. Without them, a position number is not comparable to anything and your trend line is fiction.

If you need Google-specific, geo-locked, device-locked SERPs at volume, that is a dedicated search-results API rather than a general scraper, and the trade-offs are spelled out in theSerpApi comparison. For everything else — Bing-agnostic visibility, "did we appear at all", competitor coverage, answer-engine citations — a general search endpoint plus a scrape is enough.

On-page audits: one fetch, structured output

A technical audit over your own site is a discovery step and an extraction step. Discover withmap (URLs only, one credit), then pull the pages in batches of up to 50 withbatch/scrape. Ask for json and links and you get structure instead of markup to parse yourself:

# 1. discover the site
curl -s https://fastcrawl.net/api/v1/map/ \
  -H "Authorization: Bearer ***" \
  -d '{"url":"https://example.com","max_urls":500}' | jq -r '.links[]' > urls.txt

# 2. audit 50 at a time
curl -s https://fastcrawl.net/api/v1/batch/scrape/ \
  -H "Authorization: Bearer ***" \
  -d "$(jq -n --argjson u "$(jq -R . urls.txt | jq -s '.[0:50]')" \
        '{urls:$u, formats:["json","links"], fetchMode:"http"}')" \
  | jq -r '.results[] | select(.success) | .json.title'

Note fetchMode: "http". Your own pages are server-rendered; paying a browser to wait for JavaScript you don't need just costs seconds, not credits, but at 500 pages seconds add up. The tiering logic is the same one inHTTP-only vs browser rendering: start cheap, escalate only when a page proves it needs it. auto does that decision for you.

For fields a generic extractor won't hand you — whether the FAQ schema is present, whether the H1 matches the title tag, whether the pricing table mentions a new tier — usePOST /api/v1/extract with a JSON schema. It costs 1 credit per page and returns validated JSON, so the audit becomes a table you query rather than a report you read:

curl -s https://fastcrawl.net/api/v1/extract/ \
  -H "Authorization: Bearer ***" \
  -d '{
    "url": "https://example.com/pricing",
    "prompt": "On-page SEO fields for this page",
    "schema": {
      "type": "object",
      "properties": {
        "title":         {"type": "string"},
        "h1":            {"type": "string"},
        "title_matches_h1": {"type": "boolean"},
        "word_count":    {"type": "integer"},
        "plans":         {"type": "array", "items": {"type": "string"}}
      }
    }
  }'

For spot checks without writing code, themeta tag checker andsitemap checker do the single-URL version in a browser tab.

Competitor monitoring: diff, don't re-read

The expensive habit in competitor tracking is re-reading everything. Add"changeTracking" to your formats and the API compares against your previous scrape of that URL and returns metadata.change_information: a change_status offirst, unchanged, added, removed ordifferent, plus a line diff.

curl -s https://fastcrawl.net/api/v1/scrape/ \
  -H "Authorization: Bearer ***" \
  -d '{"url":"https://competitor.com/pricing","formats":["changeTracking"],"maxAge":0}' \
  | jq '.metadata.change_information'

Diff against clean markdown, not raw HTML. Raw HTML changes every time an ad rotates or a build ID bumps, and you get a daily "change" that means nothing. This is also why monitoring belongs in the scraping layer rather than in your database: the same normalization pass that strips boilerplate for LLM input is what stabilizes the hash.

Push the schedule off your own machine. Monitors are on the paid plan: give it a URL, a cadence, an hour, a timezone and a webhook, and it re-scrapes and POSTs only when content differs.

curl -s https://fastcrawl.net/api/v1/monitors/ \
  -H "Authorization: Bearer ***" \
  -d '{"url":"https://competitor.com/changelog","schedule":"daily",
       "hour":9,"timezone":"Asia/Hong_Kong",
       "webhook_url":"https://your.app/hooks/seo"}'

Then wake an agent on the webhook payload instead of on the clock: summarize what moved, classify it (price change? new feature? copy polish?), route only the material ones. Themonitors vs polling post covers that pattern in detail, including when a cron loop is still the right call.

The credit math for a monthly SEO run

Every endpoint costs one credit per call and failed calls are free, so an SEO workload is easy to price. Take a realistic setup: 150 tracked queries checked daily, 300 of your own pages audited weekly, 40 competitor pages monitored daily.

• Rank checks: 150 × 30 = 4,500 credits
• On-page: 300 × 4 = 1,200 credits
• Competitors: 40 × 30 = 1,200 credits

About 6,900 credits a month. That fits the $5 Go plan (5,000 credits) with overage metered at $0.25 per 1,000, so roughly $5.50 all in — versus a suite seat at $100+. The free tier is the wrong shape for this job: 2,000 credits is fine, but the 100 requests per day cap means you cannot run 150 daily rank checks at all. Daily frequency on any real keyword set is a paid-plan decision, not a credits decision.

The bigger lever is frequency, and it is the same one inplanning your scraping budget. Rankings do not move hourly. Weekly for the long tail, daily for the money terms, and you cut the rank-check line by two thirds with no loss of signal anyone would notice.

When to keep the subscription

DIY wins when your tracked set is bounded, your cadence is human (daily or weekly), and you already have somewhere to put results — a Postgres table, a sheet, a Slack channel. It loses when you need historical index volumes, click-through data from Search Console at scale, backlink graphs, or multi-country localized rank grids. Building a backlink index is not a weekend; buying that part is fine while you scrape the parts that are just fetch-and-diff.

The pragmatic mix most teams land on: keep the suite for the industry-wide data you cannot reproduce, and run your own monitors for the 40 pages you actually care about. Those are the ones you want answered in an hour rather than on the vendor's refresh schedule.