Web scraping for market research: a practical pipeline

Market research is the one scraping job where the data is already public, the questions are already written, and the work is entirely mechanical. You have a competitor set. You want the same fields from all of them, on a schedule, in a place you can query. That is a pipeline, not a research project.

Most teams build it the other way round: a pile of one-off scrapers, each written for the page it was pointed at, each breaking independently when a layout changes. The pipeline version is four steps that stay the same no matter who the competitors are: map the source,extract a fixed schema, monitor for change, andstore the result. This is what each step looks like with real calls.

Decide what you are surveying before you fetch anything

Market research scraping fails when the question is vague. "Understand the competition" is not a query. "Which of these 40 vendors added a free tier in the last 90 days" is a query, and it tells you the exact fields you need: vendor name, plan names, price points, feature flags, snapshot date.

Write that field list down first. Every downstream decision follows from it — how often you need to re-scrape, which pages matter, whether you need a browser at all. A pricing page is usually static HTML, so HTTP-only fetching is enough and costs you a fraction of a rendered page. A pricing table that loads from an API in the browser needs a real render. Same research question, different cost profile, decided by the field list.

Keep the set small and named. Forty vendors you actually track beats four hundred URLs you never look at. If you are surveying a market rather than a named set of companies, you are really building a directory, which is a crawl problem, not a scrape problem.

Step 1: map each source to find the pages that hold the fields

You rarely want the whole site. You want the five or six page types that carry the fields: pricing, product, changelog, careers, docs changelog, and the blog index. Themap endpoint returns the URL inventory for a domain so you can filter to those page types instead of crawling everything:

curl -X POST https://fastcrawl.net/api/v1/map \
  -H "Authorization: Bearer ***" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://example-vendor.com","max_urls":200}'

The response is a JSON array of URLs under links. Grep that list for/pricing, /changelog, /docs and you have your target set — the filtering happens in your code, on a list, which is far cheaper than discovering the same thing by scraping four hundred pages and throwing most of them away.

Run map once per vendor and store the result. It is the only part of the pipeline that changes slowly, so re-mapping monthly is plenty. Everything after this is a fixed URL list and a schema.

Step 2: extract one schema across every source

This is the step that makes the data comparable. Scraping raw HTML gives you forty different markup dialects that you then have to normalise by hand. Extracting to a schema gives you forty rows of the same shape, which is the actual asset. The extract endpoint takes a URL and a JSON Schema and returns the schema filled in:

curl -X POST https://fastcrawl.net/api/v1/extract \
  -H "Authorization: Bearer ***" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example-vendor.com/pricing",
    "prompt": "Extract the public pricing plans for this product. Use the plan name shown on the page, the monthly price in USD, and the customer-facing feature bullets verbatim.",
    "schema": {
      "type": "object",
      "properties": {
        "vendor": { "type": "string" },
        "plans": {
          "type": "array",
          "items": {
            "type": "object",
            "properties": {
              "name": { "type": "string" },
              "monthlyPriceUsd": { "type": "number" },
              "featureBullets": { "type": "array", "items": { "type": "string" } }
            },
            "required": ["name"]
          }
        }
      },
      "required": ["vendor", "plans"]
    }
  }'

Two details matter here. First, the prompt carries the disambiguation rules ("the public pricing plans", "the price shown on the page") while the schema carries the structure. Keep them separate and a field rename is a one-line change rather than a rewrite. Second, the schema is the same for every vendor, so your downstream SQL is one table, not one parser per company.

For a set of 40 vendors, send them as batches rather than 40 sequential requests. Thebatch endpoint (POST /api/v1/batch/scrape with aurls array) takes up to 50 URLs per call at 10-wide concurrency, and grouping by page type keeps a failure on one vendor's pricing page from poisoning a run.

Step 3: let monitors do the repeated work, not your cron

The field list is stable; the values are not. Pricing changes, features ship, a free tier appears. The naive version of this step is a cron job that re-scrapes everything nightly and diffs in your own code. The version that survives contact with production is a scheduled monitor that fires a webhook only when the content hash actually changes:

curl -X POST https://fastcrawl.net/api/v1/monitors \
  -H "Authorization: Bearer ***" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example-vendor.com/pricing",
    "schedule": "daily",
    "timezone": "Asia/Hong_Kong",
    "hour": 7,
    "webhook_url": "https://your-app.example.com/hooks/vendor-change"
  }'

The monitor stores a snapshot, compares on the next run, and reports back throughGET /api/v1/monitors/ with last_change_detected, or viaPOST /api/v1/monitors/{id}/run if you want to force a check while you are debugging. Your application code only ever sees "this page changed", which means you re-run extraction on the handful of pages that moved instead of all 40 every night.

That is the same change detection pattern asmonitoring websites with AI agents, pointed at a research question instead of an uptime one. The economics are the point: watching 40 pages daily costs less than re-extracting 40 pages daily, and the diff gives you a timeline for free.

Step 4: store diffs, not snapshots

Market research data is only useful as a time series. Keep a table keyed on(vendor, field, observed_at, value) and write a row only when the value changes. Six months of that table answers questions a snapshot cannot: who raised prices, who cut them, how fast the market converged on annual billing, which vendor added a feature the week a competitor announced theirs.

Extract text, not rendered pages, on the way in. Clean extraction is what makes the storage layer cheap and the LLM step affordable later, which is the same lever ascutting your LLM token bill: you pay once for structure and then query it forever. Raw HTML archives accumulate cost and give you nothing to query.

Keep it defensible

Everything described above reads public pages at low volume with a fixed cadence. That is the shape of the work that stays unremarkable: public pages only, no login walls, no personal data, no attempt to look like a human when you are not one. Respect robots.txt, keep concurrency modest, and space runs so a monitor firing daily does not hammer a small vendor's site.

Also respect the volume ceiling. A research pipeline needs a handful of page types per vendor, not a full-site crawl every night. If you find yourself needing hundreds of pages per company per day, the question has probably moved from "market research" to "mirroring someone's product", and that is a different conversation with your lawyers than it is with your scraper.