Scraping job boards at scale: patterns, pitfalls, and a working pipeline

Job boards are some of the most scraped pages on the web, and for good reason. They hold structured, high-value data — job titles, companies, locations, salaries, posting dates — that feeds job aggregators, salary studies, and market research. A pipeline that works for one board usually breaks on the next. This is how I'd build one for the Hong Kong market (headlinejobs.hk, hkgoodjobs.com, hk.jobsdb.com) on a scraping API, where the real effort sits, and the pitfalls that quietly burn credits.

Why job boards are their own animal

Unlike a blog or a marketing page, a job board is a database rendered as a list. That changes what a scraper has to do. Nearly all boards are pagination- or filter-heavy: one search page holds 10-30 listings, and the rest sit behind ?page=2, ?page=3, or an infinite-scroll feed. You are not scraping one page; you are walking a large, mostly-regular graph of pages.

Two things work in your favor. First, most boards are server-rendered — the HTML you get from a plain HTTP GET already contains the listings, so a headless browser is often a waste of compute (see when you don't need a browser). Second, the interesting content is structured and repetitive: every listing has the same fields, which makes it a clean target for JSON extraction rather than scraping-and-parsing blind HTML.

What does not work in your favor: boards change schema as they update their themes, listings expire fast (so freshness matters more than depth), and some operators slap bot walls on their search endpoints. A robust pipeline plans for all three from the start.

Step 1 — discover the listing URLs

You rarely want to start from the homepage and crawl everything. Boards have thousands of jobs and a lot of noise (category indexes, company pages, "about" pages). Find the actual listing URLs first with the map endpoint, which returns every URL reachable from a start page:

curl -X POST https://fastcrawl.net/api/v1/map/ -H "Authorization: Bearer YOUR_KEY" -H "Content-Type: application/json" -d '{"url":"https://hk.jobsdb.com/jobs-in-information-technology"}'

Filter the returned list to URLs matching your listing pattern (a /job/12345 slug, say), and you have a clean seed set. The search endpoint is a useful complement when a board hides its full listing behind a long filter chain you'd rather not reconstruct.

Step 2 — extract structured fields

Once you have listing URLs, pull the fields you actually want. The extract endpointtakes a URL, a prompt, and a JSON schema, and returns data shaped exactly to that schema — no post-processing, no fragile selectors:

curl -X POST https://fastcrawl.net/api/v1/extract/ -H "Authorization: Bearer YOUR_KEY" -H "Content-Type: application/json" -d '{"url":"https://hk.jobsdb.com/job/xyz","schema":{"type":"object","properties":{"title":{"type":"string"},"company":{"type":"string"},"location":{"type":"string"},"salary":{"type":"string"},"posted":{"type":"string"}},"required":["title","company"]}}'

The response is {"success": true, "url": "...", "data": {"title":"Senior Backend Engineer", "company":"...", ...}}. This is the step that separates a job aggregator from a pile of markdown. Because the schema is declared up front, a pricing or salary analysis downstream gets clean, comparable fields instead of text to parse.

Step 3 — run it efficiently

Per-URL calls add up fast on a board with thousands of listings. The batch scrape endpoint takes up to 10 URLs in one request, and with fetchMode: "http" you skip the browser render for boards that are already server-rendered. That combination cuts both latency and cost on the bulk pass.

Extraction is where the credit and token bill concentrates, so keep the fields you return tight. Every extra field is tokens. If you only need title, company, location, salary and posted date, ask for exactly those — the pattern in cutting your LLM token costapplies directly here. Re-extract only the listings that changed, not the whole board, every run.

Step 4 — keep it fresh with monitors

The highest-value job-board data is the new listings, not the archive. Rather than re-scraping the whole board on a timer, point a monitor at each search URL. It re-scrapes on schedule, detects changes, and fires a webhook when the content shifts:

curl -X POST https://fastcrawl.net/api/v1/monitors/ -H "Authorization: Bearer YOUR_KEY" -H "Content-Type: application/json" -d '{"url":"https://hk.jobsdb.com/jobs-in-information-technology","schedule":"daily","timezone":"Asia/Hong_Kong","hour":9,"webhook_url":"https://your.app/ingest"}'

The webhook only fires when the listing set changes, so you get new-post alerts without paying for a full re-scrape every hour. This is the change-detection pattern expanded inmonitoring websites with AI agents. Combined with the extract step, you have a pipeline that publishes a fresh, structured feed on demand.

Pitfalls that quietly eat credits

• Pagination loops. Infinite-scroll boards can hand you an ever-longer page list; cap depth and stop when a page returns no new listing IDs.
• Duplicate listings. The same job appears under several categories and repeats across re-scrapes. Dedupe on a stable key — the job ID in the URL slug, never the title.
• Blocked search endpoints. Some boards only wall the search route, not the individual job page. Scrape the listing pages and leave the search page to an occasional map or search call.
• Schema drift. When the board redesigns, your field mappings shift. Keep the schema in one place so a redesign is a one-file change, not a rewrite.

And respect the site. Identify your client, stay inside reasonable frequency, and honor robots.txt where these boards publish it. A board that blocks your IP is worse than a slower pipeline that keeps working.