Extracting structured data from PDFs at scale

Despite a decade of "PDF is dead" predictions, PDFs still hold most of the long-form data on the web: filings, research papers, government reports, datasheets, invoices, contracts, and the long tail of legacy enterprise documents. The web changed formats; the data didn't. A serious scraping stack needs a real answer for PDFs, not a pdftotext escape hatch.

This is a working pattern for turning messy, multi-page PDFs into clean structured JSON at scale with a single API call, plus the failure modes that quietly wreck naive pipelines.

Why PDFs break the obvious approach

Plain pdftotext works for one clean, text-layer PDF. It falls over on the cases that matter: scanned reports where every page is an image, multi-column research papers where reading order gets jumbled, tables that collapse into whitespace, and forms whose fields are layered over a background image. You find out about the failure when an empty string shows up in your database six hours later.

The honest answer is that there is no single extractor that handles every PDF well. The job is to pick a tier based on what the document actually is, and to keep that tier decision out of your application code. The same principle that drives HTTP-only vs browser rendering applies here: classify the document, then choose the cheapest tool that will return clean data.

The three tiers of PDF extraction

Tier 1: text layer present. The PDF was generated from a Word doc, a LaTeX source, or a report writer. Extracting per-page text is fast and cheap. pdftotext or a lightweight library does the job in milliseconds. Most invoices, filings, and modern research papers live here.

Tier 2: scanned image. The PDF is a stack of page images with no text layer. You need OCR. Tesseract handles clean scans; cloud OCR handles noisy ones, handwriting, or non-Latin scripts. This tier is 10-50x slower and 5-20x more expensive than Tier 1, and the quality is only as good as the scan.

Tier 3: complex layout. Multi-column papers, nested tables, mixed figures and footnotes, forms with field annotations. Pure OCR loses the structure. This tier wants an LLM that reads the document visually — render the page to an image, hand it to a model with a schema, and get structured fields back. It is the slowest and most expensive tier, and the only one that reliably returns usable data on the awkward 20% of documents.

A pipeline that picks the wrong tier pays for it in latency, cost, or garbage data. A pipeline that has to know in advance is fragile, because the same URL can serve a text-layer report one quarter and a scanned one the next.

One endpoint, all three tiers

The parse endpoint takes a PDF URL and returns clean markdown by default, with structured JSON on request. Internally it classifies the document and routes to the right tier, so a single call handles the easy case and the gnarly one without you writing a router. It also renders the page visually for Tier 3, so the model that reads it sees the same document a human would:

curl -X POST https://fastcrawl.net/api/v1/parse/ \
  -H "Authorization: Bearer ***" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://www.gov.hk/about/report.pdf","formats":["markdown","json"]}'

The response is page-by-page markdown with headings, paragraphs, and tables preserved, plus ajson field with title, headings, paragraphs,links, images, and wordCount. The token cost is the same as a scraped HTML page, because the markdown is what an LLM would see anyway — the saving is the boilerplate and footer noise you skip, which is the lever incutting your LLM token bill.

Pulling structured fields, not just text

Markdown is the start, not the end. For most PDF pipelines the goal is specific fields: invoice totals, line items, contract parties, citation metadata, or the named entities in a report. Theextract endpoint accepts a PDF URL, a schema, and a prompt, and returns the schema filled in:

curl -X POST https://fastcrawl.net/api/v1/extract/ \
  -H "Authorization: Bearer ***" \
  -H "Content-Type: application/json" \
  -d '{
    "url":"https://www.gov.hk/about/report.pdf",
    "schema":{
      "type":"object",
      "properties":{
        "reportTitle":{"type":"string"},
        "publishDate":{"type":"string"},
        "keyFigures":{"type":"array","items":{"type":"object","properties":{"label":{"type":"string"},"value":{"type":"string"}}}},
        "recommendations":{"type":"array","items":{"type":"string"}}
      },
      "required":["reportTitle","publishDate"]
    }
  }'

The schema stays in your code, not buried in a prompt, so a new field is a one-line change. The same call works on a clean text-layer filing and a scanned report; the endpoint handles the routing.

Patterns that scale beyond one document

One PDF is a script. A thousand is a pipeline. Three patterns get you there:

Batch by URL. The batch scrape endpoint accepts up to 10 URLs in a single call. Group your PDFs by source so a single batch covers a coherent set, and a failure on one URL doesn't poison the others.

Cache by hash. PDFs change rarely but get re-requested often, especially regulatory filings. Hash the URL plus the Last-Modified header and store the extracted JSON. Re-extract only when the hash changes. This is the same logic asmonitoring websites with AI agents, applied to documents.

Re-extract only what changed. If a filing gets amended, you usually only care about the new pages. A monitor pointed at the PDF URL detects the change and the pipeline re-parses; your code never re-fetches the unchanged 80 pages.

Failure modes worth planning for

The corner cases are where naive pipelines die:

• Password-protected PDFs. A handful of filings are encrypted. Detectencrypted: true in the response and skip with a clear log, rather than retrying forever.
• Image-only pages mid-document. A 50-page report with 48 text pages and 2 scanned appendices still needs OCR on those two. Tiered routing handles this if your endpoint does it; manual code rarely does.
• Tables that span pages. Row 20 of a table on page 7 plus row 21 on page 8 is the same record. Treat page boundaries as a render concern, not a data boundary, in your schema.
• Huge files. 500-page filings can blow context windows. Cap page count and process the rest separately, or summarize the back half with a cheaper model.

Plan for these up front and PDF extraction is a normal pipeline. Skip them and it is a month-long source of bugs.