[ CIVIC_DATA_&_POLITICAL_RESEARCH ]

Multilingual Scanned-PDF Extraction Pipeline

2026Document IntelligenceIn-depth

What it is

Indian electoral roll PDFs — Hindi, Kannada, scanned or digital — turned into searchable English records.

Client work is shown without identifying imagery

[ THE_PROBLEM ]

Why this existed

A vast amount of important public and commercial data exists only as scanned PDFs in non-Latin scripts, with broken or absent text layers. Standard PDF text extraction returns garbage, and manual data entry does not scale to millions of pages.

[ WHAT_WE_BUILT ]

What we built

A pipeline with swappable extraction backends. The default runs OCR to per-page markdown (which recovers real text even when the PDF's embedded Unicode map is broken, as it is on Kannada rolls), then a structuring pass that emits voter records with English transliteration — देवनागरी names to Latin, Devanagari numerals to digits. A vision backend handles the messiest scans. The recommended hybrid keeps OCR for name accuracy but backfills the EPIC voter-ID boxes that OCR drops on scanned pages with a targeted vision pass, merged by reading order. Output lands in Postgres with trigram indexes for fuzzy search, behind an authenticated dashboard with regional analytics. Runs at roughly $1 per 1,000 pages.

Swappable extraction backends selected per document type
OCR to per-page markdown that recovers text even when the embedded Unicode map is broken
LLM structuring pass producing typed records with English transliteration
Numeral and name transliteration across scripts
Hybrid mode: OCR for names, targeted vision pass to backfill ID fields, merged by reading order
Postgres storage with trigram indexes for fuzzy search on misspelled names
Authenticated dashboard with regional analytics
CLI for batch processing with per-backend selection

[ HOW_IT_IS_USED ]

How a company uses it

Any organisation sitting on a large archive of scanned, non-English, badly-structured PDFs — land records, court filings, historical registers, regulatory submissions — that needs them queryable. The architecture is the deliverable; the document type is a swap.

Built with

Next.jsMistral OCRClaudeDrizzleNeon Postgrespg_trgm

[ COMMON_QUESTIONS ]

Questions clients ask

What does extraction cost per page?

Roughly one dollar per thousand pages on the default backend. That number is what makes bulk digitisation of an archive viable rather than a budget line somebody has to defend — and it is why backend choice is exposed as a flag rather than hard-coded.

Will this work on our document type?

Probably, and the architecture is the reusable part. The pipeline shape — OCR to markdown, then a structuring pass against a typed schema, then indexed storage — transfers to land records, court filings, invoices, historical registers and regulatory submissions. What changes is the schema and the accuracy tuning.

How accurate is it, and how do you know?

Accuracy varies by field and by scan quality, which is exactly why the pipeline has multiple backends and a hybrid mode. The pattern we settled on came from finding that OCR was strongest on regional-script names while a vision model was strongest on boxed ID fields — so the hybrid uses each where it wins. Any serious deployment needs a labelled sample to measure against before volume processing.

[ RELATED_WORK ]