OCR + LLM document pipelines: measured accuracy or it didn't happen

Documents nobody designs for, skewed scans, faxes of faxes, handwriting in margins, still run large parts of regulated industries. I build pipelines that turn that mess into structured records: OCR for the pixels, an LLM for the judgment, and a labeled eval set standing between every change and production. The last part is the actual product.

Division of labor matters. OCR engines are superb at glyphs and hopeless at meaning; LLMs are superb at meaning and expensive at glyphs. Feed the LLM raw OCR output plus the layout signal, and ask it for typed, schema-validated fields, not prose. Structured output with validation converts 'the model usually gets it' into 'the field is present, typed, and checkable.'

The eval discipline: a labeled set of real documents (the ugly ones, oversampled), field-level accuracy tracked per field type, and deploys gated on no-regression. Field-level is the honest granularity, document-level accuracy hides the one critical field you keep missing. When accuracy is measured in the mid-to-high 90s, the remaining errors cluster exactly where a human reviewer should be looking; the eval tells you where to put the human.

The trap to avoid: demo-driven extraction. Every document pipeline looks magical on clean samples. The gap between a demo and a system is a few hundred labeled nightmares and the willingness to be graded by them, forever.