Multi-modal document parsing: PDFs to structured fields, with honest accuracy.

Multi-modal document understanding turns a PDF - prospectus, KID, factsheet, loan tape, counterparty agreement - into structured fields. The field has moved through three architectural eras, and the credible vendor names which one they are using and why.

Era 1, pre-2022, was OCR plus rules: Tesseract or Azure Form Recognizer extract text and table cells, Camelot or pdfplumber lift tables in lattice or stream mode, regex and template logic project the result into a schema. Brittle on rotated tables, multi-column layouts, charts, signatures, and scanned forms. The mechanism is fully deterministic - which is its virtue, but the failure modes are sharp and they recur in every real document set.

Era 2, 2022 to 2024, was layout-aware transformers. LayoutLMv3 jointly embeds text tokens, 2D layout positions, and image patches in a single pre-trained model. Donut is OCR-free end-to-end: a Swin Transformer encoder reads the document pixels, a BART decoder emits a structured token sequence directly. Nougat extends the recipe to academic PDFs with LaTeX-correct math. These models trade interpretability for tolerance of the messy reality of real documents.

Era 3, 2024 onward, is generalist Vision-Language Models. The frontier models read the document image directly and produce structured output through the same prompt-engineering surface as a text-only LLM. The architectural pattern - aligned image-text encoders feeding a transformer decoder - inherits the Flamingo and LLaVA recipe. Chart understanding lives in this era: the model reasons over chart pixels directly rather than going through a brittle chart-to-table OCR step.

All three eras still ship in production. The choice is not 'use the newest one'; the choice is which combination matches your document classes, your accuracy target, your latency budget, and your residency constraint.

Every PM, fund administrator, and ops lead reading this has spent years hand-keying fields from PDF documents into internal systems. ISIN from page one of the KID, SRRI box on page two, OCF from the costs table, PRIIP category from the risk section, charges narrative pasted into the suitability folder. The pain is not abstract.

The buyer-facing outcome is field-level recall on the documents that drive your regulatory reporting: PRIIPs KIDs into your distributor packs, prospectus key terms into the suitability matrix, factsheet performance and risk numbers into the marketing approval flow, AIFMD Annex IV holdings look-through into the depositary reconciliation. Concretely, for a generic input like `prospectus_ABC_2024.pdf` with issuer reference `ISSUER:DE000-ABC-...`, we are producing a typed JSON object with ISIN, SRRI, OCF, PRIIP category, performance-fee terms, and the charges narrative ready for downstream pipelines - while logging the source location of every extracted field for audit.

PRIIPs RTS Annex II specifies the exact fields a KID must carry. MiFID II Article 24 ties costs disclosure to suitability and pre-contractual information. SFDR Articles 8 and 9 define the sustainability classifications whose underpinnings must be traceable to source documents. AIFMD Annex IV reporting requires look-through to underlying holdings - which in turn requires extraction from the documents that describe those holdings.

The regulator's interest is not the parsing - it is the chain of evidence from source document to reported field. That is why field-level provenance, not document-level pass rate, is the metric that matters.

We run the extraction stack inside the client's tenancy on local or private-region inference. The documents and the structured outputs do not leave the firm; DORA Art. 28 to 30 third-party ICT risk is minimised because the model, the prompts, and the extracted fields all sit inside the client's resilience perimeter. The audit trail is created by construction - extraction is not a black box bolted onto the document warehouse, it is a logged step inside it.

We pick the model by document class. Clean, structured filings go to a layout-aware transformer because it is fast, cheap, and interpretable. Mixed-quality scans and chart-heavy material go to a generalist VLM because it tolerates the noise. Era-1 OCR plus rules stays in the stack for the cleanest, highest-volume cases where regex on extracted text is still the right tool.

Accuracy is reported per field, not per document. On a representative regulated-document set, the field-level recall typically looks like this: ISIN 99.8, SRRI 97.1, OCF 94.4, charges narrative 78.6. The narrative field is harder because it is unstructured prose and the ground truth is itself debatable. We show those four numbers explicitly in the validation report. Vendors who quote a single 99 percent figure are either measuring document-level pass on a clean test set or hiding the long tail.

Every extracted field carries provenance: page number, bounding box, model id, prompt version, confidence. The downstream pipeline can re-derive the extraction and the auditor can trace any reported field back to its pixel region. This is what 'audit trail by construction' means in practice - the metadata is not added after the fact, it is the output of the extraction step.

Inference runs on the client's GPU pool or in a private-region tenancy with a documented data flow. No document leaves the firm's perimeter. DORA Articles 28 to 30 third-party ICT risk reduces to managing one vendor (the model weights and runtime), not a chain.

Human-in-the-loop is the default for any field with confidence below the per-field threshold we calibrate against the client's tolerance. The operator reviews a queue of low-confidence extractions; their corrections feed the validation set for the next model release. The workflow is mechanised, not automated end-to-end - the distinction matters to the audit committee.

Do not run a 'drop your PDF here, get JSON' hero animation. It hides every failure mode the buyer has lived with for twenty years and instantly loses credibility.

Do not quote '99 percent accuracy' without a per-field confusion matrix. The senior PM reading the slide knows that the charges narrative is harder than the ISIN and will discount the entire page if both are reported as the same number.

Do not show a pristine demo factsheet that looks nothing like the rotated, multi-column, half-scanned reality of a real fund's marketing pack. The credible demo is on a document the buyer recognises as theirs.

Do not put a brain or a magic wand in the diagram. Show the document, then the patches, then the structured output. The buyer has been hand-keying these fields for decades; the credible promise is mechanisation, not magic.

Do not send fund documents through a third-party multi-modal API without a DORA Art. 28 register entry and a documented data residency contract. The documents themselves are the regulated artefact.