Build a document extraction pipeline with a confidence gate.

Someone receives supplier documents as email attachments. They open each one, work out what it is, rename it, file it somewhere sensible and copy a handful of fields into a spreadsheet. It takes a few minutes per document and it happens every day.

Automating this is one of the most requested pieces of work an agency will be asked for, and one of the easiest to build badly. A pipeline that extracts fields with ninety percent accuracy and writes all of them to the database has not saved anybody time. It has created a data quality problem that someone will find months later.

The difference is a confidence gate: a deliberate decision, at a defined point, about which extractions are trusted and which are held for a person. This is how to build it.

Split classification from extraction.

The instinct is one call: here is a document, tell me what it is and pull out the fields. Resist it.

Two calls work better. The first decides the document type. The second extracts against the schema for that specific type.

This is not about accuracy alone, although shorter prompts do perform better. It is about being able to locate a failure. When a single call returns the wrong supplier name on a document it also misclassified, you cannot tell which decision went wrong. When classification is separate, you know immediately whether the pipeline picked the wrong schema or extracted badly against the right one.

It also means each document type gets its own schema, and an invoice schema and a delivery note schema have very little in common.

attachment
  -> classify        -> { type: "invoice" }
  -> load schema     -> invoice fields
  -> extract         -> { supplier, number, date, total, currency }
  -> confidence gate -> pass or hold
  -> file + record

Define the schema per document type.

Keep each one small and make every field explicit about its format.

{
  "type": "object",
  "required": ["supplier_name", "document_number", "issue_date", "total"],
  "additionalProperties": false,
  "properties": {
    "supplier_name": { "type": "string", "maxLength": 200 },
    "document_number": { "type": "string", "maxLength": 60 },
    "issue_date": { "type": "string", "description": "ISO 8601 date" },
    "total": { "type": "number" },
    "currency": { "type": "string", "enum": ["EUR", "GBP", "USD"] },
    "fields_found": {
      "type": "array",
      "items": { "type": "string" }
    }
  }
}

Two of those deserve explanation.

Dates and numbers must be normalised at extraction. A document showing 05/03/2026 is ambiguous, and a total written as 1.234,56 means something different depending on the locale. Ask for ISO dates and plain decimal numbers, and state the expectation in the prompt. Do not leave this for a later step to guess at.

fields_found is the honest signal. Rather than asking the model how confident it is, ask which fields it actually located in the document. A field that is present in the output but absent from fields_found was inferred rather than read, and that distinction is far more reliable than a self-reported score.

Build the gate from things you can verify.

The temptation is to ask the model for a confidence number and threshold on it. Model confidence is not calibrated and a 0.9 does not mean what it appears to mean.

Build the gate from checks you can actually evaluate.

Check	Holds the document when
Required fields present	Any required field is missing or empty
Reported as found	A required field is absent from fields_found
Format valid	The date does not parse, or the total is not a number
Range plausible	The total is negative, or far outside normal for this supplier
Arithmetic consistent	Line items are extracted and do not sum to the total
Duplicate	Supplier and document number already exist in the record store
Type uncertain	Classification returned other or unknown

The arithmetic check is the strongest single signal available on financial documents, and it is free. If you extract the line items and the total, they either reconcile or they do not, and a mismatch means something was misread.

holds = []
if missing_required(fields):        holds.add("missing field")
if not all_reported_found(fields):  holds.add("inferred value")
if not parses_iso(fields.date):     holds.add("bad date")
if line_items and abs(sum(line_items) - fields.total) > 0.01:
                                    holds.add("total mismatch")
if exists(fields.supplier, fields.number):
                                    holds.add("duplicate")

if holds: review_queue(document, fields, holds)
else:     commit(document, fields)

Never modify the source file.

Store the original attachment untouched, and store the extracted record alongside it with a reference back.

This matters more than it seems. Extraction improves. Prompts get better, models change, a schema gains a field nobody needed at first. If the originals are intact you can re-run extraction across the archive and compare. If the pipeline renamed, converted or discarded them, that option is gone permanently.

It also settles disputes. When somebody questions a figure, the answer is the original document, not a reconstruction of what the pipeline believed.

The review queue is a feature, not a failure.

Everything the gate holds goes to a queue with three things attached: the source document, the extracted values, and the specific reasons it was held.

Make it fast

The reviewer should see the document and the proposed values together, correct a field and approve in a few seconds. A queue that requires opening two systems will not be used, and an unused queue means the held documents simply never get processed.

Record corrections

When a reviewer changes a value, store what it was and what it became. After a few weeks this is the most useful dataset you have. It tells you exactly which field on which document type is being extracted badly, and that is where prompt work should go.

Watch the rate

Track the proportion of documents held. A rate that is stable and low means the pipeline is working. A rate that climbs usually means a supplier changed their template, and that is worth knowing before somebody notices the backlog.

The review queue makes the error rate visible instead of hiding it inside a spreadsheet. A pipeline with a twelve percent hold rate and a working queue is far better than one with a claimed two percent error rate that nobody can verify.

Handle the documents that are not documents.

Real inboxes contain things your pipeline was not designed for, and each needs a decided outcome rather than an accident.

Scanned images rather than text need an OCR step first, and quality varies enough that a poor scan should hold rather than produce plausible nonsense. Multi-page attachments may contain several distinct documents, so decide whether to split them or treat them as one. Password-protected files cannot be read and should be flagged rather than retried forever. Signature images, logos and email footers arrive as attachments constantly and should be filtered before extraction so you are not paying to classify a company logo.

Test with the awkward archive, not with three clean examples.

Ask the client for a sample of what actually arrived last month, including the ones that caused problems. Then test in this order.

Start with documents whose correct values you already know, so you can measure rather than judge. Include at least one from every supplier, because templates differ far more than expected. Add the deliberately hard cases: a poor scan, a credit note with negative values, a foreign currency, a document with no line items, one in another language.

Measure two things separately. Extraction accuracy on the documents that passed the gate is what the client actually experiences. Hold rate is what determines whether the pipeline saves them any time. A pipeline that holds everything is accurate and useless.

What to hand over.

The schema for each document type, so a future developer knows what is expected. The gate rules, written as rules rather than as code comments. Where the originals are stored and how a record links back to one. How to work the review queue. And the current hold rate, so a change in it is visible as a change.

If a client is asking for document processing and you want the confidence gate designed before the extraction is built, send us a description of the documents and what has to come out of them. We will tell you which checks would carry the weight. Send us the details.