Invoice and Receipt Extraction

In the Real World · Brief · Deep learning · Advanced · 2 to 3 days

400 supplier invoices a month, retyped by hand. 1,200 historic documents and their true field values. Build the extraction pipeline before you read the walkthrough.

The situation

Bracewell Ledger does the books for small businesses. Every month around 400 supplier invoices arrive as PDFs and somebody types five fields from each into the ledger: the invoice number, the date, the net, the VAT and the total.

You have 1,200 historic documents from 6 suppliers, already parsed into a text layer with a box around every token.

The data

What the room believes

1. Extraction is a layout problem, so reading the box positions will solve it.
2. A pretrained reading model will handle a new supplier better than rules.
3. The way to trust an extraction is a confidence score.
4. The bottleneck is accuracy.

Definition of done

1. A verdict on each of the four beliefs, with the evidence.
2. An extraction pipeline scored per field, and separately on whether the whole document is right.
3. An evaluation that says something about a supplier you have never seen, and a defence of how you arranged it.
4. A rule for which documents a person still has to look at, and a defence of the rule.
5. The cost of running it, and what has to happen when a new supplier signs up.

If you want to go further

• Hold one supplier out completely and compare that number to a random split.
• When your rules fail on the held out supplier, change one thing at a time until they work, and note which thing it was.
• Try a pretrained extractive reader and look at which fields it can and cannot find.
• Find a property every genuine invoice satisfies and use it to check your own output.
• Report field accuracy and whole document accuracy separately, and see how far apart they are.

In the Real World · Deep learning · Advanced · 2 to 3 days

Bookkeeping retypes 400 supplier invoices a month. Rules that read 74.5% of the fields correctly score 0.0% on a supplier they have not seen, and the reason is not the layout. One arithmetic check then catches every wrong number before it reaches the ledger.

Section 1Problem Definition

No code yet. Document extraction is sold as a model problem and is mostly a generalisation problem with an unusually good validator sitting next to it.

The problem in one sentence

Anything you build works on the suppliers you built it against, and the only question that matters is what happens when a new one arrives.

Business objective

Post as many invoices as possible without a person, never post a wrong number, and make adding a new supplier something the bookkeeper can do rather than the developer.

Hypotheses

1. H1. Extraction is a layout problem, so reading the box positions will solve it.
2. H2. A pretrained reading model will handle a new supplier better than rules.
3. H3. The way to trust an extraction is a confidence score.
4. H4. The bottleneck is accuracy.

Section 2Data Collection

import numpy as np
import pandas as pd

tk = pd.read_csv('data/bracewell-text-layer.csv')
tr = pd.read_csv('data/bracewell-truth.csv', dtype={'invoice_number': str})
BY = {k: g.sort_values('token_index').reset_index(drop=True)
      for k, g in tk.groupby('doc_id')}

print('documents      :', len(tr))
print('suppliers      :', tr['supplier'].nunique())
print('tokens         :', len(tk))
print('tokens per doc :', round(len(tk) / len(tr), 1))
print('line items     :', round(tr['line_items'].mean(), 1), 'on average')
print('a label missing:', round(100 * (tr['label_dropped'] != 'none').mean(), 1), 'percent')

documents      : 1200
suppliers      : 6
tokens         : 35737
tokens per doc : 29.8
line items     : 3.0 on average
a label missing: 27.1 percent

Section 3Data Preprocessing

3aDuplicates and schema checks

print('documents with no tokens :', int((~tr['doc_id'].isin(tk['doc_id'])).sum()))
print('duplicate doc ids        :', int(tr['doc_id'].duplicated().sum()))
print('boxes with zero width    :', int((tk['x1'] <= tk['x0']).sum()))
print('net plus vat equals total on every document :',
      bool(((tr['net'] + tr['vat'] - tr['total']).abs() < 0.005).all()))

documents with no tokens : 0
duplicate doc ids        : 0
boxes with zero width    : 0
net plus vat equals total on every document : True

That last line is the most valuable fact in the file and section 7d spends the whole of it on the consequence.

3bHandling categorical mess

print(tr['layout'].value_counts().sort_index().to_string())
print()
print('date formats :', dict(tr['invoice_date'].apply(
    lambda s: 'slash' if '/' in s else 'dash' if '-' in s else 'word').value_counts()))
print('currency prefixed totals :',
      round(100 * tr['currency_prefix'].mean(), 1), 'percent')

layout
A    200
B    200
C    200
D    200
E    200
F    200

date formats : {'slash': 433, 'dash': 392, 'word': 375}
currency prefixed totals : 16.6 percent

3cDealing with outliers

Line item counts run from one to five and do not affect the header or the totals block. What varies usefully is vertical position, because a longer invoice pushes the totals down the page, which is why an absolute coordinate rule would be a mistake and every rule below is relative to its label.

3dHandling missing values

27.1 of documents are missing one label entirely. Not the value, the label. That is what a supplier template does when somebody edits it, and it is the single largest source of failure for a label based rule.

3eHandling skewed data

Nothing skewed matters here. Amounts span a wide range and the extraction does not care what the number is, only where it is.

3fData types and normalisation

import re

LABELS = {
    'invoice_number': ['invoice no.', 'ref:', 'document number', 'invoice #', 'no.',
                       'our reference'],
    'invoice_date': ['invoice date', 'dated', 'issue date', 'date of issue', 'date',
                     'invoice dated'],
    'net': ['subtotal', 'goods', 'net amount', 'net', 'net total', 'net value'],
    'vat': ['vat', 'vat @ 20%', 'tax', 'vat amount', 'vat total', 'value added tax'],
    'total': ['total due', 'amount payable', 'gross', 'balance due', 'total',
              'total payable'],
}
# The list as it stood before layout F was opened. This is the honest starting point.
NEW_SUPPLIER_WORDS = ['our reference', 'invoice dated', 'net value', 'value added tax',
                      'total payable']
SEEN_LABELS = {f: [l for l in v if l not in NEW_SUPPLIER_WORDS]
               for f, v in LABELS.items()}
MONEY = re.compile(r'^(?:GBP\s*)?\d[\d,]*\.\d{2}$')
FIELDS = ['invoice_number', 'invoice_date', 'net', 'vat', 'total']

def flat(doc):
    g = BY[doc]
    return [(str(r.text), r.x0, r.y0, r.x1, r.y1) for r in g.itertuples()]

def norm(field, v):
    if v is None:
        return None
    v = str(v).strip().rstrip(':')
    if field in ('net', 'vat', 'total'):
        try:
            return '%.2f' % float(v.replace(',', '').replace('GBP', '').strip())
        except ValueError:
            return None
    return v

Every extracted amount is normalised before comparison: commas stripped, currency prefix removed, two decimal places. Skipping that step makes a correct extraction look wrong on the 16.6% of documents that write GBP in front of the number.

Section 4Exploratory Data Analysis

4aTarget variable analysis

Five fields per document. A document is only useful to the ledger if all five are right, which is a much harder bar than field accuracy and is reported separately throughout.

4bNumerical variables

The coordinates are the numeric variables and they carry most of the signal. A value sits either on the same line as its label and to the right of it, or directly underneath. Both are cheap to test and section 7b measures what that is worth.

# Does the value sit beside its label, or underneath it? It varies by supplier.
for lay, g in tr.groupby('layout'):
    same = below = 0
    for doc in g['doc_id'].head(40):
        t = flat(doc)
        for i, (w, x0, y0, x1, y1) in enumerate(t):
            if w.lower() not in LABELS['invoice_number']:
                continue
            for w2, a0, b0, a1, b1 in t:
                if abs(b0 - y0) < 4 and a0 > x1 - 1:
                    same += 1
                    break
                if abs(a0 - x0) < 8 and -22 < (b0 - y0) < -4:
                    below += 1
                    break
            break
    print('layout %s  value beside the label %3d   value under it %3d' % (lay, same, below))

layout A  value beside the label  28   value under it   7
layout B  value beside the label  32   value under it   3
layout C  value beside the label   0   value under it  36
layout D  value beside the label  31   value under it   6
layout E  value beside the label   0   value under it  34
layout F  value beside the label  34   value under it   3

Two arrangements, split unevenly across the suppliers. A rule that only handles one of them is right about half the time by construction, which is roughly what the token order rule achieves.

4cCategorical variables

The layout is the categorical variable that matters and it is exactly the one that will not be available for a new supplier. That is why layout F is held out completely rather than split at random.

4dRelationships between variables

labels = ['invoice no.', 'ref:', 'document number', 'invoice #', 'no.', 'our reference']
for lay, g in tr.groupby('layout'):
    doc = g['doc_id'].iloc[0]
    words = set(BY[doc]['text'].str.lower())
    print('layout %s  %-26s labels present: %s'
          % (lay, g['supplier'].iloc[0], sorted(words & set(labels)) or 'none of them'))

layout A  Ashcombe Timber Ltd        labels present: ['invoice no.']
layout B  Pellow Office Supplies     labels present: ['ref:']
layout C  Norbury Logistics          labels present: ['document number']
layout D  Kestrel Print              labels present: ['invoice #']
layout E  Hadley Catering            labels present: ['no.']
layout F  Wrenfield Cleaning         labels present: ['our reference']

Six suppliers, six different words for the same field. A rule that looks for Invoice No. finds nothing on the supplier that writes Our reference, and the geometry is irrelevant to that failure.

4eTesting our hypotheses

4fSubgroups

The subgroup is the supplier, and the whole design of this project is that one of them is never seen during development.

Section 5Feature Engineering

5aThe leakage trap

5bNew features

5cEncoding

No encoding. Everything here is text matching and coordinate arithmetic, and section 7b is partly an argument that this is the correct amount of machinery for the problem.

5dFeature selection

The only selection that matters is which label words are in the list, and that is the finding rather than a preprocessing step.

Section 6Model Selection

Section 7Model Training

def rules_v2(doc, labels):
    # Find the label, then take the value beside it or directly underneath.
    toks = flat(doc)
    out = {}
    for f, labs in labels.items():
        got, best = None, None
        for i, (t, x0, y0, x1, y1) in enumerate(toks):
            if t.lower() not in labs:
                continue
            for j, (t2, a0, b0, a1, b1) in enumerate(toks):
                if j == i:
                    continue
                same_line = abs(b0 - y0) < 4 and a0 > x1 - 1
                below = abs(a0 - x0) < 8 and -22 < (b0 - y0) < -4
                if not (same_line or below):
                    continue
                if f in ('net', 'vat', 'total') and not MONEY.match(t2):
                    continue
                if t2.lower() in labs:
                    continue
                dist = abs(a0 - x1) if same_line else abs(b0 - y0)
                if best is None or dist < best:
                    best, got = dist, t2
        out[f] = got
    return out

TRUTH = tr.set_index('doc_id').to_dict('index')
HELD_OUT = 'F'

def score(predict, labels, tag):
    hit = {f: {'seen': [0, 0], 'held': [0, 0]} for f in FIELDS}
    for doc in tr['doc_id']:
        p, t = predict(doc, labels), TRUTH[doc]
        key = 'held' if t['layout'] == HELD_OUT else 'seen'
        for f in FIELDS:
            want = ('%.2f' % float(t[f])) if f in ('net', 'vat', 'total') else str(t[f])
            hit[f][key][1] += 1
            hit[f][key][0] += int(norm(f, p.get(f)) == want)
    seen = np.mean([100 * hit[f]['seen'][0] / hit[f]['seen'][1] for f in FIELDS])
    held = np.mean([100 * hit[f]['held'][0] / hit[f]['held'][1] for f in FIELDS])
    print('%-46s %7.1f%% %9.1f%%' % (tag, seen, held))
    return hit

print('%-46s %8s %10s' % ('method', 'seen', 'new supplier'))
score(rules_v2, SEEN_LABELS, 'Rules using the box positions')
score(rules_v2, LABELS, 'Rules using the box positions, all labels known')

7aBaselines

Every method is scored twice: on the 1,000 documents from the five suppliers used during development, and on the 200 from the supplier that was never opened.

method                                             seen new supplier
Rules on the token order                          57.4%      20.0%
Rules using the box positions                     74.5%       0.0%
Rules using the box positions, all labels known    74.5%      62.6%
Extractive reader, no rules                       18.9%      22.2%

Reading the box positions is worth 17.1 points on the suppliers it was built for, and takes a new supplier to 0.0%. Not low. Zero.

Once the new supplier’s labels are known, four of the five fields land within a few points of the known suppliers. The invoice number is the exception at 10.5%, because it is the field whose label is dropped most often and whose value has no shape a regex can recognise.

7bComparing candidates

Read down the columns rather than across. No method wins every field. Token order is best on the invoice number and hopeless on the total; box positions are the reverse. The reader is the only thing that does well on the invoice number for a supplier it has never seen, at 85.5%, because it never depended on a label in the first place.

The reader scores 65.0% on the invoice number and 0.0% on every money field. Asked what the VAT amount is, it returns a span of a form that is almost never a bare number. It was trained on paragraphs of prose and an invoice is a table with the grid lines taken out.

from transformers import AutoTokenizer, AutoModelForQuestionAnswering
import torch

QUESTIONS = {'invoice_number': 'What is the invoice number?',
             'invoice_date': 'What is the invoice date?',
             'net': 'What is the net amount before tax?',
             'vat': 'What is the VAT amount?',
             'total': 'What is the total amount payable?'}

name = 'distilbert-base-cased-distilled-squad'
tok_ = AutoTokenizer.from_pretrained(name)
mdl = AutoModelForQuestionAnswering.from_pretrained(name).eval()
ctxs = {doc: ' '.join(t[0] for t in flat(doc)) for doc in tr['doc_id']}

def ask(question, docs):
    out = {}
    for i in range(0, len(docs), 32):
        batch = list(docs[i:i + 32])
        enc = tok_( * len(batch), [ctxs[b] for b in batch], truncation=True,
                   max_length=384, padding=True, return_tensors='pt')
        with torch.no_grad():
            o = mdl(**enc)
        for bi, b in enumerate(batch):
            s = int(o.start_logits[bi].argmax())
            e = max(int(o.end_logits[bi].argmax()), s)
            out[b] = tok_.decode(enc['input_ids'][bi][s:e + 1],
                                 skip_special_tokens=True).strip()
    return out

invoice number     65.0%         85.5%
date               29.6%         25.5%
net                 0.0%          0.0%
VAT                 0.0%          0.0%
total               0.0%          0.0%

7cHyperparameter tuning

There is nothing to tune and that is worth saying plainly. The coordinate tolerances were set from the layouts used in development and never adjusted against the held out supplier, because doing so would turn them into parameters fitted on the test set.

7dFinal evaluation

None of the numbers above are good enough to post to a ledger unattended. What makes the pipeline shippable is not the extraction, it is that this particular problem comes with a way to check its own answer.

def arithmetic_ok(p):
    try:
        net, vat, total = (float(p['net']), float(p['vat']), float(p['total']))
    except (TypeError, ValueError, KeyError):
        return False
    return abs(net + vat - total) < 0.02

documents where all three amounts are right : 57.2 percent
of the wrong ones, flagged by the check     : 100.0 percent
of the flagged ones, actually wrong         : 100.0 percent
sent for review                             : 42.8 percent
wrong amounts reaching the ledger           : 0.0 percent

preds = {doc: rules_v2(doc, LABELS) for doc in tr['doc_id']}
flagged = {doc: not arithmetic_ok({f: norm(f, p.get(f)) for f in FIELDS})
           for doc, p in preds.items()}
right = {doc: all(norm(f, preds[doc].get(f)) == '%.2f' % float(TRUTH[doc][f])
                  for f in ('net', 'vat', 'total')) for doc in tr['doc_id']}

wrong = [d_ for d_ in tr['doc_id'] if not right[d_]]
passed = [d_ for d_ in tr['doc_id'] if not flagged[d_]]
print('all three amounts right :', round(100 * np.mean(list(right.values())), 1), 'percent')
print('flagged for review      :',
      round(100 * np.mean(list(flagged.values())), 1), 'percent')
print('of the wrong ones, flagged :',
      round(100 * np.mean([flagged[d_] for d_ in wrong]), 1), 'percent')
print('wrong amounts among those that passed :',
      round(100 * np.mean([not right[d_] for d_ in passed]), 1), 'percent')

Net plus VAT equals total on every genuine invoice, so any extraction where it does not is wrong. That catches 100.0% of documents with a wrong amount and flags nothing that was right. 42.8% go to a person and the rest post with an error rate of 0.0.

At 400 documents a month, retyping all five fields takes 4.0 minutes each and costs 560.00 pounds a month. Reviewing a flagged document takes the same 4.0 minutes and checking a passed one takes 1.5, which comes to 359.80 pounds. A saving of 2,402 pounds a year, and the number that matters more to a bookkeeper is that nothing wrong gets posted.

Section 8Documentation and Handoff

What to do, and who owns it

What not to do

• Do not split documents at random. It puts every supplier in training and reports a number that will not survive contact with the seventh.
• Do not conclude a new layout needs a new model. Here it needed five strings.
• Do not trust a confidence score when a constraint is available. The arithmetic was right 100.0% of the time and needed no calibration.
• Do not use a prose reading model on a form. It returned 0.0% on every amount in the file.
• Do not normalise late. 16.6% of documents prefix the amount with GBP, and a correct extraction fails an exact comparison without it.
• Do not report field accuracy alone. A ledger needs all five right, and that rate is far lower than any single field.

Reproducibility

What to take from this

• Hold out a supplier, not a sample. The failure mode in production is a new source, and a random split cannot see it.
• Most layout failures are vocabulary failures. Check which before reaching for a model.
• A domain constraint is a better validator than a confidence score. It needs no calibration and it does not drift.
• Prose models are bad at forms. An invoice is a table with the lines removed and a reader trained on paragraphs has nothing to grip.
• Field accuracy is not document accuracy. Five fields at eighty per cent each is not a document you can post.
• The goal is not full automation. It is that nothing wrong reaches the ledger, and those need different designs.

Bookkeeping wanted the retyping to stop. What the extraction alone gives is 74.5% of fields on known suppliers and nothing anybody would post unattended. What makes it shippable is a line of arithmetic that every genuine invoice satisfies and no wrong extraction does, and a label list that a bookkeeper can extend in five minutes. Neither of those is a model, and together they are the project.