Native PDF table extraction: evaluation method and limits

Start with a disclosed test set

The public worked example is one synthetic, two-page material report with five rows and 25 expected field values. It exercises decimal strings, a qualifier, a zero, a leading-zero identifier, a repeated header and an ambiguous comma. It is also a development fixture, so success on it is not independent validation.

Separate evaluation fixtures must be kept apart from that example and their provenance recorded. Natural supplier PDFs, scanned tables, arbitrary layouts and human correction effort need separate evidence. We do not infer those outcomes from a small synthetic set.

Measure fields, structure and evidence separately

Record enough to repeat the test

Retain the input hash, fixture license, expected output, parser version, settings, test timestamp and result artifact. Report the count of tested documents and fields alongside the result. Freeze expected output before evaluating a changed parser; do not modify ground truth simply to make a test pass.

For the public example, the provenance file records the fixture and published asset hashes, while the result JSON retains the native-table engine version and raw field states.

Executed synthetic evaluation: 2026-10-30

Five separate CC0 synthetic documents were run once each using Native engine native-table.v1. They are separate from the public material example. The test includes one supported layout and four negative cases, with zero natural documents.

Interpretation: the positive result applies to one two-page synthetic layout. Source reconstruction checks referenced words and their page; it is not independent geometric annotation. The wrapped-cell failure is a counterexample to assuming that completion means correctness. Negative checks passing do not mean their documents were successfully extracted.

Settings: fixed Sample / Test / Result / Unit / Method columns, local English OCR fallback on textless pages, no external model provider, one run per fixture, OS cache uncontrolled. The textless fixture contains drawn shapes and does not measure OCR recognition accuracy. The report records parser and binary hashes, platform, wall time and settings. Hosting cost, per-page cost and human correction time were not measured. These results are not a production throughput or real-world accuracy estimate.

Invoice line-item evaluation

Executed invoice checks: 2026-10-30

Twelve separate, owned CC0 synthetic invoice fixtures ran 3 times each (36 executions), with zero natural documents. Four supported layouts returned 57/57 exact field contracts, 2/2 explicit unit fields, and 57/57 fields within their authored source-cell bounds. Repeated executions are not additional independent examples.

Representative failures: the wrapped description loses its continuation. Right-aligned numbers that begin left of their headers are assigned to the wrong fields; the missing Amount warning does not identify every wrong field. A gap above 80 points ends the supported table region, so later rows can be absent. Rotated pages and unmatched headers return no rows. A textless second page remains unextracted even when page 1 succeeds.

The checks verify CSV/XLSX serialization against the extracted output separately from correctness. The harness cross-checks the sum of line amounts for supported fixtures; the product does not reconcile invoice totals. Source reconstruction alone does not prove the correct column was selected. Conservative table-region warnings can also appear for footer text, including supported examples. Passing a negative-case check means the disclosed behavior was observed, not that the invoice was extracted successfully.

Measured wall times, parser/binary hashes and settings are in the report. Local English OCR fallback was enabled for textless fixtures, which contain drawn shapes and do not measure OCR recognition accuracy. No external model provider, customer document, assessed human correction time or measured per-page infrastructure cost was used. These results do not establish a general accuracy percentage or production throughput.

Failures and evidence gaps are part of the result

The example’s density comma remains unresolved. Scanned PDFs need OCR, which this workflow does not provide. Merged cells, continuation tables without recognised headers and split rows are outside the advertised scope. Review is required even when a job completes.

No natural-document accuracy rate, calibrated confidence score, competitor superiority, customer outcome or human time saving is claimed here. A successful HTTP response or downloaded workbook would not establish any of those claims.