Keep development examples separate from validation evidence

Development examples teach the workflow

Keep validation examples separate so you can assess whether the workflow works beyond the cases used to build it.

In ParseMill, select an extracted invoice field to inspect its source, correct the value and export the reviewed workbook. Try your own invoices as well as the interactive sample. Compare upload, checking and correction time with your current way of building the same spreadsheet.

The separation applies to the whole workflow, including manual post-processing and acceptance thresholds. A held-out document is no longer untouched if its results repeatedly guide tuning, even when the underlying extraction model was never retrained.

Use a document split manifest

These counts are illustrative, not a universal sample-size recommendation. Choose a design that represents your actual source classes and decision needs. Keep closely related revisions or duplicate pages together where splitting them would leak nearly identical content across sets.

Google: Dividing datasets distinguishes training, validation and test sets and explains how repeated use can wear out evaluation data. Apply that principle to your extraction workflow’s development history.

Protect the final comparison

1. Inventory source classes, layouts and quality levels before assigning documents to sets. Record source identities and any relationships between documents.
2. Use development examples to create rules and expected outcomes. Keep final-evaluation source and reference files separate from routine tuning.
3. Freeze the candidate workflow and comparison rules before running the final set. Record failures as well as successful documents.
4. If final-set results lead to changes, label that set as development evidence for the next iteration and obtain fresh evaluation examples when needed.

W3C: Data on the Web Best Practices recommends explicit data versions. Version the split manifest and expected references so a reported result can be tied to the exact evaluation set.

Keep regression tests in their proper role

A regression fixture proves that a known case still behaves as expected. It does not establish general performance on unseen documents. Report regression pass counts separately from held-out evaluation results, even when both are useful for release decisions.

Avoid random page splitting that places near-identical pages from one report in both development and test sets. Consider the unit of independence relevant to your workflow, such as issuer template, document family or source report.

State what the evaluation supports

The fixture’s final evaluation should describe the selected workflow’s behavior on T01–T08 under the recorded conditions. It should not claim universal accuracy or hide the fact that the development set received repeated attention. Retain the failure examples for later diagnosis.

When evaluating ParseMill, reserve representative documents that have not shaped your setup choices. Combine that evidence with correction effort and source coverage to decide whether the workflow suits your incoming document stream.