Compare extraction runs without changing the test inputs

Freeze the inputs to a fair comparison

Keep the same documents and acceptance checks so an apparent improvement reflects the workflow rather than an easier test set.

In ParseMill, select an extracted invoice field to inspect its source, correct the value and export the reviewed workbook. Try your own invoices as well as the interactive sample. Compare upload, checking and correction time with your current way of building the same spreadsheet.

Use a content hash or another immutable source identifier, not just a filename. Two files named report.pdf can contain different revisions. Likewise, name the expected-output version and comparison rules used to evaluate each run.

Use a run manifest

H1 and C1 are illustrative labels, not actual hashes or configurations. Replace them with real identifiers in an executed comparison. This fixture isolates a tool-version change; a settings experiment would instead hold the tool version fixed and record the one setting changed.

W3C: Data on the Web Best Practices supports explicit version indicators and history. Apply that discipline to sources, references and outputs, not only to the final spreadsheet.

Run a controlled difference review

1. Save the source and expected reference before the experiment. Record hashes, tool version, configuration and the intended comparison.
2. Execute both runs under the documented conditions and preserve their unedited outputs. Label any failed inputs rather than removing them from the comparison.
3. Compare keyed records for added, removed and changed fields. Review each difference against the frozen source and expected reference.
4. Summarize improvements, regressions and unchanged failures separately. Record manual correction effort only as a separate follow-on measure.

Google: Dividing datasets explains the role of separate evaluation data. If repeated comparisons guide tuning, keep a fresh held-out set for the final assessment of the chosen configuration.

Handle unavoidable changes honestly

A service may update a model behind a stable interface or use nondeterministic processing. Record the information available and repeat runs when needed to characterize variation. Do not claim a single-factor experiment if the execution conditions cannot be held constant.

If the expected reference is corrected during review, rescore both outputs against the new reference. Otherwise, a reference improvement can look like a parser improvement or regression. Preserve the earlier scores with their original reference version. Add a short explanation of the reference correction and identify every score affected by it.

Make the conclusion reproducible

For the fixture, differences can be attributed to the recorded version change only within the controlled conditions and document set. A useful conclusion identifies the affected fields and source classes, not merely a larger overall score. Save the manifest with the difference report.

When comparing ParseMill workflows, use the same source package and acceptance rules. Review regressions as well as improvements before choosing the configuration that best supports your register task.