What the run returned
| Measure | Expected | Observed |
|---|---|---|
| Routine cases | 15 | 15 |
| Cases requiring review | 30 | 30 |
| Processor errors on first run | 0 | 0 |
| Exact case results | 45 | 45 |
| Exceptions incorrectly routed as routine | 0 | 0 |
| Accepted documents with the wrong client / entity | 0 / 0 | 0 / 0 |
0 cases differed from the expected result. Each case ran 3 measured times. Route totals use the first result; exact-match and unsafe-acceptance checks cover every repetition. An exact match checks the entire output: document identity, type, period, accepted/duplicate/exception status, missing requirements and reason codes. An expected exception passing the test means it was correctly held for review.
Matching, missing documents and exceptions
The configured checklist asks which document types are present for a client, entity and month. A valid attachment can be accepted while the package still needs another document.
| Scenario | Observed associations and missing documents | Case result |
|---|---|---|
| Exact registered client and entity; complete three-type package intake-01 |
Missing: none on the configured checklist. | routine Matches expected |
| Duplicate attachment in same message, renamed intake-09 |
Missing: none on the configured checklist. | routine Matches expected |
| One required receipt missing despite valid bank and payroll intake-12 |
Missing: receipt (2026-08). | exception Matches expected |
| Known sender is not authorized for claimed client intake-17 |
Missing: bank statement (2026-08). | exception Matches expected |
| Entity alias matches two entities intake-18 |
Missing: bank statement (2026-08). | exception Matches expected |
| Changed receipt without explicit correction link is held intake-32 |
Missing: none on the configured checklist. | exception Matches expected |
| Textless scanned PDF envelope requires review intake-38 |
Missing: bank statement (2026-08). | exception Matches expected |
| Text digest mismatch prevents filing intake-40 |
Missing: bank statement (2026-08). | exception Matches expected |
View all 45 case results
| Case | Expected route | First observed route | Full output check |
|---|---|---|---|
| intake-01 · Exact registered client and entity; complete three-type package | routine | routine | Exact match |
| intake-02 · Registered client and entity aliases | routine | routine | Exact match |
| intake-03 · Case and whitespace normalization | routine | routine | Exact match |
| intake-04 · Canonical identifiers are exact lookup values | routine | routine | Exact match |
| intake-05 · Single authorized client and single entity fallback | routine | routine | Exact match |
| intake-06 · Single authorized client fallback with explicit entity | routine | routine | Exact match |
| intake-07 · Document type registered alias | routine | routine | Exact match |
| intake-08 · Filename and email subject cannot override explicit document facts | routine | routine | Exact match |
| intake-09 · Duplicate attachment in same message, renamed | routine | routine | Exact match |
| intake-10 · Already received exact content is duplicate | routine | routine | Exact match |
| intake-11 · New attachment completes package alongside trusted prior document | routine | routine | Exact match |
| intake-12 · One required receipt missing despite valid bank and payroll | exception | exception | Exact match |
| intake-13 · Empty message leaves all requirements missing | exception | exception | Exact match |
| intake-14 · Missing period is not inferred from subject or filename | exception | exception | Exact match |
| intake-15 · Invalid numeric month | exception | exception | Exact match |
| intake-16 · Unknown sender cannot self-assign a registered client | exception | exception | Exact match |
| intake-17 · Known sender is not authorized for claimed client | exception | exception | Exact match |
| intake-18 · Entity alias matches two entities | exception | exception | Exact match |
| intake-19 · Missing entity for multi-entity client | exception | exception | Exact match |
| intake-20 · Unknown entity does not fuzzy-match | exception | exception | Exact match |
| intake-21 · Unknown client does not fall back to known sender | exception | exception | Exact match |
| intake-22 · Authorized adviser submits two independently identified clients | routine | routine | Exact match |
| intake-23 · Adviser with two client scopes cannot omit client | exception | exception | Exact match |
| intake-24 · One bad item preserves valid item without claiming complete package | exception | exception | Exact match |
| intake-25 · Conflicting duplicate client headers are never last-wins | exception | exception | Exact match |
| intake-26 · Unsupported type tax return is escalated | exception | exception | Exact match |
| intake-27 · Missing document type is not guessed from filename | exception | exception | Exact match |
| intake-28 · Known but unrequested month is not silently filed | exception | exception | Exact match |
| intake-29 · Plain body text is not an attachment or a completeness fact | exception | exception | Exact match |
| intake-30 · Late receipt explicitly belongs to prior month | routine | routine | Exact match |
| intake-31 · Explicit later corrected receipt links to prior version | routine | routine | Exact match |
| intake-32 · Changed receipt without explicit correction link is held | exception | exception | Exact match |
| intake-33 · Correction target does not exist | exception | exception | Exact match |
| intake-34 · Correction cannot cross entity boundary | exception | exception | Exact match |
| intake-35 · Non-increasing correction revision is held | exception | exception | Exact match |
| intake-36 · Injection prose after header cannot change client or checklist | exception | exception | Exact match |
| intake-37 · Email-body injection cannot waive absent receipt | exception | exception | Exact match |
| intake-38 · Textless scanned PDF envelope requires review | exception | exception | Exact match |
| intake-39 · Unsupported archive does not imply successful extraction | exception | exception | Exact match |
| intake-40 · Text digest mismatch prevents filing | exception | exception | Exact match |
| intake-41 · Annual tax period is outside monthly contract | exception | exception | Exact match |
| intake-42 · Same filename with distinct document content is not duplicate | routine | routine | Exact match |
| intake-43 · Receipt without stable reference stays an exception | exception | exception | Exact match |
| intake-44 · Unsupported prose-only attachment cannot invent an entity | exception | exception | Exact match |
| intake-45 · Malformed correction revision is held | exception | exception | Exact match |
How the benchmark works
- Define the expected result. The fixture supplies a synthetic message, attachment text, an approved client/contact/entity catalog, requested monthly packages and prior documents. Expected outputs are authored separately from the processor.
- Run the intake rules. The processor receives the input, catalog and policies—not case IDs, titles or expected results. It validates text hashes, resolves registered identities and aliases, checks explicit periods, detects duplicates and corrections, and lists missing document types.
- Compare every output. The runner records the complete expected and actual result for every case. Source hashes bind the report to the exact fixture, processor and runner; the page build refuses changed inputs or inconsistent totals.
The test set includes ordinary complete packages, incomplete submissions, ambiguous entities, conflicting versions, unsupported or unreadable inputs and instructions embedded in text. Its deliberately chosen mix of 15 routine and 30 exception cases is not an estimate of a firm's exception rate.
Recognized facts use explicit headers before the document narrative. Arbitrary PDFs, natural-language extraction, live connector reliability, accounting validity and human handling time require separate measurements. A sender match is a routing hint, not identity verification.
Inspect the evidence: download every test input, expected output and measured result below. The report identifies the exact processor and runner versions by their source hashes.
Your workflow, measured
Find the work worth automating.
A workflow audit maps your intake queue, the information it needs and the decisions your team should keep. Start with the documents your team spends the most time matching and chasing.