What you can do today

  • Create explicit expected values and expected-missing states for every field in an evaluation schema.
  • Run development, held-out and shadow comparisons in Optimize with source revision checks.
  • Inspect field correctness, unavailable evidence, OCR warnings and evaluation history.
  • Review drift counters over recent annotated evaluations and retain Optimize promotion/rollback decisions.
  • Correct & learn records confirmation and correction reward signals, then compares baseline and candidate field accuracy, extraction coverage, improvements and regressions on held-out annotations.
  • Inspect document-by-document outcomes before an owner accepts a feedback candidate. Held-out data cannot supply proposed label changes or qualify another evaluated candidate within that program.
  • Revalidate source revisions and approved organization context; stale feedback, rules and evaluation details cannot silently authorize reuse.

Check whether an extraction change helps

Record the expected order number and amount on separate documents. Run two rule versions, inspect incorrect and missing fields, and retain the evidence supporting a promotion decision.

Supported scope and limits

  • Monitoring covers recorded annotated evaluations. Continuous production drift detection and scheduled sampling are not available.
  • The current evaluator runs deterministic named-field extraction, not provider-backed models or arbitrary document workflows. Reward signals do not mean model training or online reinforcement learning is running.
  • Metrics describe the annotated sample and reviewer-declared document families. Synthetic examples and a small passing cohort do not establish accuracy on customer documents.
  • Elapsed execution time is recorded; infrastructure cost, business time savings and reviewer labor are not automatically measured.
  • Feedback learning uses at most 100 approved knowledge records for organization context; it does not call model providers, build embeddings or run GraphRAG.

Planned extensions

  • Task-qualified model/provider comparison and complete accepted-outcome cost.
  • Representative customer cohorts with independently reviewed annotations.
  • Layout-change detection and sampled production review.
  • Workspace alerts and release gates based on declared workflow-specific tolerances.

How to check the result

  • Declare expected values before running the candidate.
  • Count unsupported inputs and failed cases rather than removing them from the result.
  • Test missing fields, conflicting labels and changed or deleted sources.
  • Keep natural and synthetic evaluation evidence separate.