Overview
This study explores a pre-inference routing strategy for document field extraction systems. The core concept involves predicting a document's difficulty prior to extraction using inexpensive, document-based signals. This prediction then dictates whether a document is processed by a cheaper extraction model or a more robust, and typically more costly, alternative. The aim is to optimize cost efficiency without significant degradation in extraction quality.
Research Context
Traditional document-extraction systems commonly employ a single model for all documents. While straightforward, this approach can lead to inefficiencies: it may be overly expensive for documents that are easy to process and potentially less effective for documents presenting greater challenges. The research posits that by distinguishing document difficulty proactively, systems could adapt their extraction strategy, potentially yielding cost savings and maintaining, or even improving, performance for diverse document types.
Approach
The research investigates whether a document's processing difficulty can be predicted before the extraction phase. This prediction relies on readily available, document-based signals, which are designed to be inexpensive to acquire. The system then uses this prediction to route documents to one of two distinct extractors: a 'cheaper' model or a 'stronger' model. The study examined the efficacy of this routing mechanism across five document genres.
Two primary conditions were identified as critical for the effectiveness of pre-inference routing:
- The cheaper model must fail frequently enough to justify the overhead of routing.
- These failures must be predictable using visible document features, such as image quality and layout characteristics.
These conditions were translated into a practical test for evaluating the applicability of routing. The research also explored different routing mechanisms, including a simple bag-of-words router and one based on engineered features. Furthermore, the study involved a small labeled pilot to predict the suitability of routing for a given genre, and its accuracy was assessed. The robustness of the findings was tested across two different model pairs, characterized by cost differences of 5x and 3x, respectively.
Findings
The research indicates that pre-inference routing proves beneficial under specific circumstances. When both identified conditions are met – a sufficiently high failure rate of the cheaper model and predictable failures based on visible features – the calibrated router demonstrably reduces costs. Specifically, the system achieved cost reductions of 31-33% for receipts and 77% for degraded ad-buy forms. These reductions were observed while maintaining extraction quality within 0.02 F1 score of a system that consistently uses the larger, stronger model.
Conversely, the study found that routing does not provide an advantage if either of the two critical conditions is absent. Examples cited where routing was not beneficial include clean digital invoices and nutrition labels, which are generally considered easy to read and process without significant failures from simpler models.
A small labeled pilot was found to accurately predict whether routing would be effective. In the two instances where this pilot was conducted, its predictions were correct. The study also observed that a simple bag-of-words router performed comparably to routers designed with engineered features, suggesting that the primary determinant of routing effectiveness lies with the document genre itself rather than the specific router design. Interpretable features were employed to clarify which genres are amenable to routing.
A key limitation identified was that the router requires retraining for each new dataset; it does not transfer effectively across different datasets, even within the same document genre. The observed results were consistent across model pairs with varying cost differentials (5x and 3x).
Why This Matters
The findings suggest a method for optimizing the operational costs of document extraction systems. By selectively applying more expensive, robust models only when necessary, organizations could achieve substantial cost savings in processing large volumes of documents, particularly in industries handling diverse document types with varying levels of quality and complexity.
Key Limitations Mentioned by Researchers
- The router must be retrained for each dataset and does not transfer across datasets, even within the same genre.