Measuring success in intelligent document processing

Straight through processing (STP) refers to transactions that pass straight through a system from start to finish without any manual intervention. The measure is an important indicator for predicting the long-term value generation of an automation solution.

In the long run, the majority of return on investment (ROI) from an automation exercise accrues from reduction in human effort per unit of workflow and reduction in operational errors and risk incidences. One of the clearest measures of human effort reduction is the percentage of workflow that goes through error-free, without the need for any human intervention or oversight, which we term percent STP (%STP).

However, in most implementations, a high %STP is hard to achieve, owing to a wide range of reasons. To paraphrase Tolstoy, all successful automation instances are alike, but every failed automation instance fails in its own way.

We commence the journey toward improving %STP with a simple framework around comprehensively mapping instances in which automation might break down. Developing an intuition for this is key to improving %STP.

Let’s start with a 10,000-foot view of the possible scenarios that arise when a document is processed by an automation system. The document could be processed by the automation system correctly or incorrectly; further, the system can diagnose its processing as a “success” and send results downstream, or as a “failure” and escalate to a human subject matter expert. A combination of these gives us four possible, mutually exclusive scenarios, as described in the table below.

On the basis of this classification, we can now define %STP as the percentage of instances resulting in a “Correct Pass” scenario. Let’s explore each scenario.

Correct Pass and Correct Escalation

A Correct Pass is the ideal state; this describes a scenario in which a document has been successfully categorized, information has been extracted and validated, and the data has been persisted or exported. A Correct Escalation means the document did not correctly complete one or more categorization, extraction, validation or export/persistence actions. These often occur due to document-related causes or automation-related causes, or both. Document-related causes include:

• Unknown context — This document type has never been seen before and therefore the system doesn’t have the knowledge to process it.
• Readability — The document is not legible enough to successfully extract the data required. This is usually due to scan quality, font, size or other issues.
• Missing data — The data required to be extracted is not present in the document, something that usually must be rectified by the sender.

Best-in-class automation systems can identify document-related causes but solving them requires reconfiguration or retraining. Automation-related causes, on the other hand, include:

• Incorrect categorization: The document is incorrectly categorized, leading to errors and escalation.
• Incorrect extraction: The document extraction fails, leading to errors and escalation.
• Incorrect validation: The system marks valid data as invalid.

Recommended controls:

• Unknown context: A good system will auto-suggest possible formats and configurations, and minimize the manual effort required to configure the document type within the system. This might be a cloning of the closest possible option, so users must type out or demarcate minimal information.
• Readability: Experiment and empirically discover the ideal readability in terms of DPI, Font size, etc, and notify all upstream sources of the minimum specs required for documents.
• Missing data: This can only be controlled by the sender, and there’s no remediation on the recipient side for these issues.
• Incorrect categorization: This can be solved by validations. Ensuring that fields are not blank, and cross checking against similar document types can flag these situations early.
• Incorrect extraction: This should be evaluated as part of the proof of concept or trial phase to ensure the system has a high degree of extraction quality for the documents at hand. In case of repeat issues, further training might improve the extraction quality, especially if issues can be isolated to certain document types, senders or fields.
• Incorrect validation: Rule-based validations can be corrected by examining and fixing how rules have been codified. Unsupervised validations are likely to require model retraining.

Incorrect Pass and Incorrect Escalation

Incorrect Pass occurs when automation has failed, but the incorrect results of document processing have been passed on downstream. No remediation is possible in this scenario, but the right controls can help monitor for and avoid these scenarios.

Incorrect Escalations occur when a document has been successfully categorized, extracted and validated, and the data has been persisted or exported, but the system requires human intervention. Incorrect escalations can happen for many reasons and need case-by-case examination.

Recommended control: A root-cause analysis of each false break is recommended, followed by action to prevent repetition.

Document-related causes for these scenarios include: an unknown type of document, poor readability or missing data that result in the document being incorrectly processed. To control for this, test the automation system with documents that have known errors to see if they are incorrectly adjudged as correctly processed.

Automation-related causes might be

• Incorrect categorization: The automation system categorizes a document incorrectly, but the document passes the validations and is marked correctly.

Recommended control: Test extraction for similar documents and ensure that each document type’s validation in an exact set, rather than a must-have set. For example, both an invoice and a check might have a payee a date and an amount. But an invoice might have a vendor name and items as well. A well-defined validation set will ensure that a document classified as a check will have only the payee, a date and an amount, and no other data exists in this document.

• Incorrect extraction: This occurs when the wrong value was extracted from the document, but the system is unaware the value is incorrect.

Recommended controls: Cross-check validations against other documents; maker-checker model for fields with high sensitivity to mistakes.

• Incorrect validation: This occurs when incorrect data was extracted but the system reports it as valid.

Recommended control: Conduct random sampling by automating regression tests, which will ensure the outputs are exactly as expected for a defined set of inputs.

High %STP is an important metric to focus on and solve for to ensure the long-term viability of the automation process. Automation failures can be caused by several reasons. Having the right framework to exhaustively organize the points of failure is a much-needed starting point to solve them.

A veteran of the financial services industry, Prashant Vijay is currently chief executive at Romulus, which specializes in building software products that automate document-heavy operations in the financial services industry. He has spent more than two decades working at the intersection of technology and data across multiple roles and geographies. His views are informed by his experience in tech and business roles at Goldman Sachs, and his sales and product and business management roles at IHS Markit.