Human in the loop: Oversight in the age of AI-powered banking

As AI reshapes financial services, human oversight remains crucial for accuracy, fairness and compliance as use cases expand. [caption id="attachment_57379256" align="alignright" width="300"] Bank Automation News CEO JJ Hornblass with Koren Picariello of Morgan Stanley at Bank Automation Summit. (Photo/Bank Automation News)[/caption] The adoption of AI in financial services is imminent, Koren Picariello , managing director and head of generative AI strategy and execution at $6.2 trillion wealth management firm Morgan Stanley , said during a fireside chat at Bank Automation Summit 2025 earlier this month. While not every financial institution will deploy AI at the same scale, every institution needs a strategy, she said. “You can’t do nothing, especially with how quickly these technologies are evolving.” — Koren Picariello, managing director and head of generative AI strategy and execution, Morgan Stanley Morgan Stanley opted for an ambitious AI strategy, partnering with OpenAI in 2022 to develop its virtual assistant, called AI @ Morgan Stanley Assistant , which Picariello said is used by nearly 100% of the firm’s financial adviser teams. Risk factors But some FIs prefer a more cautious approach. Cleveland-based KeyBank , for example, is reluctant to deploy AI for customer-facing solutions until its models have undergone extensive testing, Dominic Cugini , chief transformation officer at the $187 billion bank, previously told Bank Automation News , predicting it would be 2026 or 2027 before the bank rolls out gen AI applications for consumers. [caption id="attachment_57378111" align="alignright" width="300"] (AI-generated)[/caption] And while AI will be necessary for complex tasks, there are many other existing technologies that can help banks become more efficient, if deployed properly, Cugini said. Robotic process automation, for one, has streamlined repetitive tasks whereas integrating AI agents while maintaining human oversight for accuracy and bias control is a challenge, Cugini said at Bank Automation Summit. For example, while AI can gather and organize data related to dispute resolution and suspicious activity reporting, cutting manual work by 50%, humans must still review and validate results, Cugini said. AI-powered optical character recognition is useful for extracting data, although AI struggles to form hypotheses, so a final human check is required to ensure accuracy and fairness, he explained. Fearing bias AI can add tremendous value to a bank’s operations, but is a double-edged sword as concerns about bias, regulatory scrutiny and compliance implications persist. “Bias is terrifying,” Cugini said. Unlike human errors, which are typically detected early, mistakes made by AI could take longer to detect, so thousands or even millions of faulty transactions could go undetected for years, he said. [caption id="attachment_57379257" align="alignright" width="300"] (AI-generated)[/caption] Minneapolis-based U.S. Bank is also taking a cautious approach to AI deployment , ensuring transparency with how it’s using consumer data, Shruti Patel , chief product officer for business banking, told BAN . “[Banks] are custodians of [a huge amount of] customers and data and money movement capabilities, so we want to make sure that we have the right framework from a usage perspective and safety perspective,” Patel said. "We want to make sure that when we deploy these AI tools…we’re making sure that we're very deliberate, that it causes no social harm and it's very unbiased in the way we deploy it," she continued. The bank is exploring AI cause use cases include customer support chatbots and fraud prevention, she said. Human error Embracing AI doesn’t negate the need for financial institutions to audit AI output to avoid costly inaccuracies, but relying on human judgment comes with its own risks which AI could help mitigate, Morgan Stanley’s Picariello said. “If the expectation is perfection, we need to move off of that immediately. Nothing is perfect. We actually all come with a bias that a human does it perfectly — we're better than the model — and we know quantitatively that that's not true.” — Koren Picariello In fact, the costliest mistakes can occur due to a lack of tech, according to industry experts. In one widely publicized example, an employee at $1.7 trillion Citi mistakenly transferred $81 trillion to a customer . [caption id="attachment_57368483" align="alignright" width="300"] Citi in April 2024 mistakenly transferred a client $81 trillion, according to a February report. (Photo/Bank Automation News)[/caption] “Citi needs better technical controls around their payment authorization policies. Those policies and technology can be developed in-house,” James McCarthy , founder and chairman of financial consulting agency McCarthy Hatch and a founding member of the Consumer Financial Protection Bureau , told BAN last month after the incident was reported. Citi was able to promptly recall the erroneous payment. All FIs can prevent costly human errors with deployment of AI, specifically agentic AI , which can flag unusual activity that could go undetected by humans, Ronak Doshi , partner at consulting firm Everest Group , told BAN last month. AI shouldn’t completely replace employees but rather help reduce human error by using it as a supplementary tool, he said. AI and compliance As banks rush to innovate with new technology, being mindful of impending regulations will be essential for success. [caption id="attachment_57378517" align="alignright" width="300"] (AI-generated)[/caption] Regulations present both a challenge and an opportunity, particularly when it comes to tech strategy, Agus Sudjianto , senior vice president of risk and technology at open-source generative AI and machine learning platform provider H2O.ai, told BAN . “Stricter compliance requirements mean banks must adopt more transparent and explainable AI solutions,” he said, adding that FIs with a risk management framework will have a competitive edge in AI deployment. H20.ai this month announced what the company calls “the industry's first risk management framework for generative AI” for safe deployment of AI, he said. “Generative AI is relatively new and introduces unique challenges like biased outputs, hallucinations and security vulnerabilities,” which traditional model risk management (MRM) frameworks are not designed to handle, Sudjianto said. [caption id="attachment_57377862" align="alignright" width="300"] (AI-generated)[/caption] H2O.ai’s MRM solution extends these principles to gen AI, he said, incorporating: Automated testing; Human calibration; and Explainability tools to ensure AI models are trustworthy and compliant. “The biggest challenges were ensuring that our system could reliably detect weaknesses in generative AI models, align AI outputs with human judgment and stay compliant with evolving regulations,” Sudjianto said. While the MRM framework is designed to automate and streamline testing and validation of AI models, it does not take the place of human auditing, he said, echoing Picariello. H20.ai has raised $256 million since being founded in 2012, with Nvidia , Wells Fargo and Nexus Ventures among its investors, Sudjianto said. “AI adoption will continue to grow [over the next three years], but regulations will tighten, requiring better transparency and compliance,” Sudjianto said. “Banks that invest in strong AI governance ... will have a competitive advantage in deploying AI safely and effectively.” — Agus Sudjianto, SVP of risk and technology, H2O.ai Human oversight for better CX Consumers feel more comfortable with a human in the loop when a transaction isn’t simple, Todd Michaud , chief executive at AI-driven intelligent automation platform HuLoop Automation , told BAN . HuLoop’s solution detects low-confidence or high-risk cases using decision thresholds, anomaly detection and risk criteria, Michaud said. When unclear patterns or red flags appear, a human expert is alerted to ensure proper oversight, he said. [caption id="attachment_57378613" align="alignright" width="300"] (Courtesy/Canva)[/caption] It is also programmed to quickly detect anomalous transactions, and refer them to human experts if a more complex decision is necessary, Michaud said. Adding a human to the loop “dramatically reduces false positives while accelerating legitimate claims, improving security and customer experience,” Michaud said. Reducing false positives also means satisfied consumers, who can become frustrated by unnecessary transaction blocks, he said. “One of our banking customers reduced dispute resolution times by 40% using HuLoop,” Michaud said, without disclosing the bank. “By automating routine verifications while escalating ambiguous cases to human analysts, they improved accuracy and customer satisfaction, reducing backlogs and ensuring fair resolutions.” A human touch Having a human involved in overseeing AI and ML models also ensures a broad perspective on their use while keeping customer interests a priority, making it an effective approach to maintaining regulatory compliance, Lynn Woosley , managing director at consulting firm Asurity Advisors , told BAN . “Sometimes it's just extra arms and legs, especially for community banks that may not have all the expertise they need in-house,” she said. Clients still demand human touch as a component of their overall experience, Joanne Wyper , executive vice president and head of digital and operations for Citizens Commercial Banking, said at Bank Automation Summit 2025. [caption id="attachment_182179" align="alignright" width="300"] (Courtesy/Citizens Bank)[/caption] That’s why the $217.5 billion Citizens modeled its Citizens Digital Butler , the bank’s AI-powered assistant launched in 2023, to resemble a human, Wyper said. "That took more complexity, and it took more time, because to try and make a commercial digital assistant personal to each client, to the multiple people that work in the organization, took quite a lot of thinking power, but it was definitely worth the squeeze,” she said. Citizens plans to link the digital butler to a large language model this year to refine its conversational abilities, she said. Automating fairness AI-powered data analytics software can be employed to test the accuracy of AI-based models, Kareem Saleh , co-founder and CEO of “fairness-as-a-service” startup FairPlay , recently told BAN . [caption id="attachment_57378679" align="alignright" width="300"] (AI-generated)[/caption] FairPlay uses AI-driven insights to detect “blind spots” in financial institutions’ credit decisioning models and is scheduled to launch an index tool in the third quarter that shows FIs how their underwriting affects consumers, Saleh said. For example, between 25% and 33% of rejected loan applications are due to biased AI models, he said. AI, if used responsibly, can also enable community banks to more efficiently evaluate and process smaller-dollar loans that without automation can be too resource-intensive to underwrite, Katie Quilligan , investor at VC firm BankTech Ventures , said during Bank Automation Summit 2025. As more banks look to deploy AI, the need for a human in the loop will only strengthen, HuLoop’s Michaud said. “AI will take on more routine activities, but human oversight will remain crucial for ethical, regulatory and strategic considerations,” he said.

Human in the loop: Oversight in the age of AI-powered banking

AI is inevitable, but humans are irreplaceable, experts say

Transactions: Extend, Visa expand partnership

A growth mindset in banking requires AI

Related Posts

United Wholesale Mortgage’s AI-powered virtual assistant closes 80K loans in first year

LeadX organizing UCC data to boost user sales

TD’s agentic AI tool doubles same-day mortgage approval rates

A growth mindset in banking requires AI

Stay Informed with Our Newsletters

EMERGING FINTECH DIRECTORY

The Buzz Podcast

SPONSORED

How AI and Product Experts Turn Fuzzy Requirements Into Focused Dev-ready Roadmaps

Is Your Technology Supplier There for You?

Hiding in Plain Sight: How to Use Data to Spot Consumer Accounts Being Used by Small Businesses

Connect

Welcome Back!

Retrieve your password