The Cloud-Based Conundrum

GonzoBankers who opted over the past few weeks for spending their spare moments outside enjoying the sporadic bursts of early spring sunshine instead of wading through the barrage of stimulating banking blogs may have missed the breaking news related to Ellie Mae’s Encompass platform that raises some questions around the stability of outsourced vendor solutions.

On March 31, Ellie Mae announced its Network Services (as part of the Encompass solution) was experiencing outages and was unavailable to users. This continued into the following day. Adding to the frustration, on the morning of April 1 an announcement sent to all users stated that access had been restored to the system. Apparently this was not true for all users, as some continued to experience issues throughout the day. The Encompass system itself was not down, just the services used through Encompass. More on that later.

This is, of course, a rotten situation, unless you are a competing mortgage LOS provider hoping to turn the tables on an active deal. Encompass has a strong presence in Cornerstone’s market with a ton of momentum right now, so it is unfortunate for both the firm and our Gonzo client base.

Also, it’s incredibly bad timing given the fact that this occurred at month-end, causing many lenders to miss end-of-month closing opportunities. But, in all fairness, Cornerstone’s past experience with Encompass clients confirms this outage is an exception instead of the norm.

What We Know

Ellie Mae initially announced there was a Distributed Denial of Service (DDoS) attack and that full functionality would be restored to all customers. The silver lining is there was no data breach and therefore all customer information is secure. That’s seriously good news. Given the fallout from recent data breaches (e.g. Target, Michaels and others), this issue would obviously be much worse if customer data were accessed.

Shortly thereafter, the official story was corrected in that it appeared to be a deliberate attack by someone with in-depth knowledge of the mortgage industry. But after further evaluation from an outside information security firm, it was determined to be an infrastructure failure due to a combination of factors related to network, hardware, software and end-user demand for service.

Wait a minute – insert screeching brakes and shattering glass sounds here!! This is even more troubling than the thought that Encompass was the victim of an outside attack. An outside attack is one thing, and companies can only go so far to ensure they are adequately prepared to handle it, but we have a publicly traded, financial software services company announcing it had a nearly-48-hour outage due to infrastructure issues? Hmmm.

Let’s not lose sight of the scope of this outage, either, as roughly 20% of all mortgage loan applications in the United States (or approximately 3 million) are processed through Encompass. This only further amplifies the fallout, because given the company’s size, everyone is going to hear about it.

Another major point here is that this outage was not limited in terms of clients and functionality affected. While it stands to reason that hosted clients would experience an issue, in-house clients weren’t safe, as their logins to the system still required an authentication back to Ellie Mae.

In either case, the system failed, leaving lenders unable to access necessary information. The functionality loss impacted all loan processing and closing tasks as clients were unable to pull credit, order flood certs, lock rates and perform automated underwriting – among other things.

However, the biggest black eye here is certainly from end customer impact, specifically customers who were trying to get their loans closed at the end of the month. With the system down and closing docs unavailable, closing dates had to be shuffled, customers were inconvenienced and there were unexpected costs on some lenders to secure alternative loan closing arrangements. This put lenders in an awkward position of doing damage control for poor service on something that was out of their control.

Considering there are no comments being given at this time outside of the official statement provided last Monday,  it would be nice to know exactly what caused this outage. The thought that this was in any way volume related is a hard pill to swallow given the paltry mortgage volumes we’ve seen recently compared to the past several years.

To Ellie Mae’s credit, it has scrambled well to stop the bleeding by re-routing volume and adding capacity across its data centers, but this clearly seems to be a short-term fix to a longer-term problem.

The truth is this article could easily be about the recent issues with Q2 or the widespread outages at Digital Insight. Another example, although a bit dated, was the major item processing outage at Jack Henry’s data center during Hurricane Sandy causing it to re-route all work from its New Jersey data centers.

Would anyone have guessed that one of the major core providers would have to close its item processing data center for weeks and shift work elsewhere? Or for that matter, would anyone have guessed that Ellie Mae would be down at month-end, leaving lenders completely unable to process loans?

What Needs to Be Done

The bottom line here is that financial institutions are outsourcing their solutions – sometimes blindly – to large, big name vendors that are supposed to ensure these issues don’t happen. But, as we know all too well, this isn’t necessarily the case.

On the surface, outsourcing can seem like such an easy play as it allows virtually all the risk, staffing and other requirements associated with in-house hosting to be pushed off on a third party provider. However, due diligence and vendor management efforts must still be done, and by outsourcing, institutions are essentially putting the vendor behind the wheel of their reputational risk.

Here are a few things that can be done to protect against this:

Know the options. On the front end, there must be a strong system selection methodology that allows for vetting significant issues that could provide ongoing operational or reputational risk. Solid outsourcing agreements must be established to ensure the financial institution is safeguarded against these issues.

Manage your Vendors. Ongoing management of the vendors and a good understanding of the vendors’ service level agreements are critical. Specifically, know how vendors measure and report their service levels. For example, recognize that a vendor can consider 99% up-time as the sum total of the up-time for its entire client base over a period of months, even if one of its clients was down the entire month.

Understand the regulators. The final consideration here is the constantly moving expectations of regulators. GonzoBankers need to diligently think about what regulators are going to ask and understand their expectations relative to documentation. Regulators are going to require adequate steps to be taken to ensure such outages don’t negatively affect operations again. This is a tall order considering these are blind spots for the vendors, and it is going to require financial institutions to ask the right questions of their vendors.

Maintaining the status quo moving forward is not an option. We can no longer assume that everything will be fine just because these responsibilities are offloaded to sometimes immature and unreliable big name vendors. We need to ensure that issues like these don’t happen again.
-dj

The best defense is a good offense.

Build that offense through a partnership with industry experts Cornerstone Advisors. Cornerstone can help you dig up the knowledge you’ll need to make smart choices in the ever-churning world of vendors and regulators.

Contact us today to talk about putting a Vendor Management Strategy in place at your institution.