$2 trillion stimulus package was signed into law with $350 billion made available through the Paycheck Protection Program to the country’s 30 million businesses that qualify. As of last Friday, the majority of large and medium-sized banks were struggling to ingest the applications - most due to performance issues related to the volume of requests. Let’s break down what an individual submission looks like, how this identifies areas of technology to invest in for US banks, and what target architecture could immediately solve 100% of onboarding issues.
Size of submission packet and volume of transactions
Since the program works on a first-come, first-serve basis, it is realistic to expect a large portion of the 30 million businesses to drive up processing demand all at once. The banks are responsible to a) facilitate inbound traffic, b) authenticate existing customers, c) ingest application data, d) validate the legitimacy of the request (e.g. KYC, AML, etc.), and e) submit an application to the government. The system used by the government, E-Tran, bears the downstream grunt of accepting the influx of data.
If the healthcare.gov performance fiasco taught the world anything is that interdependent systems should facilitate data ingestion, caching, storage, and transition in an asynchronous workflow. In other words, a bank should be able to capture the application without an immediate response from the API provided by the government. For example, healthcare.gov suffered from the tight coupling of the application portal to the local state infrastructure amongst many other issues - even by modernizing the portal, the submissions would fail due to legacy state infrastructure.
The individual submission packet is small and inconsequential in context of today’s technology and cloud provider processing throughput. Having had to go through the process ourselves, here’s what our assets looked like:
Two page PDF application provided by the bank - 530KB
PDF document with monthly payroll history - 20KB
Four quarterly federal payroll filings - 3MB
Scanned identify document of applicant (e.g. id card, front and back) - 2MB
The total submission packet weighed in at roughly 5.5 megabytes of data. Using average US internet speeds, this upload from the customer side would take half a second. Let’s follow the trail.
Investment opportunities in banking applications and infrastructure
From a transaction volume perspective, let’s go on a limb and assume that all of the 30 million small businesses had their paperwork together and ready for the program to open its doors Friday morning (unlikely). Let’s normalize and distribute those requests across the fifteen largest US banks that hold the majority of the market. Each bank would have to facilitate 2 million applications a day or roughly 60 applications per second (and 330MB/s). For comparison, Twitter has 6,000 tweets posted each second. Twitter is a company with revenues of $1 billion, while the smallest bank on the top 15 list is $9 billion. Affording the technology shouldn’t be an issue, and I’ll show later on how a cloud-based approach would even minimize the expense footprint in standing up this capability immediately.
If anything, the failure to ingest small business applications is another piece of evidence that additional modernization investment is necessary for banking. Keep in mind that the government is paying banks a 0.5% fee of the amount distributed through the program, a lost opportunity for the majority of banks that couldn’t accept applications long into Saturday.
Poor experience design compounds technical issues
In addition to unresponsive websites, reports of cumbersome user experience emerged from several banks. Unable to integrate the loan application process as a separate product into their existing lending workflow, banks stood up separate applications that worked only on specific browsers, required very specific file types for evidence submission (e.g. no support for excel and financial data), and separate credentialing and authentication workflows. During an uncertain time with unpredictable infrastructure, matters were made worse by cumbersome user interfaces that left customers crying for help even when the technology worked.
Performance-oriented architecture for small business loan decisioning
In addition to digital onboarding, cash management online, and mobile banking, we’ve designed and launched automated small business loan underwriting platforms that helped cut time fourfold from origination to cash-in-hand. Here’s an example of application architecture and infrastructure needs to alleviate the challenges documented above.
To handle large, unpredictable transaction volumes, we recommend using an asynchronous approach - accept the data, queue it, and process it asynchronously. Since the program requirements were in flux and not finalized until going live, apply an event streaming pattern to the application. This will enable your bank to set up stages of processing with normalized data and then modification can be done in increments:
as requirements surface, quickly adjust ingestion process and start accepting the data
once ingestion is running, modify validation and verification processes to match requirements without blocking the ingestion
For faster response to market, consider serving application web pages via CDN providers in a form of SPA resources instead of relying on in-house infrastructure. In other words, all high volume traffic would be handled by CDN provider and then the downstream processes (API calls, etc.) connected with the bank could be queued and throttled based on existing system throughput. This would save long term costs of scaling legacy environments to handle volumes that are seasonal in nature. Furthermore, a cloud-native approach would guarantee that the bank only pays for the resources that are being used - as soon as the demand tapers off, the environments and capabilities can be deprovisioned.
For security hardening field level encryption should be used at the edge (e.g. AWS CloudFront). Such a configuration would allow data to flow securely via streams without fear that some services could gain access to sensitive fields. Such as an approach could also be incorporated into API gateway as middle-ware.
To meet the large volume of inbound requests, consider leveraging cloud providers’ API gateway. Once captured, drop inbound data into streams such as Kafka, Kinesis or EventHub, which would ensure that none of the requests are lost - no matter the spikes and valleys of demand. Continue to asynchronous processing once captured. Processing stages would allow you to make changes to processors without impacting the chain - making changing, testing and deploying faster - even in an error scenario where impact is localized and it is possible to “replay” all the events after a fix is applied. This approach would provide resiliency to failure, changing requirements, uptime/downtime of dependent environments (such as E-Tran).
Using this architecture the bank could scale processors independently: event processors could be made with lambda functions listening to event streams. Adding additional consumers to the stream would be painless, possibly sending out notifications, integrating with internal/external systems, and so on. Lastly, such an architecture would provide the maximum speed to market with resiliency to make changes along the way.
Please don’t hesitate to reach out if we can share additional thoughts on performance architecture in banking.
