In an unprecedented era of high customer expectations, low market confidence and the constant threat of small and nimble players disrupting the market, financial institutions find themselves in a position where they must evolve or become extinct. As established players continue to be dependent on legacy technology and monolithic systems that require significant overhead and lead time to deliver even the smallest increment of value, many are struggling to satisfy the needs of their customers and shareholders. Their ability to remain competitive is diminishing.
It’s time for financial institutions to take the leap, as their technology counterparts have done, and embrace the microservices architecture movement. For many companies, such as Netflix, Amazon, Gilt and PayPal, a microservices architecture has been the foundation that has enabled them to rapidly scale, deliver value and grow to become industry leaders. In recent years, interest in microservices has increased, and according to Google Trends, it continues to rise. However, the transition is not easy, and not all companies are successful.
A good example in the banking space is the core banking system that one finds at the deepest level of the IT organization. These platforms are typically batch-based, largely monolithic, and ancient by technology standards. The challenge that typically surfaces with these monolithic platforms is that customer-facing applications must evolve faster, yet they have data dependencies on core banking. Even if your mobile banking application uses an agile methodology to go to market, change sets that impact core banking will delay the whole train from getting to the station on time. This is a perfect example where a microservices architecture layered on top of core banking could decouple the release cycles and increase agility for the organization overall.
This post explores why implementing a microservices architecture is not easy and offers advice on how financial services institutions, particularly banks, can effectively evaluate and embrace the microservices architecture movement.
What is a microservices architecture?
There are myriad books on microservices, such as Sam Newman's "Building Microservices." Martin Fowler also has written a lot on this topic. In short, microservices stems from the same ideology as Agile and DevOps by seeking to break down slow moving, monolithic systems into multiple small, independent services that are highly decoupled and self-contained to focus on a specific function or capability. This breaking down of systems into manageable pieces that operate independently in parallel allows large organizations to be more nimble and flexible to move at the speed of their smaller competitors. Additionally, microservices is a technology-agnostic approach, thus adopters can leverage different technologies. However, this doesn't mean that you should strive to have a "zoo," but that you are not bound to some frameworks and you can use appropriate technologies were applicable.
What banks need to understand about microservices
I am a big fan of microservices. Throughout my career with different companies, I recognized that a microservices architecture is quite capable of solving problems...if applied correctly. The keyword is if. I have seen many companies adopt a microservices architecture only to learn later in the process that it entailed a lot more complexity. Many organization-wide questions need to be answered immediately, and after several months some companies abandon the effort completely. Conversely, those that did know how to apply the process correctly didn't recognize that the project/product did not need to be done in a microservices environment. There are a lot of good fits for microservices, especially with companies that deal with lots of different data from various data sources. Typically, these are financial institutions, data intelligence services, IoT services, and large-scale eCommerce companies.
So what do I mean by "if applied correctly"? As nice as it sounds and as much as most developers would like to transition to a microservices environment, there can be hidden pitfalls. A microservices architecture does not make your software simpler. In fact, it makes it more complicated! There are more "moving" parts than in a monolith approach. It also is more difficult to monitor, manage, version, and decommission. Secondly, you cannot buy a microservices architecture for yourself. Microservices isn't a product. The entire company must embrace a microservices architecture to be successful or to make the most of a microservices world.
Benefits that a microservices architecture brings
If implemented correctly, microservices has a lot of benefits, including smaller units of functionality, which makes it:
Easier and faster to develop (less information in developers "cache")
Easier to refactor replace (smaller footprint)
Easier to scale (just replicate instances)
Faster to deploy (less functionality and less hard dependencies)
Smaller failure footprint (only part of a system may be down instead of whole system)
Yet, to run a microservices architecture properly, the entire company must optimize and align its operations. This is only from a development perspective, but to be successful other parts of the organization will have to be touched as well. For example:
Product owners will have to come up with smaller requirements, and as such, easier to understand with a much smaller scope. This equates to less waiting until implementation is done
Entire teams would be smaller, more agile and working with a smaller scope
There will be a strong bond with DevOps, because a larger number of services would make full automation of deployment a necessity
Good organizational communication with application support to monitor the hundreds of services. To effectively achieve this, you will need proper tools, tracing, recovery and proper alerting for your services
Living service documentation. Documentation has to be automated as well or else your endpoints would be useless even in your company
What is the best path?
To begin your transition to a microservices architecture, consider the following questions first:
Does your organization really need microservices? If you can't answer this question, seek out advice first.
Do you have the right competence in-house? Microservices is more complex than traditional monolith applications and you will need a team with deep expertise in microservices architecture. Inexperienced teams can actually produce inverse results (you'll get a totally obfuscated solution).
Do all departments agree with the shift to microservices architecture? All departments (such as application support, security) must be aligned as this is organization-wide change.
Once you are certain that a microservices architecture is a good fit for your organization and have organizational buy-in, the biggest challenge is to understand what's going to be needed and how much scope is going to be your "infrastructure" things. Most of the literature suggests, and I concur, that ripping apart existing systems into smaller pieces one by one until a whole product is a collection of microservices is the way to go. This makes you create required infrastructure as you go. A lot of companies fall into a trap early on by starting with a small greenfield project (approximately three months) and try to make a microservices proof of concept at this stage. This actually ends up with a half-baked service that has trouble with deployment, missing proper logging, tracing, and discovery. So what actually happens in these situations? The problem is that the scope is not understood up front. The business requirements might be ok and pretty clear, but non-functional requirements are unclear at this point. The usual approach to non-functional questions, such as, "How do we log?" is let's just dump to a file, we'll sort it out later. So the question becomes "later when?" At this point DevOps builds some kind of deployment (usually using old ways), application support gets their text file log monitoring, business requirements are there.
So did we get microservices? The answer is no. It's pretty much the same kind of service with less functionality and some half-baked artifacts that go with it. At this point there is a large risk actually to revert back to monolith applications/services that are more familiar. This is why we suggest making sure that your organization has a team with deep expertise in microservices architecture.
Next Steps
Microservices is not just about development. It's about security, discovery, deployment automation, aggregated logging and monitoring. This includes a lot of "infrastructure" parts that either must exist or a decision must be made before the development begins. We will cover this aspect of microservices in a future blog post.