Prepare for automatic continuous deployment
Confident in our evaluation and clarity on the prospective benefits, we set out to extend our CI/CD process with automatic continuous deployment. Our teams took the observations, researched best practices, and developed a plan to implement process changes to support sensible automation. We outlined our recommendations for process improvements specifically around:
building the architecture,
preparing a deployment playbook,
expanding and configuring test coverage,
and implementing comprehensive test coverage.
Building the architecture
Historically, our DevOps teams found a suitable operational window, shut down the application in the production environment, and then deployed the changes into the corresponding environment. However, automatic continuous deployment requires a clustered application architecture that supports hot-swappable services. Only such designs provide the capability for production instances of the application to remain in operation while some components/instances are updated in the environment. Moreover, only such an architecture offers adequate support to a continuous deployment pipeline that:
automatically migrates a release to production,
automatically verifies release functionality,
verifies a release quality and functionality,
and avoids any operational impact on the end user's experience.
We identified two critical architectural success factors.
Design interoperability into the software. To accommodate different versions running simultaneously in the production environment, implement data operations to be both backward- and forward-compatible. Some application versions share a single database or send/receive data via a message queue or data-ingestion service. The architecture must anticipate various configurations and provide partial services if an application instance encounters incomplete data sets.
Design applications to be stateless. Avoid internal state dependencies and instead pursue the following measures:
Move session storage to a dedicated Redis or relational database system.
Migrate user-generated content, such as files residing within the file system of a single web server to scalable cloud-storage services like Azure Blob storage or AWS S3.
Replace custom stateful flows in individual server instances with authentication flows that adhere to standard protocols, such as OAuth with JWT bearer tokens.
Design applications to avoid caching overuse. Frequently validate the use of custom caches. Application settings retrieved from a central location should not be cached for more than five minutes. Instead, use a key-value store such as Redis that permits multiple application instances to become stateless.
Preparing a deployment handbook
When crafting a deployment strategy for continuous delivery, be sure to consider the type of application, the architecture, and the deployment tools. An essential requirement in a continuous deployment pipeline is the ability to precisely orchestrate rolling updates among various architectural tiers for an application. As each tier update completes, the next begins work. Operational data is easily sharable data between tiers.
For most application topologies, it’s necessary to update specific tiers in a tight sequence. The database schema, for example, may need to be migrated first. Or, the caching servers must be flushed before updating the app servers. Most critically, it is entirely impractical to simultaneously apply changes to all the production application tiers and system configurations.
In the pursuit of continuous deployment, be sober about the details, the timing, and the sequencing. Services may be unavailable immediately after restarting, and updating some app components may not be instantaneous. Therefore, it is vitally important to gain the ability to disconnect a machine from a load-balanced pool before any application update. Also, automate all of the specific actions of managing machine disconnections and connections to a production pool.
For simplicity, consider a basic example of three identical services in a load balanced production pool, with the load evenly distributed to three servers.
Option 1: Disconnect Machine 1 from the production pool, update application Version 1 to Version 2, and then reconnect Machine 1 to the pool. When disconnecting Machine 2, Machine 1 will be running with app Version 2 while Machine 3 still runs Version 1. Perform the same procedure on Machine 2 and Machine 3. Since all machines presumably share the same data, in the event compatibility issues arise, data could become corrupt on all of the machines unless reliable data-commit measures are taken (which might require changes to the app and database).
Option 2: Orchestrate the deployment to disconnect half of the machines from the production pool, update the disconnected machines, and reconnect them to production. Then, disconnect the remaining machines running the older application versions, update, and reconnect them. While this approach may be convenient and maintain production continuity, the overall performance will likely be poor for users using the application. Using this approach is risky, where errors may result from miscoordination or mistiming. Additional effort is necessary to orchestrate deployment.
Option 3: Use a product like Microsoft Azure for deployment slots. Each machine is paired with another, and one machine assumes control of a production application instance for another during an update. When the update is done, a swap occurs: the machine reconnects to production and takes control of the first machine, which disconnects from production while it receives the update. A simple click of a button repeats the process for each pairing (deployment slot). If all machines running the applications have deployment slots with newer versions deploying, performing a swap on all slots results in all machines running the latest version. Azure deployment slots are also useful for rolling back versions easily without much effort. However, keep in mind that data may become corrupt if data operations compatibility issues emerge between different application versions (unless preemptive data-commit measures are in place).
EXPANDING AND CONFIGURING TEST COVERAGE
Reliable pipelines continuously deliver high-quality components and services into production, the penultimate goal. In a continuous deployment pipeline, the delivery of quality software assets requires that the team implement a succession of automated tests all along the pipeline. For comprehensive testing to support continuous deployment, the primary objectives are to detect bugs very early and prevent bugs from reaching the production environment.
Any team that embarks on a move toward continuous deployment must prioritize a smarter and more robust testing strategy. Begin by increasing coverage at the code level with a more intensive focus on unit and integration testing. Do not waste effort by duplicating tests in both the unit and end-to-end (E2E) suites since a defect found in a unit test typically appears in E2E testing. Add functionality, API, and user interface testing to capture defects that don't get sufficient coverage in unit or integration tests.
Implementing comprehensive testing
Other critical types of testing include regression, performance, security, and accessibility testing. These types of tests apply to any product development effort when building a CI/CD pipeline. Integrate these tests as part of the deployment pipeline to ensure the deployed application performs effectively.
Unit testing and end-to-end testing places more emphasis on bug prevention over detection. There is no such thing as a bug-free software application. Whenever developers change or modify software, even a small tweak generates unexpected consequences. It's incredibly exhausting to cover all operating system versions, updates, browser versions, components, services, and drivers. There is a better way. Cover the most important features with unit tests and UI E2E test cases to be confident that the main application flow works.
Regression testing ensures that proposed changes won’t break existing functionality. The purpose is to catch bugs that might inadvertently move downstream with the build or release candidate. Regression tests also keep previous bugs from making any future appearances. Be sure to update regression suites to envelop the new features built into the software with each successive release.
Performance testing detects and quantifies any variations in performance, especially concerning user experience and bulk-processing features. Additionally, this type of testing also points to the root causes of any degradations in responsiveness or processing speeds and reveal problems that inhibit applications from reaching production.
Usability testing is a type of black-box testing that determines the end user utility of an application. Accessibility, engagement, and smooth navigation contribute to higher market value for an application or website. Test software applications for the ease with which the end users can use them. If average users find that an application is difficult or tedious to use, they stop using it.
Accessibility testing is a subset of usability testing that assesses the level at which people with disabilities use the application. Results often become inputs to future application features. Accessibility evaluation is generally more formalized and tedious than usability testing. To accommodate as many potential users as possible, governments and other organizations consult web accessibility standards, such as the US Section 508 legislation and the Web Content Accessibility Guidelines (WCAG).
Security testing verifies that user data remains secure and uncorrupted. Security specialists collaborate with the developers to reveal issues and help remediate the well before the next deployment to production. Well executed security testing protocol empowers DevSecOps to bring in security testing well upstream in development and reinforces security at verification checkpoints all along the build pipeline.