Editor's Note: This post was originally published in August 2013, and has been updated for accuracy and comprehensiveness.
Today's blog post is about optimizing your development process. We will identify some red flags that you should be on the lookout for, suggest best practices, and offer insights into our development process and the related benefits.
You are in trouble if you find any of the following in your software development process:
You are deploying your applications manually, it’s a pain in the neck, and it takes a lot of time.
Your source code integrations are performed manually and conflicts occur often.
You are editing application configuration manually in staging or production environments.
You tell other developers that “it works on my machine” and the problem is in their code.
You do not have automated tests.
You often have last minute hot-fixes.
Fear not – even if you are feverishly nodding in agreement this article should help you correct your course.
A Better Way to Develop Software
Say Hello to Best Practices
Here is a general list of best practices that software developers across various industries and fields should be aware of and follow:
Use version control. There is not a single reason not to. Everyone is using and it’s easy to implement. You can use Team Foundation Server, Visual Source Safe, Subversion or even old CVS.
Automate the build. A common mistake is not to include everything in the automated build. Automated build should not only get latest source code from repository and compile it but also stop/start web site and other processes if needed, execute SQL to update database if necessary, manage application’s configuration and so on.
Your build should be self-testing. Include automated tests in the build process to catch bugs faster and with higher efficiency. At the minimum you should prepare unit tests and integration tests. Automated tests for user interfaces are also a very good idea (see magic of http://seleniumhq.org/).
Developers must commit at least every day. More frequent - better. The single most important prerequisite for a developer committing to the main branch is that their code can build correctly. This, of course, includes passing the build tests. As with any commit cycle the developer first updates their working copy to match the main branch, resolves any conflicts with the mainline, and then builds on their local machine. If the build passes, then they are free to commit the changes.
Keep the build fast. The whole point of Continuous Integration is to provide rapid feedback. A build that takes an hour is unreasonable. If you're starting a new project think about how to keep the build fast. Usual bottleneck is testing - keep your tests fast!
Test in a staging environment before deploying software on production. No, seriously, you need to test. Surprised? /sarcasm.
Make it easy for everyone to get the latest executable. Anyone involved in the project should be able to get the latest executable and be able to run it: for demonstrations, exploratory testing, or just to see what has changed this week.
Everyone should see what's happening. One of the most important things to communicate is the state of the main branch build. Build status can be reported via email or RSS feed. An even better solution is to integrate green and red lava lamps with Continuous Integration Server.
Bubble, Bubble, Build's In Trouble
Automate Deployment. To implement Continuous Integration you need to set up multiple environments (development, integration, staging, and production) that will run your build and tests and, if possible, automate as much deployment between these environments as possible. You may be doing these deployments several times a day.
Continuous Integration and Continuous Deployment
Environments
You should have a development, integration, staging, and, obviously, production environments.
Development environment usually is a developer machine with required software installed on it.
Integration environment is a separated machine that runs Continuous Integration Server and periodically runs auto-build-test scripts and reports status of builds.
While difficult, you should try to have a staging environment as similar as possible to your final production environment.
Database Continuous Integration Strategy
Database schema changes and corresponding code changes must always be deployed together. While deploying software to an environment, code files and libraries may usually be deleted or overwritten. Database files, however, must be intelligently manipulated so as not to destroy vital business data.
Successful database change management requires a consistent strategy to be applied by all team members. The ideal process for database change management would consist of:
Developers using their own local database to do their development work.
Each environment has a dedicated database. I.e. development, integration, staging, production.
Each developer maintains changes locally. When the database changes are ready to commit alongside the application source code, the developer follows these steps:
Create a change script that wraps all of the database changes into a single transactional change script.
Save the change script in to a folder in your source tree called Update.
Commit the change script along with the source code that corresponds to the change.
The continuous integration server detects changes to the source control repository and:
Builds the application code.
Executes the applications unit tests.
Executes the database create task to create a new database if needed.
Executes all of the new changes that are in source control under Update node if needed.
Executes the projects integration (data access) tests.
Marks the builds a success when all the tests pass.
Each developer runs the build script locally after receiving new schema changes scripts from the source code repository.
For more information: http://code.google.com/p/tarantino/wiki/DatabaseChangeManagement
Build and Deploy Strategy
Developer builds new version of the application locally and runs unit tests.
Developer checks in source code to version control.
Continuous integration server sees the updates and automatically calls auto-build script.
Build script runs any new update on the environment database.
Build script replaces the configuration files with the environment server configuration.
Build script compiles the DEBUG configuration of the application.
Build script runs all unit and integration tests.
If tests are OK build script publishes new version of application in integration environment.
The same changes should be made in the strategy for staging and production environments: run build script only after manual verification if everything is ready to go up to staging/production. In other hand if you have separated branches for development, staging and production in version control then you can map build scripts to run automatically for staging and production environments too (if new revision exists in corresponding branch). For production environment build script it is a good idea to add backup tasks for database and application too.
Summary
The setup of a Continuous Integration and Continuous Deployment environment adds a certain overhead to the project but the benefits far outweigh it. Collaboration between developers becomes simpler with less time spent on environment and database synchronization. Testing and deployment drops in complexity and last minute surprise issues and burning fixes practically disappear.