This is a document explaining a proposed principle for releasing code changes on the Investor Services project. I've named the concept "Releases Without Resistance" (or RWR).
This is the branch that contains the latest version of the project code. For us that is currently employee-portal, in GitFlow it is usually development.
We all know that a release consists of one or more chunks of code, each of these chunks of code was driven by a requirement and was merged into the code base with a merge request.
The term we use for establishing a list of chunks we want to deploy to production is "cutting a release".
Currently, when we cut a release, we create a new branch from a commit hash in the main branch at a point where we know a certain list of tickets are included (assumedly, those that have the status "Fixed" in JIRA).
To reiterate: we choose a place in the git tree (a hash) and give that place a name corresponding to today's date (usually something like release/2000-01-01_employee-portal). The intention being that the name serves as a label for one or more chunks of code that we want to include in our upcoming production deployment.
Next, the snapshot of the code at that hash (in the form of a branch) is deployed to the test server and the QA team begin to test it. Meanwhile, development continues on the main branch.
Usually, QA will find problems with one or more chunks of code included in the release. At this point they will re-open these tickets and the developers will begin to fix the problems.
Usually, when we fix these tickets we are forced to fix them on top of the commit hash where we originally cut the release. Because fixing them on top of the commit hash at the HEAD of the main branch would mean including any chunks of code that have been added to the main branch since the release was first cut. (Notice it's the commit hash, not the branch, that we must apply the fixes on top of—this will be important later.)
"HEAD" is a shorthand reference to the commit hash at the tip of a branch. The hash of the first commit you see when viewing the
git logof a branch is the value that HEAD references.
So when we branch off from the release branch, make a code change, and merge back in to the release branch, we have added code to the release branch that isn't in the main branch. And because the main branch also contains code that isn't in the release branch, the branches are said to have "diverged".
When we allow branches to diverge, merge conflicts may be introduced. This is a serious problem because resolving merge conflicts and combining different chunks of code essentially equates to writing new code, and that code may have bugs that weren't previously present in either piece of code. Anything that is changed about the code after it is tested undermines the validity of the results of the testing for that code.
Another serious problem is caused by the blocking nature of releases. Because the QA team can only reasonably test a small number of releases at any one time and there are a finite amount of environments to deploy releases to, one release usually blocks another release.
This isn't a problem as long as a small number of release-blocking issues are raised and the release goes out quickly. In this circumstance, by the time we're unblocked, we've probably got a reasonably small number of new code chunks in the main branch and these can be immediately included in a new release branch.
The problem comes when the release is delayed because issues are found. Especially when those issues are not found immediately. Whilst the issues in the release branch are being fixed, more and more chunks of code are being included in the main branch. This means the divergence of the branches increase and the risk of the previous two problems increase concurrently. It also means that once the release process has become unblocked, the main branch contains so many code chunks that the next release contains more than the previous release and the chance of issues being found increases too.
The risk of issues being found increases the chance that a release will be delayed, releases being delayed increases the amount of code included in the next release, and that in turn increases the risk of issues being found. It is an exponentially worsening feedback loop that ultimately leads to more work, slower turnaround, longer release cycles, and lower quality releases.
Fixing the divergence issue is easy: don't allow releases to be modified. Instead, increment the release version every time a change is made to a release using a versioning system such as SemVer.
Wouldn't this mean a never ending series of releases that ultimately fail to reach production? No, to illustrate I'll describe a typical release under this system.
It's time to cut a release, but this time rather than creating a new branch from the commit hash that contains the chunks of code we want to include, we create a tag instead.
This tag might look something like 0.1.0. The first number is the major version1, the second number is the minor version2, and the third number is the patch number3.
1 The major version being zero means that this application does not yet have a defined public interface that users have come to depend on4.
2 The minor version increments each time we release the application. Whenever one or more chunks of code are added to the code base and a release is cut, this number should be incremented.
3 The patch number is the number that will be incremented every time a release is changed during the release process.
4 It should be noted that we're probably extremely close to incrementing the major version to 1, and we may want to consider the upcoming implications of WealthFront or LendingClub using the application daily—any breaking changes will become issues for users at this point and we will need to begin carefully considering modifications to the application that change or remove existing functionality.
The next thing that happens is the release is deployed to the test environment as usual. Rather than deploying a specific branch, the release master will deploy a specific tag. The QAs will begin to test the release and the developers will continue to merge other chunks of code in to the main branch as usual.
Every time a ticket is reopened, a developer will checkout the latest tag on the main branch by version number5—then just create a new branch at that point, make the code change, and submit a merge request in to the main branch, as usual.
5 The latest tag can be easily found by checking
git tag --list. Or if the developer wants to really streamline their workflow, they could create aliases to run composite commands such as:
git checkout $(git tag -l --sort -version:refname | head -n 1)
Now, when the other developers approve the change, one extra thing will happen before the branch is merged: The developer will create a new tag incrementing the patch version number against the HEAD of the patch branch.
These merge requests for release patches should probably be clearly labelled with the label "PATCH" (or some similar precaution) to ensure that this happens.`
Once the fix has been merged to the main branch, the latest tag will again be deployed and the corresponding ticket will be moved back to "In Test" in JIRA for the QA team to re-test.
Once every ticket that was re-opened is verified and the QAs have confirmed that they have found no regression errors, the latest tag by version number is checked out, a branch is created, and a merge request is opened in to the master branch.
Once the code is merged to master, that means the QAs have confirmed every chunk of code in the release, performed a full regression test, and have signalled that they are happy for that version of the code to be released to production.
There are two key differences to this workflow from the existing one:
-
All fixes to proposed releases are merged in to the main branch with everything else, meaning there is no divergence
-
Multiple releases can be cut at the same time, by incrementing the major version number, meaning that releases no longer block each other (as much)
The advantage of the first difference should be obvious: when the release is deployed, there is no potentially complicated merge that needs to happen afterwards. This means issues introduced by combining code that opposes or conflicts with code in the main branch no longer exist. The main branch is constantly in development and everything even in release candidates is available every time any developer pulls.
The advantage of the second different is much more subtle but arguably much more important: when issues with an ongoing release are being solved, another release containing different chunks of code can be cut. Even if it doesn't get deployed and tested straight away, it means the development team can manage the size of future releases regardless of whether the ongoing release is complete.
Releases do still need to be deployed to production sequentially. But it means that releases do not impose on each other; fixes can be made to ongoing releases and upcoming releases concurrently and can potentially be tested in parallel and in isolation.
This has the amazing benefit of negating the exponentially worsening feedback loop that we described earlier. Delayed releases no longer increase the amount of code included in the next release, which means the risk of issues being found in the next release decreases, which increases the likelihood of a quick, release with few issues ultimately causing fewer delays in future. It's essentially the exact opposite!