This article is the follow-up discussion to the relevancy of formal Quality Assurance with products following the continuous delivery development and deployment model. The first article is located here
In the last article, I described some of the development and release challenges one of my teams faced following what I assume is a fairly traditional, small team process. I concluded with a brief run-down of how those challenges were mitigated through a gradual switch to a more continuous style release and delivery process.
While we were developing our own implementation of continuous release, the VP of Development tasked the dev leads to read “Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation” (Addison-Wesley Signature Series (Fowler)). It is a thought provoking book that explains continuous delivery from A – Z. One of the core concepts is a discussion on source control branches and how they should be used sparingly. The following excerpt clearly summarizes that sentiment:
“Branching should, in most circumstances, be avoided”
I’m not going to dive into the details, but the central idea is that branch usage delays feedback on the correctness, or deployability, of any given software change. The book asserts that this delay can be the source of many production defects. Instead of branches, the authors advocate pushing code changes directly into the main trunk and then deploying to production as soon as possible.
Of course, I’m leaving out a many details. In particular there is a significant amount of “process plumbing” one needs to have in place to manage source code like this while still maintaining quality. But overall, the core concept is that a developer will create, test, push, and then immediately fix (if required), all in one continuous block of time.
Take a minute to think about the concept I just stated and what it implies.
If a developer is not working in a source control branch, and always commits to master … if the deployment process is continuous and code release is immediate … then that infers the changes *just* coded are immediately pushed to a production environment (after running automated tests, of course).
If we take that statement as fact, then when does QA occur?
For certain, high value or high risk features, QA would happen right when it does now – in a distinct phase before the code is pushed to production. However, for most of the “normal” feature releases, the answer I took away from the book is that QA occurs on production, after the code is released! I must admit that I’m not quite ready to take that plunge yet. I’m sure much of my reluctance is based purely on the unknowns associated with such – in my thinking – a radical process change. That said, a senior developer approached me recently with a request to take the final step towards pure continuous integration and do just this.
As I mentioned above, we still QA within a distinct phase before deployment to production and that process does catch bugs. If I agreed to move to a fully continuous process, all those bugs would be deployed and potentially found by customers. That is not acceptable and my current thought is that we cannot migrate to a fully continuous process until we’ve figured out how to squash more of these defects. But in addition to bug squashing, the team is looking at ways to tie features to permissions, or switches such that it can be deployed to production in an “off” state. This way, the feature can undergo final QA on production, but not be consumed by the general customer base. Once QA is complete, the feature can be turned on at will.
Back to my original question: is QA still relevant? I believe yes. However, I also believe the role changes slightly. In my experience, teams that develop without dedicated QA spend a lot of effort trapping bugs. Effectively, the developer is the QA team. When such teams acquire formal Quality Assurance, their testing practices inevitably grow lax. “QA will catch it”, is a common sentiment I’ve often heard throughout my career. (I might have uttered this phrase myself once or twice, particularly during a crunch period). With continuous integration, developers need to return to a more disciplined style of testing, leaving QA to do what I believe is its true purpose: to find those hard to find, esoteric bugs that developers don’t have the time to test for themselves.