“We absolutely must leave room for doubt or there is no progress and no learning. There is no learning without having to pose a question. And a question requires doubt. People search for certainty. But there is no certainty. People are terrified–how can you live and not know? It is not odd at all. You can think you know, as a matter of fact. And most of your actions are based on incomplete knowledge and you really don’t know what it is all about, or what the purpose of the world is, or know a great deal of other things. It is possible to live and not know.” - Richard Feynman
Recently I gave an interview to Duncan Nisbet from Lets’ Test which ranged in topics from my role at Barclays, the work we are doing with Per Scholas, and my talk on “Testing for Confidence” for EuroSTAR. Thomas Hulvershorn had a great comment and observation on Facebook about the idea of providing information without weighing in on release decisions;
“There is one thing I can’t get my head around. I totally get that if your mindset is geared to establish confidence in a build, you’ll miss the point of testing. You are basically checking not testing and on top of that you are hoping that you don’t find bugs. You put that very nicely in your talks and I take that on board.
I do however struggle with the ‘we provide only information’ bit.
In my field of operation it is expected of me to provide a Sign Off recommendation purely from QA point of view. Basically outlining, what has been tested, do we think we did enough testing, what bugs were found, what bugs were left open and then as a final verdict how confident we are that the build is ready to go live.
I can’t really see anything wrong with that tbh. I feel it is our job as Test Managers to provide an objective view of the quality of the build and to say: Good to go from a functional and non functional stability point of view.
Who else should make the call if not us?”
Excellent questions and very valid points. I will try to discuss these issues in turn and break his question into parts to make my response coherent, but this should really be discussed in an online forum of some sort. Also, I would like to add that Michael Bolton has (as in most topics), covered this specific topic much more extensively than I, and is a great source of information in general.
When dealing with situations where it is “expected” to “provide a Sign Off recommendation purely from QA point of view”, I think it is important to differentiate between attesting to the activities that were conducted during testing and providing “sign off” on a release. The former is part of telling the testing story which is an essential skill to develop and gain credibility with your stakeholders, project teams, and peers. The latter, or providing “sign off”, in my opinion is not something the test team should do, particularly due to the source of the information.
Testing provides only part of the information that should be used to make release decisions, but unfortunately, too often (particularly where testing is called “QA”), there is a perception that it is a holistic view of quality. Even information about “what has been tested, do we think we did enough testing, what bugs were found, what bugs were left open” is not enough information to make “a final verdict how confident we are that the build is ready to go live.” There is also a real danger in adding fuel to the bias “fire”, by putting yourself in a position of commenting on the quality of your own work.
As a Test Manager, you might feel its your job to “provide an objective view of the quality of the build”, which is a perfectly reasonable position to take, especially if you believe that testers are stakeholders in the company and invested in the success of the project. I would assert that being invested and maintaining objectivity are not mutually exclusive, and in fact, to function properly in your role, its is crucial to mind and manage your own bias. I believe that testing should provide an objective view of the quality of the build – where I differ is my view on WHO should be forming (and communicating) that view.
So when the question arises, “Who else should make the call if not us?, my answer is simply: your stakeholders. Being invested in a project is not only measured by helping to get a release out the door. It is equally (if not more so), about knowing your role and doing the best you can at delivering value to the organization in that role. Putting the “QA” stamp of approval on a release does not leave any room for doubt , and can set an unrealistic expectation that quality has been “assured”. Is there “harm” in providing “sign off” from a testing perspective? Maybe not, but the likelihood of misinterpretation and and a false impression of certainty are too great a risk and probably outside the remit of objectivity.