“We absolutely must leave room for doubt or there is no progress and no learning. There is no learning without having to pose a question. And a question requires doubt. People search for certainty. But there is no certainty. People are terrified–how can you live and not know? It is not odd at all. You can think you know, as a matter of fact. And most of your actions are based on incomplete knowledge and you really don’t know what it is all about, or what the purpose of the world is, or know a great deal of other things. It is possible to live and not know.” – Richard Feynman
Recently I gave an interview to Duncan Nisbet from Lets’ Test which ranged in topics from my role at Barclays, the work we are doing with Per Scholas, and my talk on “Testing for Confidence” for EuroSTAR. Thomas Hulvershorn had a great comment and observation on Facebook about the idea of providing information without weighing in on release decisions;
“There is one thing I can’t get my head around. I totally get that if your mindset is geared to establish confidence in a build, you’ll miss the point of testing. You are basically checking not testing and on top of that you are hoping that you don’t find bugs. You put that very nicely in your talks and I take that on board.
I do however struggle with the ‘we provide only information’ bit.
In my field of operation it is expected of me to provide a Sign Off recommendation purely from QA point of view. Basically outlining, what has been tested, do we think we did enough testing, what bugs were found, what bugs were left open and then as a final verdict how confident we are that the build is ready to go live.
I can’t really see anything wrong with that tbh. I feel it is our job as Test Managers to provide an objective view of the quality of the build and to say: Good to go from a functional and non functional stability point of view.
Who else should make the call if not us?”
Excellent questions and very valid points. I will try to discuss these issues in turn and break his question into parts to make my response coherent, but this should really be discussed in an online forum of some sort. Also, I would like to add that Michael Bolton has (as in most topics), covered this specific topic much more extensively than I, and is a great source of information in general.
When dealing with situations where it is “expected” to “provide a Sign Off recommendation purely from QA point of view”, I think it is important to differentiate between attesting to the activities that were conducted during testing and providing “sign off” on a release. The former is part of telling the testing story which is an essential skill to develop and gain credibility with your stakeholders, project teams, and peers. The latter, or providing “sign off”, in my opinion is not something the test team should do, particularly due to the source of the information.
Testing provides only part of the information that should be used to make release decisions, but unfortunately, too often (particularly where testing is called “QA”), there is a perception that it is a holistic view of quality. Even information about “what has been tested, do we think we did enough testing, what bugs were found, what bugs were left open” is not enough information to make “a final verdict how confident we are that the build is ready to go live.” There is also a real danger in adding fuel to the bias “fire”, by putting yourself in a position of commenting on the quality of your own work.
As a Test Manager, you might feel its your job to “provide an objective view of the quality of the build”, which is a perfectly reasonable position to take, especially if you believe that testers are stakeholders in the company and invested in the success of the project. I would assert that being invested and maintaining objectivity are not mutually exclusive, and in fact, to function properly in your role, its is crucial to mind and manage your own bias. I believe that testing should provide an objective view of the quality of the build – where I differ is my view on WHO should be forming (and communicating) that view.
So when the question arises, “Who else should make the call if not us?, my answer is simply: your stakeholders. Being invested in a project is not only measured by helping to get a release out the door. It is equally (if not more so), about knowing your role and doing the best you can at delivering value to the organization in that role. Putting the “QA” stamp of approval on a release does not leave any room for doubt , and can set an unrealistic expectation that quality has been “assured”. Is there “harm” in providing “sign off” from a testing perspective? Maybe not, but the likelihood of misinterpretation and and a false impression of certainty are too great a risk and probably outside the remit of objectivity.
Thanks for following up on this Keith.
I like to elaborate a bit on this “how confident we are that the build is ready to go live.”
On our end the overall sign off is much more than just the QA (or better Test) report.
It contains checklists from the Producer (aka Project Manager) in regards to his tasks for preparing the build and providing support assets for publishing on the App Stores. Also involved are the Backend Server team, the Marketing people, Operations Manager and so on.
Based on all these inputs the final ‘Go’ call is with our Stakeholders on highest company level.
From my own experience in the past 18 years working in various companies and sectors I do however acknowledge that if something goes wrong in production the first shout goes out to the test team ‘how has this been missed in testing?’.
This is the point where I start agreeing with your POV and that’s why this discussion is helpful and thought provoking.
Although the eyes turn to the test team first, it is very rare that the testers have missed very obvious issues.
The Test Manager is usually the one who follows up and investigates the issue
further and traces the root cause of the problem. Experience shows that it very rarely is a miss in testing but a problem affecting the whole production chain.
Test Teams are in danger of constantly having to justify themselves. We are permanently soul searching how we can prevent issues like from happening in the future, while follow ups on production side (‘why was the bug there in the first place?’, ‘why was there a misunderstanding of the feature scope?’…) are less frequent or handled with the same scrutiny.
Often a vicious circle starts forming:
Pressure on Test is met with Test Teams writing more scripts to safe guard themselves and appear to be accountable. This leads to less intelligent testing and less quality of the tests which leads to more production fails and pressure on Test.
This goes on and on until a point where the time it takes to execute these ‘tests’ takes ages and the shout for test automation is heard.
While great for checking existing functionality Test Automation takes away precious resources that could be used to do some actual testing.
Nobody is happy at the end, nobody wins.
(for the sake of the argument I’m displaying a black and white view here on Test automation. Done right it definitely has its important place in testing.)
Let’s look a bit more on the bright side:
You wrote, “that testers are stakeholders in the company and invested in the success of the project”.
I am very lucky to say that this is 100% the case for the company I work for.
I work with an internal team that we have hired and built from scratch and that we regard as equal and valued part of the production team.
We all are invested in the company and we love and care for the products we produce. That sounds a bit idealistic and a like a cheap sales pitch but it is something that you find quite frequently in the Games and Entertainment sector.
I have recently moved into the role of Operations Management in our company.
In this role my team and I are responsible for getting our products out in time, in scope, quality and budget.
I’m currently redefining the role of our Project Managers and also encourage
our Test team to make and plan time for exploratory testing with a specifically
Games tailored approach that we developed.
I hope having worked in QA for so long that I can be driving change across the production chain and avoid the problems as mentioned above.
Keith – even though you are objective in presenting your facts and telling the testing story, stakeholders expect test teams / managers “opinion”. When you provide your opinion you will add to the bias. You may not “sign off” but you are still influencing and in the right way.