The debate around ISO Standard 29119 has intensified after James Christie gave his talk at the Association for Software Testing conference this year, which started a movement organized by the International Society for Software Testing and a petition to stop the publication of ISO Standard 29119. The following is a transcription of a roundtable discussion between veterans of the software testing industry to set the context for the opposition.
Keith Klain (KK) – Thanks everyone for joining, I wanted to set some context for the discussion for people who might not be familiar with ISO 29119 or the petition to stop its publication. James, you gave a talk about 29119 at CAST this year which spawned the recent movement, can you give us a brief overview of your thoughts on the standard?
James Christie (JC) – What lots of people object to is it’s an attempt by one faction to define itself as being the very embodiment of responsible, professional testing. The ISO working group have effectively defined those who disagree with them as being irrelevant at best and by implication, when you see some of the marketing material that’s pushing ISO 29119, that those who don’t comply as irresponsible and unprofessional.
If enough big companies and governments insist its suppliers are compliant, then its going to force testing down to a common and low standard, it will just be a commodity that will be easily bought and sold. It will be low quality, low status work, the sort of profession that I don’t want to work in.
Iain McCowatt (IM) – And there’s precedent for that, in terms of standards getting this kind of adoption. If you look at ISO 9000 as the example, that was pushed heavily by the UK government. The Department of Trade and Industry made grants to companies in order to help them get registered through ISO 9000, and it soon became something that you needed to register through in order to sell to the government, to the Ministry of Defense, and to many other companies.
It became a cost of doing business for many companies if they wanted to do business in Europe, and it was seized on by Offtel, as part of their regulatory regime in telcos in the UK. Ultimately, there were more than 1 million companies registered worldwide by the early 2000s. So standards can be seized on by politicians and regulators and pushed down people’s throats in that way.
Michael Bolton (MB) – The NIST in the United States is thoroughly cognizant of that too, in their communication, they refer to the fact that sure, standards are voluntary and not mandatory, but that gets slippery when a regulatory refers to them as an example of guidance documentation or recommendation of guidance documentation, or mandatory. The NIST is aware of that, and it’s interesting to see the extent to which they are concerned about it or not. The specific words are,
“Still another classification scheme distinguishes between voluntary standards, which by themselves impose no obligations regarding use, and mandatory standards. A mandatory standard is generally published as part of a code, rule or regulation by a regulatory government body and imposes an obligation on specified parties to conform to it. However, the distinction between these two categories may be lost when voluntary consensus standards are referenced in government regulations, effectively making them mandatory” standards.”
I’ve certainly seen clients who are basically victims of goal displacement where extra time and extra energy is spent conforming to the standard and that’s a distraction from the work of actually doing testing.
Ilari Aegerter (IA) – For those testers who are wondering why they should care about ISO 29119. Imagine you were a ballet dancer, and the “ballet standardization body” has decided that you now have to wear a straight jacket for your performances – and they just happened to be in the straight jacket selling business. I believe that this will significantly the same effect with the 29119 standard. So if you were a ballet dancer, would you object to this standard? Of course you would, and in the same way, a software tester needs to object to imposing bodies of the standards working group.
Griffin Jones (GJ) – The FDA has been really very good in terms of not propagating and imposing standards on our particular industry. And I find it really interesting that at the moment, FDAs been reluctant to cite external standards around software and they are really letting the industry develop and observing what’s happening instead of attempting to standardize around a particular set of documents or practices.
KK – Do you think that is directly about the ISO standard, or can you expand on that a little bit?
GJ – I’m in a strange place, because my regulator is saying “whoa, lets not standardize right now, lets observe what’s happening in the industry, and write guidance’s that open up the possibility of other practices”, as opposed to ISO 29119, which is attempting to impose across all industries a single way of doing things.
MB – Interestingly the FDA is in the position where they’re supervising the work of stuff that might rapidly kill people.
GJ – They’re also personally and organizationally responsible to a political body and the public for their choices and actions, both in what they chose to regulate and chose not to regulate. So that degree of responsibility I think reshapes the way that make their choices as opposed to ISO, which does not have that degree of responsibility.
IM – Griffin, do you think their (the FDA) stance would change if another large scale public failure?
GJ – That is the pattern of FDA, they tend to be driven by large, public failures of that kind that drives new regulation, and however they move relatively slowly, and with deliberation, so I wouldn’t expect a knee-jerk reaction.
KK – To add to that point, I’ve managed dozens of regulatory audits in financial services, and just recently, in the last two to three years I’ve started to see the ISO 829 standard, particularly in Asia, as a template for reviewing our test approach. So when we had failures, particularly high or medium profile ones that would get picked up by the regulators, and they would come in and audit us, they would use that as a template for review. I’ve actually had one regulator have the standard printed off in the review, so seeing the expansion with 29119 gave me real cause for concern.
IA – Well one thing is that it’s expansive but restricting at the same time. If you are using agile, then just goodnight, there is no way such a production machine of paper piles would ever fit into the agile mindset that wants to eliminate that sort of wasteful behavior like the production of procedural blocking stones. So it is expansive but also very restrictive and doesn’t fit into many contexts that are to be found in the software testing business.
IM – To Keith’s point, I’ve worked with a lot of clients, I’ve worked with a lot of project and program managers who take the view, well, there is a standard, why don’t you use it. And I think that’s exactly the kind of reaction we will expect to see either amongst the politicians or regulators if there is a disaster, or we would expect to see play out in court if an organization were sued over the quality of their software. And it came to light that they hadn’t tested within “internationally agreed standards”, I think that could potentially quite damaging for them.
JC – That’s something they have been pushing explicitly in their presentations, this appeal to fear. What would we do if there were big problems and you were asked why didn’t you follow the international standard.
IM – It is the classic “fear, uncertainty and doubt” strategy.
KK – That’s an interesting point though, because if you look at something like Healthcare.gov or any other high profile failures that impacted the public from a non-safety critical perspective. There hasn’t been a backlash to or swing towards this kind of stuff, even though I was half expecting it with Healthcare.gov, but it hasn’t happened yet. So although I think fear is a strong driver for how they are presenting this, but so far we haven’t seen it.
MB – Well they move so slowly when they are changing stuff, it takes them a couple years to catch up to this sort of thing. One of the things that’s really interesting, If you look at the presentation materials, they’ve remained remarkably consistent over the last six or seven years. To the degree that one wonders, if they’ve been paying attention to anything that happens in the marketplace at all.
They don’t actually cite this stuff, they don’t cite disasters in particular, they just have this sort of vague, ominous, “well, what IF you run into some kind of disaster”, and I think part of the reason for that perhaps, is that some of the companies who are promoting the standard so aggressively are exactly the ones that failed.
KK – Well, I’ve brought this up before about Healthcare.gov, that both vendors that built the system, one of them actually has a quality and certification practice to help your business meet standards. They are some level of CMM or TMM maturity, and both have fully compliance to international standards on their websites, and that’s one of their marketing angles. And clearly that’s proven to be completely useless when it comes to actually managing testing.
GJ – A lot of these big disasters are not actually software failures per se, or software testing failures, if you dig into James Reasons books about gigantic organizational failures, you discover that its not really the technical people at the pointy end of the stick that are the issue, it’s the large organizational, cultural factors that are causing the processes and the people to behave in ways that ultimately lead to these types of disasters.
KK – That’s a really interesting point, Griffin, so what you’re basically saying is that adherence to process and standards can drive dysfunctional behavior in an organization.
GJ – Absolutely, it creates the appearance of compliance and safety. The organization becomes lax, and what happens is, in the Swiss cheese model, one of these opportunities for disaster occurs, goes through all your safety systems and no one catches it and you end up with planes falling from the sky or the chemical plant emitting a toxic plume. But these are not technical failures per se, these are really organizational issues, and what we are seeing from the technical side are merely symptoms not causes.
IM – This is one thing that really scares me about standardization, this implicit assumption that the process will save us, and that if the process is good, the software will be good.
GJ – When I talk to executives, senior executives, especially risk management people, they are in those positions because the understand that those systems are fallible and they are always looking for “how will the plant blow up” even though everything looks and appears to be good. When you talk to senior executives, they get that – hopefully. Middle management and the organizations that are at the operational level tend to have less awareness of the fallibility of their processes and systems, and they tend to believe what their gauges and dials and processes say.
JC – This is really interesting the way this conversation has developed. Have any of you read the Cynefin framework? It’s a bit too complicated to go into now, but it helps us make sense of our problem by deciding which of four spaces in a quadrant it fits into. The spaces are Obvious, Complicated, Complex and Chaotic. There’s a huge temptation to think our problem belongs in the Obvious space, where everything is predictable and ordered, when our problem is Complicated or Complex. The danger is you just fall over into the Chaotic zone. Standards are a good fit for the Obvious zone, but not the others and they are where most software is developed.
KK – To shift gears a little bit, what do you all think about some of the arguments made in favor of the standard like common language across industry and methodologies?
MB – This whole common language trope is really quite silly. Because what it ignores is the fact that as a common language for testing, as different people from different disciplines come together on a project, the subject matter experts, the program managers, the development managers, the developers, the testers. They are all bringing different languages in every time, they’re all bringing different domain languages all the time and certainly when it comes to the business they are doing.
So what if there is a common language for testing? The programmers, the project managers will not have had the ISO standard training, and who’s to say how they should speak? Its at the stage where its as if they are trying to declare a world common language for testing, ok, lets make it Hindi! Lets make it Chinese!
IA – The standardization of language has examples of failures and it also addresses a non-problem that the clarification of what you mean is done in human interaction. That’s done actually very efficiently if people are paying attention, and having an external body dictating some sort of terminology which is not even to the knowledge of everyone involved is just a complete failed attempt for something for which there is no need for.
KK – I hear this same argument from people who are trying to centralize testing or create a “center of excellence” from vendors who are interested in consolidating everything, and this common language drive comes up quite a bit. And frankly in my twenty years of doing this, I have never once seen a project have a problem because we all didn’t use the same testing terminology. That is a phantom problem that leads into a secondary issue here, which is why should people outside of testing care about this standard?
IM – I think, Keith, the bigger problem will arise from people thinking they are speaking the same language but actually they are not. If you look at part one of the standard, the level of language that it’s defining is really quite trivial. Lets face it, we can all pretty much agree at a very trivial level what system integration testing is, and we’d all happily use the term. But I’ve run into all kinds of problems when people have different assumptions about by what they mean by system integration testing, which systems are they talking about, what integration points are they talking about, and so on and so forth.
MB – But wouldn’t a standard help? The devils advocate question, wouldn’t a standard language help in that circumstance, and if not, why not?
IM – So Michael how does a standard language help define the scope of system integration testing in any particular project I have in my portfolio? It doesn’t. Because the standards people have absolutely no concept as to my context, or what we’re trying to achieve, or what systems we are integrating. We can only work that actually by working together as people and working through it and working to get some common understanding around a particular project.
MB – Here’s a justification I heard from one consultant, and honestly this is true. She said to me, when I go into a new company they are using all these different terms, and it’s so hard to figure out what people are saying. My response to her was, boy, if that’s your biggest problem as a consultant, you’re going to face an awfully rough ride.
Michael Bolton is a consulting software tester and co-author (with senior author James Bach) of Rapid Software Testing, a methodology and mindset for testing software expertly and credibly in uncertain conditions and under extreme time pressure. Michael has 25 years of experience testing, developing, managing, and writing about software. http://www.developsense.com
Iain McCowatt is one of the founders of the ISST, and the author of the petition to stop ISO 29119. His day job is as a director in a bank, helping large enterprise IT programmes to solve complex testing problems and gain insight into the quality of their software. http://exploringuncertainty.com/blog/
Griffin Jones is an agile tester, trainer, and coach, who provides consulting on context-driven software testing and regulatory compliance to companies in regulated and unregulated industries. Owner of Congruent Compliance, Griffin has been participating in software development for over twenty-five years. http://www.congruentcompliance.com
James Christie has over 30 years of experience in IT, covering development, IT audit, information security management, project management and testing. He is now a self-employed testing consultant based in Scotland. http://clarotesting.wordpress.com/
Keith Klain is the CEO of Doran Jones Testing and has over 20 years of multinational experience in enterprise-wide testing programs, Keith has built and managed global test teams for financial services and IT consulting firms in the US, UK, and Asia Pacific.www.qualityremarks.com
Ilari Henrik Aegerter is President of the International Society for Software Testing where he wants to bring back common sense into testing and oppose wasteful practices. He has been in software testing for the past 10 years, most of the time as a manager of testers. http://www.ilari.com/
James Denman writes, edits, and manages the production of content for SearchSoftwareQuality.com. His job is one part editor, one part reporter, one part copywriter, and three parts whatever else needs doing. http://searchsoftwarequality.techtarget.com