The debate around ISO Standard 29119 has intensified after James Christie gave his talk at the Association for Software Testing conference this year, which started a movement organized by the International Society for Software Testing and a petition to stop the publication of ISO Standard 29119. The following is a transcription of a roundtable discussion between veterans of the software testing industry to set the context for the opposition.
James Denman (JD) – So is the problem that ISO 29119 is standardizing wrong, or is it that there cant be standards that work?
Michael Bolton (MB) – Well certainly the former, and imagine for example, and this is an metaphor that comes to me from James Bach, so imagine that somebody declared, somebody in the restaurant business declared that there was only one kind of food. One kind of cuisine, we’re going to mandate one kind of cuisine. I’m wondering how the other cuisines would react to that and also wonder how diners would react to that.
This (ISO 29119) is supposed to cover all software development projects, and the veiled threat is that showed up in one of the promotional pieces, was that if you’re working on you’re own in a garage, then this shouldn’t effect you, but otherwise it should. Well, that to me is mischief.
I run a web start up, a little start up company, I am about to sell services to a large institution that takes regulation and standardization very seriously. Well, first of all they don’t discern very well whether this is good standard or not, but should the way I work in my small outfit with my small number of programmers and even smaller number of testers, should I displace the goal of producing high quality software with the goal of producing volumes of documentation and following a process model that doesn’t fit the way we actually build software. It doesn’t make sense to me.
Griffin Jones (GJ) – To elaborate upon your point, Michael, there’s goal displacement also if you look at 29119, I think what’s happening is the test group is attempting to impose its standard across all the other software development disciplines. How can you implement 29119 without impacting product development, the developers, the project management people and all the other roles creating the product?
And yet, did they involve any one of those other roles in the creation of their standard? Did they address those other stakeholders concerns? I don’t see it.
Keith Klain (KK) – That’s really an excellent point, Griffin, and I tried to pick up on this earlier, but the interdisciplinary nature of developing software means you cannot put testing in a box and treat it as a kind of factory. And that creates a big problem when you look at testing as something that can be isolated, and can have disastrous on one, the quality of the software you are trying to produce, but as well, all the other things that happening around it by wasting money, creating useless documentation, goal displacement for the testing group, etc.
It has a much wider implication that just testing, and I’m waiting for when the software development or agile community is going to join the fray on this, because you cannot implement this standard (29119) without having profound affect on development, design, project management, it just can’t be done.
Iain McCowatt (IM) – I think a more realistic outcome Keith, is that testers simply become irrelevant. Imagine a test team saying, we’re now following 29119 and you developers can’t do agile anymore because we won’t be able to test it. You’d be laughed out of the organization.
GJ – The analogy I like to use is, since testing is about finding information, it’s the nervous system of the body. So if 29119 suddenly implements a process that doesn’t allow the nervous system to integrate with the rest of the body, something bad is probably going to happen to the body.
James Christie (JC) – But isn’t there a danger that some big companies or maybe government departments will insist on 29119 compliance, because it’s the only standard they can fix on, there isn’t a development one that they can focus on, so they will insist that suppliers be compliant with that, and therefore, developments would have to comply with that. I noticed a LinkedIn discussion the other day, there was a test manager bemoaning the fact that they were having to comply with 29119 because their customer was insisting on it.
Ilari Aegerter (IA) – There are other examples of that in the certification business. In Switzerland, the country I’m from, many large corporations, many of them in the financial and insurance sector actually won’t allow people on a tester interview unless they are certified. And I anticipate the same thing will happen with 29119, just on a larger scale because people are willing to refer to something they don’t understand even though it won’t do the job very well.
MB – I’m not sure that any of you have seen this:
ALL – (laughter)
KK – I think this is one of the big dangers here, James (Denman), bringing this back full circle to your point, about is 29119 bad or are standards bad, and I think it’s a bit of both. What you’ll find is that the companies that have ability and resources to make themselves compliant to the standard will spend all their time doing that. And particularly when you’re looking at the public sector where you’ve seen this, in that industry where people are compliant to the standard and deliver terrible services.
I’ve seen this as well in the financial services industry, where they don’t know anything about the actual work, but can see that we ticked all the boxes. So it looks like, “can I show that I’ve met the standard of what a test case is meant to look like?” Absolutely. Can I show you I ran thousands of them? Absolutely. Was that testing worthwhile? Was it good testing? Did we get any interesting information? Well, that’s a much harder question to answer, and the standard will not in any way help answer that question, and in fact, will mask, and has a strong possibility of actually clouding your judgment. Because a standard, particularly in knowledge work, gives the appearance of it being good.
IM – That’s right Keith, it’s a placebo. Testing can be hard and testing can be complex, but a lot of people who buy testing want it to be straightforward and simple and the kind of thing you can wrap up in a document. That’s the real need that I see fulfilled, it (29119) takes something that’s hard work and involves understanding people and their needs and desires, and it tries to reduce it to some thing that fits on a page in a process model.
And that sells, because a lot of the people who buy this stuff don’t want to understand it, they want simplicity, they want an easy answer.
GJ – It sells also because it works – up to a point.
JC – A major part of my objection to 29119 is because of my background not just in testing, but when I worked as an auditor and also as an information security manager. When I worked in security, one of the banes of my life was auditors and the ISO 2700 family of security standards. The internal auditors would come in and they would expect a common standard driven approach to security, and one of my clients was a pharmaceuticals company that was dealing with the FDA and another one was a high street retailer that had to move really fast.
And the auditors would refuse to recognize the importance of the context the client was in. They were focusing on the ISO standard and the internal variants on the standard. It was a pain for us and was extremely damaging for the client too, especially the retailer who found that key technical staff were being tied up in gold plating security work, particularly against irrelevant risks. They weren’t available to help them move fast in the marketplace and so it was creating real commercial risks addressing irrelevant IT security risks.
When I worked as an auditor, it’s a difficult job, because you have to audit areas with which you’re not familiar. And a standard offers something reassuring to cling on to, auditors are always looking for a benchmark, a basis for their audit, and a standard can provide that and it can give them easy answers. But it doesn’t tell them the right questions they should be asking in a particular situation.
Auditors do cling on to standards, and it stops them growing. It keeps them at poor quality, inexperienced auditors who are just running through the same script over and over again, and the way I see ISO 29119 being pushed, it’s appealing to that school of poor quality and diminishes the role of the good auditor.
GJ – Hear, hear!
IM – It’s interesting to note, James that you mentioned security standards, as its very interesting to note that there is no standard for hacking. Nor is there a standard for writing bugs in applications.
MB – Right! You read my mind, Iain!
ALL – (Laughter)
IM – It amazes me that whilst it’s obvious that you can’t have a standard for putting bugs in that the difficulty of standardising finding then isn’t similarly obvious.
MB – I’m sure there’s a committee working on a hacking and a bug making standard, Iain.
ALL – (Laughter)
KK – Now that’s a standard I could get behind!
JC – One of my colleagues, a fellow security manager made a very revealing comment. It was a very cynical comment, he said, our job was not to protect the clients, it was to protect our company. It was to protect our company’s reputation so that we had our backsides covered if there was a problem. It wasn’t to protect the client. And there is similar sort of mindset with the 29119 lobby, we’re selling peace of mind. It won’t get you better software, but you’ll be bulletproof when it comes to the investigation of why things went horribly wrong because you are following an internationally agreed standard.
GJ – It creates an appearance, but I assert that it’s a paper mache shield for the organization. When the bad thing eventually happens, it won’t matter.
Michael Bolton is a consulting software tester and co-author (with senior author James Bach) of Rapid Software Testing, a methodology and mindset for testing software expertly and credibly in uncertain conditions and under extreme time pressure. Michael has 25 years of experience testing, developing, managing, and writing about software.http://www.developsense.com
Iain McCowatt is one of the founders of the ISST, and the author of the petition to stop ISO 29119. His day job is as a director in a bank, helping large enterprise IT programmes to solve complex testing problems and gain insight into the quality of their software. http://exploringuncertainty.com/blog/
Griffin Jones is an agile tester, trainer, and coach, who provides consulting on context-driven software testing and regulatory compliance to companies in regulated and unregulated industries. Owner of Congruent Compliance, Griffin has been participating in software development for over twenty-five years. http://www.congruentcompliance.com
James Christie has over 30 years of experience in IT, covering development, IT audit, information security management, project management and testing. He is now a self-employed testing consultant based in Scotland. http://clarotesting.wordpress.com/
Keith Klain is the CEO of Doran Jones Testing and has over 20 years of multinational experience in enterprise-wide testing programs, Keith has built and managed global test teams for financial services and IT consulting firms in the US, UK, and Asia Pacific.www.qualityremarks.com
Ilari Henrik Aegerter is President of the International Society for Software Testing where he wants to bring back common sense into testing and oppose wasteful practices. He has been in software testing for the past 10 years, most of the time as a manager of testers. http://www.ilari.com/
James Denman writes, edits, and manages the production of content for SearchSoftwareQuality.com. His job is one part editor, one part reporter, one part copywriter, and three parts whatever else needs doing. http://searchsoftwarequality.techtarget.com