An open letter to the ISTQB

Date: April, 25 2013

To: ISTQB BOD

Cc: ISTQB Governance Working Group

Subject: Open Letter to the ISTQB

To whom it may concern;

Recently a discussion transpired over Twitter regarding the validity and governance of the Foundation level exam you offer through your training partners. Rex Black, a current board member and past president of the ISTQB, was involved in the exchanges and made the following comments in response to my queries about whether there have there ever been problems with the certifications validity, specifically the reliability coefficient:

@RBCS: ASTQB works with professional exam consultants (psychometricians) to ensure statistical validity

@RBCS: They are reviewed continuously by ASTQB. Nothing is perfect, but exams are constantly perfected

@RBCS: NDA prevents a detailed response. Is “nothing is perfect” not clear enough? :-)

@RBCS: Non-disclosure agreements prevent detailed answers; I have answered as directly as I can.

Rex did not answer my questions purportedly due to an non-disclosure agreement. Per the ISTQB website, you are a “non-profit association” dedicated “to continually improve and advance the software testing profession”. It would lead me to believe that your values of “openness” and “integrity” would mean answers to those questions are vital to maintaining your charter.

So I appeal to this board for answers to what seem to be straightforward questions:

1) Have there ever been issues with the ISTQB Foundation exam reliability coefficient reviewed by your exam consultants Kryterion?

2) Have the reliability co-efficients consistently shown, since the inception of the ISTQB’s certification program, that results on the certification exams accurately measure the testers’ knowledge of the syllabi?

3) Have there ever been any other issues with the validity of the exams?

4) How often do those external reviews take place?

5) Are the results of Kryterion’s (or a third party’s) independent evaluations publicly available?

Rex suggested that acknowledging potential obstacles for testers due to issues with the Foundation certification was akin to being “prejudice (sic) against the 300,000 people who have ISTQB certs.” I would assert that not answering basic questions about threats to the certifications validity does those 300,000 people a greater disservice.

Thank you for your help in getting these questions answered.

Best regards.

Keith Klain

  • http://twitter.com/charrett Anne-Marie Charrett

    Hi Keith,

    can you explain some of the terminology in your blog post?

    1) What is the ISTQB Foundation exam reliability coefficient?

    2) Who are Kryterion and why are they involved?

    cheers

    Anne-Marie

    • KeithKlain

      Thanks, Anne-Marie, the reliability co-efficient is used to assess whether someone has mastered the learning objectives of an exam. Per Rex Black, Kryterion is the firm that the ISTQB uses to validate its exams. Thanks – KK

      • http://twitter.com/imccowatt Iain McCowatt

        Kryterion is also a firm that provides examination services (e.g. electronic exams): http://www.kryteriononline.com

    • http://twitter.com/charrett Anne-Marie Charrett

      Anyone who wants to understand a little background on this, I found this website gives a basic overview. http://fcit.usf.edu/assessment/basic/basicc.html

  • http://twitter.com/rajeshmathur rajesh mathur

    Hi Keith,

    Thanks for raising the questions to ISTQB. I raised similar questions few years ago through my (now hacked and deactivated) blog. Obviously there was no response.

    “Can someone call the producer of ScamCity please?”

    Cheers
    Rajesh

  • Edward Bishop

    ISTQB-watchers may be interested in my editorial in the current issue of Professional Tester magazine. http://professionaltester.com/files/PT-issue20.pdf, see page 3.

  • http://twitter.com/YorkyAbroad Simon Morley

    Keith, I like the ambition to get answers. But.. you’re making an assumption that the organisation’s values/interpretation of “openness” and “integrity” are similar to yours. If they’re not then they might not feel any urge to answer.

    I would be interested in seeing an analysis / example of how some example exam questions relate to the learning objectives of the syllabus. Having looked at the foundation level syllabus (and its learning objectives) I’m dismayed at the ambiguity and unclear assumptions being made… So, seeing how a connection between a given learning objective and a set/range of example questions would be useful.

    But, that doesn’t mean the organisation would feel compelled to give out such information – in fact, having looked at one syllabus I’d be inclined to not reveal it (if I was them)….

  • Arslan Ali

    For people seeking out the Training / Questions / Syllabus comparison scenarios, here is an exercise that we carried out:; and the reason why we should sign the Petition!

    We recently did two (2) “Poll” based exercises on LinkedIn; we placed two questions with their multiple choices as answers from ISTQB Sample question paper.

    As no one was aware that the questions were asked from the Sample Paper, the results were really surprising. 80% of professionals, even the certified ones did not select the right answer as per the Sample Paper!

    In addition, it also triggered a discussion regarding why most of the experienced testers and professionals never selected the right answer? The conclusion was very clear; we were answering on the basis of (a) The Real Experience and (b) On the basis of context that the question was providing;

    Question 1:
    Which of the following, if observed in reviews and tests, would lead to problems (or conflict) within teams? Where Testers and Reviewers:

    Not CURIOUS enough to find bugs – Votes 2 (16%)
    Not QUALIFIED enough to find bugs – Votes 1 (8%)
    REPORT bug as person not product CRITIC – Votes 5 (41%)
    EXPECT bugs are already fixed by developers – Votes 1 (8%)
    Does not FOLLOW-UP the reported bugs – Votes 3 (25%)

    Question 2:
    Which of the following statements is the MOST valid goal for a test team?
    Enough component testing was executed – 3 Votes (21%)
    Cause as many failures as possible… – 4 Votes (28%)
    Prove that all bugs are identified – 4 Votes (28%)
    Rest of bugs will not cause crashes – 3 Votes (21%)

    My question is;

    why is there a distribution of votes? Why people tend to think differently in contrast to the “Best Practices” posed by the Sample Paper and given Syllabus? What I believe, is that the certification is “trying” to create a trend on the basis of their set “Best Practices” and is actually forcing everyone to prepare the exam and then do the certification; which is wrong!

    Similarly, often I have seen people using the same set of questions in interviews and are raising the bar on the basis of the certification answers as the “right” answers, even though the interviewee is providing a correct answer on the basis of his/her experience and the given context.

    So we need to bring in that change!

  • Pingback: Certifiable – Fighting the fights worth fighting… | Quality Remarks

  • Pingback: Strange Bedfellows | Quality Remarks

  • Pingback: The value of the ISTQB certificates | Test Pappy