When setting up a new test project, the question usually soon arises as to which documents should be created. Here, one’s own quality management system (QMS) or, for example, that of the customer can provide the answer, if a QMS exists at all.
From IEEE 829 to ISO/IEC/IEEE 29119
Many testers know and follow the IEEE 829 “IEEE Standard for Software Test Documentation”. There are extensive materials based on IEEE 829. One can find easily adaptable examples on the web or in books. There are presentations, YouTube videos, … And so I notice again and again that even interested testers/test managers (especially if they have been in the business for a long time) have not noticed that the beloved IEEE 829 has been superseded. Continuing to follow IEEE 829 is certainly better than testing and documenting without a concept. However, for new projects or whenever the question of “state of the art” might arise downstream (regulated market, failed contract work, etc.), it should be considered to implement the current standard. Since 2013, IEEE 829 has been replaced by ISO/IEC/IEEE 29119-3 “Software and systems engineering – Software testing – Test documentation”. And now is the best time to switch or get in, because ISO/IEC/IEEE 29119-3:2021 is freshly released in October and available for purchase. However, 29119-3 does not come alone. As can easily be surmised, it is the third part of a series of standards. Those who want to be even better also put 29119-2:2021 alongside it, which explains how to get to the documents. Those who want to remove any doubt will obtain all 5 parts of the standard. If the costs are too high (approx. 200€ per part of the standard) or if you have (unfounded) respect for reading a standard, you should read the excellent book “ISO 29119 – Understanding and applying software testing standards” by Matthias Daigl and Rolf Glunz. It begins with the promising words, “A book about a new standard and how to apply it in practice sounds attractive and exciting – though certainly not for everyone equally.” In addition to an enthusiastic introduction to the standard, the book makes the case for why the standard is worth the price and will quickly provide a return on investment (ROI). In other words, after purchasing the book, you will want to obtain at least individual parts of ISO/IEC/IEEE 29119 after all (this is what happened to me). And you can not get around the acquisition. Unlike nouns of the ISO 25000 series, for which detailed content is published here https://iso25000.com, there is no such thing for the ISO 29119 series.
Contents of ISO/IEC/IEEE 29119-3:2021
The standard describes the following documents:
- Test policy
- Organizational test practices
- Test plan
- Test status report
- Test completion report
- Test model specification
- Test case specification
- Test procedure specification
- Test data requirements
- Test environment requirements
- Test data readiness report
- Test environment readiness report
- Actual results
- Test results
- Test execution log
- Incident report
First, all document types are explained in terms of content. In another section, all information elements are listed in terms of the necessity of their use. These differentiations exist:
- mandatory
- recommended
- possible
Then follows an example for each document type for an agile and a classic project. Finally, to facilitate the transition from IEEE 829, a mapping between both standards is part of ISO/IEC/IEEE 29119-3. The explanatory chapters refer to each other and indicate which information could also be incorporated into other documents, should one decide to omit individual documents. Sensible tailoring while maintaining the spirit of the standard is thus always already considered and conveyed. Decisive innovation in this version: The concept “test model” replaces the concept “test condition”. I assume that this will be discussed extensively in the professional world, so it will not be a topic in this article. The standard is also particularly suitable as a basis for structuring an order if software testing is to be handled by a contractual partner.
Criticism of ISO/IEC/IEEE 29119-3:2021
Despite all the enthusiasm, however, as a quality assurance specialist I must also express criticism:
- The required information is listed in great detail.This gives the feeling that further information is superfluous. In the area of test execution documentation, however, I miss a reference to the version of the test object (System Under Test, Application Under Test, Device Under Test, etc.). Here and there, imprecise version designations are used. In projects with iterations, however, experience has taught me that exact build designations are worth their weight in gold for tracing test, error and repair progressions.
- The fact that the term “incident report” hides the commonly known “bug report” is introduced late in the descriptive chapters. This is surprising, since “bug-tracking tools” are then written about as a matter of course.
- As a QA engineer, I read and review documents holistically. In the case of 29119-3:2021, I was disturbed by linguistic anomalies. Especially in the examples, there are nested tapeworm sentences, faulty grammar, and carelessness errors (typos).
- In the descriptive chapters there is no good differentiation between product risks in the sense of “risk based testing” and project risks of the development or test project.Example risks are for the most part only the description of situations that can lead to damage instead of the actual damage. For example, “Incomprehensible manual” is not a harm. “Wasted fertilizer” is a damage that can be caused by an incomprehensible manual, among other things. But a risk is defined as risk = severity of damage x probability of occurrence of damage. So without a concretely named damage, I cannot name a risk. Certainly, ISO/IEC/IEEE 29119-3 is not a standard for risk management. However, incorrect examples make it more difficult for many people to think in terms of risks, which they are already unaccustomed to.
- In the introduction of the examples it is explained that they are not consistent with each other. Each is to be seen as a single one on its own. I find this inconvenient since the documents are described for two example companies outlined earlier. For clarity, consistency within each example company would already be convenient. Presumably one did not want to expose oneself to the danger of having overlooked inconsistencies.Because the examples are partly already inconsistent in themselves. However, the experienced tester will recognize this and correct it in his own documents.
Conclusion
The 29119-3 is very practical and brings detailed examples that can be easily adapted and used for agile or classic projects. I find the standard useful and helpful and this judgment is not limited by the mentioned criticisms.