2009 Second International Conference on Computer and Electrical Engineering

An Investigation of the Weaknesses of the ISO 9126 International Standard

Rafa E. Al-Qutaish
Al Ain University of Science and Technology - Abu Dhabi Campus, Abu Dhabi, P.O. Box: 112612, UAE e-mail: rafa@ieee.org
AbstractSince 2005 and up-to-date, the International Organization for Standardization (ISO) is ongoing to update the current ISO 9126 international standard on software product quality measurement. However, this current standard will be replaced by the quality measurement division (ISO 25020, ISO 25021, ISO 25022, ISO 25023, and ISO 25024) of the upcoming ISO 25000 series of international standards on software quality requirements and evaluation (SQuaRE). This paper presents an investigation of the weaknesses of the current ISO 9126 by collecting various viewpoints of interested researchers and practitioners. Furthermore, this investigation will tackle the potential solutions of such weaknesses to be taken into account when preparing the new quality measurement division of the ISO 25000 series of international standards. Keywords-ISO 9126; ISO 25000; software quality; software product

I.

INTRODUCTION

will collect a set of weaknesses from our previous work [8, 9, 10, 11, 12, 13] and from other researchers and practitioners [14, 15, 16, 17, 18, 19, 20]. Furthermore, this paper will provide a set of proposed suggestions to overcome these weaknesses. Thus, these proposed suggestions can be taken into account on the development of the up-coming quality measurement division (ISO 25020, ISO 25021, ISO 25022, ISO 25023, and ISO 25024) of the ISO 25000 (SQuaRE: Software Product Quality Requirements and Evaluation) International series of standards. The rest of this paper is organized as follows: section 2 presents an overview of the ISO 9126 International Standard and its contents. Section 3 provides a general overview of the quality measurement division of the ISO 25000 series os international standards. Section 4 analyses and investigates the weaknesses of the ISO 9126. Finally, section 5 concludes the paper and gives some suggestions for the improvement of the upcoming ISO 25020n on quality measurement which is a part of the ISO 25000 series of international standards. II. ISO 9126: AN OVERVIEW The ISO 9126 series of standards now consists of one international standard (IS) and three technical reports (TR), that is: ISO IS 9126-1 on Software Product Quality Model [2]. ISO TR 9126-2 on Software Product External Quality Metrics [3]. ISO TR 9126-3 on Software Product Internal Quality Metrics [4]. ISO TR 9126-4 on Software Product Quality in-Use Metrics [5]. The first document of the ISO 9126 series (Quality model) contains two-parts quality model for software product quality [5]: Internal and External Quality Model. Quality in-Use Model. The first part of the two-parts quality model determines six characteristics in which they are subdivided into twentyseven sub-characteristics for internal and external quality [2]. These sub-characteristics are a result of internal software attributes and are noticeable externally when the software is used as a part of a computer system, see Figure 1. Whereas, the second part of the two-part model indicates four quality in-use characteristics [2], see Figure 2. However, the definitions of each of these characteristics and subcharacteristics are available in the ISO 9126-1 [2].

In 1991, ISO published its first international consensus on the terminology for the quality characteristics for software product evaluation, which is called ISO 9126 on Software Product Quality Characteristics and Guidelines for their Use [1]. Afterwards, from 2001 to 2004, ISO published an expanded four-part version, containing both the ISO quality models and inventories of proposed metrics for these models. The current version of the ISO 9126 series now consists of one International Standard [2] and three Technical Reports [3-5]. Currently, ISO has recognized a need for further enhancements to ISO 9126, primarily as a result of advances in the field of information technologies and changes in environment [6]. Therefore, ISO is now working on the next generation of software product quality standards which will be referred to as Software Product Quality Requirements and Evaluation (ISO 25000). This series of standards will replace the current ISO 9126 and ISO 14598 series, and will consist of five divisions [7], each of them may contain one or more documents: ISO 2500n on Quality Management. ISO 2501n on Quality Model. ISO 2502n on Quality measurement. ISO 2503n on Quality Requirements. ISO 2504n on Quality Evaluation. In this paper we will investigate the weaknesses of the current ISO International standard (ISO 9126). However, we
978-0-7695-3925-6/09 $26.00 2009 IEEE DOI 10.1109/ICCEE.2009.83 275

Figure 1. ISO 9126 quality model for external and internal quality (characteristics and sub-characteristics).

Figure 2. ISO 9126 quality model for quality in-use (characteristics).

The second [3], third [4] and fourth [5] parts of the ISO 9126 series provide the following information: Sets of metrics for each external quality subcharacteristic, internal quality sub-characteristic and quality in-use characteristic. Explanations of how to apply and use these sets of metrics. Examples of how and where to apply these metrics during the software product lifecycle. III. THE QUALITY MEASUREMENT DIVISION OF ISO 25000: AN OVERVIEW

One of the objectives of this new ISO 25000 series (and what makes it different from the current ISO 9126 series) is the harmonization of its contents with the software measurement terminology of ISO 15939 [10] which itself based on the ISO metrology terminology [21]. Figure 3 shows the structure of the quality measurement division (ISO 2502n) series that is to replace the current four-part ISO 9126 series of standards [22]. This proposed

quality measurement division (ISO 2502n) will consist of five documents: ISO 25020 on Measurement Reference Model and Guide. ISO 25021 on Quality Measure Elements. ISO 25022 on Measurement of Internal Quality. ISO 25023 on Measurement of External Quality. ISO 25024 on Measurement of Quality in Use. The work on ISO 25000 is being carried out by Working Group 6 (WG6) of the software and system engineering subcommittee (SC7) of the ISO/IEC joint technical committee (JTC1) on Information Technology, that is, ISO/IEC JTC1/SC71.

The author is a Canadian representative and committee member of the ISO/IEC Joint Committee 1 Subcommittee 7 (JTC1/SC7) Work Group 6 on Software Evaluation and Measurement.

276

Figure 3. The structure of the measurement division (ISO 2502n series).

IV.

THE WEAKNESSES OF ISO 9126

In this section we will discuss the weaknesses of the ISO 9126. However, these weaknesses will be classified into two classes, that is, the weaknesses which we have identified during our previous work, and the weaknesses which are discovered by other researchers and practitioners. A. Self-Discovered Weaknesses During our previous research, we have found the following weaknesses of ISO 9126: 1) The terminology needs to be harmonized with the ISO International Vocabulary of Basic and General Terms in Metrology [8]. 2) The metrics in ISO 9126 parts 2, 3, and 4 are completely merged and need to be classified into base and derived metrics to be aligned with the ISO 15939 [9]. 3) It is to be noted that the ranges of the results of many of the metrics in ISO 9126 are between 0 and 1. Therefore, it is easy to convert them to percentage values. However, from our point of view, these results would be easier to understand if they were ranked in terms of qualitative values; for example, for task completion, if the percentage result is 100%, then the completion of the task is labeled excellent; if the result is 80%, then the completion of the task is labeled very good; and so on [10, 13]. 4) When mapping the ISO 9126 to ISO 12207, many of the ISO 9126 quality metrics referred to ISO 12207 processes. However, as known, each process in ISO 12207 contains a number of different activities. Thus, it is more usable for the ISO 9126 users to refer to the activities of the ISO 12207 [11, 12]. B. The Weaknesses Discovered by other Researchers The following are the weaknesses which have been identified by other researchers: 1) Jung et al. [14] stated that including the six compliance sub-characteristics leads to two possible

2)

3)

results. The first possibility is that the compliance sub-characteristic for each characteristic belongs to its characteristic as defined in ISO 9126. The second possibility is that the six compliance subcharacteristics constitute a separate dimension [14]. Furthermore, their results reveal ambiguities in the way that ISO 9126 is structured in terms of characteristics and sub-characteristics. Chua and Dyson [15] - in their investigation uncovered some inherent weaknesses in the ISO 9126 quality model, particularly with regards to the Usability characteristic. To make the model simpler to use for educators, who may not be usability experts, they propose that this characteristic should be extended to include more specific factors such as consistency, simplicity, legibility and color use. It is also suggested that a Help sub-characteristic be included as part of Usability, mainly to ensure that this important factor should not be neglected. In addition, we propose the inclusion of user satisfaction as a global characteristic to summarize the general impact of the system on the user in their specific educational context and given their specific requirements. With these improvements, ISO 9126 could be a useful model for evaluating the quality of e-learning systems [14]. Based on the ISO 9126 standard, as well as an investigation of other models proposed by various Quality Measure Elements (ISO 25021) Measurement of External Quality (ISO 25023) Measurement of Internal Quality (ISO 25022) Measurement of Quality in Use (ISO 25024) easurement Reference Model and Guide (ISO 25020) authors, Abran et al. [16] stated that revised structure for a usability model need to be proposed, including the addition of two characteristics to the ISO 9126 standard which had been selected as their baseline: learnability and security. Then, specific metrics for

277

4)

5)

6)

7)

these characteristics were presented using the structured hierarchy of ISO 9126 [16]. Begh [17] stated that he could not apply the ISO 9126 metrics directly because they are imprecise. In addition, the project team in his study devoted considerable effort to redefining the metrics and applying the formalized approach with counting rules and profiles [17]. The complex attributes based on standards required a careful examination of the standards, but they eventually turned out to be easier to define [17]. Koscianski and Costa [18] stated that the normalization of metrics and scores must accomplish the following: every metric used must be mapped to the same reference scale [18]. The sample metrics of ISO/IEC 9126-2 and ISO/IEC 9126-3 generally return values in the range [0, 1], where 1 means the best possible result. Furthermore, qualitative metrics may use the same concept [18]. Behkamal et al. [19] stated that the ISO 9126 quality model is a general one. In addition, they noted that in order to apply this standard, it needs to be customized [19]. Erazo [20] has analyzed the ISO 9126 quality model and presented the results of an actual evaluation, illustrating each characteristic with their respective valued aspects. V. CONCLUSION

processes, development process and software qualification testing activity, respectively, Fig. 3 for an example on where the ISO 9126 internal quality metrics could be measured within the ISO 12207 verification activities. Ignore the compliance sub-characteristic for each characteristic in ISO 9126 quality model. Additional empirical work is necessary for clarity and validates the ISO 9126 quality model structure in terms of characteristics and sub-characteristics. The usability characteristic should be extended to include more specific sub-characteristics such as consistency, simplicity, legibility, color use, and help. Include the addition of two characteristics to the ISO 9241-11 standard which had been selected as their baseline: learnability and security. Then, specific metrics for these characteristics need to be presented using the structured hierarchy of ISO 9126. Redefine the ISO 9126 metrics and apply a formalized approach with counting rules and profiles to the ISO 9126 metrics. Map each of the ISO 9126 metrics to the same reference scale. Make the ISO 9126 be ready for customization to be used in any domain. The ISO 9126 quality model characteristics need to be ranked by their usability and importance. REFERENCES

Nowadays, ISO is ongoing to update the current ISO 9126 international standard on software product quality measurement. However, this current standard will be replaced by the upcoming quality measurement division (ISO 25020, ISO 25021, ISO 25022, ISO 25023, and ISO 25024) of the ISO 25000 series of international standards on software quality requirements and evaluation (SQuaRE). This paper presented an investigation on the weaknesses of the current ISO 9126 international standard by collecting different viewpoints from our previous researches and from interested researchers and practitioners. However, this paper has tackled the following suggestion to resolve such weaknesses: Ensure that the terminology on software product quality measurement is fully aligned with the classic measurement terminology in the sciences and in engineering. Identify and classify the ISO 9126 metrics into base and derived metrics to make it easy to determine which base metrics should be collected to be used subsequently in computing the other derived metrics. Convert the result of the ISO 9126 metrics to percentage values to be more understandable. To overcome the problem on the mapping between ISO 9126 and ISO 12207, a cross-reference numbers from ISO 12207 can be used. For example, the cross-reference number 5.3.9 is referring to primary

[1] ISO, Software Product Evaluation - Quality Characteristics and Guidelines for Their Use (ISO/IEC IS 9126). Geneva, Switzerland: International Organization for Standardization 1991. [2] ISO, Software Engineering - Product Quality - Part 1: Quality Model (ISO/IEC IS 9126-1). Geneva, Switzerland: International Organization for Standardization, 2001. [3] ISO, Software Engineering - Product Quality - Part 2: External Metrics (ISO/IEC TR 9126-2). Geneva, Switzerland: International Organization for Standardization, 2003. [4] ISO, Software Engineering - Product Quality - Part 3: Internal Metrics (ISO/IEC TR 9126-3). Geneva, Switzerland: International Organization for Standardization, 2003. [5] ISO, Software Engineering - Product Quality - Part 4: Quality in-Use Metrics (ISO/IEC TR 9126-4). Geneva, Switzerland: International Organization for Standardization, 2004. [6] M. Azuma, "SQuaRE: The Next Generation of ISO/IEC 9126 and 14598 International Standards Series on Software Product Quality," Proc. of the European Software Control and Metrics Conference (ESCOM01), London, UK, 2001, pp. 337-346. [7] ISO, Software Engineering - Software Product Quality Requirements and Evaluation (SQuaRE) - Guide to SQuaRE (ISO/IEC 25000), Geneva, Switzerland: International Organization for Standardization, 2005.

278

[8] A. Abran, R. E. Al-Qutaish, and J.-M. Desharnais, "Harmonization issues in the updating of the ISO standards on software product quality," Metrics News: Journal of the Software Metrics Community, Vol. 10, No. 2, 2005, pp. 3544. [9] A. Abran, R. E. Al-Qutaish, J.-M. Desharnais, and N. Habra, "An information model for software quality measurement with ISO standards," Proc. of the International Conference on Software Development (SWDC-REK'05), Reykjavik, Iceland, 2005, pp. 104-116. [10] A. Abran, R. E. Al-Qutaish, and J. Cuadrado-Gallego, "Analysis of the ISO 9126 on Software Product Quality Evaluation from the Metrology and ISO 15939 Perspectives WSEAS Transactions on Computers, Vol. 5, No. 11, 2006, pp. 2778-2786. [11] R. E. Al-Qutaish, "Measuring the Software Product Quality during the Software Development Life-Cycle: An International Organization for Standardization Standards Perspective," Journal of Computer Science, Vol. 5, No. 5, 2009, pp. 392-397. [12] R. E. Al-Qutaish and H. Jalfawareh, "Mapping Between ISO 9126 on Software Product Quality Metrics and ISO 12207 on Software Life Cycle Processes," Proc. of the International Arab Conference on Information Technology (ACIT'06), Yarmouk University, Irbid, Jordan, 2006. [13] A. Abran, R. E. Al-Qutaish, and J. Cuadrado-Gallego, "Investigation of the Metrology Concepts in ISO 9126 on Software Product Quality Evaluation," Proc. of the 10th International Conference on Computers (ICComp'06), Vouliagmeni, Athens, Greece, 2006, pp. 864-872. [14] H.-W. Jung, S.-G. Kim, and C.-S. Chung, "Measuring Software Product Quality: A Survey of ISO/IEC 9126," IEEE Software, Vol. 21, No. 5, 2004, pp. 89-92.

[15] B. B. Chua and L. E. Dyson, "Applying the ISO 9126 Model to the Evaluation of an e-Learning System," Proc. of the 21st Conference of the Australasian Society for Computers in Learning in Tertiary Education (ASCILITE'04), Perth, Australia, 2004, pp. 184-190. [16] A. Abran, A. Khelifi, W. Suryn, and A. Seffah, "Consolidating the ISO Usability Models," Proc. of 11th International Software Quality Management Conference, Glasgow, Scotland, UK, 2003. [17] J. Boegh, "Certifying Software Component Attributes," IEEE Software, Vol. 23, No. 3, 2006, pp. 74 81. [18] A. Koscianski and J. C. Bracarense Costa, "Combining Analytical Hierarchical Analysis with ISO/IEC 9126 for a Complete Quality Evaluation Framework," Proc. of the 4th IEEE International Symposium and Forum on Software Engineering Standards (ISESS'99), Curitiba, Brazil, 1999, pp. 218-216. [19] B. Behkamala, M. Kahanib, and M. K. Akbar, "Customizing ISO 9126 quality model for evaluation of B2B applications," Information and Software Technology, Vol. 51, No. 3, 2009, pp. 599-609. [20] S. Riascos-Erazo, "Evaluation of the Quality (norm ISO 9126) in Applications Educational Multimediales," FORMATEX Magazine, 2006, Vol. 5, pp. 832-836. [21] ISO, International Vocabulary of Basic and General Terms in Metrology (VIM). Geneva, Switzerland: International Organization for Standardization, 1993. [22] ISO, Software Engineering - Software Product Quality Requirements and Evaluation (SQuaRE) - Quality Measure Elements (ISO/IEC TR 25021). Geneva, Switzerland: International Organization for Standardization, 2007.

279