An Investigation of the Weaknesses of the ISO 9126 International ...

Figure 1. ISO 9126 quality model for external and internal quality (characteristics and sub-characteristics).
Figure 2. ISO 9126 quality model for quality in-use (characteristics).
The second [3], third [4] and fourth [5] parts of the ISO
9126 series provide the following information:
• Sets of metrics for each external quality subcharacteristic,
internal quality sub-characteristic and
quality in-use characteristic.
• Explanations of how to apply and use these sets of
metrics.
• Examples of how and where to apply these metrics
during the software product lifecycle.
III.
THE QUALITY MEASUREMENT DIVISION OF ISO
25000: AN OVERVIEW
One of the objectives of this new ISO 25000 series (and
what makes it different from the current ISO 9126 series) is
the harmonization of its contents with the software
measurement terminology of ISO 15939 [10] which itself
based on the ISO metrology terminology [21].
Figure 3 shows the structure of the quality measurement
division (ISO 2502n) series that is to replace the current
four-part ISO 9126 series of standards [22]. This proposed
quality measurement division (ISO 2502n) will consist of
five documents:
• ISO 25020 on Measurement Reference Model and
Guide.
• ISO 25021 on Quality Measure Elements.
• ISO 25022 on Measurement of Internal Quality.
• ISO 25023 on Measurement of External Quality.
• ISO 25024 on Measurement of Quality in Use.
The work on ISO 25000 is being carried out by Working
Group 6 (WG6) of the software and system engineering
subcommittee (SC7) of the ISO/IEC joint technical
committee (JTC1) on Information Technology, that is,
ISO/IEC JTC1/SC7 1 .
1 The author is a Canadian representative and committee
member of the ISO/IEC Joint Committee 1 – Subcommittee
7 (JTC1/SC7) – Work Group 6 on Software Evaluation and
Measurement.
276

Figure 3. The structure of the measurement division (ISO 2502n series).
IV. THE WEAKNESSES OF ISO 9126
In this section we will discuss the weaknesses of the ISO
9126. However, these weaknesses will be classified into two
classes, that is, the weaknesses which we have identified
during our previous work, and the weaknesses which are
discovered by other researchers and practitioners.
A. Self-Discovered Weaknesses
During our previous research, we have found the
following weaknesses of ISO 9126:
1) The terminology needs to be harmonized with the
ISO International Vocabulary of Basic and General
Terms in Metrology [8].
2) The metrics in ISO 9126 parts 2, 3, and 4 are
completely merged and need to be classified into
base and derived metrics to be aligned with the ISO
15939 [9].
3) It is to be noted that the ranges of the results of many
of the metrics in ISO 9126 are between 0 and 1.
Therefore, it is easy to convert them to percentage
values. However, from our point of view, these
results would be easier to understand if they were
ranked in terms of qualitative values; for example,
for task completion, if the percentage result is 100%,
then the completion of the task is labeled “excellent”;
if the result is 80%, then the completion of the task is
labeled “very good”; and so on [10, 13].
4) When mapping the ISO 9126 to ISO 12207, many of
the ISO 9126 quality metrics referred to ISO 12207
processes. However, as known, each process in ISO
12207 contains a number of different activities.
Thus, it is more usable for the ISO 9126 users to
refer to the activities of the ISO 12207 [11, 12].
B. The Weaknesses Discovered by other Researchers
The following are the weaknesses which have been
identified by other researchers:
1) Jung et al. [14] stated that including the six
compliance sub-characteristics leads to two possible
results. The first possibility is that the compliance
sub-characteristic for each characteristic belongs to
its characteristic as defined in ISO 9126. The second
possibility is that the six compliance subcharacteristics
constitute a separate dimension [14].
Furthermore, their results reveal ambiguities in the
way that ISO 9126 is structured in terms of
characteristics and sub-characteristics.
2) Chua and Dyson [15] - in their investigation -
uncovered some inherent weaknesses in the ISO
9126 quality model, particularly with regards to the
Usability characteristic. To make the model simpler
to use for educators, who may not be usability
experts, they propose that this characteristic should
be extended to include more specific factors such as
consistency, simplicity, legibility and color use. It is
also suggested that a Help sub-characteristic be
included as part of Usability, mainly to ensure that
this important factor should not be neglected. In
addition, we propose the inclusion of user
satisfaction as a global characteristic to summarize
the general impact of the system on the user in their
specific educational context and given their specific
requirements. With these improvements, ISO 9126
could be a useful model for evaluating the quality of
e-learning systems [14].
3) Based on the ISO 9126 standard, as well as an
investigation of other models proposed by various
Quality Measure Elements (ISO 25021)
Measurement of External Quality (ISO 25023)
Measurement of Internal Quality (ISO 25022)
Measurement of Quality in Use (ISO 25024)
easurement Reference Model and Guide (ISO 25020)
authors, Abran et al. [16] stated that revised structure
for a usability model need to be proposed, including
the addition of two characteristics to the ISO 9126
standard which had been selected as their baseline:
learnability and security. Then, specific metrics for
277

these characteristics were presented using the
structured hierarchy of ISO 9126 [16].
4) Bøegh [17] stated that he could not apply the ISO
9126 metrics directly because they are imprecise. In
addition, the project team in his study devoted
considerable effort to redefining the metrics and
applying the formalized approach with counting rules
and profiles [17]. The complex attributes based on
standards required a careful examination of the
standards, but they eventually turned out to be easier
to define [17].
5) Koscianski and Costa [18] stated that the
normalization of metrics and scores must accomplish
the following: every metric used must be mapped to
the same reference scale [18]. The sample metrics of
ISO/IEC 9126-2 and ISO/IEC 9126-3 generally
return values in the range [0, 1], where 1 means the
best possible result. Furthermore, qualitative metrics
may use the same concept [18].
6) Behkamal et al. [19] stated that the ISO 9126 quality
model is a general one. In addition, they noted that in
order to apply this standard, it needs to be
customized [19].
7) Erazo [20] has analyzed the ISO 9126 quality model
and presented the results of an actual evaluation,
illustrating each characteristic with their respective
valued aspects.
V. CONCLUSION
Nowadays, ISO is ongoing to update the current ISO
9126 international standard on software product quality
measurement. However, this current standard will be
replaced by the upcoming quality measurement division
(ISO 25020, ISO 25021, ISO 25022, ISO 25023, and ISO
25024) of the ISO 25000 series of international standards on
software quality requirements and evaluation (SQuaRE).
This paper presented an investigation on the weaknesses
of the current ISO 9126 international standard by collecting
different viewpoints from our previous researches and from
interested researchers and practitioners. However, this paper
has tackled the following suggestion to resolve such
weaknesses:
• Ensure that the terminology on software product
quality measurement is fully aligned with the classic
measurement terminology in the sciences and in
engineering.
• Identify and classify the ISO 9126 metrics into base
and derived metrics to make it easy to determine
which base metrics should be collected to be used
subsequently in computing the other derived metrics.
• Convert the result of the ISO 9126 metrics to
percentage values to be more understandable.
• To overcome the problem on the mapping between
ISO 9126 and ISO 12207, a cross-reference numbers
from ISO 12207 can be used. For example, the
cross-reference number 5.3.9 is referring to “primary
processes”, ‘development process’ and ‘software
qualification testing’ activity, respectively, Fig. 3 for
an example on where the ISO 9126 internal quality
metrics could be measured within the ISO 12207
verification activities.
• Ignore the compliance sub-characteristic for each
characteristic in ISO 9126 quality model.
• Additional empirical work is necessary for clarity
and validates the ISO 9126 quality model structure
in terms of characteristics and sub-characteristics.
• The usability characteristic should be extended to
include more specific sub-characteristics such as
consistency, simplicity, legibility, color use, and
help.
• Include the addition of two characteristics to the ISO
9241-11 standard which had been selected as their
baseline: learnability and security. Then, specific
metrics for these characteristics need to be presented
using the structured hierarchy of ISO 9126.
• Redefine the ISO 9126 metrics and apply a
formalized approach with counting rules and profiles
to the ISO 9126 metrics.
• Map each of the ISO 9126 metrics to the same
reference scale.
• Make the ISO 9126 be ready for customization to be
used in any domain.
• The ISO 9126 quality model characteristics need to
be ranked by their usability and importance.
REFERENCES
[1] ISO, Software Product Evaluation - Quality Characteristics
and Guidelines for Their Use (ISO/IEC IS 9126). Geneva,
Switzerland: International Organization for Standardization
1991.
[2] ISO, Software Engineering - Product Quality - Part 1: Quality
Model (ISO/IEC IS 9126-1). Geneva, Switzerland:
International Organization for Standardization, 2001.
[3] ISO, Software Engineering - Product Quality - Part 2:
External Metrics (ISO/IEC TR 9126-2). Geneva, Switzerland:
International Organization for Standardization, 2003.
[4] ISO, Software Engineering - Product Quality - Part 3: Internal
Metrics (ISO/IEC TR 9126-3). Geneva, Switzerland:
International Organization for Standardization, 2003.
[5] ISO, Software Engineering - Product Quality - Part 4: Quality
in-Use Metrics (ISO/IEC TR 9126-4). Geneva, Switzerland:
International Organization for Standardization, 2004.
[6] M. Azuma, "SQuaRE: The Next Generation of ISO/IEC 9126
and 14598 International Standards Series on Software Product
Quality," Proc. of the European Software Control and Metrics
Conference (ESCOM’01), London, UK, 2001, pp. 337-346.
[7] ISO, Software Engineering - Software Product Quality
Requirements and Evaluation (SQuaRE) - Guide to SQuaRE
(ISO/IEC 25000), Geneva, Switzerland: International
Organization for Standardization, 2005.
278

[8] A. Abran, R. E. Al-Qutaish, and J.-M. Desharnais,
"Harmonization issues in the updating of the ISO standards on
software product quality," Metrics News: Journal of the
Software Metrics Community, Vol. 10, No. 2, 2005, pp. 35-
44.
[9] A. Abran, R. E. Al-Qutaish, J.-M. Desharnais, and N. Habra,
"An information model for software quality measurement
with ISO standards," Proc. of the International Conference on
Software Development (SWDC-REK'05), Reykjavik, Iceland,
2005, pp. 104-116.
[10] A. Abran, R. E. Al-Qutaish, and J. Cuadrado-Gallego,
"Analysis of the ISO 9126 on Software Product Quality
Evaluation from the Metrology and ISO 15939 Perspectives
WSEAS Transactions on Computers, Vol. 5, No. 11, 2006,
pp. 2778-2786.
[11] R. E. Al-Qutaish, "Measuring the Software Product Quality
during the Software Development Life-Cycle: An
International Organization for Standardization Standards
Perspective," Journal of Computer Science, Vol. 5, No. 5,
2009, pp. 392-397.
[12] R. E. Al-Qutaish and H. Jalfawareh, "Mapping Between ISO
9126 on Software Product Quality Metrics and ISO 12207 on
Software Life Cycle Processes," Proc. of the International
Arab Conference on Information Technology (ACIT'06),
Yarmouk University, Irbid, Jordan, 2006.
[13] A. Abran, R. E. Al-Qutaish, and J. Cuadrado-Gallego,
"Investigation of the Metrology Concepts in ISO 9126 on
Software Product Quality Evaluation," Proc. of the 10th
International Conference on Computers (ICComp'06),
Vouliagmeni, Athens, Greece, 2006, pp. 864-872.
[14] H.-W. Jung, S.-G. Kim, and C.-S. Chung, "Measuring
Software Product Quality: A Survey of ISO/IEC 9126," IEEE
Software, Vol. 21, No. 5, 2004, pp. 89-92.
[15] B. B. Chua and L. E. Dyson, "Applying the ISO 9126 Model
to the Evaluation of an e-Learning System," Proc. of the 21st
Conference of the Australasian Society for Computers in
Learning in Tertiary Education (ASCILITE'04), Perth,
Australia, 2004, pp. 184-190.
[16] A. Abran, A. Khelifi, W. Suryn, and A. Seffah,
"Consolidating the ISO Usability Models," Proc. of 11th
International Software Quality Management Conference,
Glasgow, Scotland, UK, 2003.
[17] J. Boegh, "Certifying Software Component Attributes," IEEE
Software, Vol. 23, No. 3, 2006, pp. 74 – 81.
[18] A. Koscianski and J. C. Bracarense Costa, "Combining
Analytical Hierarchical Analysis with ISO/IEC 9126 for a
Complete Quality Evaluation Framework," Proc. of the 4th
IEEE International Symposium and Forum on Software
Engineering Standards (ISESS'99), Curitiba, Brazil, 1999, pp.
218-216.
[19] B. Behkamala, M. Kahanib, and M. K. Akbar, "Customizing
ISO 9126 quality model for evaluation of B2B applications,"
Information and Software Technology, Vol. 51, No. 3, 2009,
pp. 599-609.
[20] S. Riascos-Erazo, "Evaluation of the Quality (norm ISO
9126) in Applications Educational Multimediales,"
FORMATEX Magazine, 2006, Vol. 5, pp. 832-836.
[21] ISO, International Vocabulary of Basic and General Terms in
Metrology (VIM). Geneva, Switzerland: International
Organization for Standardization, 1993.
[22] ISO, Software Engineering - Software Product Quality
Requirements and Evaluation (SQuaRE) - Quality Measure
Elements (ISO/IEC TR 25021). Geneva, Switzerland:
International Organization for Standardization, 2007.
279