Research Article

Impact of ISO 9001 on Software Quality

By Chris FitzGibbon MMS, CSQE, CQMgr, CQA
Published in: Capital Quality News, October, 1998.

Many benefits have been attributed to ISO 9001 registration. However, the existing body of anecdotal literature on the application of ISO 9001 to software organizations lacks systematic measurement and multi-organization comparisons. It provides very little empirical evidence to support the claim that ISO 9001 registered quality systems result in better software project outcomes.

The objectives of this study were to compare the projects of ISO 9001 registered and non-ISO 9001 software organizations to:

1. identify any differences in project characteristics,
2. compare project outcomes,
3. examine the perceptions toward the project characteristics examined, and
4. measure the perceived impact of ISO 9001 registration on project outcome.

Project outcome was measured in terms of schedule overrun and project manager satisfaction with the project. An insufficient number of respondents provided budgetary data to provide a valid statistical comparison of that success factor.

Based on an extensive literature review and several interviews with software project managers, quality auditors and consultants, a long list of project characteristics was reduced to just eight. The project characteristics examined were: project planning, software testing, the coding process, project record keeping, design reviews and code inspections, customer involvement in the software development process, involvement of software engineers in project planning, and project manager power. Several of the factors used to compare ISO 9001 and non-ISO organizations were not requirements of the standard but were considered by industry experts to be good measures for comparison.

Data was collected from the project managers of 20 projects from ISO 9001 registered organizations and the projects of 32 non-ISO 9001 organizations. All respondents worked in Canada. Despite many claims to the contrary, this study found no significant differences between the ISO 9001 and non-ISO samples that could be explained by any of the eight project characteristics examined.

The levels of project manager satisfaction were equal among respondents from both groups. However, there was a significant difference in their project outcomes: the projects from the ISO 9001 sample averaged significantly shorter schedule overruns than the non-ISO 9001 sample.

The perceptions of the respondents toward the importance of the eight project characteristics also revealed some significant differences. The ISO 9001 group placed more importance on design reviews and code inspections, project planning and the involvement of software engineers in the planning process. The non-ISO 9001 respondents attributed more importance to project manager power and the involvement of customers in development activities. Both samples were similar in their ranking of software testing, record keeping and the coding process.

When asked whether any differences in these software project characteristics could be attributed to ISO 9001 registration, the respondents from the ISO 9001 organizations were very positive. Many attributed better project outcomes to the more formalized processes associated with registration to the standard. A few respondents stated that their ISO 9001 registration required very few changes to their already mature project management practices.

A surprising picture was revealed from the non-ISO 9001 data. Only two of the respondents from the non-ISO 9001 group thought that ISO 9001 registration would have a positive impact on project outcome. The results clearly show contrasting opinions on the contribution of ISO 9001 registration to project success.

The study concluded with the surprising finding that no significant differences among the two groups could be explained by the project characteristics examined. The projects of ISO 9001 organizations, however, were more likely to be delivered on schedule. There were differences in the perceived importance of some software project activities and the study identified opposing views on the perceived impact of ISO 9001 registration on project outcomes.

This study was published through Carleton University and follows a book co-authored by the researcher on the application of the ISO 9001 standard to small and medium-sized software organizations. Chris FitzGibbon is the Vice President and a quality system consultant with Orion Canada Inc.. He also serves as the Quality Manager at AMITA Corporation, an ISO 9001 registered software organization. Chris is a certified Quality Manager (CQMgr), Software Quality Engineer (CSQE) and Quality Auditor (CQA). He can be contacted at (613) 563-9000 or via email at chris@orioncanada.com.