Research Article

Are Companies Earning Return on Their Investment in ISO 9000 Registration? A Review of the Empirical Evidence

By Chris FitzGibbon MMS, CSQE, CQMgr, CQA

Introduction

Many benefits have been attributed to ISO 9001 registration. However, the existing body of anecdotal literature on the application of ISO 9001 to software organizations lacks systematic measurement and multi-organization comparisons. It provides very little empirical evidence to support the claim that ISO 9001 registered quality systems result in better software project outcomes.

This article provides a review of the empirical research on the effects of ISO 9001 registration. The studies are divided into two parts:

1. research specifically pertaining to software organizations and
2. research on non-software organizations.

A Review of Empirical Research on the Effects of ISO 9001 Registration on Software Development Organizations

A 1994 study of German ISO 9001 software development organizations (Stelzer, 1995) and subsequent research on European ISO 9001 software development organizations (Stelzer, Mellis, and Herzwurm, 1996) comprise the body of empirical research on this issue.

ISO 9000 as an Indicator of Software Quality

This 1994 survey of the first twenty ISO 9001 registered software organizations in Germany examined the extent to which ISO 9001 registration was an indicator of the use of software quality practices (Stelzer, 1995). The study concentrated on five quality system elements: code reviews and inspections, software testing, product and process measurements, measurement of quality costs, and demonstration of quality improvements.

Based on data from 18 respondents, the study found significant variations in the implementation of an ISO 9001 compliant quality system. The research concluded that ISO 9001 registration is not an indicator of the strength of a software developer’s products, processes, or its quality system. The researchers added that ISO 9001 registration is not a guarantee that the certified quality system fulfils all the requirements specified in the standard. Some of the participants in the study reported that their ISO 9001 compliant quality system did not adequately address the five quality system elements examined.

The study also found that ISO 9001 registration met the expectations of 47% of the respondents and partially met the expectations of another 53%. None of the participants in the study reported that registration to ISO 9001 was a failure.

The results of this study indicate that the value of ISO 9001 registration for software development organizations was not consistent. The researcher identified the requirement for further research.

Software Process Improvements Achieved through ISO 9001

The objective of this study was to gain a better understanding of the software process improvements achieved through registration to ISO 9001 (Stelzer, Mellis, and Herzwurm, 1996).

Based on data from 36 European software developers, interviews with ISO 9000 auditors, and discussions with consultants working in the software industry, the researchers identified ten key success factors for software process improvement via ISO 9000. Success factors describe the aspects that the respondents considered to be the most helpful when implementing an ISO 9000 quality system (Stelzer, Mellis, and Herzwurm, 1996: 6). The sequence of the ten factors in Table 1 does not indicate their importance.

Table 1: Success factors for software process improvement
via ISO 9000

Only two of the ten key success factors identified in the study are explicit requirements of ISO 9001:

1. the definition and documentation of the status quo, and
2. internal audits.

The list of success factors reveals that software organizations typically take a more comprehensive approach to achieve software process improvement than the minimum requirements of the ISO 9001 standard (Stelzer, Mellis, and Herzwurm, 1996: 8).

The research attributed several benefits to ISO 9001 registration. The majority of the software organizations reported achieving substantial improvements during the implementation of the standard. The development by software organizations of a clearly defined organizational structure and procedures helped identify and address process weaknesses. Formal customer perception surveys provided unbiased and unprejudiced feedback on the quality of a company’s products and services. Internal quality audits improved job satisfaction and provided an opportunity for staff to discuss the quality system’s strengths and weaknesses. These benefits and the preparation of guidelines, checklists, templates and tools led to the conclusion that ISO 9001 registration contributes to improved efficiency, shorter development times and lower costs.

Software organizations with established processes before the implementation of an ISO 9001 quality system did not achieve significant software process improvements through ISO 9001 registration. These organizations only documented the status quo and sought certification.

Although only one third of respondents could quantify the benefits of ISO 9001 registration, nearly all participants reported that they would favour implementing an ISO 9000 quality system once again.

Canadian Study on the Impact of ISO 9001 on Software Quality

The objectives of this study were to compare the projects of ISO 9001 registered and non-ISO 9001 software organizations to:

1. identify any differences in project characteristics,
2. compare project outcomes,
3. examine the perceptions toward the project characteristics examined, and
4. measure the perceived impact of ISO 9001 registration on project outcome.

Project outcome was measured in terms of schedule overrun and project manager satisfaction with the project. An insufficient number of respondents provided budgetary data to provide a valid statistical comparison of that success factor.

Based on an extensive literature review and several interviews with software project managers, quality auditors and consultants, a long list of project characteristics was reduced to just eight. The project characteristics examined were: project planning, software testing, the coding process, project record keeping, design reviews and code inspections, customer involvement in the software development process, involvement of software engineers in project planning, and project manager power. Several of the factors used to compare ISO 9001 and non-ISO organizations were not requirements of the standard but were considered by industry experts to be good measures for comparison.

Data was collected from the project managers of 20 projects from ISO 9001 registered organizations and the projects of 32 non-ISO 9001 organizations. All respondents worked in Canada. Despite many claims to the contrary, this study found no significant differences between the ISO 9001 and non-ISO samples that could be explained by any of the eight project characteristics examined.

The levels of project manager satisfaction were equal among respondents from both groups. However, there was a significant difference in their project outcomes: the projects from the ISO 9001 sample averaged significantly shorter schedule overruns than the non-ISO 9001 sample.

The perceptions of the respondents toward the importance of the eight project characteristics also revealed some significant differences. The ISO 9001 group placed more importance on design reviews and code inspections, project planning and the involvement of software engineers in the planning process. The non- ISO 9001 respondents attributed more importance to project manager power and the involvement of customers in development activities. Both samples were similar in their ranking of software testing, record keeping and the coding process.

When asked whether any differences in these software project characteristics could be attributed to ISO 9001 registration, the respondents from the ISO 9001 organizations were very positive. Many attributed better project outcomes to the more formalized processes associated with registration to the standard. A few respondents stated that their ISO 9001 registration required very few changes to their already mature project management practices.

A surprising picture was revealed from the non-ISO 9001 data. Only two of the respondents from the non- ISO 9001 group thought that ISO 9001 registration would have a positive impact on project outcome. The results clearly show contrasting opinions on the contribution of ISO 9001 registration to project success.

The study concluded with the surprising finding that no significant differences among the two groups could be explained by the project characteristics examined. The projects of ISO 9001 organizations, however, were more likely to be delivered on schedule. There were differences in the perceived importance of some software project activities and the study identified opposing views on the perceived impact of ISO 9001 registration on project outcomes.

Summary

Table 2: Summary of the empirical research on the effects of ISO 9001 registration on software development organizations

A Review of Empirical Research on the Effects of ISO 9000 on Non-Software Organizations

This review of the empirical research on the effects of ISO 9000 on non-software organizations provides: a comparison of the ISO 9000 registration experiences of US and Foreign-owned firms (Ebrahimpour, M., Withers, B. E., and Hikmet, N., 1997), the costs and benefits of ISO 9000 registration (Atwater and Discenza 1993) and the effects of ISO 9000 registration on business performance (Seddon, Davis, Loughran and Murrel, 1993; Surrey University, 1992; Manchester Business School, 1995).

The ISO 9000 Registration Experiences of US and Foreign-Owned Firms

This research provides a comprehensive study of firms operating in the United States which have successfully achieved ISO 9000 registration (Ebrahimpour, M., Withers, B. E., and Hikmet, N., 1997). Specifically, it compares the registration experiences of 541 US-owned and foreign-owned manufacturing firms to identify the key elements associated with successful ISO 9000 registration.

The key reasons for seeking ISO 9000 registration were to achieve corporate objectives and to increase market share. Of the twenty possible areas for improvement provided by the researchers, respondents from both groups expected the highest degree of improvement as a result of ISO 9001 registration to occur in the areas of product design, product quality, communication, process design, public image, and supplier relations.

Internal quality audits, document control, design control, corrective action, and training were identified as the most demanding elements of the ISO 9000 registration process. This result is consistent with the fact that ISO 9000 registration hinges upon the development, implementation, and documentation of comprehensive systems for controlling processes.

Inadequate documentation was identified as the primary barrier to implementing ISO 9000. Poor communication was also highly rated as a barrier to implementation; however, inadequate support from top management was not a serious barrier to becoming ISO 9000 compliant. As an explanation, the researchers suggested that all respondents had successfully registered as a result of existing senior management support. The survey results also indicate that both groups had low expectations that ISO 9000 registration would lead to improvements in productivity or market share.

The research found no significant differences between US and foreign-owned firms registered to ISO 9000.

The Costs and Benefits of ISO Registration

This study was based on data from 29 high technology manufacturers in Colorado Springs (Atwater and Discenza 1993). Among the research questions was an investigation of the costs and benefits of ISO 9000 registration.

Seventy-one percent of the respondents felt that the benefits associated with ISO 9000 registration were greater than or equal to its costs. Approximately half the sample reported that their ISO 9000 registration had resulted in substantial improvements to operating efficiency, customer satisfaction, competitive position, and the time required to train new employees. A third noted a significant reduction in scrap and rework. Forty-five percent reported the high cost to implement the system as well as the development and maintenance of acceptable documentation were major obstacles to ISO 9000 registration.

All the respondents cited customer requirement as their main reason for pursuing ISO 9000 registration.

The Impact of BS5750 Registration on Organizational Performance

Research conducted for Vanguard Consulting Inc., a consulting company in England, measured the associated costs and benefits of registration to BS5750, the British Standard Institute’s predecessor to ISO 9000 (Seddon, Davis, Loughran and Murrel, 1993; Seddon, 1996). The study used seven-point Likert scales to examine the perceptions of 647 representatives from British organizations that had successfully implemented BS5750.

The study was based on the nine business indicators contained in Table 3. Fifteen percent of the organizations surveyed reported an improvement on all the business indicators and attributed this improvement to their compliance to the BS5750 standard. Approximately one third of the respondents reported improvements in market share and their ability to attract new customers. Sixty-nine percent reported an improvement in procedural efficiency, but the results for improvements in the measurable aspects of efficiency (costs, waste, etc.) were approximately half that value. A quarter of the respondents attributed reductions in overall costs to their registration to the standard; however, 15% of the organizations perceived costs to have worsened. Regarding the benefits of installing a quality system, 69% reported improvements in efficiency, 37% saw reductions in waste, almost half experienced improved customer service, and 31% reported improved staff motivation. Table C summarizes these findings.

Table 3: The improvements and worsening attributed to BS5750 registration

The study found that the organizations that achieved the least benefit from their BS5750 registration had undertaken the initiative to address customer demand or to increase market share. The reasons for becoming BS5750 registered identified by the organizations that achieved the most value from the standard were to improve operational efficiency, customer service and the reliability of its products and services.

The Effect of ISO 9000 Registration on Key Business Ratios

In 1990, Lloyd’s Register Quality Assurance (LRQA), an internationally accredited ISO 9000 registrar, commissioned Surrey University to research the effect of ISO 9000 registration on key business ratios such as profitability and sales (Surrey University, 1993; Seddon, 1996). A total of 222 companies from the mechanical engineering manufacturing sector in England participated. All had been registered to ISO 9001 or ISO 9002 by LRQA. The results of the survey were compared with the 1992 industry averages.

The study examined six business ratios. The profit margins and return on capital employed (ROCE) of the ISO 9000 companies averaged more than double the industry average. ROCE relates the pre-tax profits to the long term investment in a company. On average, the sample reported 40% higher sales per employee and a profit per employee figure more than three times the industry average. Capital employed per employee, a measure of the level of investment in the company, was approximately double the industry average. And, asset turnover, a ratio of business capital investment to output, was higher. Table 4 summarizes these findings.

Table 4: A comparison of the business ratios of ISO 9000 companies and the industry average

Data were also collected on the benefits of implementing an ISO 9000 compliant quality system. The research found that 69% reported the implementation of an ISO 9000 quality system improved productivity. Approximately half reported reductions in waste and improvements to staff motivation. Forty percent saw reductions in costs and 73% reported improvements in customer service.

The survey revealed that 81% of companies obtained registration to increase market share or address their customers’ demand for ISO 9000 registration.

The Reasons for Seeking ISO 9000 Registration and its Results

In 1995, the Manchester Business School conducted research on behalf of SGS Yarsley, an internationally accredited ISO 9000 registrar, that examined the reasons for obtaining ISO 9000 registration and its impact on businesses (Manchester Business School, 1995). A total of 1190 questionnaires were completed by ISO 9000 registered companies in the manufacturing, construction and service industries.

The driving force behind the majority of the decisions to pursue ISO 9000 registration was external pressure. Seventy-seven percent of respondents sought registration to the standard in anticipation of future customer requirement. Maintaining market share, pressure from existing customers and marketability were also identified. Internal factors included improvements to the consistency, cost reductions, efficiency and quality of operations. Table 5 summarizes the main reasons for seeking ISO 9000 registration.

Table 5: The reasons for seeking ISO 9000 registration

Overall, 99% of respondents claimed that they benefited from ISO 9000 registration. Better management control was identified as the most important benefit followed by the improved awareness and elimination of procedural problems. Other internal benefits included improved efficiency and better induction of new staff. ISO 9000 was also seen as a useful marketing tool, a means of improving customer service and satisfaction, and a vehicle to improve market share. Two thirds of the respondents said "keeping existing customers" was a significant benefit, although those for whom this was the only benefit were generally dissatisfied with the standard. Table 6 provides a summary of the benefits of ISO 9000 registration.

Table 6: The benefits of ISO 9000 registration

Overall, 53% of the respondents were satisfied with the impact of ISO 9000, 26% were neither satisfied nor dissatisfied and 21% were dissatisfied. While the majority of those surveyed were satisfied with their experience with ISO 9000, the research revealed several difficulties and areas of concern. The time, volume of paperwork and costs required to implement and maintain an ISO 9000 compliant quality system were identified as the major problems with registration. Table 7 summarizes the common problems associated with ISO 9000 registration.

Table 7: Common problems with ISO 9000 registration

Why Organizations Choose Not to Seek ISO 9000 Registration

In a second study conducted by Manchester Business School, 140 responses from organizations of all sizes were used to identify the reasons for not seeking ISO 9000 registration (Manchester Business School, 1995). The results included time commitments and the volume of paperwork, as well as the cost of implementing and maintaining the standard. Table 8 summarizes these findings. The researchers reported that many service sector organizations felt that ISO 9000 was not relevant and companies with less than 25 employees were particularly concerned with the costs of implementation.

Table 8: Reasons for not seeking ISO 9000 registration

Swedish Study on Expected and Actual Returns on ISO 9000 Registration

Research published in 1993 examined data collected from 23 multinational organizations with operations in Sweden. Table 9 and 10 summarize the results.

Table 9: Expected benefits compared to actual benefits from ISO 9000 registration

Table 10: Improvements Realized through ISO 9000 Registration

The Impact of ISO 9000 Registration on Slovenian Organizations

Researchers in Slovenia received responses to their survey from 117 of that country’s 355 organizations with ISO 9001 or ISO 9002 registration. The purpose of the study was to determine the improvements attributed to ISO 9000 by those organizations with registration. The 7 point Likert scale used in the study is defined below.

Likert Scale Measuring the Influence of ISO 9000:

1 – the situation has considerably worsened

2 – the situation has worsened

3 – there was minimal worsening

4 – the situation is unchanged

5 - there was minimal improvement

6 - the situation has improved

7 - the situation has considerably improved

Table 11 contains the measure, the average rating and the percentage of respondents who indicated they did not measure that parameter.

Table 11: The influence of ISO 9000 Realized through ISO 9000 Registration

Summary

Table 12: Summary of the empirical research on the effects of ISO 9000 registration on non-software organizations.

References

Atwater, J. B., and Discenza, R., "ISO 9000: Implications for U.S. Business". 1993 Proceedings of the Annual Meeting of the Decision Sciences Institute, 1993: 1784-1786.

Bailetti, A. J., and FitzGibbon, C., ISO 9001 Registration for Small and Medium Sized Software Enterprises. Ottawa: Carleton University Press, 1995.

Ebrahimpour, M., Withers, B. E., and Hikmet, N., "Experiences of US- and Foreign-owned firms: A New Perspective on ISO 9000 Implementation". International Journal of Production Research, 1997: 569-576.

FitzGibbon, Chris, ISO 9001 Registration: Lessons Learned by Canadian Software Companies. Proceedings of the Fifth International Conference on Management of Technology. February 27-March 1, 1996, Miami, Florida, p. 193-201.

FitzGibbon, Chris, An Analysis of the Difference in Software Project Management Practices Between Organizations Registered to ISO 9001 and Non-Registered Organizations. Ottawa: Carleton University, 1998.

Johansson, Christer, Lindgren, Mats, and Lissgärde, Peter, Att Lyckas Med ISO 9000: En ingående studie av 23 certifierade företag. Stockholm Sweden: Ord & Form AB, 1993.

Manchester Business School, ISO 9000 - Does it Work?. Yarsley, England: SGS International Certification Services(Yarsley), 1995.

Pivka, Marjan, Ursie, Dusko, Quality Systems Between Theory and Practice Slovenian Experiences. Slovenia: Faculty of Business and Economics, University of Maribor, 1998.

Seddon, J., Davis, R., Loughran, M., and Murrell, R., BS5750 Implementation and Value Added: A Survey of Registered Companies. Buckingham, England: Vanguard Consulting Ltd., 1993.

Seddon, J., "ISO 9000 Implementation and Value-added: Three Case Studies". Vanguard Consulting: 1996.

Stelzer, D., "Interpretations of ISO 9000 when Certifying Software Quality Systems. Findings of an Empirical Survey in Germany". Extended abstract of: "Interpretation der ISO 9000-Familie bei der Zertifizierung von Qualitätsmanagementsystemen für die Softwareentwichlung". Norbert Ruppenthal, Ulrich Sigor (Hrsg.): Qualitätsmanagement und Software: ISO 9000 - Softwareentwicklung - Ethik - Analysen - Tools. Beiträge vom adi QM/IT Expertentreffen 1994. Münster 1995, S. 15-31.

Stelzer, D., Mellis, W., and Herzwurm, G., "Software Process Improvement via ISO 9000? Results of two surveys among European software houses". Proceedings of the 29th Hawaii International Conference on System Sciences, January 3-6, 1996.

Surrey University, Fitter Finance: The Effects of ISO 9000 on Business Performance. Croydon, England: Lloyd’s Register Quality Assurance (LRQA), 1993.