veryard projects - innovation for demanding change

Six Sigma

veryard projects > systems thinking > six sigma
about six sigma veryard projects opinion on this page elsewhere
Six Sigma is a statistical measure of the reliability of a process.  By extension, the term is also used to refer to a statistical approach to process control and improvement.
> more

Six Sigma can be used when a process is repeated millions of times within a manageable period. In an appropriate context, it can be extremely powerful and effective. However, it is widely misunderstood and often misused.
> more

Six Sigma has been widely used in high-volume manufacturing situations, including computer hardware, and has delivered notable quality successes for such companies as Motorola and Texas Instruments. However, its potential value for computer software is more difficult to realise.
> more
Six sigma can be a useful tool for process improvement. However, it should not be used indiscriminately. There are many situations where six sigma measurement is silly or wrong, and may distract managers from more meaningful metrics.

We strongly recommend that the six sigma technique be used primarily for failure metrics that can be directly related to customer (or end-user) satisfaction, or to high-level safety and security requirements.
high value
dubious value
  • High volume
  • Simple repeatable process
  • Stable
  • Low volume
  • Complex process
  • Volatile
  • > what is six sigma?

    > what does six sigma measure?

    > pitfalls of six sigma

    > a true story

    > use of six sigma for software projects

    > resources and links

    Software quality management

    Measurement and control


    veryard projects - innovation for demanding change

    Six Sigma - An Introduction

    veryard projects > software quality > six sigma > introduction

    Six sigma is a statistical technique for the analysis of quality metrics, whose original application was in high volume manufacturing situations. On this page, we look at the appropriate and inappropriate uses of the technique, and consider how it may be applied to software projects.

    Please accept our apologies for the poor quality of the embedded GIF images.

    veryard projects - innovation for demanding change

    Six Sigma - Statistics

    veryard projects > software quality > six sigma > statistics

    New standards for measurable quality are being set constantly. Figure 1 gives an idea of current levels of achievement in various industries. Systems with the lowest failure rates found today register a defect rate of 3.4 per million. Thankfully, the likelihood of being killed in an air crash is substantially lower still.

    The most quality conscious companies, notably Motorola, are now setting this level of achievement as their goal. It derives from a level of performance that is managed in relation to plus or minus six standard deviations about the mean. This is known as "Six Sigma."

    Six sigma is a mechanism for attacking variation in product and the process that leads to it. How much variation is acceptable? Is a one per cent error rate acceptable?

    Figure 1: Failure Rates in Various Industries

    Many large airports have 200 flights landing each day. A one per cent error rate in landing means that every day two planes will miss the runway. This is clearly unacceptable. If, reluctantly, we accept that two misses in eight years is in some way unavoidable then we are accepting an error rate of 1 in 292,000 (3.4 errors per million). This is the six sigma level.

    With model-based development, a medium-sized model could easily have around 292,000 objects. So we might extrapolate and suggest that six sigma performance demands that there be no more than one error in every medium-sized project. As Motorola points out, a six sigma program is a major step towards defect-free operation.

    Figure 2: Six Sigma

    Six sigma is based on statistical measures in which sigma is one standard deviation about the mean. Six sigma as defined by Motorola, however, is not a simple matter of managing within plus or minus six standard deviations. Taken literally, that level of control would allow for error rates of only 0.002 per million. Instead, the six sigma approach accepts that the mean is not fixed but can drift up and down. It therefore allows a plus or minus 1.5 sigma shift in the mean as the drift within its span of control.

    Six sigma is therefore concerned with managing both the upper and lower limits of specification and the drift in the mean.

    The principle is illustrated in figure 2 where the normal distribution curve is shown along with two others to indicate the point to which a 1.5 sigma shift in the mean takes the curve.

    veryard projects - innovation for demanding change

    Six Sigma measures reliability

    veryard projects > software quality > six sigma > reliability

    Six sigma is a measure of the reliability or predictability of a process. It is particularly appropriate when a process is repeated millions of times within a manageable period.

    Reliability can be roughly expressed as an absence of failure.

    We need to distinguish between the concepts of fault and failure. A failure is an event where a system departs from requirements or expectations (predictions). A fault is a defect that may cause failures. The failure is therefore a symptom that there is a fault somewhere. Note that the fault may be in the software, in the operating instructions, or somewhere else. Thus a design defect may be a fault.

    Failures are not always noticed by the users, and not always reported even when noticed. We therefore also need to distinguish failures from reported failures.

    The relationships between the three concepts are shown in the following diagram.

    Figure 3. Fault / Failure Relationships

    A fault may exist for a long time without causing a reported failure (either because the right combination of inputs never occurs, or because nobody notices or cares). Another fault may cause thousands of different failures, and it may take some time for the software engineers to demonstrate that all these failures are due to a single fault.

    A failure may be detected by special monitoring software. Some systems may be designed to be self-monitoring. However, such automatic monitoring is only likely to pick up certain classes of failure.

    Sometimes it may take several faults acting together to cause a failure. Performance failures may result from the accumulation of many small faults.

    To sum up: You can count either faults or failures, but don’t mix them up.

    Failure metrics are preferred over fault metrics for one simple reason: they tend to be much easier to relate to customer satisfaction, whereas fault metrics tend to be internal engineering-focused.
    more Notes on Failure

    veryard projects - innovation for demanding change

    Pitfalls of six sigma

    veryard projects > software quality > six sigma > pitfalls

    Like any tool, six sigma can be used inappropriately. To the man with a hammer, everything appears to be a nail.
    Defining metrics from producer’s perspective A common failing for engineers is to define quality metrics that cannot be related to customer satisfaction. This is particularly the case with fault metrics. 
    Concentrating on the product, not the process The results of the inspection of a product (such as a complex piece of software) may sometimes be expressed in six sigma terms.

    This means that the inspectors are counting not failures (of the process) but faults (in the product).

    Implicitly, of course, they may be counting failures in the production process. But this approach may be of limited value in quality improvement, because the process errors are aggregated, and therefore difficult to trace. 

    Unreliable testing process There are two ways to get a good score on a six sigma measurement of your manufacturing process. One is to have an excellent manufacturing process. The other is to have an inadequate testing process.

    (ISO 9001 addresses this pitfall explicitly. Clause 4.11 demands that test processes be calibrated.)

    Insufficient volumes for meaningful statistics If you only make a few hundred deliveries a year, it will take thousands of years to demonstrate conformity to six sigma standards (although it may take rather less time to demonstrate non-conformity). For such situations, six sigma measures may be meaningless.

    Note that a single software model, with half a million objects and no known defects, is not large enough to demonstrate six sigma quality. You would need a series of such models before you could claim six sigma quality.

    Measuring unimportant things One way of getting enough things for a statistically significant sample is to decompose the work into very small items.

    Consider an organization producing documentation. They may produce dozens of documents per year, containing thousands of pages and millions of words. To get statistically significant error rates, it may be necessary to count the number of incorrect words.

    The trouble with measuring quality at this minute level of granularity is that they may miss the wood for the trees. All the words may be correct, but the document as a whole may not be fit for purpose.

    more General pitfalls of software metrics

    veryard projects - innovation for demanding change

    Six sigma - a true story

    veryard projects > software quality > six sigma > true story

    A diversified multinational corporation once experienced some success with a metrication programme in a manufacturing division, using six sigma measurements. Top management then decreed that the entire company should use six sigma measurement as a quality tool, and should achieve six sigma levels of quality.

    As a quality tool, six sigma is hungry for large volumes of data, counting in millions. Fine if you have high unit production, or a highly repeatable process. But many departments did not.operate at these volumes. This meant looking for data to feed the tool - often with little or no relevance to customer satisfaction or value - with a distorting effect on quality management.

    For example, a group of technical writers was reduced to counting spelling mistakes in large documents.

    If you are a technical writer, or an occasional user of documentation, and you cannot think of any quality metric which have greater significence or value than the number of spelling miskates, you definately need some consultancy. Click here, urgintely.

    veryard projects - innovation for demanding change

    Use of six sigma for software

    veryard projects > software quality > six sigma > software


    Software Development

    • There are some processes that have been automated by software tools, such as code generation. We may count the number of times (per million) that the software tool generates defective code. This may well approach six sigma quality, provided we have a clear understanding of what counts as defective code, and what technique we shall use for determining the occurrences of defective code.

    • There are some specific processes that need to be carried out at a very low level of granularity, such as the definition of certain classes of objects in a respository or component catalogue. The correctness of these objects (against some objective criteria to be determined) may provide a measure of the reliability of the associated analysis and design processes.

    • The reliability of the testing process may be measured in terms of the correct application of a large volume of test cases.

    • The reliability of such processes as configuration management and version control can be measured by the frequency of objects going astray, or the wrong versions being installed.

    Software Operation

    • The quality of a delivered application system in use can be measured by the frequency that the system produces the wrong business results.
    veryard projects - innovation for demanding change

    Six Sigma - Resources and Links

    veryard projects > software quality > six sigma > resources

    more i-six-sigma


    Thanks to Ian Macdonald and Graham Shevlin.

    Author details

    Richard Veryard is a technology consultant, based in London.


    home page

    contact us

    Original material written October 1994.
    This page last updated on September 16th, 2002
    Copyright © 1994, 2002 Veryard Projects Ltd