Estimation

Estimation - Confidence and Contingency

veryard projects > project management > estimation

estimation - improving the accuracy

project estimation

links

In a paper published in 1992 and based on experience at Shell, Russo & Schoemaker argue that people almost always overrate their ability to make accurate predictions. Even in areas directly relevant to a person’s professional expertise, most people believe themselves to have more knowledge and predictive skill than they in fact have. Many of their findings show astonishingly large discrepancies between actual ability and self-assessed ability.

They argue that professionals need to calibrate their own knowledge and skills, to know when to be certain, and when to have doubts. Unnecessary doubts may incapacitate the professional, and create an impression of incompetance, but unjustified complacency can be equally disastrous in the long run.

Software project managers are required to estimate the size of a project. They will usually add a percentage for ‘contingency’, to allow for their uncertainty. However, if their estimates are overconfident, these ‘contingency’ amounts may be insufficient, and significant risks may be ignored. Sometimes several such ‘contingency’ amounts may be multiplied together, but this is a clumsy device which can lead to absurdly high estimates, while still ignoring significant risks. In some cases, higher management will add additional ‘contingency’, to allow for the fallibility of the project manager. Game-playing around 'contingency' is rife.

Project managers should be trained to be more closely aware of the limitations of their own estimation powers, which will lead to project plans that explicitly match the project capabilities and risks, as well as contingency plans that should be more meaningful.

author
profile

> forecasting
> measurement & control
> project management
> risk management
> quality management

project
management
notions

> shell
> overconfidence
> causes of overconfidence
> overcoming overconfidence
> reference

> risk analysis
> project contingency
> cautious estimate, confident action
> conclusions

Shell's estimation capabilities

veryard projects > project management > estimation > shell

A paper published in 1992 used Shell to illustrate best practice in estimation.

In the 1970s, geologists at Shell were excessively confident when they predicted the presence of oil or gas. They would for example estimate a 40% chance of finding oil, but when ten such wells were actually drilled, only one or two would produce. This overconfidence cost Shell considerable time and money. Shell embarked on a training programme, which supposedly enabled the geologists to be more realistic about the accuracy of their predictions. Following this programme, when Shell geologists predicted a 40% chance of finding oil, four out of ten were successful.

Shell

Overconfidence

veryard projects > project management > estimation > overconfidence

According to Russo & Schoemaker, people almost always overrate their ability to make accurate predictions. Even in areas directly relevant to a person’s professional expertise, most people believe themselves to have more knowledge and predictive skill than they in fact have. Many of their findings show astonishingly large discrepancies between actual ability and self-assessed ability.

Causes of Overconfidence

Why are people generally overconfident? There is a cocktail of reasons:

Availability	People have difficulty imagining all the ways events could unfold. Psychologists call this the availability bias: out of sight equals out of mind. Therefore, we tend to assume that everything will go normally. (Question: if ten independent CSFs each have a confidence level of 90%, what is the probability of at least one of them failing? Answer: 65%)
Anchoring	Most people start with a best guess, and then not adjust away from it sufficiently. Their best guess ‘anchors’ their thinking, and prevents them imagining circumstances that might invalidate it. This is also known as the Confirmation Bias.
Hindsight	Hindsight (in which we convince ourselves we always knew, or could have known, what would happen) makes us believe that the world is more predictable than it really is.
Groupthink	Sometimes, when a majority in a group hold a particular opinion, minority opinions may be witheld for fear of unpopularity or looking foolish.

Overcoming Overconfidence

Here are some techniques for gauging (or improving the ability to gauge) the accuracy of one’s predictions:

Accelerated feedback	A training programme where professionals are given large numbers of case studies to estimate, so that they can themselves see how frequently they get the answers right, enables the professional to calibrate her own estimation skills.
Counterargumentation	Make assumptions explicit, and get others to challenge them.
Deanchoring	Instead of starting with one best guess, a better technique is to start with the highest possible and lowest possible value. Then look for assumptions that can be explicitly introduced to narrow the range.
Fault trees	Draw hierarchical diagrams of everything that can go wrong.
Scenarios	Produce two or three alternative ‘scripts’ for how the project might unfold.
Group	Decision-making in groups (especially following specific techniques, such as Delphi, designed to avoid the dangers of groupthink) can be less susceptible to overconfidence.
Awareness	Sometimes simply being aware of the dangers is enough. An exercise to demonstrate to a person that her expertise is less than she thinks may moderate her overconfidence.

References

J.E. Russo & P.J.H. Schoemaker, "Managing Overconfidence" Sloan Management Review Winter 1992, Vol 33 No 2, pp 7-19

Project Estimation

veryard projects > project management > estimation > project estimation

Cautious Estimation, Confident Action

Although a project manager should have many doubts during the planning and estimation phase of the project, this does not mean a hesitant and unconfident attitude should last into the project itself. On the contrary, because all the possible doubts have been expressed and eliminated during the planning stage, the project manager should have the highest possible confidence in the robustness of the plan, be able to convince higher management to support it, and be able to motivate herself and her team to achieve it.

Calibration

Project planning requires, not only the ability to make reasonably accurate estimates, but to understand the limitations of these estimates, and to explicitly plan for their inaccuracy.

Software project managers need to calibrate their estimation skills. The example of Shell geologists (together with other examples cited by Russo & Schoemaker) shows that focused training can help. This could include both training in the different techniques of estimation (and estimation critiques), together with case studies for project managers to learn how good their estimation skills really are.

Project Risk Analysis

veryard projects > project management > estimation > project risk analysis

At a given stage in a project, there is a given level of knowledge and uncertainty about the outcome and cost of the project. The probable cost can typically be expressed as a skewed bell curve, since although there is a minimum cost, there is no maximum cost.

cost curve

There are several points of particular interest on this curve:

Minimum	The lowest possible cost.
Mode	The most likely cost. This is the highest point on the curve.
Median	The midway cost of n projects. (In other words, n/2 will cost less than the median, and n/2 will cost more.)
Average	The expected cost of n similar projects, divided by n.
Reasonable maximum	The highest possible cost, to a 95% certainty.
Absolute maximum	The highest possible cost, to a 100% certainty.

On this curve, the following sequence holds: minimum < mode < median < average < reasonable maximum < absolute maximum Note the following points: • The absolute maximum cost may be infinite, although there is an infinitesimal tail. For practical purposes, we can take the reasonable maximum cost. However, the reasonable maximum may be two or three times as great as the average cost.

• Most estimation algorithms aim to calculate the mode. This means that the chance that the estimates will be achieved on a single project is much less than 50%, and the chances that the total estimates will be achieved on a series of projects is even lower. (In other words, you do not gain as much on the roundabouts as you lose on the swings.)

• A tall thin curve represents greater certainty, and a low broad curve represents greater uncertainty.

• Risk itself has a negative value. This is why investors demand a higher return on risky ventures than on ‘gilt-edged’ securities. Financial number-crunchers use a measure of risk called ‘beta’.

• Therefore, any information that reduces risk has a positive value.

This analysis yields the following management points: • A project has a series of decision points, at each of which the sponsors could choose to cancel.

• Thus at decision point k, the sponsors choose between continuation, which is then expected to cost Rk, and cancellation, which will cost Ck.

• The game is to reduce risk as quickly as possible, and to place decisions at the optimal points.

• This means we have to understand what specific information is relevant to the reduction of risk, plan the project so that this information emerges as early as possible, and to place the decision points immediately after this information is available.

Project Contingency

veryard projects > project management > estimation > project contingency

One meaning of the word ‘contingency’ is a plan for a specific alternative scenario. If all goes well, we will follow Plan A, but if X happens, we will follow Plan B. Plan B is known as a contingency plan. If Plan B is more expensive, then the project manager needs a contingency budget: in other words, a budget that will only need to be spent if X occurs.

Ideally, all contingency budgets should be tied to specific risks. For a project manager to ask for extra budget, to be spent if anything goes wrong, without being able to state the content, is to avoid planning for contingency altogether. After all, there is no objective basis for asking for a 20% contingency budget rather than 15% or 40%. Some projects overrun by 200% or more: would this justify the project manager asking for a 200% contingency budget?

It has long been observed that people will acknowledge general error more easily than specific error. Thus a project manager will admit that some of her estimates may be wrong, but will strongly defend each individual estimate. This is not perverse, but a natural human characteristic.

Furthermore, there is a practical limit to the amount of analysis and contingency planning that can be done upfront. At some point, the project manager and her superiors have to accept a plan at a given level of detail and completeness, and make a start with the project itself.

Nonetheless, the error of insufficient risk analysis and contingency planning is rather more frequent than the opposite error of excessive risk analysis and contingency planning.

Three Notions of Contingency
Risk Management
Crisis Management

veryard projects > project management > estimation