How To Measure Intuition in Agile Project Planing and Estimation

Developers will often estimate the time and cost of some work using their intuition. This implies that intuition acts like a measuring device.

Further, the statistical revolution brought us the concept that all measurements and measuring tools have inherent uncertainty and error, and attempts to deal with it.

Students typically learn this concept in physics class by measuring the acceleration of gravity. When the student runs a sequence of experiments to determine the acceleration of gravity they get a collection of different results. From that collection of data, they learn how to estimate the “true” acceleration of gravity and the error bound associated with their estimate.

Using that concept, I would like to tackle the problem of how to measure intuition with respect to agile project management.

A simple thought experiment

Let’s use the following thought experiment to illustrate how to measure “intuition”.

Suppose I predicted the amount of time it takes to finish some collection of user stories. Also suppose that I gave my confidence (measured in percent) in achieving these results within that time.

By comparing my predictions against what actually happens, we could estimate the quality of my intuition.

For example, estimates predicted at a 50% level mean that I expect to make correct predictions 50% of the time and incorrect predictions 50% of the time. Therefore, if I get 100% of my 50% predictions right then I incorrectly assigned my predictions to the 50% confidence level, but if I get 50% of my 50% predictions correct then I accurately assessed them.

You can apply the same reasoning for the 60% level, 70% level, etc …

For the sake of illustration, suppose that I tabulated the results of my predictions for user stories along with the results in the following table.

Story

Predicted Time

Confidence Level

Actual Time

Result

1

8 hours

50%

8 hours

Success

2

8 hours

50%

9 hours

Fail

3

8 hours

60%

8 hours

Success

4

8 hours

60%

8 hours

Success

5

8 hours

60%

8 hours

Success

6

8 hours

60%

9 hours

Fail

7

8 hours

70%

8 hours

Success

8

8 hours

70%

8 hours

Success

9

8 hours

70%

8 hours

Success

10

8 hours

70%

9 hours

Fail

11

8 hours

80%

8 hours

Success

12

8 hours

80%

8 hours

Success

13

8 hours

80%

8 hours

Success

14

8 hours

80%

8 hours

Success

15

8 hours

80%

9 hours

Fail

16

8 hours

90%

8 hours

Success

17

8 hours

100%

8 hours

Success

From this we can gather the following information

  • I correctly predicted 1/2 (50%) at the 50% confidence level
  • I correctly predicted 3/4 (75%) at the 60% confidence level
  • I correctly predicted 3/4 (75%) at the 70% confidence level
  • I correctly predicted 4/5 (80%) at the 80% confidence level
  • I correctly predicted 1/1 (100%) at the 90% confidence level
  • I correctly predicted 1/1 (100%) at the 100% confidence level

Let’s look at the regression line associated with this data.

figure_1

In the figure above, the x axis represents “true” accuracy while the y axis represents predicted accuracy. Each point represents intuition at a particular “confidence level”. The dashed line represents “perfect” intuition; so, we ideally want
point as close to the dashed line as possible. The blue line is the regression line associated for all the points and represents a persons overall intuition.

Through this interpretive framework, the data suggest that I generally have under-confident estimates.

Now, suppose the results ended up looking like the following, instead:

  • I correctly predicted 1/2 (50%) at the 50% confidence level
  • I correctly predicted 2/4 (50%) at the 60% confidence level
  • I correctly predicted 2/4 (50%) at the 70% confidence level
  • I correctly predicted 3/5 (60%) at the 80% confidence level
  • I correctly predicted 1/1 (100%) at the 90% confidence level
  • I correctly predicted 1/1 (100%) at the 100% confidence level

The chart would then change to the following:

figure_2

In this case, the regression line suggests that I have over-confident estimates.

Some Caveats

I used the discussion and examples purely for illustrative purposes. I want to appeal more to your intuition rather than providing something very mathematically rigorous.

Potential Applications

I can imagine many different applications to this framework. A couple applications from the top of my head include:

  • Suppose that we had a poker planning session and we had a difference between how people scored a user story. A project manager could use the quality of someones intuition to make decisions about project planning.
  • Someone could use their the regression line to help calibrate their own intuition (similar to how scientists have to calibrate their instruments). If someone knew that they had a tendency to over-estimate or under-estimate at a certain
    confidence level then they could theoretically use that information to train their intuition.
  • Suppose our team failed to meet our estimates 3 times in a row. However, suppose that we also only had a 50% confidence in those estimates. In this case, we can still consider our estimates as correct because we had a 1/8 (12%) chance of making 3 incorrect estimates in a row.
  • Failure to meet estimates becomes a source of information that helps improve estimates. Since we’ve treated estimates as a random variable, we’ve acknowledged that uncertainty and error exist. However, we also have a way to measure it, and use it to make predictions.

Conclusion

This is all pretty much theoretical, but I think that it might have useful applications. I will spend time thinking about it and I will continue to publish my thoughts and results.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s