Surprise tracking

One of the best pieces of advice I've heard about planning was to project yourself forward in time and imagine that the plan has just failed. Are you surprised? If not, your plan needs work.

I think surprise is a seriously undervalued intuition, because it can roll up a whole lot of different factors into a prediction in a very quick, intuitive way. Other ways of accessing those fast predictions ("do you think it will succeed?", "do you think it will fail?", "what risks can you think of?") all seem to get bogged down in biases like optimisim or social pressure, or end up being more a test of imagination than prediction. I wrote about using the worst unsurprising case exactly because I think surprise is uniquely powerful for linking prediction and risk.

But something I only thought of recently is that it might be useful to use this as an ongoing measurement. So instead of just asking "how surprised would you be if this plan failed" at the start of the plan, you could ask on a regular basis throughout the plan's life. Instead of having just one value to work with, you now have a trend. Ideally, that trend should be towards more surprise, or at worst the same. If not, it's probably a sign that your plan is in trouble.

It could work well for more specific predictions as well, like "how surprised would you be if this part of the project took longer than expected", or "how surprised would you be if we got feedback that our software was too hard to use". Over the lifetime of a project, a bunch of similar surprise tracking questions could paint a pretty interesting graph of a whole team's intuitions about the project's success.

It's already very popular to continuously track certain metrics over a project's life, but these are usually objective quantities like burn rate, server capacity, or number of users. Tracking subjective metrics seems like it could be pretty useful too, as long as those metrics were predictive. I think surprise tracking would be a good foundation for that.