I always wondered at the arbitrariness of the 95% confidence level. What if two sets of data show they have a 94% chance of being significantly different? Do we just assume they are not?

A couple of decades ago, I had an interesting conversation with a petroleum engineer who was doing reservoir modelling to predict future performance of an oilfield. After doing this for about a decade, he came to the conclusion that his modelling was reasonably accurate 70% of the time. And the other 30%, , which was usually too optimistic in estimating future production, was a source of frustration for him.

I did a little math for him to show the cost of doing the modelling was saving petroleum companies in the long run — even with the 30% failure rate.

In other words, we should expect some failure in whatever we do. We just need to know the risk and consequences of the risk. When we really understand these factors, then we can make appropriate adjustments to mitigate the risk or its consequences.

For example, a 30% failure in reservoir modelling will not result in a loss of life or degradation of the environment. There is a monetary cost for sure. But the price of oil usually plays a bigger part of the risk than the decision whether to model or not model the reservoir.

Nice article. Too bad most of the world won’t have much an idea of what you are talking about. We do need to become better educated with statistics.