What do probabilities measure, and why does it matter?

Wikipedia defines measurement as “the assignment of numbers to objects or events. It is a cornerstone of most natural sciences, technology, economics, and quantitative research in other social sciences”.

For example, a gram measures weight, a meter measures distance, and a liter measures volume.

So, what does a percent measure?

Well, it can measure at least 3 different things

  • frequency
  • belief
  • vagueness

Each measure gives rise to different disciplines.

A brief history

The history of probability and statistics is really the history of people. For that reason, I’d like to focus on the people involved in various school of thought.

The Frequentists

The frequentist tradition views probability as the result of calculating the ratio of “successes” vs “total outcomes”. Pierre Fermat, Blaise Pascal, Galileo, Gaus, and Jacob Bernoulli devised methods associated with this view of probability.

As a consequence of this view, probability only made sense when applied to large collections of objects or events.  For example, according to this view, it does not make sense to talk about the probability of a single coin flip. You could only talk about the frequency of head vs tails given a large collection of coin flips.

The Bayesians

The Bayesian tradition views probability as a measure of partial knowledge.

Consider the example of a coin flip. If we flip a coin then that coin will either land heads or tails. There is no in-between. However, we can have a partial belief in the outcome, even though the outcome can only be heads or tails.

Thomas Bayes developed this theory from a very simple equation we now know as Baye’s Theorem. With Baye’s theorem we can essentially measure the likelihood of an individual event if it is conditional on something else.

In the 1920s, John Maynard Keynes built on that foundation by developing methods that we now know as “objective” Bayesianism. Under this school of thought, given the same amount of information, everyone should have the same belief which we can measure with probability.

Around the same time, Frank Ramsey and Bruno de Finetti  proposed that you cannot really measure what people should believe. You can only measure what they actually do believe. As a result, probabilities measured the subjective beliefs of an individual. Naturally, these methods belong to “subjective” Bayesianism.

Fuzzy Logic

Lotfali Zadeh invented fuzzy logic (fuzzy set theory) starting in the 1960s as a means of measuring “truth”.

Lotfali Zadeh asked what would happen if we had partial (fuzzy) truth. That leads to very different methods and approaches.

Consider the case of a pot of cooking rice. We want to know if the rice is finished. However, we don’t really have a clear distinction of when the rice is cooked or when it is partially cooked. In this case, we say that the rice is sort of cooked because the distinction between the two categories are “fuzzy”.

Why these distinctions matters?

These are not distinctions without a difference. Improper use of probabilistic language can actually lead to confusion and bad decisions.

Consider the following scenario.

Using random sampling, you can say something about the parameters of a distribution within a certain accuracy and precision. In the frequentist tradition, we measure accuracy with error bounds and precision with confidence intervals.

Consider the error bound as a measure of the result’s reliability and the confidence interval as a measure of the method’s reliability.

For example, I could say that 50% of California voters will vote yes on a proposition within a 3% error bound and 95% confidence interval. In this case, the 3% error bound says that the true average is actually somewhere between 47% and 53% and that the method I used will be correct 95 times out of 100.

However, I have seen people state that the 95% confidence interval means that they are “95% sure” that the parameter actually lies within that interval. That is completely wrong.

This might seem like a minor distinction, but that distinction has major consequences in decision making: calculating a degree of belief is very different from calculating a confidence interval.

I’ll leave that discussion to another blog post.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s