Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Regex comments on 0 And 1 Are Not Probabilities - Less Wrong

34 Post author: Eliezer_Yudkowsky 10 January 2008 06:58AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (128)

Sort By: Old

You are viewing a single comment's thread. Show more comments above.

Comment author: Regex 20 August 2015 09:35:55AM 1 point [-]

As someone who doesn't know much beyond basic statistics, in what way are 0 or 1 probabilities? Isn't it just axiomatic truth at that point? In that sense saying zero and one are probabilities is just saying 'certain' or 'impossible' as far as I understand it. Situations where an event will definitely or definitely not occur doesn't seem to be consistent with the idea of randomness which I've understood probability to revolve around.

I suppose the alternative would be that we'd have to assume every mathematical proof has infinite evidence if we wanted to get anywhere productive- after all axioms are assumed to be true. It doesn't make much sense to need evidence in that scenario- except perhaps the probability of error and mistake? That isn't particularly calculable and would actually change from person to person.

Using one and zero makes sense to me as a matter of assumed or proven truths, but I'm still unsure how that makes it a probability.

Comment author: Epictetus 20 August 2015 02:30:48PM 1 point [-]

Situations where an event will definitely or definitely not occur doesn't seem to be consistent with the idea of randomness which I've understood probability to revolve around.

"Event" is a very broad notion. Let's say, for example, that I roll two dice. The sample space is just a collection of pairs (a, b) where "a" is what die 1 shows and "b" is what die 2 shows. An event is any sub-collection of the sample space. So, the event that the numbers sum to 7 is the collection of all such pairs where a + b = 7. The probability of this event is simply the fraction of the sample space it occupies.

If I rolled eight dice, then they'll never sum to seven and I say that that event occurs with probability 0. If I secretly rolled an unknown number of dice, you could reasonably ask me the probability that they sum to seven. If I answer "0", that just means that I rolled more than one and fewer than eight dice. It doesn't make the process less random nor the question less reasonable.

If you treat an event as some question you can ask about the result of a random process, then 1 and 0 make a lot more sense as probabilities.

For the mathematical theory of probability, there are plenty of technical reasons why you want to retain 1 and 0 as probabilities (and once you get into continuous distributions, it turns out that probability 1 just means "almost certain").

Comment author: Regex 21 August 2015 08:36:41AM 0 points [-]

This is what I meant by something being a proven truth- within the rules set one can find outcomes which are axiomatically impossible or necessary. The process itself may be random, but calling it random when something impossible didn't happen seems odd to me. The very idea that 1 may be not-quite-certain is more than a little baffling, and I suspect is the heart of the issue.

Comment author: Epictetus 21 August 2015 02:01:03PM 1 point [-]

The very idea that 1 may be not-quite-certain is more than a little baffling, and I suspect is the heart of the issue.

If 1 isn't quite certain then neither is 0 (if something happens with probability 1, then the probability of it not happening is 0). It's one of those things that pops up when dealing with infinity.

It's best illustrated with an example. Let's say we play a game where we flip a coin and I pay you $1 if it's heads and you pay me $1 if it's tails. With probability 1, one of us will eventually go broke (see Gambler's ruin). It's easy think of a sequence of coin flips where this never happens; for example, if heads and tails alternated. The theory holds that such a sequence occurs with probability 0. Yet this does not make it impossible.

It can be thought of as the result of a limiting process. If I looked at sequences of N of coin flips, counted the ones where no one went broke and divided this by the total number of possible sequences, then as I let N go to infinity this ratio would go to zero. This event occupies an region with area 0 in the sample space.

Comment author: Regex 22 August 2015 03:01:15PM 0 points [-]

If the limit converges then it can hit 0 or 1. Got it. Thank you.

Comment author: StellaAthena 20 August 2015 08:14:17PM 1 point [-]

Formally, probability is defined via areas. The basic idea is that the probability of picking an element from a set A out of a set B is the ratio of the areas of A to B, where "area" can be defined not only for things like squares but also things like lines, or actually almost every* subset of R. So, lets say you want to randomly select a real number from the interval [0,1] and want to know the odds it falls in a set, S. The area of [0,1] is 1, so the answer is just the area of S.

If S={0}, then S has area zero. If S=[0,1), then S has area 1. Not only are both of these theoretical possibilities, they are practical ones too. There are real world examples of probability zero events (the only one that comes to mind involves QM though so I don't want to bother with the details).

Now, notice that this isn't the same thing as "impossible". Instead, it means more like "it won't happen I promise even by the time the universe ends". The way I tend to think about probability zero events is that they are so unlikely they are beyond the reach of the principle that as the number of trials increases, events become expected. For any nonzero probability, there is a number of trials, n, such that once you do it n times the expected value becomes greater than 1. That's not the case with probability zero events. Probability 1 events can then be thought of as the negation of probability 0 events.

*not actually "almost every" in a formal sense, but "almost any" in a "unless you go try to build a set that you can't measure it probably has a well defined area" sense

Comment author: Regex 21 August 2015 08:24:39AM 1 point [-]

That seems a solid enough explanation, but how can something of probability zero have a chance to occur? How then do you represent an impossible outcome? It seems like otherwise 'zero' is equivalent to 'absurdly low'. That doesn't quite jive with my understanding.

Comment author: StellaAthena 21 August 2015 09:37:41PM 1 point [-]

Impossible things also have a probability of zero. I totally understand that this seems a bit unintuitive, and the underlying structure (which includes things like infinities of different sizes) is generally pretty unintuitive at first. Which is kinda just saying "sorry, I can't explain the intuition," which is unfortunately true.

Comment author: Regex 22 August 2015 02:47:33PM 0 points [-]

I'm just going to think of it as taking the limit as evidence approaches infinity. Because a probability next to zero and zero are identical, zero then is a probability?

Comment author: Stephen_Cole 22 August 2015 12:07:25AM 1 point [-]

I think one of the clearest expositions on these issues is ET Jaynes. The first three chapters (which is some of the relevant part) can be found at http://bayes.wustl.edu/etj/prob/book.pdf.

Comment author: Regex 22 August 2015 02:39:02PM 0 points [-]

"Not Found

The requested URL /etj/prob/book.pdf. was not found on this server."

Comment author: arundelo 22 August 2015 02:59:40PM 2 points [-]

Fixed Jaynes link (no trailing period).

Comment author: Regex 22 August 2015 03:01:47PM 0 points [-]

Ah. Thanks!

Comment author: Stephen_Cole 22 August 2015 03:17:11PM 0 points [-]

Oops. Thanks for the fix!