x = y

            x² = x*y

     x² - y² = x*y - y²

(x+y)(x-y) = y(x-y)

         x+y = y

         y+y = y

         2*y = y

            2 = 1

The above is an incorrect "proof" that 2=1. Even for those who know where the flaw is, it might seem reasonable to react to the existence of this "proof" by distrusting mathematical reasoning, which might contain such flaws that lead to erroneous results. But done properly, mathematical reasoning does not look like this "proof". It is more explicit, making each step obviously correct that an incorrect step cannot meet the standard. Let's take a look at what would happen when attempting to present this "proof" following this virtue:

I present an algorithm I designed to predict which position a person would report for an issue on TakeOnIt, through Bayesian updates on the evidence of other people's positions on that issue. Additionally, I will point out some potential areas of improvement, in the hopes of inspiring others here to expand on this method.

For those not familiar with TakeOnIt, the basic idea is that there are issues, represented by yes/no questions, on which people can take the positions Agree (A), Mostly Agree (MA), Neutral (N), Mostly Disagree (MD), or Disagree (D). (There are two types of people tracked by TakeOnIt: users who register their own opinions, and Experts/Influencers whose opinions are derived from public quotations.)

The goal is to predict what issue a person S would take on a position, based on the positions registered by other people on that question. To do this, we will use Bayes' Theorem to update the probability that person S takes the position X on issue I, given that person T has taken position Y on issue I:

Really, we will be updating on several people T_j taking positions T_y on I:

Suppose we are building an agent, and we have a particular utility function U over states of the universe that we want the agent to optimize for. So we program into this agent a function CalculateUtility that computes the value of U given its current knowledge. Then we can program it to make decisions by searching through its available actions for the one that maximizes its expectation for its result of running CalculateUtility. But wait, how will an agent with this programming behave?

Suppose the agent has the opportunity (option A) to arrange to falsely believe the universe is in a state that is worth utility u_FA but this action really leads to a different state worth utility u_TA, and a competing opportunity (option B) to actually achieve a state of the universe that has utility u_B, with u_TA < u_B < u_FA. Then the agent will expect that if it takes option A that its CalculateUtility function will return uFA, and if it takes option B that its CalculateUtility function will return u_B. u_FA > u_B, so the agent takes option A, and achieves a states of the universe with utility u_TA which is worse than the utility u_B it could have achieved if it had taken option B. This agent is not a very effective optimization process¹. It would rather falsely believe that it has achieved its goals than actually achieve its goals. This sort of problem² is known as wireheading.

Let us back up a step, and instead program our agent to make decisions by searching through its available actions for the one whose expected results maximizes its current calculation of CalculateUtility. Then, the agent would calculate that option A gives it expected utility u_TA and option B gives it expected utility u_B. u_B > u_TA, so it chooses option B and actually optimizes the universe. That is much better.

So, if you care about states of the universe, and not just your personal experience of maximizing your utility function, you should make choices that maximize your expected utility, not choices that maximize your expectation of perceived utility.

1. We might have expected this to work, because we built our agent to have beliefs that correspond to the actual state of the world.

2. A similar problem occurs if the agent has the opportunity to modify its CalculateUtility function, so it returns large values for states of the universe that would have occurred anyways (or any state of the universe).

In his Coding Horror Blog, Jeff Atwood writes about the Monty Hall Problem and some variants. The classic problem presents the situation in which the game show host allows a contestant to choose one of three doors, one of which opens to reveal a prize while the other two reveals goats. The host then opens one of the other doors, reliably choosing one that has a goat, and invites the contestant to switch to the remaining unopened door. The problem is to determine the probability of winning the prize by switching and staying. The variants deal with cases in which the host does not reliably choose a door with a goat, but happens to do so.

Jeff cites Monty Hall, Monty Fall, Monty Crawl (PDF) by Jeff Rosenthal, which explains why the variants have different probabilities in terms of the "Proportionality Principle", which the appendix acknowledges to be a special case of Bayes' Theorem.

One of Jeff's anonymous commenters presented the Monty Maul Problem:

Hypothetical Situation:

The Monty Maul problem. There are 1 million doors. You pick one, and the shows host goes on a bloodrage fueled binge of insane violence, knocking open doors at random with no knowledge of which door has the car. He knocks open 999,998 doors, leaving your door and one unopened door. None of the opened doors contains the car.

Are your odds of winning if you switch still 50/50, as outlined by the linked Rosenthal paper? It seems counter-intuitive even for people who've wrapped their head around the original problem.

If you take as absolute the problem's statement the host is randomly knocking doors open, then yes, the fact that only goats were revealed is strong evidence that only goats were available because you picked the door with the prize, which, when combined with the low prior probability that you picked the door with the prize, gives equal probability to either of the unopened doors having the prize.

However, the fact that only goats were revealed is also strong evidence that the host deliberately avoided opening the door with the prize, and therefor switching is a winning strategy. After all, the probability of this happening if the host really is choosing doors randomly is 2 in a million, but it is guaranteed if the host deliberately opened only doors with goats.

Note that this principal still applies in variants with fewer doors. Unless there is an actual penalty for switching doors (which could happen if the host only sometimes offers the opportunity to switch, and is more likely to do so when the contestant chooses the winning door), any uncertainty about the host choosing doors randomly implies that it is a good strategy to switch.