Before I read Probability is in the Mind and Probability is Subjectively Objective I was a realist about probabilities; I was a frequentest. After I read them, I was just confused. I couldn't understand how a mind could accurately say the probability of getting a heart in a standard deck of playing cards was not 25%. It wasn't until I tried to explain the contrast between my view and the subjective view in a comment on Probability is Subjectively Objective that I realized I was a subjective Bayesian all along. So, if you've read Probability is in the Mind and read Probability is Subjectively Objective but still feel a little confused, hopefully, this will help.

I should mention that I'm not sure that EY would agree with my view of probability, but the view to be presented agrees with EY's view on at least these propositions:

• Probability is always in a mind, not in the world.
• The probability that an agent should ascribe to a proposition is directly related to that agent's knowledge of the world.
• There is only one correct probability to assign to a proposition given your partial knowledge of the world.
• If there is no uncertainty, there is no probability.

And any position that holds these propositions is a non-realist-subjective view of probability. 

Imagine a pre-shuffled deck of playing cards and two agents (they don't have to be humans), named "Johnny" and "Sally", which are betting 1 dollar each on the suit of the top card. As everyone knows, 1/4 of the cards in a playing card deck are hearts. We will name this belief F₁; F₁ stands for "1/4 of the cards in the deck are hearts.". Johnny and Sally both believe F₁. F₁ is all that Johnny knows about the deck of cards, but sally knows a little bit more about this deck. Sally also knows that 8 of the top 10 cards are hearts. Let F₂ stand for "8 out of the 10 top cards are hearts.". Sally believes F₂. John doesn't know whether or not F₂. F₁ and F₂ are beliefs about the deck of cards and they are either true or false.

So, sally bets that the top card is a heart and Johnny bets against her, i.e., she puts her money on "Top card is a heart." being true; he puts his money on "~The top card is a heart." being true. After they make their bets, one could imagine Johnny making fun of Sally; he might say something like: "Are you nuts? You know, I have a 75% chance of winning. 1/4 of the cards are hearts; you can't argue with that!" Sally might reply: "Don't forget that the probability you assign to '~The top card is a heart.' depends on what you know about the deck. I think you would agree with me that there is an 80% chance that 'The top card is a heart' if you knew just a bit more about the state of the deck."

To be undecided about a proposition is to not know which possible world you are in; am I in the possible world where that proposition is true, or in the one where it is false? Both Johnny and Sally are undecided about "The top card is a heart."; their model of the world splits at that point of representation. Their knowledge is consistent with being in a possible world where the top card is a heart, or in a possible world where the top card is not a heart. The more statements they decide on, the smaller the configuration space of possible worlds they think they might find themselves in; deciding on a proposition takes a chunk off of that configuration space, and the content of that proposition determines the shape of the eliminated chunk; Sally's and Johnny's beliefs constrain their respective expected experiences, but not all the way to a point. The trick when constraining one's space of viable worlds, is to make sure that the real world is among the possible worlds that satisfy your beliefs. Sally still has the upper hand, because her space of viably possible worlds is smaller than Johnny's. There are many more ways you could arrange a standard deck of playing cards that satisfies F₁ than there are ways to arrange a deck of cards that satisfies F₁ and F₂. To be clear, we don't need to believe that possible worlds actually exist to accept this view of belief; we just need to believe that any agent capable of being undecided about a proposition is also capable of imagining alternative ways the world could consistently turn out to be, i.e., capable of imagining possible worlds.

For convenience, we will say that a possible world W, is viable for an agent A, if and only if, W satisfies A's background knowledge of decided propositions, i.e., A thinks that W might be the world it finds itself in.

Of the possible worlds that satisfy F₁, i.e., of the possible worlds where "1/4 of the cards are hearts" is true, 3/4 of them also satisfy "~The top card is a heart." Since Johnny holds that F₁, and since he has no further information that might put stronger restrictions on his space of viable worlds, he ascribes a 75% probability to "~The top card is a heart." Sally, however, holds that F₂ as well as F₁. She knows that of the possible worlds that satisfy F₁ only 1/4 of them satisfy "The top card is a heart." But she holds a proposition that constrains her space of viably possible worlds even further, namely F₂. Most of the possible worlds that satisfy F₁ are eliminated as viable worlds if we hold that F₂ as well, because most of the possible worlds that satisfy F₁ don't satisfy F₂. Of the possible worlds that satisfy F₂ exactly 80% of them satisfy "The top card is a heart." So, duh, Sally assigns an 80% probability to "The top card is a heart." They give that proposition different probabilities, and they are both right in assigning their respective probabilities; they don't disagree about how to assign probabilities, they just have different resources for doing so in this case. P(~The top card is a heart|F₁₎ really is 75% and P(The top card is a heart|F₂) really is 80%.

This setup makes it clear (to me at least) that the right probability to assign to a proposition depends on what you know. The more you know, i.e., the more you constrain the space of worlds you think you might be in, the more useful the probability you assign. The probability that an agent should ascribe to a proposition is directly related to that agent's knowledge of the world.

This setup also makes it easy to see how an agent can be wrong about the probability it assigns to a proposition given its background knowledge. Imagine a third agent, named "Billy", that has the same information as Sally, but say's that there's a 99% chance of "The top card is a heart." Billy doesn't have any information that further constrains the possible worlds he thinks he might find himself in; he's just wrong about the fraction of possible worlds that satisfy F₂ that also satisfy "The top card is a heart.". Of all the possible worlds that satisfy F₂ exactly 80% of them satisfy "The top card is a heart.", no more, no less. There is only one correct probability to assign to a proposition given your partial knowledge.

The last benefit of this way of talking I'll mention is that it makes probability's dependence on ignorance clear. We can imagine another agent that knows the truth value of every proposition, lets call him "FSM". There is only one possible world that satisfies all of FSM's background knowledge; the only viable world for FSM is the real world. Of the possible worlds that satisfy FSM's background knowledge, either all of them satisfy "The top card is a heart." or none of them do, since there is only one viable world for FSM. So the only probabilities FSM can assign to "The top card is a heart." are 1 or 0. In fact, those are the only probabilities FSM can assign to any proposition. If there is no uncertainty, there is no probability.

The world knows whether or not any given proposition is true (assuming determinism). The world itself is never uncertain, only the parts of the world that we call agents can be uncertain. Hence, Probability is always in a mind, not in the world. The probabilities that the universe assigns to a proposition are always 1 or 0, for the same reasons FSM only assigns a 1 or 0, and 1 and 0 aren't really probabilities.

In conclusion, I'll risk the hypothesis that: Where 0≤x≤1, "P(a|b)=x" is true, if and only if, of the possible worlds that satisfy "b", x of them also satisfy "a". Probabilities are propositional attitudes, and the probability value (or range of values) you assign to a proposition is representative of the fraction of possible worlds you find viable that satisfy that proposition. You may be wrong about the value of that fraction, and as a result you may be wrong about the probability you assign.

We may call the position summarized by the hypothesis above "Modal Satisfaction Frequency theory", or "MSF theory".

Previously: Seeing Red: Dissolving Mary's Room and Qualia, A Study of Scarlet: The Conscious Mental Graph

When we left off, we'd introduced a hypothetical organism called Martha whose actions are directed by a mobile graph of simple mental agents. The tip of the iceberg, consisting of the agents that are connected to Martha's language centers, we called the conscious subgraph. Now we're going to place Martha into a situation like Mary's Room: we'll say that a large unconscious agent of hers (like color vision) has never been active, we'll grant her an excellent conscious understanding of that agent, and then we'll see what happens when we activate it for the first time.

But first, there's one more mental agent we need to introduce, one which serves a key purpose in Martha's evolutionary history: a simple agent that identifies learning.

Strategic Reliabilism is an epistemological framework that, unlike other contemporary academic theories, is grounded in psychology and seeks to give genuine advice on how to form beliefs. The framework was first laid out by Michael Bishop and J.D. Trout in their book Epistemology and the Psychology of Human Judgment. Although regular readers here won’t necessarily find a lot of new material here, Bishop and Trout provide a clear description of many of the working assumptions and goals of this community. In contrast to standard epistemology, which seeks to explain what constitutes a justified belief, Strategic Reliabilism is meant to explain excellent reasoning. In particular, reasoning is excellent to the extent it reliably and efficiently produces truths about significant matters. When combined with the Aristotelian principle that good reasoning tends to produce good outcomes in the long run (i.e. rationalists should win), empirical findings about good reasoning gain prescriptive power. Rather than getting bogged down in definitional debates, epistemology really is about being less wrong.

The book is an easily read 150 pages, and I highly recommend you find a copy, but a chapter-by-chapter summary is below. As I said, you might not find a lot of new ideas in this book, but it went a long ways in clarifying how I think about this topic. For instance, even though it can seem trivial to be told to focus on significant problems, these basic issues deserve a little extra thought.

If you enjoy podcasts, check out lukeprog’s interview with Michael Bishop. This article provides another overview of Strategic Reliabilism, addressing objections raised since the publication of the book.

Sequel to: Seeing Red: Dissolving Mary's Room and Qualia

Seriously, you should read first: Dissolving the Question, How an Algorithm Feels From Inside

In the previous post, we introduced the concept of qualia and the thought experiment of Mary's Room, set out to dissolve the question, and decided that we were seeking a simple model of a mind which includes both learning and a conscious/subconscious distinction. Since for now we're just trying to prove a philosophical point, we don't need to worry whether our model corresponds well to the human mind (though it would certainly be convenient if it did); we'll therefore pick an abstract mathematical structure that we can analyze more easily.

Essential Background: Dissolving the Question

How could we fully explain the difference between red and green to a colorblind person?

Well, we could of course draw the analogy between colors of the spectrum and tones of sound; have them learn which objects are typically green and which are typically red (or better yet, give them a video camera with a red filter to look through); explain many of the political, cultural and emotional associations of red and green, and so forth... but it seems that the actual difference between our experience of redness and our experience of greenness is something much harder to convey. If we focus in on that aspect of experience, we end up with the classic philosophical concept of qualia, and the famous thought experiment known as Mary’s Room¹.

Mary is a brilliant neuroscientist who has been colorblind from birth (due to a retina problem; her visual cortex would work normally if it were given the color input). She’s an expert on the electromagnetic spectrum, optics, and the science of color vision. We can postulate, since this is a thought experiment, that she knows and fully understands every physical fact involved in color vision; she knows precisely what happens, on various levels, when the human eye sees red (and the optic nerve transmits particular types of signals, and the visual cortex processes these signals, etc).

One day, Mary gets an operation that fixes her retinas, so that she finally sees in color for the first time. And when she wakes up, she looks at an apple and exclaims, "Oh! So that's what red actually looks like."²

Now, this exclamation poses a challenge to any physical reductionist account of subjective experience. For if the qualia of seeing red could be reduced to a collection of basic facts about the physical world, then Mary would have learned those facts earlier and wouldn't learn anything extra now– but of course it seems that she really does learn something when she sees red for the first time. This is not merely the god-of-the-gaps argument that we haven't yet found a full reductionist explanation of subjective experience, but an intuitive proof that no such explanation would be complete.

The argument in academic philosophy over Mary's Room remains unsettled to this day (though it has an interesting history, including a change of mind on the part of its originator). If we ignore the topic of subjective experience, the arguments for reductionism appear to be quite overwhelming; so why does this objection, in a domain in which our ignorance is so vast³, seem so difficult for reductionists to convincingly reject?

Veterans of this blog will know where I'm going: a question like this needs to be dissolved, not merely answered.

ETA: As stated below, criticizing beliefs is trivial in principle, either they were arrived at with an approximation to Bayes' rule starting with a reasonable prior and then updated with actual observations, or they weren't.  Subsequent conversation made it clear that criticizing behavior is also trivial in principle, since someone is either taking the action that they believe will best suit their preferences, or not.  Finally, criticizing preferences became trivial too -- the relevant question is "Does/will agent X behave as though they have preferences Y", and that's a belief, so go back to Bayes' rule and a reasonable prior. So the entire issue that this post was meant to solve has evaporated, in my opinion. Here's the original article, in case anyone is still interested:

Pancritical rationalism is a fundamental value in Extropianism that has only been mentioned in passing on LessWrong. I think it deserves more attention here. It's an approach to epistemology, that is, the question of "How do we know what we know?", that avoids the contradictions inherent in some of the alternative approaches.

The fundamental source document for it is William Bartley's Retreat to Commitment. He describes three approaches to epistemology, along with the dissatisfying aspects of the other two:

• Nihilism. Nothing matters, so it doesn't matter what you believe. This path is self-consistent, but it gives no guidance.
• Justificationlism. Your belief is justified because it is a consequence of other beliefs. This path is self-contradictory. Eventually you'll go in circles trying to justify the other beliefs, or you'll find beliefs you can't jutify. Justificationalism itself cannot be justified.
• Pancritical rationalism. You have taken the available criticisms for the belief into account and still feel comfortable with the belief. This path gives guidance about what to believe, although it does not uniquely determine one's beliefs. Pancritical rationalism can be criticized, so it is self-consistent in that sense.

Read on for a discussion about emotional consequences and extending this to include preferences and behaviors as well as beliefs.

Meta-Proposal of which this entry is a subset:

The Shortcut Reading Series is a series of less wrong posts that should say what are the minimal readings, as opposed to the normal curriculum, that one ought to read to grasp most of the state of the art conceptions of humans about a particular topic. Time is finite, there is only so much one person can read and thus we need to find the geodesic path to epistemic enlightenment and show it to Less Wrong readers.

Exemplar:

“How not to be a Naïve Computationalist”, the Shortcut Reading Series post in philosophy of mind and language:

This post’s raison d’etre is to be a guide for the minimal amount of philosophy of language and mind necessary for someone who ends up thinking the world and the mind are computable (such as Tegmark, Yudkowsky, Hofstadter, Dennett and many of yourselves) The desired feature which they have achieved, and you soon will, is to be able to state reasons, debugg opponents and understand different paradigms, as opposed to just thinking that it’s 0 and 1’s all the way down and not being able to say why.

This post is not about Continental/Historical Philosophy, about that there have been recommendations in http://lesswrong.com/lw/3gu/the_best_textbooks_on_every_subject/

The order is designed.

What is sine qua non, absolutely necessary, is in bold and OR means you only have to read one, the second one being more awesome and complex.

Language and Mind:

• 37 Ways words can be Wrong - Yudkowsky
• Darwin Dangerous Idea Chapters 3,5, 11, 12 and 14 - Daniel Dennett
• On Denoting - Bertrand Russell
• On What There Is - Quine
• Two Dogmas of Empiricism - Quine
• Namind and Necessity - Kripke OR Two Dimensional Semantics - David Chalmers
• “Is Personal Identity What Matters?” - Derek Parfit
• Breakdown of Will - Part Two (don’t read part 3) George Ainslie
• Concepts of Consciousness 2003 - Ned Block
• Attitudes de dicto and de se - David Lewis- Phil Papers 1
• General Semantics - David Lewis - Phil Papers 1
• The Stuff of Thought, Chapter 3 “Fifty Thousand Innate Concepts” - Steve Pinker
• Beyond Belief - Daniel Dennett in Intentional Stance
• The Content and Epistemology of Phenomenal Belief - David Chalmers
• Quining Qualia OR I Am a Strange Loop OR Consciousness Explained - Dan & Doug
• Intentionality - Pierre Jacob - Stanford Encyclopedia Phil
• Philosophy in the Flesh - Lakoff  & Johnson - Chap 3,4, 12, 21,24 and 25. 

What you cannot find here you probably will on Google or Library.nu (if anyone has a link to Beyond Belief (EDIT: Found it!), post it, it is the only hard to find one)

Congratulations, you are now officially free from the Naïve philosophical computationalism that underlies part of the Less Wrong Community. Your computationalism is now wise and well informed.

Feel free now to delve into some interesting computational proposals such as

• Consciousness as Integrated Information - Giulio Tononi
• What is Thought - Eric Baum
• Good and Real - Gary Drescher
• The Mathematical Universe Hipothesis - Max Tegmark

Dealing with complexity is an inefficient and unnecessary waste of time, attention and mental energy. There is never any justification for things being complex when they could be simple. - Edward de Bono

There are many realms and domains in which the quote above should not be praised. But I think I have all philosophy majors with me when I say that there must be a simpler way to get to the knowledge level we reach upon graduation.

Finally, having wasted substantial amounts of time reading those parts that should not be read of philosophy, and not intending to do the same mistake in other areas, I ask you to publish a selection of readings in your area of expertise, The Sequences are a major rationality shortcut, and we need more of that kind.

If you want to carry a brimming cup of coffee without spilling it, you may want to "change" your goal to instead primarily concentrate on humming. This is an example of a general pattern. It sometimes helps to focus on a nearby artificial goal rather than your actual goal. Let me call that strategy "gamification". There is a business strategy, also named "gamification", of adding game mechanics to a website in order to achieve various business goals. This is related but different. Here I'm referring to a strategy for problem solvers.

We sometimes fail, and sometimes one failure is very similar to another failure. That is, there are characteristic ways that we fail. One of the primary ways that we can improve is to learn our failure modes and create external structures (pieces of paper, software tools) that check, protect against, or head off those forms of failure.

For example, imagine this plan of checklist improvement:

1. Change your normal way of working to include an explicit checklist (that starts empty).
2. When you make a mistake:
   1. Analyze what went wrong
   2. Try to generalize the particular incident to a category
   3. Add an item to your checklist.

This is very simple and generic, but it is reasonable to believe that if you carefully and diligently followed this plan, your reliability would go up (with diminishing returns because your errors are also your opportunities for improvement).  I have not read Mayo, but her "error-theoretic" philosophy of science might be applicable here.

We can try to build a correspondence between failure modes, and game mechanics that attempt to cope for that failure mode.