Thanks, I have yet to learn the relevant math. I only have a very vague idea about Bayesian probability at this point so my use of "update" might very well be wrong, as you indicated.
There are some things I am particularly confused about when it comes to probability. I am looking forward to the time when I start learning probability theory (currently doing Calculus).
Just a two examples:
I don't understand why you can't just ignore some possible outcomes. We are computationally limited agents after all. Thus if someone comes a long and tells me about a certain possibility that involves a huge amount of utility but doesn't provide enough (how much would be enough anyway?) evidence, my intuition is to just ignore that possibility rather than to assign 50% probability to it. After all, if humanity was forced to throw a fair coin and heads would imply doom then I'd be in favor of doing everything to ensure that the coin doesn't land heads (e.g. a research project that figures out how to throw it in a certain way that maximizes the probability of it landing tails). But that would be crazy because anyone could come along and make unjustified predictions that involve huge utility stakes and I would have to ignore all lower utility possibilities even if there is a lot of empirical evidence in favor of them.
Something else I am confused about is how probability is grounded. What rules are used to decide how much certain evidence can influence my probability estimates? I mean, if using probability theory only shifts my use of intuition towards the intuitive assignment of numerical probability estimates of evidence, in favor of a possibility, then it might as well distort my estimates more than trusting my intuition about the possibility alone. Because 1.) if using my intuition to decide how much each piece of evidence changes the overall probability, I can be wrong on each occasion rather than just one (accumulation of error) 2.) humans are really bad with numbers, and forcing them to put a number on something vague might in some cases cause them to over or underestimate their confidence by many orders of magnitude.
I only have a very vague idea about Bayesian probability at this point so my use of "update" might very well be wrong
I think most people use "update" colloquially, i.e. something along the lines of "what you just said appears to constitute evidence that I wasn't previously aware of, and I should change my beliefs accordingly". I don't know how often rationalists actually plug these things into a formula.
I don't understand why you can't just ignore some possible outcomes.
This is the problem of Pascal's Mugging - I think ...
What did Will mean? To take an idea seriously is “to update a belief and then accurately and completely propagate that belief update through the entire web of beliefs in which it is embedded,” as in a Bayesian belief network (see right).
Belief propagation is what happened, for example, when I first encountered that thundering paragraph from I.J. Good (1965):
Good’s paragraph ran me over like a train. Not because it was absurd, but because it was clearly true. Intelligence explosion was a direct consequence of things I already believed, I just hadn’t noticed! Humans do not automatically propagate their beliefs, so I hadn’t noticed that my worldview already implied intelligence explosion.
I spent a week looking for counterarguments, to check whether I was missing something, and then accepted intelligence explosion to be likely (so long as scientific progress continued). And though I hadn’t read Eliezer on the complexity of value, I had read David Hume and Joshua Greene. So I already understood that an arbitrary artificial intelligence would almost certainly not share our values.
Accepting my belief update about intelligence explosion, I propagated its implications throughout my web of beliefs. I realized that:
I had encountered the I.J. Good paragraph on Less Wrong, so I put my other projects on hold and spent the next month reading almost everything Eliezer had written. I also found articles by Nick Bostrom and Steve Omohundro. I began writing articles for Less Wrong and learning from the community. I applied to Singularity Institute’s Visiting Fellows program and was accepted. I quit my job in L.A., moved to Berkeley, worked my ass off, got hired, and started collecting research related to rationality and intelligence explosion.
My story surprises people because it is unusual. Human brains don’t usually propagate new beliefs so thoroughly.
But this isn’t just another post on taking ideas seriously. Will already offered some ideas on how to propagate beliefs. He also listed some ideas that most people probably aren’t taking seriously enough. My purpose here is to examine one prerequisite of successful belief propagation: actually making sure your beliefs are connected to each other in the first place.
If your beliefs aren’t connected to each other, there may be no paths along which you can propagate a new belief update.
I’m not talking about the problem of free-floating beliefs that don’t control your anticipations. No, I’m talking about “proper” beliefs that require observation, can be updated by evidence, and pay rent in anticipated experiences. The trouble is that even proper beliefs can be inadequately connected to other proper beliefs inside the human mind.
I wrote this post because I'm not sure what the "making sure your beliefs are actually connected in the first place" skill looks like when broken down to the 5-second level.
I was chatting about this with atucker, who told me he noticed that successful businessmen may have this trait more often than others. But what are they doing, at the 5-second level? What are people like Eliezer and Carl doing? How does one engage in the purposeful decompartmentalization of one's own mind?