To those who are saying things like "Eliezer, someone will get power anyway and they'll probably be worse than you, so why not grab power for yourself?", and assuming for the sake of argument that we're talking about some quantity of power that Eliezer is actually in a position to grab: If you grab power and it corrupts you, that's bad not only for everyone else but also for you and whatever your goals were before you got corrupted. Observing that other people would be corrupted just as badly defuses the first of those objections to power-grabbing, but not the second.

Bo, the point is that what's most difficult in these cases isn't the thing that the 10-year-old can do intuitively (namely, evaluating whether a belief is credible, in the absence of strong prejudices about it) but something quite different: noticing the warning signs of those strong prejudices and then getting rid of them or getting past them. 10-year-olds aren't specially good at that. Most 10-year-olds who believe silly things turn into 11-year-olds who believe the same silly things.

Eliezer talks about allocating "some uninterrupted hours", but for me a proper Crisis of Faith takes longer than that, by orders of magnitude. If I've got some idea deeply embedded in my psyche but am now seriously doubting it (or at least considering the possibility of seriously doubting it), then either it's right after all (in which case I shouldn't change my mind in a hurry) or I've demonstrated my ability to be very badly wrong about it despite thinking about it a lot. In either case, I need to be very thorough about rethinking it, both because that way I may be less likely to get it wrong and because that way I'm less likely to spend the rest of my life worrying that I missed something important.

Yes, of course, a perfect reasoner would be able to sit down and go through all the key points quickly and methodically, and wouldn't take months to do it. (Unless there were a big pile of empirical evidence that needed gathering.) But if you find yourself needing a Crisis of Faith, then ipso facto you *aren't* a perfect reasoner on the topic in question.

Wherefore, I at least don't have the *time* to stage a Crisis of Faith about every deeply held belief that shows signs of meriting one.

I think there would be value in some OB posts about resource allocation: deciding which biases to attack first, how much effort to put into updating which beliefs, how to prioritize evidence-gathering versus theorizing, and so on and so forth. (We can't Make An Extraordinary Effort every single time.) It's a very important aspect of practical rationality.

Unfortunately, the capabilities of an omnipotent being are themselves not very well defined. Suppose we want to determine whether "The Absolute is an uncle" is meaningful. Well, says the deranged Hegelian arguing the affirmative, of course it is: we just ask our omnipotent being to take a look and see whether the Absolute is an uncle or not.

Butbutbutbut, you say, we can't tell it how to do that, whereas we can tell it how to check whether there's a spaceship past the cosmological horizon. But can we really? I mean, it's not like we know how to make that observation, or we'd be able to make it ourselves. What's the difference between this and checking whether the Absolute is an uncle? "Well, we know what it means to check whether there's a spaceship past the cosmological horizon, but not what it means for the Absolute to be an uncle." Circular argument alert!

It does *feel* like there's a difference that we can use, but trying to formulate it exactly seems to lead to a circular definition.

(No one is really going to defend "The Absolute is an uncle", but there certainly are people prepared to claim that the existence of an afterlife is testable because dead people might discover it, or because God could tell you whether it's there or not; and I don't think any sort of logical positivist would agree.)

Interesting aesthetic question raised by Caledonian's comment: "not beckoning, but drowning" versus "not wading, but drowning". I think the latter would have worked much better, but presumably C. thought it too obvious and wanted to preserve more of Stevie Smith's semantics. :-)

Arthur, what would keeping a time coordinate buy you in your scenario? Suppose, simplifying for convenience, we have A -> B -> C -> B [cycle], and suppose each state completely determines its successor. What advantage would there be to labelling our states (A,0), (B,1), (C,2), (B,3), (C,4), etc., instead of just A,B,C? Note that there's no observable difference between, say, (B,1) and (B,3); in particular, no memory or record of the past can distinguish them because those things would have to be part of state B itself.

I think David Deutsch has a similar unsorted-pile-of-block-slices view of the world. I don't know if either was influenced by the other.

You can make positions relative in ways other than using pairwise distances as your coordinates. For instance, just take R^4n (or R^11n or whatever) and quotient by the appropriate group of isometries of R^4 or R^11 or whatever. That way you get a dimension linear in the number of particles. The space might be more complicated topologically, but if you take general relativity seriously then I think you have to be prepared to cope with that anyway.

So, in Eliezer's example of triangles in 2-space, we start off with R^6; letting E be the group of isometries of R^2 (three-dimensional: two dimensions for translation, one for rotation, and we also have two components because we can either reflect or not), it acts on R^6 by applying each isometry uniformly to three pairs of dimensions; quotienting R^6 by this action of E, you're left with a 2-dimensional quotient space.

Of course you end up with the same result (up to isomorphism) this way as you would by considering pairwise distances and then noticing that you're working in a small subset of the O(N^2)-dimensional space defined by distances. But you don't have to go via the far-too-many-dimensional space to get there.

But ... suppose the laws of physics are defined over a quotient space like this. From the anti-epiphenomenal viewpoint, I wonder whether we should consider the quantities in the original un-quotiented space to be "real" or not. Consider quantum-mechanical phase or magnetic vector potential, which aren't observable (though other things best thought of as quotients of them are). Preferring to see the quotiented things as fundamental seems to me like the same sort of error as Eliezer (I think rightly) accuses single-world-ists of.

But ... the space of distance-tuples (appropriately subsetted) and the space of position-tuples (appropriately quotiented) are the *same* space, as I mentioned earlier. So, how to choose? Simplicity, of course. And, so far as we can currently tell, the laws of physics are simpler when expressed in terms of positions than when expressed in terms of distances. So, for me and pending the discovery of some newer better way of expressing the state space that supports our churning quantum mist, sticking with absolute positions seems better for now.

Yeah, but when playing actual Taboo "rational agents should WIN" (Yudkowsky, E.) and therefore favour "nine innings and three outs" over your definition (which would also cover some related-but-different games such as rounders, I think). I suspect something like "Babe Ruth" would in fact lead to a quicker win.

None of which is relevant to your actual point, which I think a very good one. I don't think the tool is all *that* nonstandard; e.g., it's closely related to the positivist/verificationist idea that a statement has meaning only if it can be paraphrased in terms of directly (ha!) observable stuff.

Lee, I'm confident that you'd find that "97 is approximately 100" seems more natural to most people than "100 is approximately 99". As for the percentage differences, (1) why should the percentage difference be the thing to focus on rather than the absolute difference, and (2) why do it that way around? (Only, I think, because of the effect I mentioned above: when you say "X is approximately Y" you're implicitly suggesting Y as a standard of comparison, because it's useful for that purpose one way or another.)

Tiiba, I don't think what I described is a bias, but perhaps I didn't explain it well. I'm proposing that in phrases like "X is approximately Y" and "X is like Y", the connectives are not intended to be taken as symmetrical relations like "differs little from"; rather, they mean something like "If you want to know about X, it may be useful to think about Y instead". And I don't see anything wrong with that, as such.

Let me give an analogy from a field where bias is quite effectively eliminated: pure mathematics. Mathematicians have various notations they use to express relationships of the form "this function is bigger than that one for large x". One of them, written something like "f ~ g", means "the ratio f/g tends to 1 in whatever limiting case we're interested in" (n -> oo, x -> 0, whatever). This really is a symmetrical relation; f ~ g if and only if g ~ f. But if you ask mathematicians which of "x^3+17x^2-25x+1 ~ x^3" and "x^3 ~ x^3+17x^2-25x+1" is more *natural* then I bet they'll quite consistently go for the former.

Now, if you want to call it a "bias" every time some term that looks symmetrical is used asymmetrically as a matter of convention or convenience, fair enough. I'd prefer to reserve "bias" for cases where the asymmetrical usage actually causes, or is a symptom of, error. As I say, I'm sure there's plenty of error caused by typicality heuristics; but I don't see that the asymmetry in the use of phrases like "is like" is, or indicates, an error.

(What "wrong question" do you think is being answered here?)

Has it been established that people who prefer "98 is approximately 100" to "100 is approximately 98" or "Mexico is like the US" to "the US is like Mexico" do so because, e.g., they think 98 is *nearer* to 100 than vice versa? It seems to me that "approximately 100" and "like the US" have an obvious advantage over "approximately 98" and "like Mexico": 100 is a nice-round-number, one that people are immediately familiar with the rough size of and that's easy to calculate with; the US is a nation everyone knows (or thinks they do).

I bet there really is a bias here, but that observation doesn't strike me as very good evidence for it. The rival explanations are too good. (The example about disease in ducks and robins is much better.)

Jeffrey wrote: To me, this specific exercise reduces to a simpler question: Would it be better (more ethical) to torture individual A for 50 years, or inflict a dust speck on individual B? Gosh. The only justification I can see for that equivalence would be some general belief that badness is simply independent of numbers. Suppose the question were: Which is better, for one person to be tortured for 50 years or for everyone on earth to be tortured for 49 years? Would you really choose the latter? Would you not, in fact, jump at the chance to be the single person for 50 years if that were the only way to get that outcome rather than the other one?

In any case: since you now appear to be conceding that it's possible for someone to prefer TORTURE to SPECKS for reasons other than a childish desire to shock, are you retracting your original accusation and analysis of motives? ... Oh, wait, I see you've explicitly said you aren't. So, you know that one leading proponent of the TORTURE option actually *does* care about humanity; you agree (if I've understood you right) that utilitarian analysis can lead to the conclusion that TORTURE is the less-bad option; I assume you agree that reasonable people can be utilitarians; you've seen that one person explicitly said s/he'd be willing to be the one tortured; but in spite of all this, you don't retract your characterization of that view as shocking; you don't retract your implication that people who expressed a preference for TORTURE did so because they want to show how uncompromisingly rationalist they are; you don't retract your implication that those people don't appreciate that real decisions have real effects on real people. I find that ... well, "fairly shocking", actually.

(It shouldn't matter, but: I was not one of those advocating TORTURE, nor one of those opposing it. If you care, you can find my opinions above.)