Doug raises another good point. Related to what I said earlier, I think people really do functionally have prior probability=1 on some propositions. Or act as if they do. If "The Bible is the inerrant word of God" is a core part of your worldview, it is literally impossible for me to convince you this is false, because you use this belief to interpret any facts I present to you. Eliezer has commented before that you can rationalize just about anything; if "God exists" or "The Flying Spaghetti Monster exists" or "reincarnation exists" is part of the machinery you use to interpret your experience, in a deep enough way, your experiences can't disprove it.

Eliezer: for 'better' vs 'frooter,' of course you're right. I just would have phrased it differently; I've been known to claim that the word 'better' is completely meaningless unless you (are able to) follow it with "better at or for something." So of course, Jadagul_real would say that his worldview is better for fulfilling his values. And Jadagul_hypothetical would say that his worldview is better for achieving his values. And both would (potentially) be correct. (or potentially wrong. I never claimed to be infallible, either in reality or in hypothesis). But phrasing issues aside, I do believe that I think this happens more often than you think it happens.

Sebastian Hagen: That's actually a very good question. So a few answers. First is that I tend to go back and forth on whether by 'happiness' I mean something akin to "net stimulation of pleasure centers in brain," or to "achievement of total package of values" (at which point the statement nears tautology, but I think doesn't actually fall into it). But my moral code does include such statements as "you have no fundamental obligation to help other people." I help people because I like to. So I lean towards formulation 1; but I'm not altogether certain that's what I really mean.

Second is that your question, about the sociopath pill, is genuinely difficult for me. It reminds me of Nozick's experience machine thought experiment. But I know that I keep getting short-circuited by statements like, "but I'd be miserable if I were a sociopath," which is of course false by hypothesis. I think my final answer is that I'm such a social person and take such pleasure in people that were I to become a sociopath I would necessarily be someone else. That person wouldn't be me. And while I care about whether I'm happy, I don't know that I care about whether he is.

Of course, this all could be "I know the answer and now let me justify it." On the other hand, the point of the exercise is to figure out what my moral intuitions are...

Steven: quite possibly related. I don't think they're exactly the same (the classic comic book/high fantasy "I'm evil and I know it" villain fits A2, but I'd describe him as amoral), but it's an interesting parallel.

Eliezer: I'm coming more and more to the conclusion that our main area of disagreement is our willingness to believe that someone who disagrees with us really "embodies a different optimization process." There are infinitely many self-consistent belief systems and infinitely many internally consistent optimization processes; while I believe mine to be the best I've found, I remain aware that if I held any of the others I would believe exactly the same thing. And that I would have no way of convincing the anti-Occam intelligence that Occam's Razor was a good heuristic, or of convincing the psychopath who really doesn't care about other people that he 'ought' to. So I hesitate to say that I'm right in any objective sense, since I'm not sure exactly what standard I'm pointing to when I say 'objective.'

And I've had extended moral conversations with a few different people that led to us, eventually, concluding that our premises were so radically different that we really couldn't have a sensible moral conversation. (to wit: I think my highest goal in life is to make myself happy. Because I'm not a sociopath making myself happy tends to involve having friends and making them happy. But the ultimate goal is me. Makes it hard to talk to someone who actually believes in some form of altruism).

Paul Crowley: remember that US markets are much larger than, say, the US economy. From the article:

It depends on the comparison. U.S. GDP is $12 trillion, the total value of traded securities (debt and equity) denominated in U.S. dollars is estimated to be more than $50 trillion, and the global value of traded securities is about $165 trillion.

And $10 trillion isn't where they are now, it's where they will be in four years or so. So while it's a bloody large amount of money, it's unlikely to be more than, say, 5% of traded securities on the market. And that doesn't include stuff like currency holdings.

Eliezer: I'm finding this one hard, because I'm not sure what it would mean for you to convince me that nothing was right. Since my current ethics system goes something like, "All morality is arbitrary, there's nothing that's right-in-the-abstract or wrong-in-the-abstract, so I might as well try to make myself as happy as possible," I'm not sure what you're convincing me of--that there's no particular reason to believe that I should make myself happy? But I already believe that. I've chosen to try to be happy, but I don't think there's a good 'reason' for it.

On the other hand, maybe I right now am the end result you're looking for. In which case, yes, I do tip cabdrivers; no, I don't cheat; and usually I'd pull the kid off, if there weren't much risk to me.

Joesph: I don't think I added more constraints, though it's a possibility. What extra constraints do you think I added?

As for not salvaging it, I can see why you would say that, but what word should be used to take its place? Mises commented somewhere in On Human Action that we can be philosophical monists and practical dualists; I believe that everything is ultimately reducible to (quasi?-)deterministic quantum physics, but that doesn't mean that's the most efficient way to analyze most situations. When I'm trying to catch a ball I don't try to model the effect of the weak nuclear force on the constituent protons. When I'm writing a computer program I don't even try to simulate the logic gates, much less the electromagnetic reactions that cause them to function. And when I'm dealing with people it's much more effective to say to myself, "given these options, what will he choose?" This holds even though I could in principle, were I omniscient and unbounded in computing power, calculate this deterministically from the quantum state of his brain.

Or, in other words, we didn't throw out Newtonian mechanics when we discovered Relativity. We didn't discard Maxwell when we learned about quantum electrodynamics. Why should this be different?

Joseph Knecht: I think you're missing the point of Eliezer's argument. In your hypothetical, to the extent Eliezer-as-a-person exists as a coherent concept, yes he chose to do those things. Your hypothetical is, from what I can tell, basically, "If technology allows me to destroy Eliezer-the-person without destroying the outer, apparent shell of Eliezer's body, then Eliezer is no longer capable of choosing." Which is of course true, because he no longer exists. Once you realize that "the state of Eliezer's brain" and "Eliezer's identity" are the same thing, your hypothetical doesn't work any more. Eliezer-as-a-person is making choices because the state of Eliezer's brain is causing things to happen. And that's all it means.

Eliezer: I'll second Hopefully Anonymous; this is almost exactly what I believe about the whole determinism-free will debate, but it's devilishly hard to describe in English because our vocabulary isn't constructed to make these distinctions very clearly. (Which is why it took a 2700-word blog post). Roland and Andy Wood address one of the most common and silliest arguments against determinism: "If determinism is true, why are you arguing with me? I'll believe whatever I'll believe." The fact that what you'll believe is deterministically fixed doesn't affect the fact that this argument is part of what fixes it.

Interestingly (at least, I think it's interesting), I'd always felt that way about time, before I learned about quantum mechanics. That's what a four-dimensional spacetime means, isn't it? And so science fiction stories that involve, say, changing the past have never made any sense to me. You can't change the past; it is. And no one can come from the future to change now, because the future is as well. Although now that I think about it more, I realize how this makes slightly more sense in this version of many-worlds than it does in a collapse theory.

Eliezer: why uncountably infinite? I find it totally plausible that you need an infinite-dimensional space to represent all of configuration space, but needing uncountability seems, at least initially, to be unlikely.

Of course, it would be the mathematician who asks this question...