All of ScottAaronson's Comments + Replies

(1) Well, that's the funny thing about "should": if copyable entities have a definite goal (e.g., making as many additional copies as possible, taking over the world...), then we simply need to ask what form of reasoning will best help them achieve the goal. If, on the other hand, the question is, "how should a copy reason, so as to accord with its own subjective experience? e.g., all else equal, will it be twice as likely to 'find itself' in a possible world with twice as many copies?" -- then we need some account of the subjective e... (read more)

(1) I agree that we can easily conceive of a world where most entities able to pass the Turing Test are copyable. I agree that it's extremely interesting to think about what such a world would be like --- and maybe even try to prepare for it if we can. And as for how the copyable entities will reason about their own existence -- well, that might depend on the goals of whoever or whatever set them loose! As a simple example, the Stuxnet worm eventually deleted itself, if it decided it was on a computer that had nothing to do with Iranian centrifuges. We... (read more)

shminux: I don't know any way, even in principle, to prove that uncertainty is Knightian. (How do you decisively refute someone who claims that if only we had a better theory, we could calculate the probabilities?) Though even here, there's an interesting caveat. Namely, I also would have thought as a teenager that there could be no way, even in principle, to "prove" something is "truly probabilistic," rather than deterministic but with complicated hidden parameters. But that was before I learned the Bell/CHSH theorem, which does pr... (read more)

Alright, consider the following questions:

• What's it like to be simulated in homomorphically encrypted form (http://en.wikipedia.org/wiki/Homomorphic_encryption)---so that someone who saw the entire computation (including its inputs and outputs), and only lacked a faraway decryption key, would have no clue that the whole thing is isomorphic to what your brain is doing?

• What's it like to be simulated by a reversible computer, and immediately "uncomputed"? Would you undergo the exact same set of experiences twice? Or once "forwards" a

Well, I can try to make my best guess if forced to -- using symmetry arguments or any other heuristic at my disposal -- but my best guess might differ from some other, equally-rational person's best guess. What I mean by a probabilistic system's being "mechanistic" is that the probabilities can be calculated in such a way that no two rational people will disagree about them (as with, say, a radioactive decay time, or the least significant digit of next week's Dow Jones average).

Also, the point of my "Earth C" example was that symmetry ... (read more)

Well, all I can say is that "getting a deity off the hook" couldn't possibly be further from my motives! :-) For the record, I see no evidence for a deity anything like that of conventional religions, and I see enormous evidence that such a deity would have to be pretty morally monstrous if it did exist. (I like the Yiddish proverb: "If God lived on earth, people would break His windows.") I'm guessing this isn't a hard sell here on LW.

Furthermore, for me the theodicy problem isn't even really connected to free will. As Dostoyevsky ... (read more)

Wei, I completely agree that people should "directly attack the philosophical problems associated with copyable minds," and am glad that you, Eliezer, and others have been trying to do that! I also agree that I can't prove I'm not living in a simulation --- nor that that fact won't be revealed to me tomorrow by a being in the meta-world, who will also introduce me to dozens of copies of myself running in other simulations. But as long as we're trading hypotheticals: what if minds (or rather, the sorts of minds we have) can only be associated wi... (read more)

The relevant passage of the essay (p. 65) goes into more detail than the paraphrase you quoted, but the short answer is: how does the superintelligence know it should assume the uniform distribution, and not some other distribution? For example, suppose someone tips it off about a third Earth, C, which is "close enough" to Earths A and B even if not microscopically identical, and in which you made the same decision as in B. Therefore, this person says, the probabilities should be adjusted to (1/3,2/3) rather than (1/2,1/2). It's not obvious wh... (read more)

As a point of information, I too am only interested in predicting macroscopic actions (indeed, only probabilistically), not in what you call "absolute prediction." The worry, of course, is that chaotic amplification of small effects would preclude even "pretty good" prediction.

"Even if we could not, by physical law, possibly know the fact, this still does not equate to the fact having inherent unknowability."

I think the sentence above nicely pinpoints where I part ways from you and Eliezer. To put it bluntly, if a fact is impossible for any physical agent to learn, according to the laws of physics, then that's "inherently unknowable" enough for me! :-) Or to say it even more strongly: I don't actually care much whether someone chooses to regard the unknowability of such a fact as "part of the map"... (read more)

In both cases, the question that interests me is whether an external observer could build a model of the human, by non-invasive scanning, that let it forecast the probabilities of future choices in a well-calibrated way. If the freebits or the trillions of bouncing molecules inside cells served only as randomization devices, then they wouldn't create any obstruction to such forecasts. So the relevant possibility here is that the brain, or maybe other complex systems, can't be cleanly decomposed into a "digital computation part" and a "micr... (read more)

Hi Eliezer,

(1) One of the conclusions I came to from my own study of QM was that we can't always draw as sharp a line as we'd like between "map" and "territory." Yes, there are some things, like Stegosauruses, that seem clearly part of the "territory"; and others, like the idea of Stegosauruses, that seem clearly part of the "map." But what about (say) a quantum mixed state? Well, the probability distribution aspect of a mixed state seems pretty "map-like," while the quantum superposition aspect seems pr... (read more)

Just as a quick point of information, these arguments are all addressed in Sections 2.2 and 3.1. In particular, while I share the common intuition that "random" is just as incompatible with "free" as "predictable" is, the crucial observation is that "unpredictable" does not in any way imply "random" (in the sense of governed by some knowable probability distribution). But there's a broader point. Suppose we accepted, for argument's sake, that unpredictability is not "fundamental to freedom" (whatever we take "freedom" to mean). Wouldn't the question of whether human choices are predictable or not remain interesting enough in its own right?

shminux: Thanks so much for compiling these notes and quotes! But I should say that I thought the other LW thread was totally fine. Sure, lots of people strongly disagreed with me, but I'd be disappointed if LW readers didn't! And when one or two people who hadn't read the paper got things wrong, they were downvoted and answered by others who had. Kudos to LW for maintaining such a high-quality discussion about a paper that, as DanielVarga put it, "moves in a direction that's very far from any kind of LW consensus."

Hi Paul. I completely agree that I see no reason why you couldn't "get a functional human out of a brain scan" --- though even there, I probably wouldn't convert my failure to see such a reason into a bet at more than 100:1 odds that there's no such reason. (Building a scalable quantum computer feels one or two orders of magnitude easier to me, and I "merely" staked $100,000 on that being possible --- not my life or everything I own! :-) )

Now, regarding "whether there can be important aspects of your identity or continuity of exp... (read more)

"Intertemporal solidarity is just as much a choice today as it will be should teleporters arrive."

I should clarify that I see no special philosophical problem with teleportation that necessarily destroys the original copy, as quantum teleportation would (see the end of Section 3.2). As you suggest, that strikes me as hardly more perplexing than someone's boarding a plane at Newark and getting off at LAX.

For me, all the difficulties arise when we imagine that the teleportation would leave the original copy intact, so that the "new" and ... (read more)

"But calling this Knightian unpredictability 'free will' just confuses both issues."

torekp, a quick clarification: I never DO identify Knightian unpredictability with "free will" in the essay. On the contrary, precisely because "free will" has too many overloaded meanings, I make a point of separating out what I'm talking about, and of referring to it as "freedom," "Knightian freedom," or "Knightian unpredictability," but never free will.

On the other hand, I also offer arguments for why I think u... (read more)