Robin Z:

I think it is not only a clash of intuitions, since the success rates of pre-scientific theory and folk psychology are poor. This should urge caution in keeping concepts that seem to give rise to much confusion. I would argue that the default attitude towards pre-scientific concepts that have been shrouded in confusion for thousands of years, with still no clarity in sight, should be to avoid them when possible.

When you say that you haven't seen evidence that puts "soul" on shaky grounds, do you mean that assuming determinism and what we know of human physiology that you believe there are still good reasons for positing the existence of a soul? If so, please explain what you mean by the term and why you think it is still a valuable concept. I think the notion of "soul" arose out of ignorance about the nature of living things (and how human beings were different from non-humans in particular) and that it not only serves no positive purpose but causes confusion, and I suspect that "choice" would never have arisen in anything like the form it did if we were not also confused about things like "free will" and the nature of thought.

Regarding "fire" and that it persists as a term, the only aspects of fire that continue to exist from ancient times are descriptions of its visual appearance and its obvious effects (that is, just the phenomenology of fire). Everything else about the concept has been abandoned or re-explained. Since the core of the concept (that it has the distinctive visual appearance it does and "it burns stuff") remain, it is reasonable to keep the term and redefine the explanation for the core aspects of the concept.

The case with choice is quite different, as the phenomenology is much more complex, it is less direct, and it is "mental" and not visual (and thus much more likely to be confused/confusing). Things like "fire" and "thunder" and "rain" and "sun" can easily be re-explained since the phenomenology was reasonably accurate and that was the basis for the concept, but we don't all agree on what is meant by choice or what the concept is supposed to explain.

Eliezer:

If determinism is true, what is the difference between "I could reach state X by taking action Y, if I wanted" and "I could reach state X by taking action Y, if 1=2"?

If determinism is true, then I couldn't have wanted to do X, just as it couldn't have been the case that 1=2, so the conditional is vacuously true, since the antecedent is false and must be false.

This doesn't seem to me like a plausible explanation of possibility or could-ness, unless you can explain how to distinguish between "if I wanted" and "if 1=2" without mentioning possible worlds or possibility or even reachability (since reachability is what you are defining).

Nick, your example confuses more than it clarifies. What exactly is the choice? Brain processes that occur in 0 < t < 1? Brain processes occurring in that slice that have certain causal relations with future actions? Conscious brain processes occurring such that...? Conscious brain processes occurring such that ... which are initiated by (certain) other brain processes?

You speak as if "choice" means something obvious that everybody understands, but it only has such a meaning in the sense that everybody knows what is meant by "soul" (which refers to a non-existent thing that means something different to practically everybody who uses it and usually results in more confusion than clarification).

@Jagadul:

by "constraints", I meant that Eliezer specified only that some particular processes happening in the brain are sufficient for choice occurring, which my example refuted, to which you added the ideas that it is not mere happening in the brain but also the additional constraints entailed by concepts of Eliezer-the-person and body-shell-of-Eliezer and that the former can be destroyed while the latter remains, which changes ownership of the choice, etc.

Anyway, I understand what you're saying about choice as a higher-level convenience term, but I don't think it is helpful. I think it is a net negative and that we'd do better to drop it. You gave the thought, "given these options, what will he choose?", but I think the notion of choice adds nothing of value to the similar question, "given these options, what will he do?" You might say that it is different, since a choice can be made without an action occurring, but then I think we'd do better to say not "what will he choose?" but something more like "what will he think?", or perhaps something else depending on the specifics of the situation under consideration.

I believe there's always a way of rephrasing such things so as not to invoke choice, and all that we really give up is the ability to talk about totally generic hypothetical situations (where it isn't specified what the "choice" is about). Whenever you flesh out the scenario by specifying the details of the "choice", then you can easily talk about it more accurately by sidestepping the notion of choice altogether.

I don't think that "choice" is analogous to Newtonian mechanics before relativity. It's more akin to "soul", which we could have redefined and retrofitted in terms of deterministic physical processes in the brain. But just as it makes more sense to forget about the notion of a soul, I think it makes more sense to forget about that of "choice". Just as "soul" is too strongly associated with ideas such as dualism and various religious ideas, "choice" is too strongly associated with ideas such as non-determinism and moral responsibility (relative to some objective standard of morality). Instead of saying "I thought about whether to do X, Y, or Z, then choose to do X, and then did X", we can just say "I thought about whether to do X, Y, or Z, then did X."

@Constant:

I think "choice" is closer to "caloric" than "heat", because I don't believe there is any observable mundane phenomenon that it refers to. What do you have in mind that cannot be explained perfectly well without supposing that a "choice" must occur at some point in order to explain the observed phenomenon?

Jadagul: I'm saying that Eliezer's explanation of what a choice is is not a sufficient condition. You suggested some additional constraints, which I would argue may be necessary but are still not sufficient conditions for a choice occurring.

My key point, though, as Schizo noted, was that I don't think the concept should be salvaged, any more than phlogiston or caloric should have been salvaged.

Robin, I don't think "whose brain it is" is really a meaningful and coherent concept, but that is another topic.

My general point was that Eliezer seemed to be saying that certain things occurring in his brain are sufficient for us to say that he made a choice and is morally responsible for the choice. My example was intended to show that while that may be a necessary condition, it is not sufficient.

As for what I actually believe, I think that while the notions of choice and moral responsibility may have made sense in the context in which they arose (though I have strong doubts because that context is so thoroughly confused and riddled with inconsistencies), they don't make sense outside of that context. Freewill and choice (in the west at least) mean what they mean by virtue of their place in a conceptual system that assumes a non-physical soul-mind is the ultimate agent that controls the body and that what the soul-mind does is not deterministic.

If we give up that notion of a non-physical, non-deterministic soul-mind as the agent, the controller of the body, the thing that ultimately makes choices and is morally responsible (here or in the afterlife), I think we must also give up concepts like choice and moral responsibility.

That is not to say that they won't be replaced with better concepts, but those concepts will relate to the traditional concepts of choice and moral responsibility as heat relates to caloric.

Eliezer said:

I am not saying that choice is an illusion. I am pointing to something and saying: "There! Right there! You see that? That's a choice, just as much as a calculator is adding numbers! It doesn't matter if it's deterministic! It doesn't matter if someone else predicted you'd do it or designed you to do it! It doesn't matter if it's made of parts and caused by the dynamics of those parts! It doesn't matter if it's physically impossible for you to have finally arrived at any other decision after all your agonizing! It's still a choice!"

--

Does it not follow from this that if I installed hardware into your brain that completely took control of your brain and mapped my thoughts onto your brain (and the deliberations in your brain would be ones that I initiated) that you would still be choosing and still be morally responsible? If I decided to kill someone and then forced you to carry out the murder, you would be morally responsible. I find this hard to accept.

Nick: note that I said "conscious thoughts" and not "thoughts", and I specified that the individual is not aware of the inputs/outputs from/to the actuators/sensors and has no control over them.

Nick, I don't understand what you mean by random. There is nothing in the slightest random (as I understand the term) in the scenario I gave. The primary difference between the two cases is that in one case you believe you are effecting action via your conscious thoughts (but you aren't) and in the other you do not believe you are effecting action via your conscious thoughts. In both cases, your actions are fully determined by what is going on in your brain; it's just that your conscious thoughts are irrelevant to what happens (and you are deluded about this in one of the two scenarios).

Eliezer: why do you say John-1 (the "coward") is morally responsible if under your scenario it was physically impossible for him to act as John-2 did given his initial physical conditions? (If it were not impossible, then his actions wouldn't have been fully determined by his initial physical condition.)

To possibly confuse matters a little more, here's a thought experiment that occurred to me for some reason. I'd be curious to hear what anybody who says that determinism does not undermine moral responsibility, or who makes the even stronger claim that there is absolutely no conflict between determinism and moral responsibility, has to say about the following:

You wake up in front of a schoolhouse, where you've just taken a nap, and discover that your body has been encased in thick metal armor that has actuators at all the joints and is covered in sensors (nobody else did it; this just happens to be the one in a google^^^google chance of this spontaneously happening -- so there's nobody else to "blame"). You are not strong enough to move the armor or break free, but the sensors and actuators are wired up to your brain such that the sensors send their data to certain parts of your brain that you have no conscious awareness of, and the actuators respond to signals from some (perhaps the same) part of your brain whose happenings you are also not conscious of.

The schoolhouse is burning, cherubic youth are screaming, and you could probably save a child or two. But of course, you are not physically capable of doing anything except going along for the ride and doing whatever the armor does based on the firings in your brain that you have no control over or awareness of.

Let's say that the armor turns and runs. Are you -- the person inside -- morally responsible?

If under normal circumstances one's actions were totally predetermined, does one have any more ability to choose than the individual in the armor does? If not, how do you assert that the John-1 would be morally responsible but armored John-A1 would not be morally responsible?

I'm not sure what I think about determinism and moral responsibility, but I have a difficult time understanding how these two topics could have no relation to each other, as some people in this thread seem to believe.

AI researchers of previous eras made predictions that were wildly wrong. Therefore, human-level AI (since it is a goal of some current AI researchers) cannot happen in the foreseeable future. They were wrong before, so they must be wrong now. And dawg-garn it, it seems like some kind of strange weirdo religious faith-based thingamajiggy to me, so it must be wrong.

Thanks for a good laugh, Mr. Horgan! Keep up the good work.