A prereminiscence: It's like it was with chess. We passed through that stage when AI could beat most of us to where it obviously outperforms all of us. Only for cultural output in general. People still think now, but privately, in the shower, or in quaint artisanal forms as if we were making our own yogurts or weaving our own clothes. Human-produced works are now a genre with a dwindling and eccentric fan base more concerned with the process than the product.

It was like the tide coming in. One day it was cutely, clumsily trying to mimic that thing we do. Soon after it was doing it pretty well and you watched it with admiration as if it were a dog balancing on a ball. Then soon it could do it well enough for most purposes, you couldn't help but admit. Then as good as all but the best, even to those who could tell the difference. And then we were suddenly all wet, gathering our remaining picnic and heading for higher ground, not only completely outclassed but even unable to judge by how far we were being outclassed. Once a chess machine can beat everybody every time, who is left to applaud when it gets twice again as good?

If it had been a war, we would have had a little ceremony as we brought down our flag and folded it up and put it away, but because it happened as quickly and quietly as it did and because we weren't sure whether we wanted to admit it was happening, there was no formal hand-off. One day we just resignedly realized we could no longer matter much, but there was so much more now to appreciate and we could see echoes and reflections of ourselves in it, so we didn't put up a fight.

Every once in a while someone would write an essay, Joan Didion quality, really good. Or write a song. Poignant, beautiful, original even. Not maybe the best essay or the best song we'd seen that day, but certainly worthy of being in the top ranks. And we'd think: we've still got it. We can still rally when we've got our backs to the wall. Don't count us out yet. But it was just so hard, and there was so much else to do. And so when it happened less and less frequently, we weren't surprised.

And our curiosity left us, too. We half-remembered a time when we would have satisfied curiosity by research and experiment (words that increasingly had the flavor of “thaumaturgy,” denoting processes productive though by unclear means). But nowadays curiosity is déclassé. It suggests laziness (why not just ask it?)… or poverty (oh, you can’t afford to ask it; you want one of us to ask it)—an increasing problem as we less and less have something to offer in trade that it desires and lacks.

The book in the Chinese Room directs the actions of the little man in the room. Without the book, the man doesn't act, and the text doesn't get translated.

The popcorn map on the other hand doesn't direct the popcorn to do what it does. The popcorn does what it does, and then the map in a post-hoc way is generated to explain how what the popcorn did maps to some particular calculation.

You can say that "oh well, then, the popcorn wasn't really conscious until the map was generated; it was the additional calculations that went into generating the map that really caused the consciousness to emerge from the calculating" and then you're back in Chinese Room territory. But if you do this, you're left with the task of explaining how a brain can be conscious solely by means of executing a calculation before anyone has gotten around to creating a map between brain-states and whatever the relevant calculation-states might be. You have to posit some way in which calculations capable of embodying consciousness are inherent to brains but must be interpreted into being elsewhere.

The book in the room isn't inert, though. It instructs the little guy on what to do as he manipulates symbols and stuff. As such, it is an important part of the computation that takes place.

The mapping of popcorn-to-computation, though, doesn't do anything equivalent to this. It's just an off-to-the-side interpretation of what is happening in the popcorn: it does nothing to move the popcorn or cause it to be configured in such a way. It doesn't have to even exist: if you just know that in theory there is a way to map the popcorn to the computation, then if (by the terms of the argument) the computation itself is sufficient to generate consciousness, the popcorn should be able to do it as well, with the mapping left as an exercise for the reader. Otherwise you are implying some special property of a headful of meat such that it does not need to be interpreted in this way for its computation to be equivalent to consciousness.

This reminds me of my intuitive rejection of the Chinese Room thought experiment, in which the intuition pump seems to rely on the little guy in the room not knowing Chinese, but that it's obviously the whole mechanism that is the room, the books in the room, etc. that is doing the "knowing" while the little guy is just a cog.

Part of what makes the rock/popcorn/wall thought experiment more appealing, even given your objections here, is that even if you imagine that you have offloaded the complex mapping somewhere else, the actual thinking-action that the mapping interprets is happening in the rock/popcorn/wall. The mapping itself is inert and passive at that point. So if you imagine consciousness as an activity that is equivalent to a physical process of computation you still have to imagine it taking place in the popcorn, not in the mapping.

You seem maybe to be implying that we have underinvestigated the claim that one really can arbitrarily map any complex computation to any finite collection of stuff (that this would e.g. imply that we have solved the halting problem). But I think these thought experiments don't require us to wrestle with that because they assume ad arguendo that you can instantiate the computations we're interested in (consciousness) in a headful of meat, and then try to show that if this is the case, many other finite collections of matter ought to be able to do the job just as well.

More on gratitude, how it works, and how to best take advantage of it: Notes on Gratitude

Wants are emergent, complex forms of pain and pleasure. They are either felt or they are not felt, and reason only comes in at the stage of deciding what to do about them.

Are you really certain that one's desires are just givens that one has no rational influence over? I'm skeptical.

https://www.lesswrong.com/posts/aQQ69PijQR2Z64m2z/notes-on-temperance#Can_we_shape_our_desires_

An influential ethical philosopher is on his way to address a conference of wealthy donors about effective altruism. His rhetorical power and keen arguments are such that he can expect these donors to reach deep and double their donations to yet worthier causes after his talk. On his way to the conference, however, he comes across a child drowning in a pond. He is the only one around who can save this child, but to do so, he would have to jump in the pond, ruin his humble but respectable second-hand suit, and miss the train to the conference. While he would certainly have a good excuse to give to the donors, before he would have the opportunity to do so they would probably leave the conference feeling resentful at the waste of their valuable time, rather than generous and inspired. He figures he can save more lives by letting the child drown and instead catching his train to the conference, so he turns his back on the pond.