The sequence is rationality-informed but also picks up things from folk wisdom, religious traditions, etc. when that seems helpful. It references cogsci studies and insights when those are available.

In Firefox, you can hover over the timestamp next to the name of the commenter, right-click (or whatever it is you Mac people do), and select Bookmark Link... to bookmark a comment.

Alternatively, you can click on the timestamp and then use whatever mechanism your OS/Browser allows to bookmark the page you go to as a result of the click.

I posit: high enough that you’re slightly overoptimistic about stuff you can’t control, so that for stuff where confidence itself makes the difference, you squeak in.

FWIW, see Notes on Optimism, Hope, and Trust for more on this hypothesis, including William James's speculations.

All this sounds wonderful, but reminds me of people who have amazing systems to play the stock market and leverage $1,000 to $1,000,000. Of all the people with all their systems, a few lucky ones hit the jackpot by chance, while the majority muddle through or lose it all. The lucky ones assume the market has acknowledged their financial genius and go on to tell us all about it.

If 5% of glioblastoma patients uncannily survive, how much am I supposed to update on hearing that one of those patients did some combination of plausible but undertested interventions during their recovery?

A prereminiscence: It's like it was with chess. We passed through that stage when AI could beat most of us to where it obviously outperforms all of us. Only for cultural output in general. People still think now, but privately, in the shower, or in quaint artisanal forms as if we were making our own yogurts or weaving our own clothes. Human-produced works are now a genre with a dwindling and eccentric fan base more concerned with the process than the product.

It was like the tide coming in. One day it was cutely, clumsily trying to mimic that thing we do. Soon after it was doing it pretty well and you watched it with admiration as if it were a dog balancing on a ball. Then soon it could do it well enough for most purposes, you couldn't help but admit. Then as good as all but the best, even to those who could tell the difference. And then we were suddenly all wet, gathering our remaining picnic and heading for higher ground, not only completely outclassed but even unable to judge by how far we were being outclassed. Once a chess machine can beat everybody every time, who is left to applaud when it gets twice again as good?

If it had been a war, we would have had a little ceremony as we brought down our flag and folded it up and put it away, but because it happened as quickly and quietly as it did and because we weren't sure whether we wanted to admit it was happening, there was no formal hand-off. One day we just resignedly realized we could no longer matter much, but there was so much more now to appreciate and we could see echoes and reflections of ourselves in it, so we didn't put up a fight.

Every once in a while someone would write an essay, Joan Didion quality, really good. Or write a song. Poignant, beautiful, original even. Not maybe the best essay or the best song we'd seen that day, but certainly worthy of being in the top ranks. And we'd think: we've still got it. We can still rally when we've got our backs to the wall. Don't count us out yet. But it was just so hard, and there was so much else to do. And so when it happened less and less frequently, we weren't surprised.

And our curiosity left us, too. We half-remembered a time when we would have satisfied curiosity by research and experiment (words that increasingly had the flavor of “thaumaturgy,” denoting processes productive though by unclear means). But nowadays curiosity is déclassé. It suggests laziness (why not just ask it?)… or poverty (oh, you can’t afford to ask it; you want one of us to ask it)—an increasing problem as we less and less have something to offer in trade that it desires and lacks.

The book in the Chinese Room directs the actions of the little man in the room. Without the book, the man doesn't act, and the text doesn't get translated.

The popcorn map on the other hand doesn't direct the popcorn to do what it does. The popcorn does what it does, and then the map in a post-hoc way is generated to explain how what the popcorn did maps to some particular calculation.

You can say that "oh well, then, the popcorn wasn't really conscious until the map was generated; it was the additional calculations that went into generating the map that really caused the consciousness to emerge from the calculating" and then you're back in Chinese Room territory. But if you do this, you're left with the task of explaining how a brain can be conscious solely by means of executing a calculation before anyone has gotten around to creating a map between brain-states and whatever the relevant calculation-states might be. You have to posit some way in which calculations capable of embodying consciousness are inherent to brains but must be interpreted into being elsewhere.

The book in the room isn't inert, though. It instructs the little guy on what to do as he manipulates symbols and stuff. As such, it is an important part of the computation that takes place.

The mapping of popcorn-to-computation, though, doesn't do anything equivalent to this. It's just an off-to-the-side interpretation of what is happening in the popcorn: it does nothing to move the popcorn or cause it to be configured in such a way. It doesn't have to even exist: if you just know that in theory there is a way to map the popcorn to the computation, then if (by the terms of the argument) the computation itself is sufficient to generate consciousness, the popcorn should be able to do it as well, with the mapping left as an exercise for the reader. Otherwise you are implying some special property of a headful of meat such that it does not need to be interpreted in this way for its computation to be equivalent to consciousness.

This reminds me of my intuitive rejection of the Chinese Room thought experiment, in which the intuition pump seems to rely on the little guy in the room not knowing Chinese, but that it's obviously the whole mechanism that is the room, the books in the room, etc. that is doing the "knowing" while the little guy is just a cog.

Part of what makes the rock/popcorn/wall thought experiment more appealing, even given your objections here, is that even if you imagine that you have offloaded the complex mapping somewhere else, the actual thinking-action that the mapping interprets is happening in the rock/popcorn/wall. The mapping itself is inert and passive at that point. So if you imagine consciousness as an activity that is equivalent to a physical process of computation you still have to imagine it taking place in the popcorn, not in the mapping.

You seem maybe to be implying that we have underinvestigated the claim that one really can arbitrarily map any complex computation to any finite collection of stuff (that this would e.g. imply that we have solved the halting problem). But I think these thought experiments don't require us to wrestle with that because they assume ad arguendo that you can instantiate the computations we're interested in (consciousness) in a headful of meat, and then try to show that if this is the case, many other finite collections of matter ought to be able to do the job just as well.