Just thought I'd throw this out there:

TabooBot: Return D if opponent's source code contains a D; C otherwise.

To avoid mutual defection with other bots, it must (like with real prudish societies!) indirectly reference the output D. But then other kinds of bots can avoid explicit reference to D, requiring a more advanced TabooBot to have other checks, like defecting if the opponent's source code calls a modifier on a string literal.

A better summary of Aaronson's paper:

I want to know:

Were Bohr and Compton right or weren’t they? Does quantum mechanics (specifically, say, the No-Cloning Theorem or the uncertainty principle) put interesting limits on an external agent’s ability to scan, copy, and predict human brains and other complicated biological systems, or doesn’t it?

EY is mentioned once, for his work in popularizing cryonics, and not for anything fundamental to the paper. Several other LW luminaries like Silas Barta and Jaan Tallinn show up in the acknowledgements.

If you have followed Aaronson at all in the past couple years, the new stuff begins around section 3.3, page 36. His definition of "freedom" is at first glance interesting, and may dovetail slightly with the standard reduction of free will.

I think steelmanning would instead be if you listed more realistic dangers of that place rather than more extreme dangers

I think you missed what was going on there. In the hypothetical, Feynman's mom was concerned about the plague and for the steelman Feynman corrected it to TB. The assumption there is that TB is a more realistic threat than the plague.

I think the 1 character per second speed is even done with EEGs that are much better than consumer grade equipment.

It could be possible to do better but it probably won't be easy.

I played a bit with Emotiv and find a maximum of one character-per-second pretty believable - at least, if you stick to actual brain signals and not signals from face muscles ( and even with face muscles one character per second seems in the right ballpark).

Yes. We both tweaked matters so that the example became a steelmanning. You changed what Richard said. I changed what his mom said. We both changed something, and after either or both of our changes, it was an example of steelmanning.

Brain-computer interfaces for the disabled have been tried. There's plenty of academic work on the topic. For some people who are completely paralyzed the technology allows them to communicate by typing 1 character per second.

We're talking about Feynman steelmanning, not me.

Feynman would have been steelmanning if she had made a worse argument to begin with yet he responded to it and a better one.

That would work too. Note that I was making what he did steelmanning by way of making the original argument worse - we're working on opposite ends but I think we agree on definition.

As far as I can tell, that's simply taking a facile objection seriously.

Steelmanning would be if she said that he could get, say, bubonic plague from her - and then he addressed not only that but also concerns about tuberculosis.