I love this quote. But this...

declarations of high confidence mainly tell you that an individual has constructed a coherent story in his mind

...strikes me as a highly confident declaration for which the quoted is simultaneously urging me to be skeptical.

I'd imagine the book lays out his case as to why I ought listen to his counsel. I'd be interested to dig into this.

I think this argument is misleading.

Re "for game theoretical reasons", the paperclipper might take revenge if it predicted that doing so would be a signalling-disincentive for other office-supply-maximizers from stealing paperclips. In other words, the paperclip-maximizer is spending paperclips to take revenge solely because in its calculation, this actually leads to the expected total number of paperclips going up.

The twenty fiveth virtue of rationality is "Be faithless to humanity and betray it to the Way".

Thanks for this. One quibble: I think it's misleading to say actions are rated according to what they achieve. Consider dancing. What dancing achieves, first and foremost, is dancing. More generally, some features of actions may be valued or disvalued as terminal values, independent of any (non-tautological) consequences. It's not hard to imagine patterns of behavior for which this is the most elegant explanation.

LessWrong updates the truth to fit its priors.

Them's fightin' words, Random Profound Deep Wisdom Generator!

Interesting. Is this kinda like a minimum complexity of outcome requirement?

It has near maximal computational capacity, but that capacity isn't being "used" for anything in particular that is easy to determine.

This is actually a very powerful criteria, in terms of number of false positive and negatives. Sadly, the false positives it DOES have still far outweigh the genuine positives, and includes all the WORST outcomes (aka, virtual hells) as well.

Are you implying that rational agents can be successfully blackmailed by other agents that simulate them inaccurately? (This does seem plausible to me, and is an interesting rare example of accurate knowlage posing a hazard.)

Thanks for the response. Yes, it depends on how much interaction I have with human beings and on the kind of people I interact with. I'm mostly interested in my own case, of course, and I interact with a fair number of fairly diverse, fairly intelligent human beings on a regular basis.

If you're a social butterfly who regularly talks with some of the smartest people in the world, the AI will probably struggle

Ah, but would it? I'm not so sure, that's why I made this post.

Yes, if everyone always said what I predicted, things would be obvious, but recall I specified that random variation would be added. This appears to be how dream characters work: You can carry on sophisticated conversations with them, but (probably) they are governed by algorithms that feed off your own expectations. That being said, I now realize that the variation would have to be better than random in order to account for how e.g. EY consistently says things that are on-point and insightful despite being surprising to me.

Believing that a theory is true that says "true" is not a thing theories can be is obviously silly.

Oh okay. This is a two-part misunderstanding.

I'm not saying that theories can't be true, I'm just not talking about this truth thing in my meta-model. I'm perfectly a-okay with models of truth popping up wherever they might be handy, but I want to taboo the intuitive notion and refuse to explicate it. Instead I'll rely on other concepts to do much of the work we give to truth, and see what happens. And if there's work that they can't do, I want to evaluate whether it's important to include in the meta-model or not.

I'm also not saying that my theory is true. At least, not when I'm talking from within the theory. Perhaps I'll find certain facets of the correspondence theory useful for explaining things or convincing others, in which case I might claim it's true. My epistemology is just as much a model as anything else, of course; I'm developing it with certain goals in mind.

I were talking about physical systems, not physical laws. Computers, living cells, atoms, the fluid dynamics of the air... "Applied successfully in many cases", where "many" is "billions of times every second"

The math we use to model computation is a model and a tool just as much as computers are tools; there's nothing weird (at least from my point of view) about models being used to construct other tools. Living cells can be modeled successfully with math, you're right; but that again is just a model. And atoms are definitely theoretical constructs used to model experiences, the persuasive images of balls or clouds they conjure notwithstanding. Something similar can be said about fluid dynamics.

I don't mean any of this to belittle models, of course, or make them seem whimsical. Models are worth taking seriously, even if I don't think they should be taken literally.

Then ZFC is not one of those cores ones, just one of the peripheral ones. I'm talking ones like set theory as a whole, or arithmetic, or Turing machines.

The best example in the three is definitely arithmetic; the other two aren't convincing. Math was done without set theory for ages, and besides we have other foundations available for modern math that can be formulated entirely without talking about sets. Turing machines can be replaced with logical systems like the lambda calculus, or with other machine models like register machines.

Arithmetic is more compelling, because it's very sticky. It's hard not to take it literally, and it's hard to imagine things without it. This is because some of the ideas it constitutes are at the core cluster of our categories, i.e. they're very sticky. But could you imagine that some agent might a) have goals that never require arithmetical concepts, and b) that there could be models that are non-arithmetical that could be used toward some of the same goals for which we use arithmetic? I can imagine ... visualise, actually, both, although I would have a very hard time translating my visual into text without going very meta first, or else writing a ridiculously long post.