If you think you might be in a solipsist simulation, you might try to add some chaotic randomness to your decisions. For example, go outside under some trees and wait till any kind of tree leaf or seed or anything hits your left half of the face, choose one course of action. If it hits the other half of your face, choose another course of action. If you do this multiple times in your life, each of your decisions will depend on the state of the whole earth and on all your previous decisions, since weather is chaotic. And thus the simulators will be unable to get good predictions about you using a solipsist simulation. A potential counterargument is that they analyze your thinking and hardcode this binary random choice, i.e. hardcode the memory of the seed hitting your left side. But then there would need to be an intelligent process analyzing your thinking to try and isolate the randomness. But then you could make the dependence of your strategy on randomness even more complicated.

Nice. I have a suggestion how to improve the article. Put a clearly stated theorem somewhere in the middle, in its own block, like in academic math articles.

Why do you hate earworms? To me, they are mildly pleasant. The only moments when I wish I didn’t have an earworm happening at that moment is when I’m trying to remember another tune and the earworm for musicianship purposes and the earworm prevents me from being able to do that.

Well, now I'm wondering - is neural network training chaotic?

This is awesome, I would love more posts like this. Out of curiosity, how many hours have you and your colleague spent on this research.

In my personal experience, exposure therapy did help me with the fear of such "extreme" risks.

In the very beginning of the post, I read: "Quick psychology experiment". Then, I read: "Right now, if I offered you a bet ...". Because of this, I thought about a potential real life situation, not a platonic ideal situation, that the author is offering me this bet. I declined both bets. Not because they are bad bets in an abstract world, but because I don't trust the author in the first bet and I trust them even less in the second bet.

If you rejected the first bet and accepted the second bet, just that is enough to rule you out from having any utility function consistent with your decisions.

Under this interpretation, no it doesn't.

Could you, the author, please modify the thought experiment to indicate that it is assumed that I completely trust the one who is proposing the bet to me? And, maybe discuss other caveats too. Or just say that it's Omega who's offering me the bet.

So you say humans don't reason about the space and objects around them by keeping 3d representations. You think that instead the human brain collects a bunch of heuristics what the response should be to a 2d projection of 3d space, given different angles - an incomprehhensible mishmash of neurons like in an artificial neural network that doesn't have any CNN layers for identifying the digit by image, and just memorizes all rules for all types of pictures with all types of angle like a fully connected layer.

I guess I was not clear enough. In your original post, you wrote "On one hand, there are countably many definitions ..." and "On the other hand, Cantor's diagonal argument applies here, too. ...". So, you talked about two statements - "On one hand, (1)", "On the other hand, (2)". I would expect that when someone says "One one hand, ..., but on the other hand, ...", what they say in those ellipses should contradict each other. So, in my previous comment, I just wanted to point out that (2) does not contradict (1) because countable infinity + 1 is still countable infinity.

take all the iterations you need, even infinitely many of them

Could you clarify how I would construct that?

For example, what is the "next cardinality" after countable?

I didn't say "the next cardinality". I said "a higher cardinality".