Common knowledge means I know, and I know that you know, and I know that you know that he knows, and she knows that I know that you know that he knows, and so on -- any number of iterations.

Each child sees 99 muddy foreheads and therefore knows n >= 99. Each child can tell that each other child knows n >= 98. But, e.g., it isn't true that A knows B knows C knows that n >= 98; only that A knows B knows C knows that n>=97: each link in the chain reduces the number by 1. So for no k>0 is it common knowledge that n>=k.

Maybe I'm confused, in the 'muddy children puzzle' it seems it would be common knowledge from the start that at least 98 children have muddy foreheads. Each child sees 99 muddy foreheads. Each child could reason that every other child must see at least 98 muddy foreheads. 100 minus their own forehead which they cannot see minus the other child's forehead which the other child cannot see equals 98.

What am I missing?

Desire is a contract you make with yourself to be unhappy until you get what you want.

– Naval Ravikant

You can see more results here: Image Annotation Viewer

Judging generously, but based on only about two dozen or so image captions, I estimate it gives a passably accurate caption about one third of the time. This may be impressive given the simplicity of the model, but it doesn't seem unreasonably effective to me, and I don't immediately see the relevance to strong AI.

Let's say you precommit to never paying off blackmailers. The advantage of this is that you are no longer an attractive target for blackmailers since they will never get paid off. However if someone blackmails you anyway, your precommitment now puts you at a disadvantage, so now (NDT)you would act as if you had a precommitment to comply with the blackmailers all along since at this point that would be an advantageous precommitment to have made.

No, Mr. Shepard, with respect, (that) is not the moral of the story. The moral of the story is that, if you have grounds to believe there is a ferocious predator at large, don't appoint as your sole watchman a twelve-year-old child whom you have resolved to ignore.

• Mitchell and Webb prosecuting attorney, from the sketch, "The boy who cried wolf"

Recall an old joke:

A man flying in a hot air balloon realizes he is lost. He reduces his altitude and spots a man in a field down below. He lowers the balloon further and shouts, "Excuse me, can you tell me where I am?" The man below says, "Yes, you're in a hot air balloon, about 30 feet above this field." "You must be an mathematician," says the balloonist. "I am. How did you know?" "Everything you told me is technically correct, but it's of no use to anyone."

Generating artificial gravity on spaceships using centrifuges is a common idea in hard-sci-fi and in speculation about space travel, but no-one seems to consider them for low gravity on e.g. Mars. Am I mistaken in thinking that all you'd need to do is build the centrifuge with an angled floor, so the net force experienced from gravity and (illusory) centrifugal force is straight "down" into it?

I realise there'd be other practical problems with centrifuge-induced artificial gravity on Mars, since it's full of dust and not the best environment, but that doesn't seem to be the right kind of objection to explain it never being brought up where I've seen it.

What do you mean? He narrated the whole sequence before the explosion, and fell to his knees at the moment Voldemort supposedly died, which is coincident with the explosion. I don't see a problem, let alone one that would be fixed by shifting the narrative back 20 seconds.

This is really interesting. I thought I understood it and I wanted to verify that by trying to summarize it (and maybe help others too) but now I'm not so sure...

Edit: Just to save anybody the reading time, my reasoning is false below. After sleeping on it, I see my mistake. Nothing below the "False Thermodynamic Miracles" subheading made sense to me yesterday because I thought the purpose of the setup was to have an "off switch" on the simulated AI under the false belief (letting it see the result of the signal after some time period). I get it now though. Maximizing "[P(no miracle) * C] + [P(miracle) * u(A) given miracle]" is the same as maximizing "u(A) given miracle". So the AI will act as if the miracle happened, because there's no cost associated with those actions if it didn't happen—only benefits if it did.

As I understand it, the basic premise is that the AI has a utility function u(w,A) where "w" is whether or not the specific thermodynamic miracle (TM) occurred and A represents a regular/normal input to a utility function (state of the universe, action taken, or whatever).

u(w,A) = C (a low constant) if w is false (TM did not happen), and u(w,A) is essentially "u(A) given w" when w is true (ie. it's a "normal" kind of utility function, dominated by things other than the TM, though it still has to incorporate the fact that the TM happened into its model of the world).

So, the signal is sent, and it calculates its expected utility for either scenario. If w = false, u(w,A) = a low constant, and if w = true, it determines an action A* that maximizes the utility function u(A)|w. As long as 2^-100 * u( A* ) > C, it starts taking action A*.

Is this right so far?

Then, after "ON" has gone through, it must adapt itself and backtrack on its previous behaviour.

This mechanism seems underspecified to me (though maybe it's obvious to others). Why would it backtrack (or what do you mean by backtrack)? Eventually, it observes that w = false (that "ON" went through unchanged) and that its actions are no longer beneficial, so it just stops doing anything, right? The process terminates or it goes to standby?

If we want the AI to actually backtrack, wouldn't we need to modify the utility function so that it takes a configuration parameter as well? u(w,A,X) s.t. X = true iff the AI is in its initial configuration, and set it so that u(F,A,F) = 0, but u(F,A,T) = C ? (this probably counts as just an implementation detail, but I figured I'd mention it anyway)

The part that makes me feel like I didn't actually understand it at all is:

However, the only possible world that it cares about, now, is one in which the "ON" signal was overwritten by a thermodynamic miracle... into exactly the same "ON" signal. As long as the possibility of this is non-zero, the AI will behave as if it's certain.

Can you explain this a bit more? Do you mean that even after it observes "ON", it doesn't stop acting as if w = true? That P(TM that overwrites ON with ON) * u(A)|w > C ? If that's the case, then it would never backtrack, right? So it's essentially a full simulation of an AI under the assumption w, but with the knowledge that w is incredibly unlikely, and no built-in halting condition?

Thanks