I had to look up whether robins are migratory.

My thoughts exactly. Seeing that statement, I must absolutely AGREE with the second part, and only politely point out that he should rephrase the first part, working "probability" and "working hypothesis" into it.

Salient point: why you mention arrangements of inscriptions and contents at all? That is what confuses me. Either the arrangements matter at some point - such as inscribing - in which case there had been a lie when the king labeled an (apparently?) empty box with "This box contains the key." (not "this box doesn't contain the dagger", which would have been true), or not at all, in which case I reiterate my previous question.

I'm trying to stay levelheaded about King Richard. What I meant was that there seems to be extraneous details here - about the order things were done in, first inscribe ("key is here", on an empty(?) box), then put dagger in, or that it was written, not spoken. Many comments only enforce the importance of that.

The "real" answer seems to be one that effectively makes all kinds of communication useless, and what I've spent so much time on was trying to pin down the borders of this insanity, some marker saying "abstract logic application to real life* not allowed past this point".

*) the use of physical boxes binding the riddle to "real life"

There is something I don't understand about the "fooling programmers -> shutdown" idea - how "It looks like it should be possible"

Am I correct assuming this requires an AI that:

1. Implements changes in discrete iterations

2. Understands how it itself works, both on low and very high levels (getting fractal here)

3. Can do the same for its more advanced iteration

4. Monitors the more advanced iteration (a simulation or a live deployment) for a certain behavior, preferably in real time or faster

5. Has the concept of deception (shutdown condition, AND a salient factor in other calculations)

6. At all times allows itself to be killed by its previous iteration (accidental Robin Hood/Friar Tuck hacks come to mind)

7. Has a full-proof definition of "deceiving programmers" which:

a) remains consistent with programmers' intentions across iterations, immune to drift

• not just from n-1 to n, but from 0 to n - you could have iteration 0 also watch over n, but would it understand n?

b) remains applicable across iterations, cannot be bypassed or cheated

• "the device category is now 'data processor', not 'human output device', so- [process terminated]"

c) allows n to give different output than in n-1 (else why bother changing?)

• just having n-1 evaluate n's output might not suffice: n-1 might need to change itself first to understand it

d) does not have too significant an impact on improvement process

• metaanalysis on own shutdown logs causing shutdown?

e) has a satisfactory solution for the concept of manipulating people, which somehow doesn't paralyze the system

• situation: output data that is true but would be misinterpreted by / cause a glitch in some other system the results will be fed to, leading people to act upon the wrong results of in a way desirable by the AI

• "It is not deceit if the format change is communicated along with the output data. Or somewhere in the status update PDFs sent 40 times a second."

• "this output is not false, it is merely unrelated"

• "but that would make the programmers' business competitors draw wrong conclusions... wait, that is allowed? Moving non-programmers to Resource.Actuator category, calculating new self-improvement strategy."

For the purposes of the universe, the first coin flip not only doesn't matter, it never existed. The magicker himself will claim - and show any amount of supporting evidence - that it landed tails. He will not even remember casting a spell, unless he'd have cast one in this case, too, but for a different reason (if he'd have cast anyway regardless of the toss outcome, he'd get looped, recalling himself making the toss, but not yet casting; or being confused why he recalls casting, but there is no change in the toss outcome).

Time between the toss and casting, when things were different than they are now? What perspective would that need? Looking at the universe from outside the universe?

The term "obsolete" as used here confuses me. It seems to imply a purpose, one that individuals - or humanity - or whatever other "intelligence collective" there may be - could get behind. What might that purpose be? Not survival, is it?

Definitely yes. It's not like killing babies is inherently wrong (*), it just is under most circumstances. I was thinking more along the lines of euthanasia of babies you've discovered have been prepared for use in biological warfare... but my mind tends to go into bad places. Let's not get any further into that.

*) unless you use absolute values for wrong, in which case it definitely is, but so is breathing

Breaking #24 of the Evil Overlord List makes me wince, too, even if it's a jester doing it. Not sure if that's the main point, though, but then, none of the proposed explanation for how the king could pull his "riddle" off without at any point lying feel entirely right to me, so, unless someone offers to help me, I shall have to take your advice and not let myself get entangled in the "complex and detailed logic", when the answer might as well be "BS".