Comment author: 16 May 2016 10:20:27AM *  0 points [-]

I keep having trouble thinking of probabilities when I'm to be copied and >=1 of "me" will see red and >=1 of "me" will see green. My thought is that it is 100% likely that "I" will see red and know there are others, once-mes, who see green, and 100% likely vice-versa. Waking up to see red (green) is exactly the expected result.

I do not know what to make of this opinion of mine. It's as if my definition of self - or choice of body - is in superposition. Am I committing an error here? Suggestions for further reading would be appreciated.

Comment author: 23 January 2016 03:54:38AM 0 points [-]

There isn't correlation between these inscriptions and implied contents (since he could have put the key and dagger in either box), but there /is/ correlation between {the inscriptions and contents} and the king's honesty. The king didn't lie and he wouldn't have put inscriptions and contents into such an arrangement that would make it true that he lied. This puts a constraint on how he could arrange the inscriptions and contents.

Comment author: 18 April 2016 03:23:10AM 0 points [-]

Salient point: why you mention arrangements of inscriptions and contents at all? That is what confuses me. Either the arrangements matter at some point - such as inscribing - in which case there had been a lie when the king labeled an (apparently?) empty box with "This box contains the key." (not "this box doesn't contain the dagger", which would have been true), or not at all, in which case I reiterate my previous question.

Comment author: 17 April 2016 02:46:00PM 1 point [-]

There's a lot of value in that. Sometimes it's best not to go down the rabbit hole.

Whatever the technicalities might be, the jester definitely followed the normal, reasonable rules of this kind of puzzle, and by those rules he got the right answer. The king set it up that way, and set the jester up to fail.

If he'd done it to teach the jester a valuable lesson about the difference between abstract logic and real life, then it might have been justified. But he's going to have the jester executed, so that argument disappears.

I think we can all agree, The King is definitely a dick.

Comment author: 18 April 2016 03:18:41AM *  1 point [-]

I'm trying to stay levelheaded about King Richard. What I meant was that there seems to be extraneous details here - about the order things were done in, first inscribe ("key is here", on an empty(?) box), then put dagger in, or that it was written, not spoken. Many comments only enforce the importance of that.

The "real" answer seems to be one that effectively makes all kinds of communication useless, and what I've spent so much time on was trying to pin down the borders of this insanity, some marker saying "abstract logic application to real life* not allowed past this point".

*) the use of physical boxes binding the riddle to "real life"

In response to Ethical Injunctions
Comment author: 15 April 2016 10:26:40AM *  0 points [-]

There is something I don't understand about the "fooling programmers -> shutdown" idea - how "It looks like it should be possible"

Am I correct assuming this requires an AI that:

1. Implements changes in discrete iterations

2. Understands how it itself works, both on low and very high levels (getting fractal here)

3. Can do the same for its more advanced iteration

4. Monitors the more advanced iteration (a simulation or a live deployment) for a certain behavior, preferably in real time or faster

5. Has the concept of deception (shutdown condition, AND a salient factor in other calculations)

6. At all times allows itself to be killed by its previous iteration (accidental Robin Hood/Friar Tuck hacks come to mind)

7. Has a full-proof definition of "deceiving programmers" which:

a) remains consistent with programmers' intentions across iterations, immune to drift

• not just from n-1 to n, but from 0 to n - you could have iteration 0 also watch over n, but would it understand n?

b) remains applicable across iterations, cannot be bypassed or cheated

• "the device category is now 'data processor', not 'human output device', so- [process terminated]"

c) allows n to give different output than in n-1 (else why bother changing?)

• just having n-1 evaluate n's output might not suffice: n-1 might need to change itself first to understand it

d) does not have too significant an impact on improvement process

• metaanalysis on own shutdown logs causing shutdown?

e) has a satisfactory solution for the concept of manipulating people, which somehow doesn't paralyze the system

• situation: output data that is true but would be misinterpreted by / cause a glitch in some other system the results will be fed to, leading people to act upon the wrong results of in a way desirable by the AI

• "It is not deceit if the format change is communicated along with the output data. Or somewhere in the status update PDFs sent 40 times a second."

• "this output is not false, it is merely unrelated"

• "but that would make the programmers' business competitors draw wrong conclusions... wait, that is allowed? Moving non-programmers to Resource.Actuator category, calculating new self-improvement strategy."

In response to Timeless Physics
Comment author: 15 April 2016 01:40:44AM 0 points [-]

An omnipotent magicker decides to flip a coin, and the coin lands heads. Afterwards, the magicker changes every particle in the universe to what it would be had the coin landed tails -- including those in his own brain. Is it true that in the past, the coin landed heads, even though this event is epiphenomenal?

I realize that the magicker is violating the laws of entropy, and that in the real world there are no magickers. I also realize that for the purposes of anyone in the universe, the first coin flip doesn't and couldn't possibly matter, because it was epiphenomenal. But I'm still curious what the answer to my question is.

In response to comment by on Timeless Physics
Comment author: 15 April 2016 09:53:23AM *  0 points [-]

For the purposes of the universe, the first coin flip not only doesn't matter, it never existed. The magicker himself will claim - and show any amount of supporting evidence - that it landed tails. He will not even remember casting a spell, unless he'd have cast one in this case, too, but for a different reason (if he'd have cast anyway regardless of the toss outcome, he'd get looped, recalling himself making the toss, but not yet casting; or being confused why he recalls casting, but there is no change in the toss outcome).

Time between the toss and casting, when things were different than they are now? What perspective would that need? Looking at the universe from outside the universe?

In response to Ethical Injunctions
Comment author: 22 October 2008 07:01:12PM -1 points [-]

Eliezer, I'm aware of nanotech. And I know you think the human race is obsolete when AI comes along. And I also think that you might be right, and that people like you might have the power to make it so.

But I also believe that if the rest of the human race really thought that was a possibility, you'd be burned at the stake.

Do you have any regard for the opinions of humanity at all? If you were in the position of having an AI in front of you, that you had convinced yourself was friendly, would you let it out of the box without bothering to consult anyone else?

In response to comment by on Ethical Injunctions
Comment author: 15 April 2016 06:55:05AM *  0 points [-]

The term "obsolete" as used here confuses me. It seems to imply a purpose, one that individuals - or humanity - or whatever other "intelligence collective" there may be - could get behind. What might that purpose be? Not survival, is it?

In response to Ethical Injunctions
Comment author: 20 October 2008 11:31:52PM 7 points [-]

"Would you kill babies if it was the right thing to do? If no, under what circumstances would you not do the right thing to do? If yes, how right would it have to be, for how many babies?"

I would have answered "yes"; eg., I would have set off a bomb in Hitler's car in 1942, even if Hitler was surrounded by babies. This doesn't seem to be a case of corruption by unethical hardware; the benefit to *me* from setting off such a bomb is quite negative, as it greatly increases my chance of being tortured to death by the SS.

In response to comment by on Ethical Injunctions
Comment author: 15 April 2016 06:31:43AM *  0 points [-]

Definitely yes. It's not like killing babies is inherently wrong (*), it just is under most circumstances. I was thinking more along the lines of euthanasia of babies you've discovered have been prepared for use in biological warfare... but my mind tends to go into bad places. Let's not get any further into that.

*) unless you use absolute values for wrong, in which case it definitely is, but so is breathing

Comment author: 13 April 2016 01:54:06PM 2 points [-]

I can't speak for Eliezer's intentions when he wrote this story, but I can see an incredibly simple moral to take away from this. And I can't shake the feeling that most of the commenters have completely missed the point.

For me, the striking part of this story is that the Jester is shocked and confused when they drag him away. "How?!" He says "It's logically impossible". The Jester seems not to understand how it is possible for the dagger to be in the second box. My explanation goes as follows, and I think I'm just paraphrasing the king here.

1- If a king has two boxes and a means to write on them, then he can write any damn thing on them that he wants to. 2- If a king also has a dagger, then he can place that dagger inside one of the two boxes, and he can place it in whichever box he decides to place it in.

That's it. That's the entire explanation for how the dagger could "possibly" be inside the second box. It's a very simple argument, that a five year old could understand, and no amount of detailed consideration by a logician is going to stop this simple argument from being true.

The jester, however, thought it was impossible for the dagger to be in the second box. Not just that it wasn't there, but that it was IMPOSSIBLE. That's how I read the story, anyway. He used significantly more complicated logic, and he thought that he'd proven it impossible. But it only takes a moment's reflection to see that he's wrong.

Some of the comments above have tried to work out what was wrong with Jester's logic, and they've explained the detailed and subtle flaws in his reasoning. That's great - if you want to develop a deep understanding of logic, self-referential statements, and mathematical truth values (and lets be fair, I suppose most of us do), but in the context of the sequences on rationality, I think there's a much better lesson to learn.

Remember: rationalists are supposed to WIN. We're supposed to develop reasoning skills that give us a better and more useful understanding of reality. So the lesson is this: don't be seduced by complex and detailed logic, if that logic is taking you further and further away from an accurate description of reality. If something is already true, or already false, then no amount of reasoning will change it.

Reality is NOT required to conform to your understanding or your reasoning. It is your reasoning that should be required to conform to reality.

Comment author: 14 April 2016 11:12:14AM 0 points [-]

Breaking #24 of the Evil Overlord List makes me wince, too, even if it's a jester doing it. Not sure if that's the main point, though, but then, none of the proposed explanation for how the king could pull his "riddle" off without at any point lying feel entirely right to me, so, unless someone offers to help me, I shall have to take your advice and not let myself get entangled in the "complex and detailed logic", when the answer might as well be "BS".

Comment author: 13 April 2016 10:47:14AM *  0 points [-]

The king wrote "This box contains the key." on the 2nd box, before putting the dagger in. Did the second box contain the key as well as the dagger?

Comment author: 17 July 2013 04:21:42AM 0 points [-]

In which case, if Bellini ever references anything self-referential, the idea that he always tells the truth is not a statement about the physical world. It's likely that the origin of the paradox is that the claim that Bellini always tells the truth and the rest of the scenario are contradictory.

Comment author: 13 April 2016 10:33:56AM 0 points [-]

I notice we're somehow not debating what Bellini always telling the truth means for the truth value of the inscribed text which may have had no meaning to him?

View more: Next