Salemicus comments on A Story of Kings and Spies - Less Wrong

22 Post author: Joshua_Blaine 11 June 2014 11:54PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (33)

You are viewing a single comment's thread. Show more comments above.

Comment author: gwern 12 June 2014 12:04:51AM *  11 points [-]

Usual problem of betting on existential risks: it's hard to see how it works. How is Orin going to be punished, exactly, if he's a liar or agent provacateur in the pay of the foreign powers intending to wear down the King? You've constructed this case to be easier than the usual prediction market scenario by making the King accept the bet and locking up Orin so he can either be rewarded in the case of military victory* or punished if all is quiet along the Don, yet is still beatable by the rival countries: 1000 coins in quiet payments to the spy isn't much to wage economic warfare on a small kingdom and also make it less likely to respond correctly in the future.

  • what does victory look like in the case of dealing with an existential risk? A rogue AI isn't going to send convenient Terminators which can be outfought in an epic showdown and then a grateful President thank Orin.

(Also, is this really isomorphic to existential risks? The King believes implicitly that there are going to be invasions in the future, while most people deny the possibilities of existential risk entirely. A more apt parable would be an Incan brought before Atahualpa warning that these gods bearing gifts are perhaps not as nice as they might appear but are representatives of a rapacious power never before known...)

Comment author: Salemicus 12 June 2014 12:15:16PM *  6 points [-]

How is Orin going to be punished, exactly, if he's a liar or agent provacateur in the pay of the foreign powers intending to wear down the King?

Well...

The King reached out and grabbed Orin's shoulder, looking into his eyes with his own, and smiled wide.

My impression from that rather chilling sentence is that, if Orin is wrong, he's going to be investigated extensively (and at leisure - we don't have to make an urgent decision any more). If he proves to be a liar, not just honestly mistaken, then a 1000 coin fine is going to be the least of his worries.

Comment author: gwern 12 June 2014 03:59:49PM 8 points [-]

That would be breaking the terms of the agreement: Orin pays 1000 coins on the installment plan if he's wrong, end of story. If the king casually breaks his agreements with informants, well, now he has another problem...

Comment author: Lumifer 12 June 2014 04:58:45PM 3 points [-]

That would be breaking the terms of the agreement: Orin pays 1000 coins on the installment plan if he's wrong, end of story.

Not quite. If Orin turns out to be wrong but had no malicious intent, then yes, he just pays 1000 coins. On the other hand, if Orin deliberately mislead the king, I don't recall the terms of the agreement including immunity against charges of treason.

Comment author: gwern 12 June 2014 05:55:49PM 3 points [-]

If you can prove malicious intent, then why the need for the bet in the first place...?

Comment author: Lumifer 12 June 2014 05:57:37PM *  7 points [-]

You can't prove it now, you may be able to prove it (or its absence) in a few days. If there is none, Orin just pays his 1000 coins.

The point of the bet was to properly incentivize Orin in the present.

Comment author: gwern 12 June 2014 10:51:15PM *  -1 points [-]

How do you prove it in a few days? 'Oh, no army appeared' said Orin. 'My friend must have been wrong or the date was pushed back. Still, as an honest man, I stand by our deal though it beggar me.'

Comment author: Lumifer 12 June 2014 11:52:58PM 3 points [-]

How do you prove it in a few days?

Are you asking me how the plot can play out in a fictional story? :-D

Here's one possibility -- the king's large and effective network of spies and informants will send the word that the Northern Kingdom executed a disinformation campaign against the king using a fellow named Orin...

Comment author: gwern 13 June 2014 02:25:09PM 0 points [-]

Here's one possibility

Possibility is not good enough. And in any case, my proposed defeater can be implemented by exactly two people: a volunteer and a rich benefactor, and so it is vastly more likely to be undiscovered by spies & informants than an actual attack. The king is unsure his spy network will uncover every attack, so a fortiori, he is very unsure that my proposed scheme would be detected.

Comment author: Lumifer 13 June 2014 02:36:51PM 4 points [-]

Possibility is not good enough.

Not good enough for what?

Frankly, I don't see towards which point are you driving. This is a fable about, basically, an exercise in game theory. You don't like the story? You think it misleads? If you were king you would have behaved differently?

Comment author: gjm 12 June 2014 04:17:12PM 5 points [-]

I don't think that's right.

If he's wrong he pays 1000 coins. If he's guilty of treason he gets executed (or whatever). Nothing in the story suggests to me that the king is giving some sort of immunity even if Orin was deliberately misleading.

Comment author: Joshua_Blaine 16 June 2014 07:11:09PM 2 points [-]

This is correct. Betting, as a policy, helps distinguish between Orin(correct) and Orin(wrong), but is really only useful for eliminating Orin(spy) because it's a novel method that the King expects spies to yet be unprepared for, and is easily investigated if circumvented.

Imagine, If Orin is wrong and yet mysteriously has all his debts re-paid and shop re-purchased shortly after being punished, some eyebrows would be raised.

Comment author: gwern 16 June 2014 09:29:08PM 0 points [-]

Imagine, If Orin is wrong and yet mysteriously has all his debts re-paid and shop re-purchased shortly after being punished, some eyebrows would be raised.

I'm sure they would be, but alas, that will happen long after the kingdom has been conquered after being driven into financial ruin by listening to foreign informants who work for their enemies.

Comment author: gwern 16 June 2014 09:28:10PM -1 points [-]

If he's guilty of treason he gets executed (or whatever).

As I pointed out, that's not possible to find. Also, I think 'treason' usually only applies to foreigners.

Comment author: gjm 16 June 2014 11:15:48PM 2 points [-]

that's not possible to find.

I'm confused. I'm sure it's not possible to determine for certain whether Orin is guilty of treason, but whyever should it be impossible to know, say, beyond reasonable doubt?

I think 'treason' usually only applies to foreigners.

Quite the reverse, I think. E.g., definition 2a in the OED says

high treason or treason proper: Violation by a subject of his allegiance to his sovereign or to the state.

(Definition 1a is "The action of betraying; betrayal of the trust undertaken by or reposed in any one; breach of faith, treacherous action, treachery." which is very broad and includes 2a as a special case. 1b is the very special case "treason of the clerks == trahison des clercs".)

Comment author: gwern 18 June 2014 03:47:07AM 0 points [-]

but whyever should it be impossible to know, say, beyond reasonable doubt?

How would you do that in practice? Even if you had a way to do that, how would you stop Orin from, say, going back to his patron and never leaving his native country again and enjoying the fruits of a grateful country's gratitude? There's just tons and tons of holes to this whole scenario which is why it was a bad idea if it wasn't intended to be a commentary on existential risk.

Quite the reverse, I think. E.g., definition 2a in the OED says

Yes, my bad. I meant to write 'only applies to citizens'. As a foreigner, Orin may be chargeable with things like espionage, but 'treason' makes little sense.

Comment author: gjm 18 June 2014 01:15:07PM *  0 points [-]

How would you do that in practice?

Search his house for incriminating correspondence. Talk to his bank manager and find out whether he's had a surprising influx of money recently. (If so, then in court you can ask Orin to explain where it came from; if it was really a bribe from a foreign enemy, he'll have to make up something that you may be able to refute.) Ask his neighbours whether unexpected people have been seen going in and out of his house. (If so, then maybe their descriptions match up to those of known foreign agents. Or their conversation might have been overheard.) All the same sorts of things you do when investigating any alleged crime. If a sufficient quantity of evidence of malfeasance piles up, you're done.

In any case, we seem to have shifted from "obviously the king was granting Orin immunity from prosecution for treason" to "of course it might be difficult to convict him of treason if guilty". Which, yes, it might. (So I should have said: "If he's found guilty of treason he gets executed (or whatever)"; my apologies for the inexactitude.)

how would you stop Orin from, say, going back to his patron [...]

He's already in custody.

it was a bad idea if it wasn't intended to be a commentary on existential risk.

I thought it obviously was intended to be a commentary on existential risk.

[EDITED to add: but I now see that the author has specifically said it wasn't. Oh well.]

Orin [...] never leaving his native country again [...] As a foreigner, Orin may be chargeable

I'm confused again. In the story, Orin isn't a foreigner, he's a citizen of the Kingdom under discussion. At least, there's every indication he is and none he isn't that I can see.

[EDITED shortly after posting, to make it clearer that the point of the evidence-gathering would be the aggregate evidence, not that you necessarily look for a single smoking gun, and to clarify the nature of some of the possible evidence.]