RobbBB comments on Precisely Bound Demons and their Behavior - Less Wrong

11 [deleted] 06 March 2015 08:01PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (12)

You are viewing a single comment's thread. Show more comments above.

Comment author: RobbBB 08 March 2015 07:51:43AM *  1 point [-]

To make this work, the demons will need to think in a particular mathematical language, whose primitives they take for granted and have relatively unmysterious empirical significance.

Alternatively, perhaps demons are somehow forced (or motivated) to 'do what i mean' -- they never perversely interpret the semantics of what you say. But they also don't coherently extrapolate your volition, which means they're free to perversely manipulate aspects of the situation that you didn't explicitly talk about (especially when you didn't consciously think about them either). E.g., if you give a demon the English-language instruction "pick up that bucket of water," it isn't free to come up with a suboptimal semantics (like "pick up" means "decapitate" and "bucket of water" means "all my friends"), but it is free to execute the correctly-interpreted instruction in a dangerous way (e.g., picking it up with so much speed and force that it produces a shockwave). If demons interpret the meaning of commands with maximal benevolence but choose a means to the specified end with maximal malice, then mathematically specifying everything about the means makes demons safe(r).