= 27d567bc2d48d9f40e9ccc20ea370b15)
irrational comments on Circular belief updating - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (50)
A sorcerer has two ways to manipulate people:
1) Move things around in the world.
2) Directly influence people's minds.
I'm not going to talk about option 2 because it stops people from being perfect reasoners. (If there's a subset of option 2 that still lets people be perfect reasoners, I'd love to hear it - that might be the most interesting part of the puzzle). That leaves option 1.
Here's a simple model of option 1. Nature shuffles a deck of cards randomly, then a sorcerer (if one exists) has a chance to rearrange the cards somehow, then the deck is shown to an observer, who uses it as Bayesian evidence for or against the sorcerer's existence. We will adopt the usual "Nash equilibrium" assumption that the observer knows the sorcerer's strategy in advance. This seems like a fair idealization of "moving things around in the world". What would the different types of sorcerers do?
Note that if both Bright and Dark might exist, the game becomes unpleasant to analyze, because Dark can try to convince the observer that Bright exists, which would mean Dark doesn't exist. To simplify the game, we will let the observer know which type of sorcerer they might be playing against, so they only need to determine if the sorcerer exists.
A (non-unique) best strategy for Bright is to rearrange the cards in perfect order, so the observer can confidently say "either Bright exists or I just saw a very improbable coincidence". A (non-unique) best strategy for Dark is to leave the deck alone, regardless of the observer's prior. Invisible has the same set of best strategies as Dark. I won't spell out the proofs here, anyone sufficiently interested should be able to work them out.
To summarize: if sorcerers can only move things around in the world and cannot influence people's minds directly, then Bright does as much as possible, Invisible and Dark do as little as possible, and the observer only looks at things in the world and doesn't do anything like "updating on the strength of their own beliefs". The latter is only possible if sorcerers can directly influence minds, which stops people from being perfect reasoners and is probably harder to model and analyze.
Overall it seems like your post can generate several interesting math problems, depending on how you look at it. Good work!
some models of 2:
Every person in faeri has a Phsycomagical Intuition. This has a 50% probability of always detecting if there are any sorcerers within a light minute or so, and a 50% probability of giving random results (that don't change or otherwise can be distinguished from having a functioning one). A sorcerer can expend some effort to set the non-functioning ones to whatever it wants.
The sorcerer cannot affect existing minds, but can alter any faerians priors before birth, in a probabilistic manner within normal human variation.
Hmm, the first case seems reducible to "moving things around in the world", and the second sounds like it might be solvable by Robin Hanson's pre-rationality.
How about, if Bob has a sort of "sorcerous experience" which is kind of like an epiphany. I don't want to go off to Zombie-land with this, but let's say it could be caused by his brain doing its mysterious thing, or by a sorcerer. Does that still count as "moving things around in the world"?
Well, it seems possible to set up an equivalent game (with the same probabilities etc) where the sorcerer is affecting a card deck that's shown to you.
Maybe I should have drawn the distinction differently. If the sorcerer can only affect your experiences, that's basically the same as affecting a card deck. But if the sorcerer can affect the way you process these experiences, e.g. force you to not do a Bayesian update where you normally would, or reach into your mind and make you think you had a different prior all along, that's different because it makes you an imperfect reasoner. We know how to answer questions like "what should a perfect reasoner do?" but we don't know much about "what should such-and-such imperfect reasoner do?"
I see what you mean now, I think. I don't have a good model of dealing with a situation where someone can influence the actual updating process either. I was always thinking of a setup where the sorcerer affects something other than this.
By the way, I remember reading a book which had a game-theoretical analysis of games where one side had god-like powers (omniscience, etc), but I don't remember what it was called. Does anyone reading this by any chance know which book I mean?
You might be thinking of Superior Beings by Steven Brams.
(My favourite result of this kind is that if you play Chicken with God, then God loses.)