Hmm, the first case seems reducible to "moving things around in the world", and the second sounds like it might be solvable by Robin Hanson's pre-rationality.

Is it sane for Bob and Daisy to be in such a positive or negative feedback loop? How is this resolved?

It is not sane.

If you use a belief (say A) to change the value of another belief (say B), then depending on how many times you use A, you arrive at different values. That is, if you use A or A,A,A,A as evidence, you get different results.
It would be as if:

P(B|A) <> P(B|A,A,A,A)

But the logic underlying bayesian reasoning is classical, so that A <-> A+A+A+A, and, by Jaynes' requirement IIIc (see page 19 of The logic of science):

P(B|A) = P(B|A,A,A,A)

Like cousin_it, I'm assuming no mind tampering is involved, only evidence tampering

Is it sane for Bob and Daisy to be in such a positive or negative feedback loop? How is this resolved?

I don't think the feedback loops exist. Bob saying "the fact that I believe in bright is evidence of Bright's existence" is double-counting the evidence; deducing "and therefore, Bright exist" doesn't bring in any new information.

It's not that different from saying "I believe it will rain tomorrow, and the fact that I believe that is evidence that it is rain tomorrow, so I'll increase my degree of belief. But wait, that makes the evidence even stronger!".

If Bob and Daisy took the evidence provided by their belief into account already, how does this affect my own evidence updating? Should I take it into account regardless, or not at all, or to a smaller degree?

Just ignore the whole "belief in dark is evidence against dark" thing, Daisy already took that information into account when determining her own belief, you don't want to double count it.

Treat it the same way as you'd treat hearing Bob tell you that in Faery, the sky is Blue, and Daisy telling you that in Faery, the sky is Green.

A sorcerer has two ways to manipulate people:

1) Move things around in the world.

2) Directly influence people's minds.

I'm not going to talk about option 2 because it stops people from being perfect reasoners. (If there's a subset of option 2 that still lets people be perfect reasoners, I'd love to hear it - that might be the most interesting part of the puzzle). That leaves option 1.

Here's a simple model of option 1. Nature shuffles a deck of cards randomly, then a sorcerer (if one exists) has a chance to rearrange the cards somehow, then the deck is shown to an observer, who uses it as Bayesian evidence for or against the sorcerer's existence. We will adopt the usual "Nash equilibrium" assumption that the observer knows the sorcerer's strategy in advance. This seems like a fair idealization of "moving things around in the world". What would the different types of sorcerers do?

Note that if both Bright and Dark might exist, the game becomes unpleasant to analyze, because Dark can try to convince the observer that Bright exists, which would mean Dark doesn't exist. To simplify the game, we will let the observer know which type of sorcerer they might be playing against, so they only need to determine if the sorcerer exists.

A (non-unique) best strategy for Bright is to rearrange the cards in perfect order, so the observer can confidently say "either Bright exists or I just saw a very improbable coincidence". A (non-unique) best strategy for Dark is to leave the deck alone, regardless of the observer's prior. Invisible has the same set of best strategies as Dark. I won't spell out the proofs here, anyone sufficiently interested should be able to work them out.

To summarize: if sorcerers can only move things around in the world and cannot influence people's minds directly, then Bright does as much as possible, Invisible and Dark do as little as possible, and the observer only looks at things in the world and doesn't do anything like "updating on the strength of their own beliefs". The latter is only possible if sorcerers can directly influence minds, which stops people from being perfect reasoners and is probably harder to model and analyze.

Overall it seems like your post can generate several interesting math problems, depending on how you look at it. Good work!

Adding some structure to this hypothetical: At time t=0, Bob and Daisy have certain priors for their beliefs on sorcery, which they have not adjusted for this argument. Bob's belief was Position 1, with reduced strength, and Daisy's was Position 3, with greater strength.

I'll call your argument A0.

At time t=1, Bob and Daisy are both made aware of A0 and its implications for adjusting their beliefs. They update; Bob's belief in 1 increases, and Daisy's belief in 3 decreases.

More arguments:

A1: If Position 1 is true, then Bright is likely to cause you to increase your belief in him, therefore increasing your belief in Bright is evidence for Position 1.

A1': Corollary: Decreasing your belief in Bright is evidence against Position 1.

A2:If Position 3 is true, then Dark is likely to cause you to decrease your belief in him, therefore decreasing your belief in Dark is evidence for Position 3.

A2': Corollary: Increasing your belief in Dark is evidence against Position 3.

At time t=2, Bob and Daisy are exposed to A1 and A2, and their converses A1' and A2'. If they believe these, they should both increase credence for Positions 1 and 3, following A1 and A2, then increase credence for Position 1 and decrease it for Position 3, following A1 and A2', then follow A1 and A2 again, etc. This might be difficult to resolve, as you mention in your first question.

However, there is a simple reason to reject A1 and A2: Their influence is totally screened off! Bob and Daisy know why they revised their beliefs, and it was because of the valid argument A0. Unless Bright and Dark can affect the apparent validity of logical arguments (in which case your thoughts can't be trusted anyway), A0 is valid independent of which position is true. This action moves them to begin a feedback loop, but stop after a single iteration.

There is a valid reason they might want to continue a weaker loop.

A3: That you have encountered A0 is evidence for the sorcerers whose goals are served by having you be influenced by A0.

A3': That you have encountered A3 is evidence for the sorcerers whose goals are served by having you be influenced by A3.

A3'': That you have encountered A3'' is evidence for the sorcerers whose goals are served by having you be influenced by A3''.

etc.

But this is only true if they didn't reason out A0 or A3 for themselves, and even then A3', A3'', etc. should be considered obvious implications of A3 for a well-reasoned thinker. (In fact, A3 is properly more like "That you have encountered a valid argument is evidence for the sorcerers whose goals are served by having you be influenced by that argument.") So that adds at most one more layer, barring silly Tortoise-and-Achilles arguments.

Given all that, for your second question, you still should take their beliefs into account, but possibly to a slightly lesser degree.

A point I'm confused on: when you, based on A0, update based on their A0-updated belief, are you double-counting A0? If so, you should update to lesser degree. But is that so?

I believe he's not assuming similar priors.

With respect to question 1, Aumann's Agreement Theorem would require that if they are acting rationally as you stated and with common knowledge, they would have to agree. That being the case, according to the formalism, question 2 is ill-posed.

Your proposed state of affairs could hold if they lack common knowledge (including lack of common knowledge of internal utility functions despite common knowledge of otherwise external facts and including differing prior probabilities). To resolve question 2 in that case you would have to assign probabilities to the various forms that the shared and unshared knowledge could take, to determine which state of affairs most probably prevails. For example, you may use your best estimation and determine that the state of affairs that prevails in Faerie is similar to the state of affairs that prevails in your own world/country/whatever, in which case you should weight the evidence provided by each person similar to how you would rate it if you were surveying your fellow countrymen. This is all fairly abstract because approaching such a thing formally is currently well outside our capabilities.

Sir Karl Popper argued against induction and against authority in Conjectures and Refutations (and in most of his books). Provisional trust in scientific claims can be useful, but all conjectures are subject to critical examination and perhaps refutation. Popper said the scientific method is defined by use of falsifiability, not by authoritty.

Just realised you're the post author, so: Thanks for posting this, it's something I've wondered about in relation to myself, as well. :)

1: No tentacles

But imagine that we also have independent evidence that Earth is 1 million years old.

This reminds me of something Eliezer once said--"How would I explain the event of my left arm being replaced by a blue tentacle? The answer is that I wouldn't. It isn't going to happen." We do not observe a young (even 10^6) Earth, and by suggesting the possibility of one as counterevidence against the strength of the 'a priori' reasoning I advocated, you must be smuggling in a circular assumption that young Earth models have significant probability.

Your argument as I understand it is roughly that since my a priori reasoning would fail in young Earth scenarios, then that reasoning is unreliable. But if our prior for young Earth scenarios is extremely low, then it will only very rarely happen that my reasoning will fail in that particular way. Therefore for your argument to go through, you would have to place a high prior probability on young Earth scenarios.

To put it another another way: If observing a young Earth would be evidence against my a priori reasoning, then by conservation of expected evidence, our actual observation of a non-young Earth must be evidence in favour of that reasoning.

People in a modern day situation, and LW'ers in particular, are better placed to understand that 'naturalistic' explanations are preferable, and that magic ones should incur huge complexity penalties. Therefore we should have low priors on young Earths, because most of our probability will be concentrated in models where intelligent life arises from nonintelligent (hence slow) processes as opposed to intelligent (e.g. God) processes.

Moreover, the more intelligent the process that generated us, the more we push the explanatory buck back onto that process. God is an extreme case where the mystery of the apparent improbability of human intelligence is replaced with the mystery of the apparent improbability of divine intelligence. But even less extreme cases like superhumanly (but still decidedly 'finitely intelligent') entities simulating us incurs a penalty for passing the explanatory buck back. Natural selection is so elegant and formidable because it is an existing behaviour of what we already observe.

2: Insanity screens off charity

Creationists like to cite irreducible organs

If--even before accepting evolution by natural selection--you can put an extremely high probability on creationists spewing objections like 'irreducible organs' regardless of the veracity of evolution by natural selection, then you can pretty much write off the counterarguments of creationists, because your observation of creationists making these arguments is extremely weak evidence that there is anything to these counterarguments. Now, this is circular if your only reason for not taking creationists seriously is that they are wrong natural selection/irreducible organs, but there's 'any number' of other reasons to suspect creationists are engaging in motivated cognition.

The same way that if natural selection provides a substantial portion of the explanation of evolution, then, you need look no further, if cognitive biases/sociology provide a substantial portion of or even all of the explanation for creationists talking about irreducible organs, then their actual counterarguments are screened off by your prior knowledge of what causes them to deploy those counterarguments; you should be less inclined to consider their arguments than a random string generator that happened to output a sentence that reads as a counterargument against natural selection.