Sorry, but you can't get around the fact that humans are not well equipped to compute probabilities. We can't even state what our priors are in any reasonable sense much less compute exact probabilities.

As a result using probabilities has come to be associated with having some kind of model. If you've never studied the question and are asked how likely you think it is there are intelligent aliens you say something like "I think it's quite likely". You only answer with a number if you've broken it down into a model (chance life evolves * average time to evolve intelligencechance of disaster..).

Thus, saying something like "70% chance" indicates to most people that you are claiming your knowledge is the result of some kind of detailed computation and can thus be seen as an attempt to claim authority. You can't change this rule on your own.

Thankfully, there are easy verbal alternatives. "Ehh, I guess I would give 3:1 odds on it" and many others. But use of chance/probability language isn't it.

Uhh, why not just accept that you aren't and can never be perfectly rational and use those facts in positive ways.

Bubbles are psychologically comforting and help generate communities. Rationalist bubbling (which ironically includes the idea that they don't bubble) probably does more to build the community and correct other wrong beliefs than almost anything else.

Until and unless rationalist take over society the best strategy is probably just to push for a bubble that actively encourages breaking other (non-rationalist) bubbles.

This error has been corrected; thank you for pointing this out!

I actually did a bit more research, and it really seems like flu vaccine efficacy in healthy adults is more like 70% (and sometimes as high as 90%), despite the fact that the average efficacy of the vaccine throughout the population is around 60%. The reason that efficacy in healthy adults is so high, relative to the average efficacy, is that efficacy in the elderly is around 30-40%.

Also, note that about 42% of the US population gets flu shots on any given year. So, if 10% of people on average get the flu, and the vaccine is 60% efficacious throughout the population, then we can write the following equations, defining sick1 as the event in which a person who was vaccinated gets the flu, and sick2 as the event in which a person who was not vaccinated gets the flu:

p(sick1) x 0.42 + p(sick2) x 0.58 = 1 x 0.10

p(sick1) = p(sick2) x 0.60

Solving this system of equations, we get:

p(sick1) = 0.0721

p(sick2) = 0.120 (previous typo: had been written as 0.0120)

The practical implication of this is that the conservative analysis conducted in this report, and shown in Figure 1, assumes that around 5.7% (rather than a more realistic 10 or 12%) of the population will catch the flu in any given year.

Omitting death seems like a big deal. Very crudely, it looks like p=10^-4. It's said that society values each life at $5M, so that's E=-$500 already, but each individual likely values their own life a bit higher.

Well it can't still be instrumental rationality anymore. I mean suppose the value being minimized is overall suffering and you are offered a (non-zero probability one time...and you know there are no other possible infinitary outcomes) threat that if you don't believe some false claim X god will create an infinite (no other infinite outcomes) amount of suffering. You know before the choice to believe the false claim that no effect of believing it will increase expected suffering to overwhelm the harm of not believing it.

But the real rub is what do you say about the situation where the rocks turn out to be rocks cleverly disguised as people. You still have every indication that your behavior convincing yourself is an attempt to believe a false statement but it is actually true.

Does the decision procedure which says whatever you want it to normally say but makes a special exception that you can deceive yourself if (description of this situation which happens to identify it uniquely in the world).

In other words is it a relation to truth that you demand. In which case the rule gets better whenever you make exceptions that happen (no matter how unlikely it is) in the actual world to generate true and instrumentally useful beliefs. Or is it some notion of approaching evidence?

If the latter you seem to be committed to the existence of something like Carnap's logical probability, i.e., something deducible from pure reason that assigns priors to all possible theories of the world. This is a notoriously unsolvable (in the sense that it doesn't have one) unsolvable problem.

At the very least can you state some formal conditions that constrain a rule for deciding between actions (or however you want to model it) that captures the constraint you want?

Would discovering that a wavefunction collapse postulate exists be evidence for simulation? A simulation that actually computed all Everett branches would demand exponentially more resources, so a simulator would be more likely to prune branches either randomly (true or pseudo-) or according to some criterion.

One thing that confuses me about these discussions, and I'm very willing to be shown where my reasoning is wrong, is that there seems to be an implicit assumption that the simulators must follow any of the rules they've imposed upon us. If I simulate a universe powered by the energy generated by avacados, would the avacado beings try to spot an avacado limit, or an order to the avacados?

A simulator could have a completely different understanding as to how the universe works.

I would guess the argument against this is that why else would we be simulated if not to be a reflection of the universe above? I'm not sure I buy this, or necessarily assign a high probability to it.

"Good people are consequentialists, but virtue ethics is what works," is what I usually say when this topic comes up. That is, we all think that it is virtuous to be a consequentialist and that good, ideal rationalists would be consequentialists. However, when I evaluate different modes of thinking by the effect I expect them to have on my reasoning, and evaluate the consequences of adopting that mode of thought, I find that I expect virtue ethics to produce the best adherence rate in me, most encourage practice, and otherwise result in actually-good outcomes.

But if anyone thinks we ought not to be consequentialists on the meta-level, I say unto you that lo they have rocks in their skulls, for they shall not steer their brains unto good outcomes.

I'd also like to point out the Cartesian barrier is actually probably a useful feature.

It's not objectively true in any sense but the relation between external input, output and effect is very very different than that between internal input (changes to your memories say), output and effect. Indeed, I would suggest there was a very good reason that we took so long to understand the brain. It would be just too difficult (and perhaps impossible) to do so at a direct level the way we understand receptors being activated in our eyes (yes all that visual crap we do is part of our understanding).

Take your example of a sensor aimed at the computer's memory circuit. Unlike almost every other situation there are cases that it can't check it's hypothesis against because such a check would be logically incoherent. In other words certain theories (or at least representations of them) will be diagonalized against because the very experiments you wish to do can't be effected because that 'intention' itself modifies the memory cells in such a way as to make the experiment impossible.

In short the one thing we do know is that assuming that we are free to choose from a wide range of actions independently of the theory we are trying to test and that how we came to choose that action is irrelevant is an effective strategy for understanding the world. It worked for us.

Once the logic of decision making is tightly coupled with the observations themselves the problem gets much harder and may be insoluble from the inside, i.e., we may need to experiment on others and assume we are similar.