Perhaps it is sometimes rational to prefer agreeing with your friends over being rational :-) ?

Without overfitting, the robot has the goal of shooting at what it sees blue. It achieves its goal. What I get from the article is that the human intelligence mis interprets the goal. Here I see the definition of a goal to equal what the program is written to do, hence it seems inevitable that the robot wll achieve its goal (if there is a bug in the code that misses shooting a blue object every 10 days, then this should be considered part of the goal as well, since we are forced to define the goal in hindsight, if we have to define one)

Shouldn't the human intelligence part be considered part of the source code? With its own goals/value functions? Otherwise it will be just a human watchig the robot kind of thing.

Should we then draw different conclusions from their experiments?

I assume you mean if you only saw one of them (knowing the researcher's intentions ineither case)? In that case, I would say yes. For the first, the N is random, while for the second N is the smallest N were r>=60. In the second case, the question is: what is the probability that the cure rate will ever reach 60%, while the first case answers the Q: what is the cure rate probability accoding to a sample on N=100

Yes, I would say, draw very differenct conclusions since you ar answering very different questions!

When you described your possible experiment, you raised the probability that your theory is provable/disprovable, hence you raised the probability of it being rational.

Your desciption didn't raise the probability of your theory being correct, it raised the probability of it being a theory!

To me the biggest problem with Judaism, Christianity, and Islam is that they do not seem to provide one clear answer as to why God created the universe in the first place. Given this, I have no way of changing probabilities when the world seems cruel or contradicting, since they do not claim the world as perfect. This of course doesn't depend on what my prior is.

For evolution, I find a weakness (I am not an expert on the subject) that related to being able to explain all outcomes equally. If an animal feature seems in perfection with survival, this is due to evolution, if a feature isn't, this is a proof of no God, hence evolution. Shouldn't an imperfect human featuring a blind spot eventually get extinct? Not necessarily. What if it was extinct? Then it's evolution.

I find explaining by evolution is not disprovable, at least of the (seemingly infinite) millions and millions of years.