In brainstorming, a common piece of advice is to let down your guard and just let the ideas flow without any filters or critical thinking, and then follow up with a review to select the best ones rationally. The concept here is that your brain has two distinct modes of operation, one for creativity and one for analysis, and that they don't always play well together, so by separating their activities you improve the quality of your results. My personal approach mirrors this to some degree: I rapidly alternate between these two modes, starting with a new idea, then finding a problem with it, then proposing a fix, then finding a new problem, etc. Mutation, selection, mutation, selection... Evolution, of a sort.

Understanding is an interaction between an internal world model and observable evidence. Every world model contains "behind the scenes" components which are not directly verifiable and which serve to explain the more superficial phenomena. This is a known requirement to be able to model a partially observable environment. The irrational beliefs of Descartes and Leibniz which you describe motivated them to search for minimally complex indirect explanations that were consistent with observation. The empiricists were distracted by an excessive focus on the directly verifiable surface phenomena. Both aspects, however, are important parts of understanding. Without intangible behind-the-scenes components, it is impossible to build a complete model. But without the empirical demand for evidence, you may end up modeling something that isn't really there. And the focus on minimal complexity as expressed by their search for "harmony" is another expression of Occam's razor, which serves to improve the ability of the model to generalize to new situations.

A lot of focus is given to the scientific method's demand for empirical evidence of a falsifiable hypothesis, but very little emphasis is placed on the act of coming up with those hypotheses in the first place. You won't find any suggestions in most presentations of the scientific method as to how to create new hypotheses, or how to identify which new hypotheses are worth pursuing. And yet this creative part of the cycle is every bit as vital as the evidence gathering. Creativity is so poorly understood compared to rationality, despite being one of the two pillars, verification with evidence being the other, of scientific and technological advancement. By searching for harmony in nature, they were engaging in a pattern-matching process, searching the hypothesis space for good candidates for scientific evaluation. They were supplying the fuel that runs the scientific method. With a surplus of fuel, you can go a long way even with a less efficient engine. You might even be able to fuel other engines, too.

I would love to see some meta-scientific research as to which variants of the scientific method are most effective. Perhaps an artificial, partially observable environment whose full state and function are known to the meta researchers could be presented to non-meta researchers, as objects of study, to determine which habits and methods are most effective for identifying the true nature of the artificial environment. This would be like measuring the effectiveness of machine learning algorithms on a set of benchmark problems, but with humans in place of the algorithms. (It would be great, too, if the social aspect of scientific research were included in the study, effectively treating the scientific community as a distributed learning algorithm.)

Am I correct to paraphrase you this way: maximizing EX and maximizing P(X > a) are two different problems.

I think it is more interesting to study how to be simultaneously supermotivated about your objectives and realistic about the obstacles. Probably requires some dark arts techniques (e.g. compartmentalization). Personally I find that occasional mental invocations of quasireligious imagery are useful.

In other words, laziness and overconfidence bias cancel each other out, and getting rid of the second without getting rid of the first will cause problems?

Isn’t that growth mindset? (Is growth mindset not rational?)

I wrote a post arguing that what is irrational overconfidence for an individual can be good for society. (In short, scientific knowledge is a public good, individual motivations to produce it is likely too low from a group perspective, and overconfidence increases individual motivation so it's good.)

To extend this a bit, if society pays people to produce scientific knowledge (in money and/or status), then overconfident people would be willing to accept a lower "salary" and outcompete more rational individuals for the available positions, so we should expect that most science is produced by overconfident people. (This also applies to any other attribute that increases motivation to work on scientific problems, like intellectual curiosity.) As a corollary, people who produce science about rationality (e.g., decision theorists) are probably more overconfident than average, people who work at MIRI are probably more overconfident than average, etc.

I think that very much depends on what you mean with rationality. The kind of rationality this community practices leads for better or worse to a bunch of people holding contrarian beliefs that certain things are possible that general society doesn't consider to be possible.

In HPMOR you have "do the impossible", "heroic responsibility" and "having something to protect" all as part of the curriculum.

As a data point to support this, overconfidence correlates positively with income.

Rationality and common sense might be bad for your chances of achieving something great, because you need to irrationally believe that it's possible at all.

That is true.

If you want to achieve something great, don't be a skeptic about it. Be utterly idealistic.

Well, umm... there is the slight issue of cost. If you are deliberately choosing a high-risk strategy to give yourself a chance of a huge payoff, you need to realize that the mode of outcomes is you failing. Convincing yourself that you are destined to become a famous actress does improve your chances of getting into the movies, but most people who believe this will end up as waitresses in LA.

It's like "If you want to become a millionaire, you need to buy lottery tickets" :-/

I don't know if I would hate on Hume this much. Hume is pretty big.

I agree with your broader point, though, I think. At the highest levels, EVERYTHING has to go right, including having the hardware, and having a super work ethic, and having a synergistic morale (an irrationally huge view of own importance, etc.)

There are a lot of ways to be irrational, and if enough people are being irrational in different ways, at least some of them are bound to pay off. Using your example, some of the people with blind idealism may get stuck to an idea that they can accomplish, but most of them fail. The point of trying to be rational isn't to do everything perfectly, but to systematically increasing your chances of succeeding, even though in some cases you might get unlucky.

Achieving something great may require your confidence in its possibility, but the reasonableness of that confidence is only discovered in hindsight. It's not uncommon to stumble upon a true belief while having begun with wrong evidence.

Well, or find a way to bottle mania.

To translate. Do not buy this LW-Yudkowsky AI mantra how hard, difficult and dangerous it is. Do it at home, for yourself, have no fear!