I found some of this hard to parse;
Just as the supernatural is a non-explanation of unexplained phenomena, superstition is a non-explanation of causal relationships; it is the recognition of them, without the attribution of cause.
not sure if it's the syllables; or the word length (or word use) or the switch between negatives and non-negatives; but this sentence was hard to read.
Try this:
Just as the idea of "supernatural" is not an explanation of mysterious phenomena, superstition is not an explanation of causal relationships; it is a recognition of a relationship; without trying to name the cause.
(not sure if it's better)
I enjoyed the read once I made sure to read it carefully a second time.
Adjusting! Thank you.
ETA: Corrected some related issues with poor phrasing, particularly implications I didn't intend to imply.
To answer your last question: The thoughts 'beautifully presented' and 'aesthetic' crossed my mind before reading your question.
Also: Another thought that crossed my mind was 'does he want to describe that rationality is kind of like human Thompson Sampling' (or more generally explore-exploit optimization)?
I have not heard of Thompson Sampling, or explore-exploit optimization. That it's a named phenomenon independent of what I considered to be rationality itself may be an issue; that's more or less explicitly my own strategy and regard for rationality, which means it may not be as generalizable as I anticipated, as I'm almost certainly engaging in typical mind fallacy without realizing it there.
The explore-exploit tradeoff is a fundamental thing in learning in complex environments (in AI this is studied in reinforcement learning). The way this often comes up for people is when ordering food (new restaurant / old favorite, favorite order / new order).
explore-exploit is no human strategy but a mathematical modelling of a specific optimization. Just in case that hasn't been clear. It is just that the specific type of rationality you described could be seen as analogous to that.
With a lot of carefully-worded caveats, yeah - I took out a section going into that in more detail because I decided it was distracting.
Outside the carefully-worded caveats, however, and to the point - treat it as an important and useful alief. You solve no problems as long as you're stuck in the "solution" that you're too stupid to solve them.
I like the vibes.
But worse, the path is not merely narrow, but winding, with frequent dead ends requiring frequent backtracking. If ever you think you're closer to the truth - discard that hubris, for it may inhibit you from leaving a dead end, and there your search for truth will end. That is the path of the crank.
I don't like this part. First, thinking that you're closER to the truth is not really a problem, it's thinking you've arrived at the truth that arguably is. Second, I think sometimes human beings can indeed find the truth. Underconfidence is just as much a sin as overconfidence, but referring to hubris in the way that you did seems like it would encourage false humility. I think you should say something more like "for every hundred ides professed to be indisputable truths, ninety nine are false", and maybe add something about how there's almost never good justification to refuse to even listen to other people's points of view.
The path of rationality is a path without destination.
I don't agree with this either, or most of the paragraph before it: there are strong trends.
The First Step
The first step on the path to truth is superstition. We all start there, and should acknowledge that we start there.
Superstition is, contrary to our immediate feelings about the word, the first stage of understanding. Superstition is the attribution of unrelated events to a common (generally unknown or unspecified) cause - it could be called pattern recognition. The "supernatural" component generally included in the definition is superfluous, because supernatural merely refers to that which isn't part of nature - which means reality -, which is an elaborate way of saying something whose relationship to nature is not yet understood, or else nonexistent. If we discovered that ghosts are real, and identified an explanation - overlapping entities in a many-worlds universe, say - they'd cease to be supernatural and merely be natural.
Just as the supernatural refers to unexplained or imaginary phenomena, superstition refers to unexplained or imaginary relationships, without the necessity of cause. If you designed an AI in a game which, after five rounds of being killed whenever it went into rooms with green-colored walls, started avoiding rooms with green-colored walls, you've developed a good AI. It is engaging in superstition, it has developed an incorrect understanding of the issue. But it hasn't gone down the wrong path - there is no wrong path in understanding, there is only the mistake of stopping. Superstition, like all belief, is only useful if you're willing to discard it.
The Next Step
Incorrect understanding is the first - and necessary - step to correct understanding. It is, indeed, every step towards correct understanding. Correct understanding is a path, not an achievement, and it is pursued, not by arriving at the correct conclusion in the first place, but by testing your ideas and discarding those which are incorrect.
No matter how much intelligent you are, you cannot skip the "incorrect understanding" step of knowledge, because that is every step of knowledge. You must come up with wrong ideas in order to get at the right ones - which will always be one step further. You must test your ideas. And again, the only mistake is stopping, in assuming that you have it right now.
Intelligence is never your bottleneck. The ability to think faster isn't necessarily the ability to arrive at the right answer faster, because the right answer requires many wrong ones, and more importantly, identifying which answers are indeed wrong, which is the slow part of the process.
Better answers are arrived at by the process of invalidating wrong answers.
The Winding Path
The process of becoming Less Wrong is the process of being, in the first place, wrong. It is the state of realizing that you're almost certainly incorrect about everything - but working on getting incrementally closer to an unachievable "correct". It is a state of anti-hubris, and requires a delicate balance between the idea that one can be closer to the truth, and the idea that one cannot actually achieve it.
The art of rationality is the art of walking this narrow path. If ever you think you have the truth - discard that hubris, for three steps from here you'll see it for superstition, and if you cannot see that, you cannot progress, and there your search for truth will end. That is the path of the faithful.
But worse, the path is not merely narrow, but winding, with frequent dead ends requiring frequent backtracking. If ever you think you're closer to the truth - discard that hubris, for it may inhibit you from leaving a dead end, and there your search for truth will end. That is the path of the crank.
The path of rationality is winding and directionless. It may head towards beauty, then towards ugliness; towards simplicity, then complexity. The correct direction isn't the aesthetic one; those who head towards beauty may create great art, but do not find truth. Those who head towards simplicity might open new mathematical doors and find great and useful things inside - but they don't find truth, either. Truth is its own path, found only by discarding what is wrong. It passes through simplicity, it passes through ugliness; it passes through complexity, and also beauty. It doesn't belong to any one of these things.
The path of rationality is a path without destination.
Written as an experiment in the aesthetic of Less Wrong. I'd appreciate feedback into the aesthetic interpretation of Less Wrong, rather than the sense of deep wisdom emanating from it (unless the deep wisdom damages the aesthetic).