As I understand the post, its idea is that a rationalist should never "start with a bottom line and then fill out the arguments".

Ooh! I see a "should" statement. Let's open it up and see what's inside!

\gets out the consequentialist box-cutter**

... its idea is that we will get worse consequences if we "start with a bottom line and then fill out the arguments."

Hmm. Is that what "The Bottom Line" says?

Let's take a look at what it says about some actual consequences:

If your car makes metallic squealing noises when you brake, and you aren't willing to face up to the financial cost of getting your brakes replaced, you can decide to look for reasons why your car might not need fixing. But the actual percentage of you that survive in Everett branches or Tegmark worlds—which we will take to describe your effectiveness as a rationalist—is determined by the algorithm that decided which conclusion you would seek arguments for. In this case, the real algorithm is "Never repair anything expensive." If this is a good algorithm, fine; if this is a bad algorithm, oh well.

In other words, it's not like you're sinning or cheating at the rationality game if you write the bottom line first.

Rather, you ran some algorithm to generate that bottom line. You selected that bottom line out of hypothesis-space somehow. Perhaps you used the availability heuristic. Perhaps you used some ugh fields. Perhaps you used physics. Perhaps you used Tristan Tzara's cut-up technique. Or a Ouija board. Or following whatever your car's drivers' manual said to do. Or your mom's caring advice.

Well, how good was that algorithm?

Do people who use that algorithm tend to get good consequences, or not?

Once your bottom line is written — once you have made the decision whether or not to fix your brakes — the consequences you experience don't depend on any clever arguments you made up to justify that decision retrospectively.

If you come up with a candidate "bottom line" and then explore arguments for and against it, and sometimes end up rejecting it, then it wasn't really a bottom line — your algorithm hadn't actually terminated. We can then ask, still, how good is your algorithm, including the exploring and maybe-rejecting? This is where questions about motivated stopping and continuation come in.

It sounds neat, but I think it is not psychologically feasible.

I agree with this. At least in a lot of contexts, you pick a side, write your bottom line, and then see how it goes. I think being rational, for people who think like me anyway, means being open to needing to toggle one's views frequently, in case you're wrong.

A little while ago, I needed to decide something that was important, complex, and it was unlikely, unfortunately, that events would ever give feedback one way or the other, as to whether I made the 'right' choice, due to the complexity and the fact there was risk and incompletely disclosed values involved.

Nevertheless, the decision was really important and the decision was weighing on me. I observed over a couple days how I made the decision. First,I chose one of the two decisions, rather randomly, just because I needed to choose already. And then I imagined defending it. "Owning" the decision and feeling responsible for it was an important step in the motivation to come up with the best arguments I couold. I imagined my family, my coworkers and my friends second-guessing me, and I kept arguing and arguing with them in my head. After a few hours of dedicated mental role play, I felt exhausted. This decision was way to difficult to defend. It would be easier, I thought, to just go the other way. So I (more half-heartedly) considered the other decision, and imagined defending that one. It turned out to be easier to defend, and some of the arguments struck me as especially compelling, so that in the end my mock-trial arguments convinced me. I felt (and feel) good about my decision, different then the one I began with.

It struck me, at the time, how similar this was to the court system. Certainly, they knew what they were doing with that. It also struck me that imagining the counter-arguments of friends and family was somewhat less effective because it tended to direct me towards making the decision that felt least 'shameful' rather than the one that would be most likely to 'win'. But due to the convolution of the problem with values, this seemed necessary.

Sometimes the decision is easier. I can just look at the facts and decide. But I don't know how often this is the case with decisions that are both really important and somewhat 'messy', as they tend to be in real life when they involve people and values and not just, for example, optimizing something with respect to a single factor.

How often do you find that your intuitive conclusion had been faulty? What do you do in this case with your intuition?

It's a good point: having a target in mind, and then searching for the best arguments for it, is a familiar experience.

The "bottom line" might be a math theorem I'm trying to prove. Or a claim in a discussion with friends that I'm trying to argue for. Or the hope that I can solve a programming problem within certain constraints. In all these cases, my thinking is directed to a goal. And I can't currently imagine (which is weak evidence) a form of thinking that isn't, yet produces good results reliably.

The rationalist ideal isn't "don't argue towards a goal". It's "always remember that until you have a rock-solid argument, the goal is desired but unproven, and to be treated as an assumption at best. Always be ready to discard or modify the goal, as evidence comes in. And when you do have a proof, have it checked by counter-motivated others."

The Bottom Line doesn't tell you to argue without a goal. It just tells you to weight all evidence correctly, for and against your goal. It may help if your goal is explicitly formulated as a question rather than a statement which might be false. But in the end, it's just about fighting biases - remembering what we've assumed without proof to explore its consequences if true, and so on.

From that post, a description of wrong arguing:

First, he writes, "And therefore, box B contains the diamond!" at the bottom of his sheet of paper. Then, at the top of the paper, he writes, "Box B shows a blue stamp," [.... and so on]; yet the clever arguer neglects all those signs which might argue in favor of box A.

(my emphasis). And a description of right arguing:

she first writes down all the distinguishing signs of both boxes on a sheet of paper, and then applies her knowledge and the laws of probability and writes down at the bottom: "Therefore, I estimate an 85% probability that box B contains the diamond."

One difference is indeed that the second arguer did not write a bottom line before writing out all the evidence. But that's misleading. The second arguer already knew that she was arguing about which box contained the diamond. She had mentally written down at the bottom "and therefore box B contains the diamond with ___ probability", and filled in the actual probability later.

Bayes-wise that is identical to writing, first, "box B contains the diamond with 50% probability" (your prior); and then modifying that number as each piece of evidence is considered. The real difference is that you must not ignore or mistreat part of the evidence, as the first arguer did.

And if you trust your lack of biases enough, you may even write down as the bottom line, "box B contains the diamond with 78% probability" (or whatever your preconceived belief is). And then, as you evaluate the evidence, instead of modifying that number, you append to the end: "I currently give this proposition such and such a probability (belief) distribution." And you modify that last statement as the evidence comes in.

The whole issue may be summarized as follows:

1. We must update on all evidence fairly, including evidence which causes us to lower the probability estimate of hypotheses we hold dear, which is a human bias to privilege.
2. To avoid this bias, it can help not to mentally give any idea such a privileged status to begin with.

Do not hold ideas dear. Hold reality dear.

Of course that should not be taken to mean you shouldn't have goals that you try to argue for! Just remember that your attempts to argue can fail, and update accordingly.

I understand the idea of the "bottom line" post a little distinctly. In my understanding it doesn't address the process of arguing (i.e. constructing verbal expressions capable of persuading others). Building an effective argument needs knowing the goal as a prerequisite, obviously. But the situation is different in the private, deciding what to believe. Quite commonly one selects one's belief on a totally inadequate basis (affect heuristics, political sympathies) and then reinforces this belief by arguments constructed with the belief in mind. This is what the post was warning against.

In the analogy with mathematical proofs: if a mathematician is reasonably certain that a theorem holds, he can go and try to find a proof. The proof is an argument presented to the public (here, other mathematicians) and should be clear, elegant and polished. But before that the mathematician must decide which theorem he should try to prove, and it would be a mistake to skip this phase, just formulate a "random" theorem and directly jump to the phase of constructing a proof. In mathematics it would be hard to succeed this way since to decide whether a proof is correct or not is relatively easy and straightforward, but outside mathematics the bottom-line approach is usually feasible and costly.

Your division of reasoning into three steps (guessing conclusion, justifying it, checking the justification) may be inevitable for small irreducible ideas where you can go through the whole process in few minutes. But most arguments are about complex hypotheses whose justification could be (and usually is) reduced to a chain of elementary inductive steps. For such hypotheses it is certainly feasible (psychologically or otherwise) to arrive at them gradually - guessing and rationalising the irreducible bits which can be easily checked, but not the hypothesis as a whole.

Related: A Rational Argument.