I think the point of the post is a bit more meta than "people should use rationality [...]". More like: "I am allowed to think that there is a right way for people to think".
I like your first question. Not having had much a chance to taboo my words, and not wanting to get lost in a maze of words, I would describe it thusly:
It is noticing a challenge. Then poking and prodding the challenge with tools until you have some ideas for how it works. Then using the ideas to poke and prod it in ways that will eliminate the wrong ideas. What remains should be ideas that you aren't sure about. You can compare those ideas with ideas you already held before the challenge, and the ones that disagree need to be tested some more. With enough effort, eventually you will have gotten rid of all the ideas that obviously don't pass your tests, and you will have a collection of ideas that you have tested that don't disagree with one another. Or ones that are clear on how they disagree, since there will always be open questions.
In the spirit of The 5-Second Level, I will name three concrete examples:
-Noticing my computer fails to wake up properly from hibernation, I note that I recently replaced my video cards, so I update video card drivers. That doesn't fix the problem, so I google it with words that I think will turn up the best results and find that some people had this problem after flashing the firmware on the video card, but that doing an RMA fixed their problem. I begin the process of RMAing, but then notice that the broader class of problems is associated with driver issues. I find a driver updater program that seems trustworthy, and find that I have 2 out-of-date drivers associated with a couple of my hard drives. I update them, and find that the computer no longer has the problem I complained about. I stop looking.
-There is a loud, distracting, unknown, short-lived beep at work, that promises to bother me all shift. I start a timer when a beep goes off and stop it when it repeats, and find that it goes off exactly every 5 minutes. I set my alarm to go off in 4 minutes, and go stand near where my best guess is at the proper time. The beep goes off, and I revise my best guess for beep source as a computer that I can't log into. I turn off the computer. There is another beep about 5 minutes later, so I set the timer, and guess again where it is. Eventually I find out there is a phone in a box with a new voicemail message.
-I'm arguing with a friend about science being the only way to know the world. He keeps drawing examples revolving around the fallibility of books and naive appeals to authority. I keep drawing examples about revolving around being right and being able to show why you are right. Eventually I stumbled upon this realization and dispelled the source of our disagreement by conceding that books aren't the only way to learn (that they aren't the best way in many cases), and that an illiterate welder who can point out where an engineer's blueprints are structurally unsound is using science, not proving that science is inferior to experience.
As you can tell, I see it as a process of weeding out the crap. If you can think of a better way to get rid of crap, I'm all ears. There's nothing sacred about the process. I'm not sure about calling it 'the arbiter of truth'. It seems as strange to me as if someone asked "Why is f(x) the arbiter for the value of sin(x)?" There are no gods here. No sentient beings that decide that the answer is right. In fact, I should stress that rationality does not guarantee that you end up at the right place, even if you did everything correctly.
Today's post, Your Rationality is My Business was originally published on April 15, 2007. A summary (from the LW wiki):
Discuss the post here (rather than in the comments of the original post).
This post is part of a series rerunning Eliezer Yudkowsky's old posts so those interested can (re-)read and discuss them. The previous post was New Improved Lottery, and you can use the sequence_reruns tag or rss feed to follow the rest of the series.
Sequence reruns are a community-driven effort. You can participate by re-reading the sequence post, discussing it, posting the next day's sequence reruns post, summarizing forthcoming articles on the wiki, or creating exercises. Go here for more details, or to discuss the Sequence Reruns.