So I've read 4 books out of 6 of RAZ, and in these reading group threads, to consolidate my understanding of philosophy behind rationality, I'll try to rewrite some of the ideas from scratch. I invite everyone to tear this apart. I'll be delighted to be corrected.

LessWrong's major focus is philosophy of language and how language relates to beliefs, truth, reality, and science. This sequence only scratches the surface of philosophy of language, while sequence A human's guide to words is a more serious treatment of the topic. It's a shame that the latter sequence appears so late in the book. In my opinion, it's much more important for ordinary people to understand how to use words and language meaningfully and productively, than to understand evolution, even if I would love everyone understand evolution too.

We can only meaningfully talk about things that are reducible to sense-impressions. Sense-impressions can be different — vision, hearing, taction, internal monologue of thoughts — but it's whatever we feel from the “inside”. We, human beings, are somehow able to memorize and recall, and analyze (i.e. break down into simpler parts), and find generalities in sense-impressions.

We see patterns in things we see, hear, touch, remember and think about. And we find out that some patterns go together with other patterns, and this allows us to predict the future. This allows us to omit details. We store succinct descriptions of things and patterns, with lossy compression, but somehow our prediction ability still remains high enough. E.g. I don't care how many black stripes or whiskers is there on this tiger, but if it matches my general compressed description of a tiger, I'd better run away quick.

Sense-impressions are the prism, through which we see the reality. There is a finite (and small) amount of information about the Universe that enters your sense organs. And even this information isn't reliable, because sense organs are complex machines that receive photons, audio waves, pressure etc. We can't see the world “directly” (whatever that would mean), we only infer information about the world from photons, audio waves, pressure.

Some of these inferences happen immediately and you don't even realize they are inferences. E.g. you see an apple on a table, so you immediately think “ah, there's an apple on the table”. Some inferences are multi-step and require conscious thinking, so they feel like inferences. E.g. you think about the physics laws you know and confidently conclude that the world is made of atoms, even though you've never seen or heard an atom.

Reality happens to be regular, and therefore information that we receive via sense organs happens to be regular too. But we can only infer reality from the information we get. So how do we “hug reality”, infer correct beliefs about reality? By making predictions. If our beliefs are correct, then we correctly anticipate a new sense-impression arriving into our consciousness. If our beliefs are incorrect, then something we do not anticipate will appear instead. If our beliefs are half-correct, then anticipations will confirm only to an extent they are correct.

So, we have patterns and these patterns enable us to predict the world. The existence of patterns allows us to talk succinctly about vast amounts of things, because we can omit details that do not influence predictions. This allows us to talk about increasingly complex subjects and also exchange information with other people via speech or writing. You just say the word “tiger” to your comrade, and they instantly know what you're talking about, even if you didn't specify which tiger, where, how many black stripes and whiskers.

Words are never meaningful by themselves, they are just sequences of letters or sounds. The world “tiger” written in English on a computer screen doesn't even resemble a tiger visually, it's just a pattern of black marks on white background, but our brain associates this word with the idea of a tiger. When it sees the word “tiger”, it knows that it ought to change anticipations in a certain way. When your comrade shouts “watch out, a tiger is behind you!”, somehow you end up anticipating a tiger behind you.

There are sense-impressions, there are anticipations of sense-impressions (predictions), there are patterns of these anticipations. Words, then, are labels you associate with these patterns. Themselves they are largely arbitrary. They don't have meaning “attached” to them. It's just if you and your comrade associate certain things with these words, you can exchange information.

Beliefs are only meaningful, when they yield predictions. Words are only meaningful, when they at some point ultimately reduce to sense-impressions. But from the “inside” meaningless and meaningful beliefs feel the same. So people constantly talk about things without realizing that they don't anticipate anything and don't refer to anything in reality.

This is what Making beliefs pay rent (in anticipated experiences) is all about.