This post is part of my Hazardous Guide To Rationality. I don't expect this to be new or exciting to frequent LW people, and I would super appreciate comments and feedback in light of intents for the sequence, as outlined in the above link. Also, note this is a STRUCTURE post, again see the above link for what that means.

Intro

Communication can be hard. Thinking can be hard. Trying to get another person to understand what you mean can be hard. Trying to explain to yourself what you mean can be hard. This sequence draws out in detail some of the things that happen in human brains that leads to these hardships, and proposes a few tactics to make it less hard.

Some Problems

When a tool becomes familiar enough, it can become forgotten. You forget a black box is a black box, and see it as a lever of reality.

What you meant is not what you said, but also, what you meant is not what you said.

I was expressing two distinct thoughts there. It wasn't very obvious. The first sentence meant, "You don't always say words that make other people think of the same things you're thinking about". The second meant, "What you mean is a different type of thing from what words you produce, and they should not be mistaken for each other".

What you meant is not what you said; the language you use is not the same thing as the cognition that produced the language. Apologies if you're getting bored.

In this sequence I'm going to make some claims about how we think and how language works, and then point to a bunch of pit-falls that can lead to confusion, never ending argument, and bad times. I believe these pitfalls have to do with inaccurate models of how language works and subtly forgetting that cognition and language are different.

(aside: what if after reading this sequence you think "Well, it's not really inaccurate models and subtly forgetting that is the cause of these problems, it's XYZ". Oh really? Why would it matter if it was the case?)

Cognition, Classification, and Inference

Your brain's job is to keep you alive. This takes a lot of guess work. Is this edible or poisons? Is that blur a tiger on its way to kill me? The usual. It sure would be useful if the brain could turn sensory input into the exact set of motor movements needed to not die. This is what the brain does. You take the outside world and make categories. Once categorized, you are use various "if... then..." rules that you previously associated with the category. If all goes well, you hear a rustle, categorize as predator, and activate "run!" This is not the only thing the brain does, but it is something the brain does.

These neural categories predates language, though they can be effected by language.

Some neural categories are loose, some are tight. Some categories immediately spawn action. The tightness, triggers, and consequences of a category are all learned through [opaque process]. Humans are basically the same considering how from more than nothing we share. Humans are nothing alike considering how far from exactly alike we are. No one will ever know your inner soul, yet you can still drive to work without being killed 100 times over in a car crash.

Remember, up to now, we have not talked about language, just the cognition of categorization. Later we talk about how language can effect how you categorize.

Sometimes, you goof. I hear a dog barking, think there's a dog around the corner, but it turns out it was just someone's ring tone. I think my new manager is going to be an ass-hole because of the face they didn't smile at me, turns out they just literally never smile and are super nice.

Pre Language seed of error [under construction]

I want to make a point about what an error in cognition feels like versus an error in modeling language.

*Ugna is relaxing under a tree when Targno appears over the hill being chased by *

[some dialoge about a caveman bringing up a shrieking pet with big teeth, Ugna say's it's "scary" Targno says it isn't]

Ugna is probably going to feel uneasy about not running. But something Ugna won't do is say, "Targno, scary by definition means something that makes me feel fear and activates my parasympathetic nervous system, this is absolutely scary!"

Ugna's mind has a quick categorization where loud shrieks get put in the "shit I should run away from". Ugna might aslo use large fangs, being huge, or moving really fast as triggers to put things in the "shit I should run away from" category. In their primitive cave language, "scary" is used to communicate "this is in the shit you should run away from category!"

For the very first time, Ugna has encountered something that shrieks loudly, has fangs, and yet isn't going to try and kill her.

It takes something special for Targno to think that Ugna is thinking about if the pet moves fast, as opposed to whether or not it can kill her.

They might still disagree, but they are disagreeing where it matters. They both really care about how to not die. There are a lot of arguments about what is scary that don't look like this.

Language

Language is more broad than spoken and written languages like English. Take sign language as an example. Despite this, I'll mostly talk about language in terms of words, which will be me thinking of things like "catch" and "derivative".

Words are not just a 1 to 1 mapping from sounds to categories in our heads. "the" and "is" are not categories. Language has it's own logic and structure to it. Verbs and nouns, subject agreement, something something I never payed attention in grade school nor took a linguistics class.

The point: cognition has it's own internal structure. Language has it's own internal structure. Language is used to try to poke in another persons brain and produce similar patterns of cognition as you did. Telepathy exists, it's just not what you expected.

This works well, a lot of the time. This fucks up a lot of the time. A deep see marine biologist has trouble talking to a mathematician because there are some concepts that just aren't in the other persons brain (how you build new concepts in a mind from old ones?). The French don't understand the Japanese because the symbols used are completely different (there are likely also conceptual differences between the cultures). A teacher has a hard time understanding kids, because even though they use the same symbol set, kids quickly make slang. Kids grew up differently, and thus a lot of words might relate to dif cognition in subtle or gross ways.

We are all so different. We are all so similar.

Bad Theories and How they got there

Nows the fun part. How do things break? Having a bad model of what language is what leads to arguing about the definition of "scary" instead of figuring out if this thing is going to kill us.

The child thinks that words have power. When I shout "hungry" the humans bring me food. You get confused when your dog doesn't stop when you tell it to. You lack theory of mind. Words not only mean things, words have power. There is no reflection.

The student thinks that words mean something. It may be the case that you don't know what they mean, but there is a right answer to this question. You know because your teacher gives you a bad score if you can't answer it "correctly".

The Greek thinks that words have meaning, and are the same thing as thoughts. There is reflection. All words and all thoughts correspond to perfectly cut logical categories. The world is syllogisms.

The poet learns that meaning is in people, not the words. A word means whatever you want it to mean, as it gets its meaning from how it's used. You see through the illusion.

The engineer learns that some words have real meanings. If they learn these meanings deeply enough, they can remake the world in their image. It is forbidden to use the words that do not mean something, and only talk about the ones that do.

The lawyer learns that once again, words have power. Not from magic, but from social contract. If you can get the right people to accept the right words, you can force other things that you want. This is all a game. Be careful not to say the wrong words when someone is listening, they might force you to make a move you don't like.

These names themselves do not form strict boundaries. In the following examples, I'll often refer to a mistake as "poet behavior" or "student behavior". That's mostly to help you have a richer web of mnemonic connections and hopefully remember this stuff more. This post is the framework, the rest will be worked examples and more subtle points. Examples of conversation, diagnostics of what went wrong, and lessons learned. Discrete skills will be pointed out, and there will be links to practice exercises to hammer in those skills. "I don't need to practice remembering that other people have different minds". Hmmm, maybe. Hard to say.

(aside: did you disagree with the names of the groups? Do you think I think something wrong? Do you not like that I unwittingly made you think of a concept you don't actually think I'm talking about? Do you think it was irresponsible to name the groups this way given common usage? The difference is interesting.)