-
The world is locked right now in a deadly puzzle, and needs something like a miracle of good thought if it is to have the survival odds one might wish the world to have.
-
Despite all priors and appearances, our little community (the "aspiring rationality" community; the "effective altruist" project; efforts to create an existential win; etc.) has a shot at seriously helping with this puzzle. This sounds like hubris, but it is at this point at least partially a matter of track record.[1]
-
To aid in solving this puzzle, we must probably find a way to think together, accumulatively. We need to think about technical problems in AI safety, but also about the full surrounding context -- everything to do with understanding what the heck kind of a place the world is, such that that kind of place may contain cheat codes and trap doors toward achieving an existential win. We probably also need to think about "ways of thinking" -- both the individual thinking skills, and the community conversational norms, that can cause our puzzle-solving to work better. [2]
-
One feature that is pretty helpful here, is if we somehow maintain a single "conversation", rather than a bunch of people separately having thoughts and sometimes taking inspiration from one another. By "a conversation", I mean a space where people can e.g. reply to one another; rely on shared jargon/shorthand/concepts; build on arguments that have been established in common as probably-valid; point out apparent errors and then have that pointing-out be actually taken into account or else replied-to).
-
One feature that really helps things be "a conversation" in this way, is if there is a single Schelling set of posts/etc. that people (in the relevant community/conversation) are supposed to read, and can be assumed to have read. Less Wrong used to be a such place; right now there is no such place; it seems to me highly desirable to form a new such place if we can.
-
We have lately ceased to have a "single conversation" in this way. Good content is still being produced across these communities, but there is no single locus of conversation, such that if you're in a gathering of e.g. five aspiring rationalists, you can take for granted that of course everyone has read posts such-and-such. There is no one place you can post to, where, if enough people upvote your writing, people will reliably read and respond (rather than ignore), and where others will call them out if they later post reasoning that ignores your evidence. Without such a locus, it is hard for conversation to build in the correct way. (And hard for it to turn into arguments and replies, rather than a series of non sequiturs.)
My problem with the "shared vocabulary" is that as you note yourself here, it implies that something has already been thought through, and it assumes that you have understood the thing properly. So for example if you reject an argument because "that's an example of a motte and bailey fallacy", then this only works if it is in fact correct to reject arguments for that reason.
And I don't think it is correct. One reason why people use a motte and bailey is that they are looking for some common ground with their interlocutor. Take one of Scott's examples, with this motte and bailey:
When the person asserts #1, it is not because they do not believe #2. It is because they are looking for some partial expression of their belief that the other person might accept. In their understanding, the two statements do not contradict one another, even though obviously the second claims a good deal more than the first.
Now Scott says that #1 is "useless," namely that even if he could theoretically accept the word "God" as applying to this, there is no reason for him to do this, because there is nowhere to go from there. And this might be true. But the fact that #2 is false does not prove that it is true. Most likely, if you work hard, you can find some #3, stronger than #1, but weaker than #2, which will also be defensible.
And it would be right to tell them to do the work that is needed. But it would be wrong to simply say, "Oh, that's a motte and bailey" and walk away.
This is not merely a criticism of this bit of shared vocabulary, so that it would just be a question of getting the right shared vocabulary. A similar criticism will apply to virtually any possible piece of shared vocabulary -- you are always assuming things just by using the vocabulary, and you might be wrong in those assumptions.
Making shared vocabulary common and explicit usually makes it faster to iterate. For example, the EA community converged on the idea of replaceability as an important heuristic for career decisions for a while, and then realized that they'd been putting too much emphasis there and explicitly toned it down. But the general concept had been floating around in discussion space already, giving it a name just made it easier to explicitly think about.