A few thoughts that have been brooding, that are vaguely relevant to your post...
One thing that I find is often disappointingly absent from LW discussions of epistemology is how much the appropriate epistemology depends on your goals and your intellectual abilities. If you are someone of median intelligence who just want to carry out a usual trade like making shoes or something, you can largely get by with recieved wisdom. If you are a researcher, your entire job consists of coming up with things that aren't already present in the market of ideas, and so using at least some local epistemology (or 'inside view', or 'figuring things out') is a job requirement. If you are trying to start a start-up, or generate any kind of invention, again, you usually have to claim to have some knowledge advantage, and so you need a more local epistemology.
Relatedly, even for any individual person, the kind of thinking I should use depends very much on context. Personally, in order to do research, I try to do a lot of my thinking by myself, in order to train myself to think well. Sure, I do engage in a lot of scholarship too, and I often check my answers through discussing my thinking with others. But I do a lot more independent thinking than I did two years ago anyway. But if I am ever making a truly important decision, such as who to work for, it makes sense for me to be much more deferential, and to seek advice of people who I know to be the best at making that decision, and then to defer to them to a fairly large degree (notwithstanding that they lack some information, which I should adjust for).
It would be nice to see people relax blanket pronouncements (not claiming this is particularly worse in this post compared to elsewhere) in order to give a bit more attention to this dependence on context.
RyanCarey writes:
If you are someone of median intelligence who just want to carry out a usual trade like making shoes or something, you can largely get by with recieved wisdom.
AFAICT, this only holds if you're in a stable sociopolitical/economic context -- and, more specifically still, the kind of stable sociopolitical environment that provides relatively benign information-sources. Examples of folks who didn't fall into this category: (a) folks living in eastern Europe in the late 1930's (especially if Jewish, but even if not; regardless of how traditional their trade was); (b) folks living in the Soviet Union (required navigating a non-explicit layer of recieved-from-underground knowledge); (c) folks literally making shoes during time-periods in which shoe-making was disrupted by the industrial revolution. It is to my mind an open question whether any significant portion of the US/Europe/etc. will fall into the "can get by largely with received wisdom" reference class across the next 10 years. (They might. I just actually can't tell.)
It's possible to be in a situation where the consensus is terrible and you have no ability to rebuild a better knowledge base. It's tragic but tragedies happen.
It seems like we're anchoring excessively on the question of sufficiency, when what matters is the net expected benefit. If we rephrase the question and ask "are there populations that are made worse off, on expectation, by more independent thought?", the answer is clearly yes, which is I think the question that we should be asking (and that fits the point I'm making).
In order to research existential risk, and to actually survive, yes, we need more thought, although this is the kind of research I had in mind in my original comment.
Independent thought helps not just for x-risk, but for personal well-being things like not joining the army, not playing the slot machines, getting off Facebook. In other words, not just for accepting action requests culturally construed as exotic, but rejecting some normal action requests. "Stable sociopolitical/economic context" is actually a fairly strong requirement, given how much the current global narrative is based on exponential growth.
The question is, what do you mean by "independent thought"? If "independent thought" means "mistrust everyone" then clearly it can be a harmful heuristic. If "independent thought" means "use your own thinking faculties to process all evidence you have, including the evidence which consists of the expression of certain beliefs by certain other people" then it's not clear to me there are significant populations that would be harmed by it. If there are, it means that the society in question has wise and benevolent leaders s.t. for large swaths of the population it is impossible to verify (given their cognitive abilities) that these leaders are indeed wise and benevolent. It seems to me that arriving at such a situation would require a lot of luck, given that the leadership of a society is ultimately determined by the society itself.
My view is that any world where the value of possible outcomes is heavy-tailed distributed (x-risk is a thing, but also more everyday things like income, and I'd guess life satisfaction) is a world where the best opportunities are nonobvious and better epistemology will thus have very strong returns.
I maybe am open to an argument that a 10th century peasant literally has no ability to have a better life, but I basically think that it holds for everyone I talk to day-to-day.
So you're saying rationality is good if your utility is linear in the quantity of some goods? (For most people it is more like logarithmic, right?) But it seems that you want to say that independent thought is usually useful...
I'm sure the 10th century peasant does have ways to have a better life, but they just don't necessarily involve doing rationality training, which is pretty obviously does not (and should not) help in all situations. Right?
Yes, it seems to me that we should care about some things linearly, though I’ll have to think some more about why I think that.
"One thing that I find is often disappointingly absent from LW discussions of epistemology is how much the appropriate epistemology depends on your goals and your intellectual abilities" - Never really thought of it that way, but makes a lot of sense.
I think it's worth asking: can we find a collective epistemology which would work well for the people who use it, even if everyone else behaved adversarially? (More generally: can we find a policy that works well for the people who follow it, even if others behave adversarially?)
The general problem is clearly impossible in some settings. For example, in the physical world, without secure property rights, an adversary can just shoot you and there's not much you can do. No policy is going to work well for the people who follow it in that setting, unless the other people play nice or the group of people following it is large enough to overpower those who don't.
In others settings it seems quite easy, e.g. if you want to make a series of binary decisions about who to trust, and you immediately get a statistical clue when someone defects, then even a small group can make good decisions regardless of what everyone else does.
For collective epistemology I think you can probably do well even assuming if 99% of people behave adversarially (including impersonating good faith actors until its convenient to defect---in this setting, "figure out who is rational, and then trust them" is a non-starter approach). And more generally, given secure property rights I think you could probably organize an entire rational economy in a way that is robust to large groups defecting strategically, yet still pretty efficient.
The adversarial setting is of course too pessimistic. But I think it's probably a good thing to work on anyway in cases where it looks possible, since (a) it's an easy setting to think about, (b) in some respects the real world is surprisingly adversarial, (c) robustness lets you relax about lots of things and is often pretty cheap.
I'm way more skeptical than you about maintaining a canonical perspective; I almost can't imagine how that would work well given real humans, and its advantages just don't seem that big compared to the basic unworkability.
It seems to me that part of the subtext here is that humans for the most part track a shared perspective, and can't help but default to it quite often, because (a) we want to communicate with other humans, and (b) it's expensive to track the map-territory distinction.
For instance, let's take the Syria example. Here are some facts that I think are tacitly assumed by just about everyone talking about the Syria question, without evaluating whether there is sufficient evidence to believe them, simply because they are in the canonical perspective:
The level of skepticism that would not default to the canonical perspective on facts like that seems - well, I don't know of anyone who seems to have actually internalized that level of skepticism of canon, aside from the President of the United States. He seems to have done pretty well for himself, if he in fact exists.
Robust collective epistemology need not look like "normal epistemology but really skeptical." Treating knowledge as provisional and tentative doesn't require a high level of skepticism. It may involve some revision to the default way humans think, but that ship had already sailed well before the enlightenment.
It seems reasonable to believe X "merely" because it is falsifiable, no one credible objects, and you've personally seen no evidence to the contrary. That protocol probably won't lead you astray, but for most interesting claims it is going to be easy for an adversary to DoS it (since even if the claim is true someone could object without compromising their own credibility) and so you are going to need to rely on more robust fallbacks.
My point isn't that you should doubt that sort of stuff strongly, it's that it seems to me to be prohibitively computationally expensive to evaluate it at all rather than passively accepting it as background observations presumed true. How, in practice, does one treat that sort of knowledge as provisional and tentative?
My best guess is that someone with the right level of doubt in social reality ends up looking like they have a substantially higher than normal level of psychosis, and ends up finding it difficult to track when they're being weird.
How, in practice, does one treat that sort of knowledge as provisional and tentative?
A belief being tentative is a property of your algorithm-for-deciding-things, not what a state of mind feels like from the inside. You can get a lot of mileage by e.g. (a) independently revisiting tentative claims with small probability, (b) responding appropriately when someone points out to you that a load-bearing tentative assumption might be wrong.
I don't think this question should be expected to have a really short answer, even if there are ironclad collective epistemology protocols. It's like saying "how, in practice, do people securely communicate over untrusted internet infrastructure?" There is a great answer, but even once you have a hint that it's possible it will still take quite a lot of work to figure out exactly how the protocol works.
Do we actually have a disagreement here? I'm saying that actually-existing humans can't actually do this. You seem to be saying that it's conceivable that future humans might develop a protocol for doing this, and it's worth exploring.
These can both be true! But in the meantime we'd need to explore this with our actually-existing minds, not the ones we might like to have, so it's worth figuring out what the heck we're actually doing.
I agree that it would take some work to figure out how to do this well.
I would say "figure out how to do this well" is at a similar level of complexity to "figure out what the heck we're actually doing." The "what should we do" question is more likely to have a clean and actionable answer. The "what do we do" question is more relevant to understanding the world now at the object level.
I'm way more skeptical than you about maintaining a canonical perspective
Which part of the post are you refering to?
I read Jessica as being pretty down on the canonical perspective in this post. As in this part:
Since then, aesthetic intuitions have led me to instead think of the problem of collective epistemology as one of decentralized coordination: how can good-faith actors reason and act well as a collective superintelligence in conditions of fog of war, where deception is prevalent and creation of common knowledge is difficult? I find this framing of collective epistemology more beautiful than the idea of a immediately deferring to a canonical perspective, and it is a better fit for the real world.
I'm on board with the paragraph you quoted.
I'm objecting to:
Try to make information and models common knowledge among a group when possible, so they can be integrated into a canonical perspective. This allows the group to build on this, rather than having to re-derive or re-state it repeatedly.
and:
This can result in a question being definitively settled, which is great for the group's ability to reliably get the right answer to the question, rather than having a range of "acceptable" answers that will be chosen from based on factors other than accuracy.
For example, in the physical world, without secure property rights, an adversary can just shoot you and there's not much you can do. No policy is going to work well for the people who follow it in that setting, unless the other people play nice or the group of people following it is large enough to overpower those who don't.
There are actual processes by which people have managed to transition from a state in which they were not protected against violence, to a state in which they were protected against some violence, or were able to coordinate via high-trust networks within a broader low-trust context. The formation of states and other polities is a real thing that happens, and to write off the relevant coordination problem as impossibly hard seems bizarre, so perhaps I'm misunderstanding you. It has historically been fairly popular to attribute this sort of transition to gods, but that seems more like reifying our confusion than actually alleviating it.
The "everyone else behaves adversarially" assumption is very pessimistic. Lots of problems are solvable in practice but insoluble if there is an adversarial majority.
Collective epistemology may be such a case. But I suspect that it's actually much easier to build robust collective epistemology than to build a robust society, in the sense that it can be done under much weaker assumptions.
Hmm. Are you assuming e.g. that we don't know who's part of a kinship group etc., so there's no a priori way of inferring who's likely to have aligned interests? That seems like an interesting case to model, but it's worth noting that the modern era is historically unusual in resembling it.
I'm just asking: can you design a system that works well, for the people who adopt it, without assuming anything about everyone else? So in particular: without making assumptions about aligned interests, about rationality, about kindness, etc.
When this harder problem is solvable, I think it buys you a lot. (For example, it may spare you from having to go literally form a new society.)
It seems that in very adversarial settings there is limited incentive to share information. Also resources used for generating object level information might have to be deployed trying to figure out the goals and of any adversarial behavior. Adversarial behavior directed at you might not be against your goals. For example an allied spy might lie to you to maintain their cover.
For these reasons I am skeptical of a productive group epistemology.
I meant to propose the goal: find a strategy such that the set H of people who use it do "well" (maybe: "have beliefs almost as accurate as if they had known each others' identities, pooled their info honestly, and built a common narrative") regardless of what others do.
There aren't really any incentives in the setup. People in H won't behave adversarially towards you, since they are using the proposed collective epistemology. You might as well assume that people outside of H will behave adversarially to you, since a system that is supposed to work under no assumptions about their behavior must work when they behave adversarially (and conversely, if it works when they behave adversarially then it works no matter what).
I think the group will struggle with accurate models of things where data is scarce. This scarcity might be due to separation in time or space between the members and the phenomenon being discussed. Or could be due to the data being dispersed.
This fits with physics and chemistry being more productive than things like economics or studying the future. In these kinds of fields narratives that serve certain group members can take hold and be very hard to dislodge.
Economics and futurism are hard domains for epistemology in general, but I'm not sure that they'd become disproportionately harder in the presence of disinformation.
I think the hard cases are when people have access to lots of local information that is hard for others to verify. In futurism and economics people are using logical facts and publicly verifiable observations to an unusual extent, so in that sense I'd expect trust to be unusually unimportant.
I was just thinking that we would be able to do better than being keynesian or believing in the singularity if we could aggregate information from everyone reliably.
If we could form a shared narrative and get reliable updates from chip manufacturers about the future of semiconductors we could make better predictions about the pace of computational improvement. Than if you assume they will be saying things with half an eye on their share price.
There might be an asymptote you can reach on doing "well" under these mildly adversarial settings. I think knowing the incentives of people helps a lot so can know when people are incentivised deceive you.
Are you assuming you can identify people in H reliably?
If you can identify people in H reliably then you would ignore everyone outside of H. The whole point of the game is that you can't tell who is who.
So what you can do is.
That said, as I don't believe in a sudden switch to utopia, I think it important to strengthen the less-adversarial parts of society, so I will be seeking those out. "Start as you mean to go on," seems like decent wisdom, in this day and age.
This isn't strong enough. The more people tend to defer to a canonical perspective, the more one can control others' actions through altering the canonical perspective.
Although, less so, to the extant that this advice...
When taking actions (e.g. making bets), use local information available only to you or a small number of others, not only canonical information. For example, when picking organizations to support, use information you have about these organizations (e.g. information about the competence of people working at this charity) even if not everyone else has this info. (For a more obvious example to illustrate the principle: if I saw Bob steal my wallet, then it's in my interest to guard my possessions more closely around Bob than I otherwise would, even if I can't convince everyone that Bob stole my wallet).
...is heeded.
Things I liked:
So I've promoted it to Featured.
I really like your promotion of fact checking :-)
Also, I'd like to especially thank you for offering the frame where every human group is potentially struggling to coordinate on collective punishment decisions from within a fog of war.
I had never explicitly noticed that people won't want their pursuit of justice to seem like unjustified aggression to "allies from a different bubble of fog", and for this reason might want to avoid certain updates in their public actions.
Like, I even had the concept of altriustic punishment and I had the concept of a fog of war, but somehow they never occured in my brain at the same time before this. Thank you!
If I was going to add a point of advice, it would be to think about being part of two or three "epistemic affinity groups". The affinity group model suggests these groups should be composed of maybe 3 to 15 people each and they should be built around a history of previous prolonged social contact. When the fog of war hits, reach out to at least one of your affinity groups!
I don't think Jessica's actually making that conflation, though:
[C]ollective superintelligences made of humans have another difficulty: they must prevent and detect disinformation. People on the internet sometimes lie, as do people off the internet. Self-deception is effectively another form of deception, and is extremely common [...]
In the past, I have found modest epistemology aesthetically appealing on the basis that sufficient coordination would lead to a single canonical perspective that you can increase your average accuracy by deferring to [...]. Since then, aesthetic intuitions have led me to instead think of the problem of collective epistemology as one of decentralized coordination: how can good-faith actors reason and act well as a collective superintelligence in conditions of fog of war, where deception is prevalent and creation of common knowledge is difficult? I find this framing of collective epistemology more beautiful than the idea of a immediately deferring to a canonical perspective, and it is a better fit for the real world.
The piece explicitly distinguishes between canonical perspective and truth, and explicitly argues that a strong reason to avoid deferring to the canonical perspective is that it is frequently made of lies.
I think the reason why this seems like a plausibly good idea instead of an obviously terrible one is that we conflate thinking with acting. You can't learn to think for yourself properly without repeatedly trying, which will necessarily involve thinking for yourself improperly. But, in the meantime, you should expect to be wrong about lots of things and should probably conform to the customs of some people with a track record of not totally failing.
Relevant: Be secretly wrong
both of these depend heavily on the subject in question.
Indeed. The answer to modesty versus contrarianism is a list of conditions concerning which to use in which situation.
In Inadequacy and Modesty, Eliezer describes modest epistemology:
I see modest epistemology as attempting to defer to a canonical perspective: a way of making judgments that is a Schelling point for coordination. In this case, the Bank of Japan has more claim to canonicity than Eliezer does regarding claims about Japan's economy. I think deferring to a canonical perspective is key to how modest epistemology functions and why people find it appealing.
In social groups such as effective altruism, canonicity is useful when it allows for better coordination. If everyone can agree that charity X is the best charity, then it is possible to punish those who do not donate to charity X. This is similar to law: if a legal court makes a judgment that is not overturned, that judgment must be obeyed by anyone who does not want to be punished. Similarly, in discourse, it is often useful to punish crackpots by requiring deference to a canonical scientific judgment.
It is natural that deferring to a canonical perspective would be psychologically appealing, since it offers a low likelihood of being punished for deviating while allowing deviants to be punished, creating a sense of unity and certainty.
An obstacle to canonical perspectives is that epistemology requires using local information. Suppose I saw Bob steal my wallet. I have information about whether he actually stole my wallet (namely, my observation of the theft) that no one else has. If I tell others that Bob stole my wallet, they might or might not believe me depending on how much they trust me, as there is some chance I am lying to them. Constructing a more canonical perspective (e.g. a in a court of law) requires integrating this local information: for example, I might tell the judge that Bob stole my wallet, and my friends might vouch for my character.
If humanity formed a collective superintelligence that integrated local information into a canonical perspective at the speed of light using sensible rules (e.g. something similar to Bayesianism), then there would be little need to exploit local information except to transmit it to this collective superintelligence. Obviously, this hasn't happened yet. Collective superintelligences made of humans must transmit information at the speed of human communication rather than the speed of light.
In addition to limits on communication speed, collective superintelligences made of humans have another difficulty: they must prevent and detect disinformation. People on the internet sometimes lie, as do people off the internet. Self-deception is effectively another form of deception, and is extremely common as explained in The Elephant in the Brain.
Mostly because of this, current collective superintelligences leave much to be desired. As Jordan Greenhall writes in this post:
We are in a collective condition of fog of war. Acting effectively under fog of war requires exploiting local information before it has been integrated into a canonical perspective. In military contexts, units must make decisions before contacting a central base using information and models only available to them. Syrians must decide whether to flee based on their own observations, observations of those they trust, and trustworthy local media. Americans making voting decisions based on Syria must decide which media sources they trust most, or actually visit Syria to gain additional info.
While I have mostly discussed differences in information between people, there are also differences in reasoning ability and willingness to use reason. Most people most of the time aren’t even modeling things for themselves, but are instead parroting socially acceptable opinions. The products of reasoning could perhaps be considered as a form of logical information and treated similar to other information.
In the past, I have found modest epistemology aesthetically appealing on the basis that sufficient coordination would lead to a single canonical perspective that you can increase your average accuracy by deferring to (as explained in this post). Since then, aesthetic intuitions have led me to instead think of the problem of collective epistemology as one of decentralized coordination: how can good-faith actors reason and act well as a collective superintelligence in conditions of fog of war, where deception is prevalent and creation of common knowledge is difficult? I find this framing of collective epistemology more beautiful than the idea of a immediately deferring to a canonical perspective, and it is a better fit for the real world.
I haven't completely thought through the implications of this framing (that would be impossible), but so far my thinking has suggested a number of heuristics for group epistemology: