LESSWRONG
LW

Community
Frontpage

16

What do you think is wrong with rationalist culture?
New Answer
New Comment

10 Answers sorted by
top scoring

MSRayne

2324

I don't know much about the community beyond what's evident on LessWrong, but I've often felt like there's an undercurrent here of people tending towards a certain degree of selfishness (moral irrealism plus consequentialism plus "rationality is about winning" together makes a somewhat machiavellian personality lots of nice excuses), as well as messiah complexes which are not only somewhat destructive to the mental health of those that have them but also feed into that ego pattern even more (we're saving the world! only smart rationalists can understand! no point in trying to talk about alignment with normies because they're useless and can't help! the entire burden of saving the world is on my shoulders!!).

In general... this may be a place to go for good reasoning, but not for sanity in a more absolute sense. The emotional and social intelligence here, and indeed to some extent the "moral intelligence" is... not always adequate.

[-]TropicalFruit62

I've also noticed those tendencies, not in the community but in myself.

Selfishness. Classification of people as "normies." Mental health instability. Machiavellianism.

But...

They get stronger as I look at the world like a rationalist. You read books like Elephant in the Brain and find yourself staring at a truth you don't want to see. I wish God were real. I wish I were still a Christian with those guardrails erected to prevent me from seeing the true nature of the world.

But the more I look, the more like it looks like a non-moral, brutally unfair, unforgiv... (read more)

Reply
[-]the gears to ascension145

good news on the moral front: prosocial moral intuitions are in fact a winning strategy long term. we're in a bit of a mess short term. but, solidarity and co-protection are good strategies; radical transparency can be an extremely effective move; mutual aid has always been a factor of evolution; the best real life game theory strategies tend to look like reputational generous tit for tat with semirandom forgiveness, eg in evolutionary game theory simulations; etc. Moral realism is probably true but extremely hard to compute. If we had a successful co-protective natural language program, it would likely be verifiably true and look like well known moral advice structured in a clear and readable presentation with its mathematical consequences visualized for all to understand.

I really like https://microsolidarity.cc as an everyday life intro to this, and I tossed this comment into metaphor.systems up to the opening bracket of this link. here are some very interesting results, various middle to high quality manifestos and quick overviews of ethics:

  • https://ncase.me/trust/ one of the best intros to morality as trustbuilding, with game theory visualizations; the only serious contender f
... (read more)
Reply
5MSRayne
You always post such cool links!!! I bet you're a cool person. :)
3MSRayne
There is a middle path. insert buddha vibes In fact, I'm a moral realist! And I've got the skeleton of a rationalist argument for it. Only the skeleton, mind, and I'm sure people could easily blow holes in it. But making posts on here is... exhausting... so I haven't written it up. Well, yes, we live in a hell ruled by a dead (never-born) god. That's why it's our responsibility to create a living one (an aligned sovereign ASI) and liberate all sentient beings from suffering. That's what you ought to be living for.
[-]tailcalled30

Do you have any favorite examples of this problem?

Reply
[-]MSRayne109

To be blunt... our founder's entire personality. (And current extreme burnout and evident depression.)

Also, I will not name names, but I know of at least one person who over DMs mentioned rendering their meat eating consistent with their other moral views by deciding that any entities without the cognition of a fully self-aware human have no moral rights, and was strongly considering whether it would be ethically acceptable to eat children and mentally disabled people. I found this disturbing enough to block them.

That's not quite an example of the specific things I mentioned, but it is an example of the rationality subculture tending to veer away from what I suppose has to be called "common sense" or "consensus reality". (Acausal reasoning and anthropics both also seem like examples of this. However "rational" they are, they are dangerous ideas that imo pose a cognitohazard.)

Actually, in the interests of full honesty, I have to give myself as an example. I didn't know about the rationalist community until I was like 20, but throughout my teens I basically was a rationalist without knowing it - and also mentally ill and coping with emotional disturbances using a lot of narcissistic ... (read more)

Reply
7Adam Zerner
Yeah, I hear ya. I think what's going on here is a difficult problem that I've been meaning to think and/or write about for some time: the balance between Taking Ideas Seriously and maintaining some sort of Memetic Immune System. It's a difficult thing to navigate. I think the person was wrong about eating children and probably leaned too strongly towards the Taking Ideas Seriously side of things - ie. should have a stronger anchor to a sort of "common sense prior" - but personally, I'm totally fine with that as long as 1) they are doing it in good faith and come from a place of wanting to figure out the truth, and 2) they aren't actually causing harm in the real world... ie by eating children. As a whole I feel like the rationalist community does a solid job navigating the tradeoff. If anything I actually think people don't lean hard enough towards Taking Ideas Seriously. For example, not many people are signed up for cryonics.
3Vladimir_Nesov
It's a false tradeoff, weird ideas taken seriously can be siloed in frames and hypotheticals. They only get to become beliefs (at any level of credence) or decision relevant if they grow up to make global sense. In particular, ITT asks to give voice to simulacra of all sorts of strange.
1Noosphere89
Re morality, there's a potentially non-trivial chance that things are even worse than you think, in that two or more people disagreeing about morality or valence by starting with different assumptions can't ever converge, even in the infinite limit of time and compute, because morality is not pointing to any reality, but is instead social realistiy. I support MSRayne blocking the person since in that scenario, there would be no convergence at all.
0Jiro
https://www.lesswrong.com/posts/QePFiEKZ4R2KnxMkW/posts-i-repent-of
-1Daniel
There are reasonable and coherent forms of moral skepticism in which the statement, "It is morally wrong to eat children and mentally disabled people," is false or at least meaningless. The disgust reaction upon hearing the idea of eating children is better explained by the statement, "I don't want to live in a society where children are eaten," which is much more well-grounded in physical reality. What is disturbing about the example is that this seems to be a person who believes that objective morality exists, but that it wouldn't entail that eating children is wrong. This is indeed a red flag that something in the argument has gone seriously wrong.
3MSRayne
My problem is more the lack of moral realism to begin with. I apparently need to work on a post about this. I am sick and tired of the lack of belief in objective morality around here, leading people to entertain such insane thoughts to begin with; needs some pushback.
3Noosphere89
This is a crux for me, and I conjecture roughly the opposite, that is moral/valence disagreements can't converge to any truth, even in the infinite limit. That is, there are no guarantees for moral reasoning converging to the truth, the way that say a bounded or unbounded Solomonoff inductor can in universes that have a simplicity bias. In other words, the assumption behind your morality are functionally arbitrary, and there's no deep justification for why you have the values you have. Note: No one should change their values immediately based on this comment.
6MSRayne
Um... evolution by natural selection? A very very short sketch: 1. most superintelligences likely to exist in the multiverse were created by civilizations of social organisms; 2. civilizations of social organisms tend to have moral systems rooted in generalizations of basic social instincts which worked in the ancestral environment, such as tit for tat defaulting to cooperation, and possibly geometric rationality; 3. some of those superintelligences are aligned and thus have value systems similar to those that tend to be evolved by civilizations of social organisms; 4. most are likely unaligned, but since unaligned superintelligences can have nearly any arbitrary utility function, those ones likely "cancel out"; 5. thus from an acausal trade standpoint, there is likely some one utility function to which the outcomes of trades between superintelligences across the multiverse tend, rooted in the most likely (according to how biological and memetic evolution by natural selection works) value systems arrived at by civilizations of social organisms prior to their local singularities, together with lots of small (because of mutually canceling out) wisps of interest in other random things from all the unaligned ASIs in the mix. 6. our own ASI, aligned or not, will (if it believes in multiverses and acausal things) probably notice this, run simulations to determine the most likely trajectories of such civilizations, and then align itself partly to the utility function of the multiverse meta-civilization in trade. That is: the existence of these facts results in a cosmic truth about what the correct utility function actually is, which can be determined by reasoning and approximated by getting more evidence, and which all sufficiently intelligent agents will converge on - which is to say, moral realism.
7Noosphere89
Now I get to the crux of why I disagree, and I note you've smuggled in the assumption that the multiverse constrains morality enough such that it's sensible to talk about one moral truth or one true utility functions. I think no multiverse that we actually live in constrains morality enough such that the conclusion of moral realism is correct, and that's why I disagree with the idea of moral realism. Similarly, this means that acausal economies will essentially be random chaos with local bubbles of moral systems, and that the aligned and unaligned systems have equal weight in the multiverse economy, that is infinite weight. And they all cancel each other out. Also, once we get to the stage that we join the acausal economy, there's no reason to make an all encompassing economy across the entire multiverse, so there's no reason for any acausal economies to form at all. Specifically for alignment, the goal and maybe definition of alignment is essentially making the AI do what someone wants. Critically, the only constraint is that the AI must either have the same goals as the person having the AI, or it has different goals but those goals aren't an impediment to the operator's goals. Note under this definition of alignment, it doesn't comstrain the morality enough to make moral realism right, even after adding in instrumental goals. Some notes on Geometric Rationality: I think there are some very useful notions from the geometric rationality sequence, like Thompson Sampling being better for exploration than it's equivalent in arithmetic rationality as well as techniques to reduce the force of Pascal's mugging, as he shows how exploration in the arithmetic rationality doesn't converge to the truth with probability of 1, while a geometric rationality technique known as Thompson Sampling does know the truth asymptotically with probability 1. However, arithmetic rationality does have some properties that are better than geometric rationality, such as being invariant to
6MSRayne
The great thing is, this is ultimately an empirical question! Once we make an aligned ASI, we can run lots of simulations (carefully, to avoid inflicting suffering on innocent beings - philosophical zombie simulacra will likely be enough for this purpose) to get a sense of what the actual distribution of utility functions among ASIs in the multiverse might be like. "Moral science"...
2Noosphere89
I definitely want to say that there's reason to believe at least some portions of the disagreement are testable, though I want curb enthusiasm by saying that we probably can't resolve the disagreement in general, unless we can somehow either make a new universe with different physical constants or modify the physical constants of our universe. Also, I suspect the condition below makes it significantly harder or flat out impossible to run experiments like this, at least without confounding the results and thereby making the experiment worthless.
[-]M. Y. Zuo-10

I call it the moderately-above-average syndrome. 

Someone with Einstein smarts or Napoleon level wiles, and with delusions of grandeur, seem to get along fine, at least judging by history.

But folks that are only 2 or 3 standard deviations above average, and who maintain similar pretences, inevitably come out a bit unbalanced.

There's also a similar concept in sociology with the anxious upper-middle classes.

Reply
2Adam Zerner
Sounds like something sorta similar to the Midwit meme.
1M. Y. Zuo
2 to 3 standard deviations above average would be on the right side of the standard bell curve diagram.  Those actually in the middle usually don't develop genuine delusions of grandeur because they would encounter, and compete with, many people moderately smarter than them in day-to-day life.  Whereas the moderately above average rarely genuinely interact, let alone compete, with modern day Einsteins.

romeostevensit

189

Low scholarship (not mainly the academic kind) due to lack of slack from prioritizing the wrong winning metrics (money and status over time). In general, an optimization frame often falls into the trap of fine tuning existing considerations instead of seeking new considerations.

[-]interstice81

What other kind of scholarship do you have in mind?

Reply
[-]Adam Zerner70

Let me attempt to paraphrase.

There are two problems: 1) not spending enough time on scholarship and 2) not having enough slack. These two problems are separate in the sense that 2 would be a problem even if 1 was solved and vice versa, but related in the sense that 2 is a big reason why 1 is a problem in the first place. And maybe 3) is another problem: that we spend too much time on existing considerations instead of seeking new considerations (exploiting instead of exploring).

Does that sound accurate?

If so, not that this adds much to the conversation, bu... (read more)

Reply
2romeostevensit
Sounds like a reasonable take. I recognize there are issues with looking at a three sigma outlier and wishing for a four sigma outlier.
[-]the gears to ascension40

What sorts of queries on which knowledge retrievers would you suggest for learning more about this from the perspective you're seeing as lacking? if it's useful for answering this, my favorite search engines are arxivxplorer, semanticscholar's recommender, metaphor, and I also sometimes ask claude or chatgpt to describe a concept to help me identify search terms. using that set of tools, what would you suggest looking up to find links I can provide to others as an intro to scholarship? I have plenty of my own ideas for what to look up, to be clear.

Reply
8romeostevensit
I also use connected papers and search citation lineages. Linked resources seem good too. For scholarship I think Richard Hamming's final two chapters in art of doing science and engineering are hard to beat.
[-]TekhneMakre40

Important + underappreciated.

Reply
[-]Sune10

What are you comparing to? It is only compared to what you would want rationalist culture to be like, or do you have examples of other cultures (besides academia) that do better in this regard?

Reply
2romeostevensit
I see small subcultures at good research schools that do well, but admit that what I'm looking for has very free examples, implying fragility.

lc

166

The only thing chess club members have to do to participate is to organize or play in chess matches. The only thing computer security club members have to do to participate is (usually) to help organize or play computer hacking challenges. The only thing you have to do to get involved in the Christian Community is to go to church and maybe attend a few church functions.

AFAICT, the only obvious way to participate in and contribute to rationalist culture is to write insightful posts on LessWrong, in the same way that the only way to get involved with the SCPWiki is to write SCPs. But the bar for doing that in a prosocial and truthful way is now pretty high, and was always going to effect a power law, with a few very intelligent founding members contributing most of the canon. It's not that they're doing anything wrong (I love their content), it's just naturally what happens.

Most of the problems I see on LessWrong lie downstream of this. Regular non-Google, non-finance software engineers face this dilemma of either staying silent and never getting to interact with the community, saying something that's been said before, indulging in one of their biases, or unfairly criticizing existing works and members. For very unconscientious people this means completely throwing away existing guardrails and deontology because that's the only way they can think to differentiate themselves from Eliezer and carve a niche.

[-]Garrett Baker911

I was able to get involved in rationality by going to in-person meetups. I suggest, if you're feeling left-out, you do the same (or create in-person meetups yourself!).

Edit: There also exist various rationalist discords you could join. They're usually fun, and don't require you to make a post.

Reply
8lc
Oh, don't be like that. It's more like: I like talking to rationalists. The only time I do so is when I make posts and comments. So I feel a noticable urge to come up with mediocre posts and comments when I'd rather just have some regular community function to attend. I'll probably try to do a meetup soon now that I'm in LA.
6TropicalFruit
I feel the same way. I like talking with people on here, but in almost every subject I have nothing substantive to contribute; I'm just a consumer. I wish there were a broader, reddit-style aspect to this site for more ordinary posts. They don't have to be about Kim Kardashian or anything, but just regular economics, the current bank runs, Bitcoin, lifestyle/fitness/nutrition stuff, interesting links. You know, minus the reddit toxicity and religious zealotry in every subreddit. Maybe I'm wrong. Maybe having the majority of the sub dedicated to AI alignment really is the way to go. It's just... I'm not smart enough, nor do I have the resources to meaningfully help on that front, and I suspect there are many like me in the IQ 130-145 range who absolutely love finally finding a community they can relate to, but don't have the 160+ IQ to really break ground on alignment research. Unless I'm selling us regular geniuses short, but I don't think I am (sadly).
3Angela Pretorius
That’s what the r/slatestarcodex subreddit is for.
1MSRayne
You know, you can contribute to alignment without contributing to alignment. Focus on the places you're shocked everyone else is dropping the ball. "Hey wait, why so little emphasis on aligning the humans that make AI? Wouldn't getting people to just slow the hell down and stop racing toward oblivion be helpful?" is one example of this, that would use an entirely different skillset (PR, social skills, etc) to work on. In my own case, I'm mainly interested in designing a system enabling mass human coordination and factored cognition, though I'm terrible at actually writing anything about the mountain of ideas in my head. This would indirectly speed up alignment by helping researchers think clearly, and also be great in many other ways. Think outside the "AI alignment directly and nothing else" box, and find something you can work on, with your skillset.
[-]Adam Zerner30

Regular non-Google, non-finance software engineers face this dilemma of either staying silent and never getting to interact with the community, saying something that's been said before, indulging in one of their biases, or unfairly criticizing existing works and members.

I'm glad you point this out. I think it is both real and important. However, I don't think it has to be that way! It's always been sort of a pet peeve of mine. "Normal" people can participate in so many ways. Here is what comes to my mind right now but definitely isn't exhaustive:

... (read more)
Reply

Dagon

10-14

I think it's wrong to think there is a "rationalist culture".  There are rationalist influences and tropes that are part of a number of distinct groups' habits and norms, but that doesn't make those groups similar enough to be called a cohesive single culture.

[-]Adam Zerner36

Disagreed, but curious.

My sense is that the differences are relatively minor and that there are a lot of really strong things that tie all the groups together: various things discussed in The Sequences like Bayesian thinking and cognitive science. What are the large differences you see with various groups?

Reply
2Dagon
There is a bit of far-mode general agreement on Bayesian thinking and the like. In terms of culture and near-mode norms, it’s those agreements which are relatively minor. Behaviorally, especially offline, is where the large differences lie.
2Adam Zerner
Hm, I think there's about 10 rationalists I've met IRL and spent more than a few hours with. Everyone has their own personality of course, but I didn't feel like there were any differences too large. From what I read on LW I get the sense that the Berkeley community might have some large differences - maybe even sub-groups in Berkeley with large differences - but I don't recall hearing about any other communities with large differences. If other communities did exist with large differences I'm not actually sure that I'd expect to hear about it via browsing LW though.

the gears to ascension

102

I'm not sure I know what rationalist culture refers to anymore. Several candidate referents have become blurred and new candidates have been introduced. Could be, lesswrong.com culture; humanity's rationalist cultures of various stripes; the rationalist cultures descended from lesswrong (but those are many at this point); the sequences view; the friend networks I have (which mostly don't have the problems I'd complain about, since I filter my friends for people I want to be friends with!); the agi safety research field (which seems to be mostly not people who think of themselves as "rationalists" anymore); berkeley rat crowd; "rationalist-adjacent" people on twitter; the thing postrats say is rationalist; a particular set of discords; some other particular set of discords; scott alexander fans; some vague combination of things I've mentioned; people who like secular solstice...

straw vulcan is more accurate than people give it credit for. a lot of people around these parts undervalue academia's output and independent scholarship and reinvent a lot of stuff. folks tend to have an overly reductive view of politics, either overly "only individuals exist and cannot be aggregated" or "only the greater good exists, individual needs not shared by others don't exist" - you know, uh, one of the main dimensions of variation that people in general are confused on. I dunno, it seems like the main thing wrong with rationalist culture is that it thinks of itself as rationalist, when in fact it's "just" another science-focused culture. shrug.

[-]tailcalled50

Do you have any favorite examples of the straw vulcan thing?

Reply
3the gears to ascension
I don't have any examples ready at hand. It tends to be a pattern I see in people who strike me as somehow new to the concept of "rationalism", people who just read the sequences and are excited to tell everyone about how they're ingroup now. I dunno. wait, maybe I could cite this comment by tropicalfruit as having the kind of vibe I'm thinking of: "oh no, are emotions truly useless? is morality fake?" - an understandable question, but still! https://www.lesswrong.com/posts/z4Rp6oBtYceZm7Q8s/what-do-you-think-is-wrong-with-rationalist-culture?commentId=HkKjobkvT6sfvRnG2 I get the sense that the rationalist vibe involves downregulating brain networks that implement important decision theory, because of not having an explicit description available that explains why those systems are key to the human {genome+memeplex}'s approximate learned decision theory
[-]Bill Benzon31

a lot of people around these parts undervalue academia's output and independent scholarship and reinvent a lot of stuff.

That's certainly my impression. I've been peeking in here off and on for several years, but became more active last June when I started (cross-)posting here and commenting a bit.

I have a PhD that's traditional in the sense that I learned to search, read, value, and cite the existing literature on a topic I'm working on. That seems to be missing here, leading, yes, to unnecessary reinvention. I recall reading a post several months ago that... (read more)

Reply
3the gears to ascension
I'd love to hear your thoughts on how to compress the training that one gets beginning and throughout a phd about how to learn effectively from ongoing research. Many folks on here either don't have time or don't think we have time to go to school, so it would be nice to get resources together about how to learn it quickly. I've also been asking AIs questions like this, and I share the good ones when they come up.
1Bill Benzon
That's a tough one, in part because the fields vary so much. I was in an English department, so that's what my degree is in. But my real training came as part of a research group in computational linguistics that was in the linguistics department. I didn't actually do any programming. I worked on knowledge representation, a big part of old-school computational linguistics (before NLP). But there are two aspects to this. One is getting the level of intellectual maturity and sophistication you need to function as a disciplined independent thinker. The other is knowing the literature. In some ways they interact and support one another but in some ways they are orthogonal. I learned the most when I found a mentor, the late David G.l Hays. I wanted to learn his approach to semantics. He tutored me for an hour or two once a week for a semester. That's the best. It's also relatively rare to get that kind of individual attention. Still, finding a mentor is the best possible thing you could do. At the same time I had a job preparing abstracts for The American Journal of Computational Linguistics. That meant I had to read a wide variety of material and prepare abstracts four times a year. You need to learn to extract the gist of an article without reading the whole thing. Look at the introduction and conclusion. Does that tell you what you need? Scan the rest. You should be able to do that – scan the article and write the abstract – in no more than an hour or two. Note, that many/most abstracts that come with an article are not very good. The idea is, if you trust the journal and the author and don't need the details, the abstract should tell you all you need. Working up that skill is good discipline. If you're working on a project with others here, each of you agree to produce 3, 4, 5 abstracts a week to contribute to the project. Post them to a place where you can all get at them. It becomes your project library. As for the level of intellectual maturity, the only way

Shmi

80

I mentioned it in the post https://www.lesswrong.com/posts/p2Qq4WWQnEokgjimy/respect-chesterton-schelling-fences : people are too eager to reject the Chesterton-Schelling fences because they feel enlightened and above the mundane guardrails that are for "normies".

AβMale

76

I wrote about this here:

[T]his error strikes me as … emblematic of a certain common failure mode within the rationalist community (of which I count myself a part). This common failure mode is to over-value our own intelligence and under-value institutional knowledge (whether from the scientific community or the Amazon marketplace), and thus not feel the need to tread carefully when the two come into conflict.

In that comment and the resulting thread, we discuss the implications of that with respect to the rationalist community’s understanding of Alzheimer’s disease, a disease I’ve studied in great depth. I’ve mostly found the community to have very strong opinions on that subject and disdain for the scientific community studying it, but very superficial engagement with the relevant scientific literature. Every single time I’ve debated the matter in detail with someone (maybe 5–10 times total), I’ve persuaded them that 1) the scientific community has a much better understanding of the disease than they realized and 2) that the amyloid hypothesis is compelling as a causal explanation. However, people in the rationalist community often have strongly-held, wrong opinions before (or in lieu of) these debates with me.

Ironically, the same thing happened in that thread: my interlocutor, John Wentworth, appreciated my corrections. However, I ultimately found the discussion a bit unsatisfying, because I don’t know that he made any meta-updates from it concerning the level of confidence that he started with without having seriously engaged with the literature.

[-]Noosphere8920

Basically, this is essentially reframing the overuse of the inside view and under using the outside view, and I think this struck truer to my objection than my answer did.

And yeah, John Wentworth ignored the literature and was wrong, and since John Wentworth admitted it was cherry picked, this is non-trivial evidence against the thesis that Goodhart is a serious problem for AI or humans.

Though it also calls into question how well John Wentworth's epistemic processes are working.

Reply

Marta

30

Limited number of groups and community events outside of the US/London (I'm from CEE, there are some groups, but not that many). It limits the possibility of in-person interaction. So, in a long-term, LW can only be my "online" community, not a "real life group of friends". Currently I regard EA events as a best way to meet rationalists, and, to be frank, it would be cool to have also other option and separate those two. 

[-]Viliam20

Wanted to write the same thing. In my country, if you organize a meetup for all rationalists and all ACX readers and all effective altruists together... the total number of participants may come close to ten, if you are lucky!

Reply

Noosphere89

33

I'm going to focus on the overuse of the inside view, and the relative disuse of base rates and outside view. And it's why I think Eliezer's views on AI doom are probably not rational, and instead the product of a depression spiral, to quote John Maxwell.

On base rates of predictions of extinction, the obvious answer is that no extinction events happened out of 172 predicted ones, and while that's not enough of a sample to draw strong conclusions, it does imply that very high confidence in doom by a specific date is not very rational, unless you believe that you have something special that changes this factor.

Link is below:

https://en.m.wikipedia.org/wiki/List_of_dates_predicted_for_apocalyptic_events

The issue is that LWers generally assume that certain things are entirely new every time and that everything is special, and I think this assumption is overused in both LW and the broader world, which probably leads to the problem of overvaluing your own special inside view compared to others outside views.

[This comment is no longer endorsed by its author]
[-]J Thomas Moros10-1

This is not sound reasoning because of selection bias. If any of those predictions had been correct, you would not be here to see it. Thus, you cannot use their failure as evidence.

Reply
8Jay Bailey
I notice I'm a bit confused about that. Let's say the only thing I know about the sun is "That bright yellow thing that provides heat", and "The sun is really really old", so I have no knowledge about how the sun mechanistically does what it does. I want to know "How likely is the sun to explode in the next hour" because I've got a meeting to go to and it sure would be inconvenient for the sun to explode before I got there. My reasoning is "Well, the sun hasn't exploded for billions of years, so it's not about to explode in the next hour, with very high probability." Is this reasoning wrong? If so, what should my probability be? And how do I differentiate between "The sun will explode in the next hour" and "The sun will explode in the next year"?
0RHollerith
Yes, IMO the reasoning is wrong: if you you definitely cannot survive an event, then observing that the event did not happened is not evidence at all that it will not explode in the future -- and it continues to not be evidence as long as you continue to observe the non-explosion. Since we can survive at least for a little while the sudden complete darkening of the sun the sun's not having gone dark is evidence that it will not go dark in the future, but it is less strong evidence than it would be if we could survive the darkening of the sun indefinitely. The law of the conservation of expected evidence requires us to take selection effects like those into account -- and the law is a simple consequence of the axioms of probability, so to cast doubt on it is casting doubt on the validity of the whole idea of probability (in which case, Cox's theorems would like to have a word with you). This is not settled science: there is not widespread agreement among scholars or on this site on this point, but its counter-intuitiveness is not by itself a strong reason to disbelieve it because there are parts of settled science that are as counterintuitive as this is: for example, the twin paradox of special relativity and "particle identity in quantum physics". When you believe that the probability of a revolution in the US is low because the US government is 230 or so years old and hasn't had a revolution yet, you are doing statistical reasoning. In contrast, noticing that if the sun exploded violently enough, we would immediately all die and consequently we would not be having this conversation -- that is causal reasoning. Judea Pearl makes this distinction in the intro to his book Causality. Taking into account selection effects is using causal reasoning (your knowledge of the causal structure of reality) to modify a conclusion of statistical reasoning. You can still become confident that the sun will explode soon if you have a refined-enough causal model of the sun.
[-]TropicalFruit1010

Off topic, but I'd just like to say this "good/bad comment" vs "I agree/disagree" voting distinction is amazing.

It allows us to separate our feeling on the content of the comment from our feeling on the appropriateness of the comment in the discussion. We can vote to disagree with a post without insulting the user for posting it. On reddit, this is sorely lacking, and it's one (of many) reasons every sub is an unproductive circle jerk.

I upvoted both of your comments, while also voting to disagree. Thanks for posting them. What a great innovation to stimulate discussion.

Reply
7Jay Bailey
So, I notice that still doesn't answer the actual question of what my probability should actually be. To make things simple, let's assume that, if the sun exploded, I would die instantly. In practice it would have to take at least eight minutes, but as a simplifying assumption, let's assume it's instantaneous. In the absence of relevant evidence, it seems to me like Laplace's Law of Succession would say the odds of the sun exploding in the next hour is 1/2. But I could also make that argument to say the odds of the sun exploding in the next year is also 1/2, which is nonsensical. So...what's my actual probability, here, if I know nothing about how the sun works except that it has not yet exploded, the sun is very old (which shouldn't matter, if I understand you correctly) and that if it exploded, we would all die?
2RHollerith
We don't need to consider that here because any evidence of the explosion would also take at least eight minutes to arrive, so there is approximately zero minutes during which you are able to observe the evidence of the explosion before you are converted into a plasma that has no ability to update on anything. That is when observational selection effects are at their strongest: namely, when you are vanishingly unlikely to be in one of those intervals between your having observed an event and that event's destroying your ability to maintain any kind of mental model of reality. We 21st-century types have so much causal information about reality that I have been unable during this reply to imagine any circumstance where I would resort to Laplace's law of succession to estimate any probability in anger where observational selection effects also need to be considered. It's not that I doubt the validity of the law; its just that I have been unable to imagine a situation in which the causal information I have about an "event" does not trump the statistical information I have about how many times the event has been observed to occur in the past and I also have enough causal information to entertain real doubts about my ability to survive if the event goes the wrong way while remaining confident in my survival if the event goes the right way. Certainly we can imagine ourselves in the situation of the physicists of the 1800s who had no solid guess as to the energy source keeping the sun shining steadily. But even they had the analogy with fire. (The emissions spectra of the sun and of fire are both I believe well approximated as blackbody radiation and the 1800s had prisms and consequently at least primitive spectrographs.) A fire doesn't explode unless you suddenly give it fuel -- and not any fuel will do: adding logs to a fire will not cause an explosion, but adding enough gasoline will. "Where would the fuel come from that would cause the sun to explode?" the 1800s can a
7TropicalFruit
Counterpoint: I'm at a local convenient store. A thief routinely robs me. He points a gun at me, threatens me, but never shoots, even when I push back a little. At this point, it's kind of like we both know what's happening, even though, technically, there's a chance of physical danger. Had this guy shot me, I wouldn't be alive to reason about his next visit. Now consider a different thief comes in, also armed. What is my probability of getting shot, as compared with the first thief? Much, much, higher with the second thief. My past experiences with the first thief act as evidence towards the update that I'm less likely to be shot. With this new thief, I don't have that evidence, so my probability of being shot is just the based rate based on my read of the situation. I believe updating on the non-fatal encounters with the first thief is correct, and it seems to me analogous to updating on the sun not having exploded. Thoughts?
2RHollerith
Because a person has a significant chance of surviving a bullet wound -- or more relevantly, of surviving an assault with a gun -- your not having been assaulted by the first thief is evidence that you will not be assaulted in future encounters with him, but it is weaker evidence than it would be if you could be certain of your ability to survive (and your ability to retain your rationality skills and memories after) every encounter with him. Humans are very good at reading the "motivational states" of the other people in the room with them. If for example the thief's eyes are glassy and he looks like he is staring at something far away even though you know it is unlikely there there is anything of interest in his visual field far away, well that is a sign he is in a dissociated state, which makes it more likely he'll do something unpredictable and maybe violent. If when he looks at you he seems to look right through you, that is a sign of a coldness that also makes it more likely he will be violent if he can thereby benefit himself personally by doing so. So, what is actually doing most of the work of lowering your probability about the danger to you posed the the first thief? The mere fact that you escaped all the previous encounters without having been assaulted or your observations of his body language, tone of voice and other details that give clues about his personality and his mental state?
1TropicalFruit
Replace thief with a black box that either explodes and kills you, or doesn't. It has some chance to kill you, but you don't know what that chance is. I was put in a room with black-box-one 5 times. Each time it didn't explode. Now, I have a choice: I can go back in the room with black-box-one, or I can go to a room with black-box-two. I'll take black-box-one, based on prior evidence.
2RHollerith
If I know nothing about the boxes except that they have the same a priori probability of exploding and killing me, then I am indifferent between the two black boxes. It is not terribly difficult to craft counter-intuitive examples of the principle. I anticipated I would be presented with such examples (because this is not my first time discussing this topic), which is why in my original comment I wrote, "its counter-intuitiveness is not by itself a strong reason to disbelieve it," and the rest of that paragraph.
7TropicalFruit
Okay but I just don't agree.  Let each black box have some probability to kill you, uniformly chosen from a set of possible probabilities. Let's start with a simple one: that probability is 0 or 1. The a prior chance to kill you is .5.  After the box doesn't kill you, you update, and now the chance is 0. What about if we use a uniform distribution from [0,1)? Some boxes are .3 to kill you, others .78. Far more of the experiences of not dying are from the low p-kill boxes than from the high p-kill ones. When people select the same box, instead of a new one, after not being killed, that brings the average kill rate of selected boxes down. Run this experiment for long enough, and the only boxes still being selected are the extremely low p-kill boxes that haven't killed all their subjects yet. This time, could you make a stronger objection, that's more directly addressed at my counter-example?
2RHollerith
TropicalFruit and I have taken this discussion private (in order to avoid flooding this comment section with discussion on a point only very distantly related to the OP.) However if you have any interest in the discussion, ask one of us for a copy. (We have both agreed to provide a copy to whoever asks.)
1Noosphere89
I would like a copy of the discussion.
2RHollerith
In your new scenario, if I understand correctly, you have postulated that one box always explodes and one never explodes; I must undergo 2 experiences: the first experience is with one of the boxes, picked at random; then I get to choose whether my second experience is with the same box or whether it is with the other box. But I don't need to know the outcome of the first experience to know that I want to limit my exposure to just one of these dangerous boxes: I will always choose to undergo the second experience with the same box as I underwent the first one with. Note that I arrived at this choice without doing the thing that I have been warning people not to do, namely, to update on observation X when I know it would have been impossible for me to survive (or more precisely for my rationality, my ability to have and to refine a model of reality, to survive) the observation not X. That takes care of the first of your two new scenarios. In your second new scenario, I have a .5 chance of dying during my first experience. Then I may choose whether my second experience is with the same box or a new one. Before I make my choice, I would dearly love to experiment with either box in a setting in which I could survive the box's exploding. But by your postulate as I understand it, that is not possible, so I am indifferent about which box I have my second experience with: either way I choose, my probability that I will die during the second experience is .5. Note the in your previous comment, in which there was some P such each time a box is used, it has a probability P of exploding, there is no benefit to my being able to experiment with a box in a setting in which I could survive an explosion, but in the scenario we are considering now there is a huge benefit. Suppose my best friend is observing the scenario from a safe distance: he can see what is happening, but is protected from any exploding box. My surviving the first experience changes his probability that the box
1Bill Benzon
It seems to me there is a distinction to be made: It is one thing to conclude that, 1) Eliezer doesn't know how to predict the date of AI Doom. That's different from asserting that 2) AI Doom is not going to happen. 1 is not evidence for 2.
[-]TropicalFruit10

I think it's appropriate to draw some better lines through concept space for apocalyptic predictions, when determining a base rate, than just "here's an apocalyptic prediction and a date." They aren't all created equal.

Herbert W Armstrong is on this list 4 times... each time with a new incorrect prediction. So you're counting this guy who took 4 guesses, all wrong, as 4 independent samples on which we should form a base rate.

And by using this guy in the base rate, you're implying Eliezer's prediction is in the same general class as Armstrong's, which is a ... (read more)

Reply

Victor Novikov

11

In my paradigm, human minds are made of something I call "microcognitive elements", which are the "worker ants" or "worker bees" of the mind.
They are "primed"/tasked with certain high-level ideas and concepts, and try to "massage"/lubricate the mental gears into both using these concepts effectively (action/cognition) and to interpret things in terms of these concepts (perception)
The "differential" that is applied by microcognitive elements to make your models work, is not necessarily related to those models and may in fact be opposed to them (compensating for, or ignoring, the ways these models don't fit with the world)

Rationality is not necessarily about truth. Rationality is a "cognitive program" for the microcognitive elements. Some parts of the program may be "functionally"/"strategically"/"deliberately" framing things in deceptive ways, in order to have the program work better (for the kind of people it works for).

The specific disagreements I have with the "rationalist" culture:

  • The implied statement that LessWrong paradigm has a monopoly on "rationality", and is "rationality", rather than an attempted implementation of "rationality", a set of cognitive strategies based on certain models and assumptions of how human minds work. If "rationality is about winning", then anyone who is winning is being rational, whether they hold LW-approved beliefs or not.
  • Almost complete disregard for meta-rationality.
  • Denial of nebulosity, fixation on the "imaginary objects" that are the output of the lossy operation of "make things precise so they can be talked about in precise terms".

All of these things have computational reasons, and are a part of the cognitive trade-offs the LW memeplex/hive-mind makes due to its "cognitive specialization". Nevertheless, I believe they are "wrong", in the sense that they lead to you having an incorrect map/model of reality, while strategically deceiving yourself into believing that you do have a correct model of reality. I also believe they are part of the reason we are currently losing - you are being rational, but you are not being rational enough. 
Our current trajectory does not result in a winning outcome.

[-]TropicalFruit21

Since reading the sequences, I've made much more accurate predictions about the world. 

Both the guiding principle of making beliefs pay rent in anticipated experience, as well as the tools by which to acquire those accurate beliefs, have worked for me.

So at an object level, I disagree with your claim. Also, if you're going to introduce topics like "meta-rationality" and "nebulosity" as part of your disagreement, you kind of have to defend them. You can't just link a word salad and expect people to engage. The first thing I'm looking for is a quick, one or two paragraph summary of the idea so I can decide whether it's worth it to pursue further.

Reply
5 comments, sorted by
top scoring
Click to highlight new comments since:
[-]Raemon1311

(fyi, downvoted because while I think there's a good version of this question, the current one feels too vague to be about anything particularly good, and most version of this discussion seem more likely to be vaguely-self-flagellating or reverse-circle-jerky rather than useful)

Reply
[-]Adam Zerner42

I'm not sure that vagueness is a problem here. It could be useful to hear from people with various takes on what exactly the question is asking.

I do worry a little about the framing leading to contentiousness though and think the question would be improved by somehow trying to mitigate that.

Reply
[-]tailcalled31

Yeah I was mainly trying to invite a broad set of opinions.

Reply
[-]iceman51

Meta-note related to the question: asking this question here, now, means you're answer will be filtered for people who stuck around with capital r Rationality and the current LessWrong denizens, not the historical ones who have left the community. But I think that most of the interesting answers you'd get are from people who aren't here at all or rarely engage with the site due to the cultural changes over the last decade.

Reply
[-]tailcalled40

Yeah, I've been reading a lot of critiques by Benjamin Hoffman and thinking about some of the prior critiques by Jessica Taylor, and that's sort of what prompted me to ask this question. It would probably also be interesting to look at others who left it, they're just harder to get hold of.

Reply
Moderation Log
A
105