Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

FormallyknownasRoko comments on Best career models for doing research? - Less Wrong

27 Post author: Kaj_Sotala 07 December 2010 04:25PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (999)

You are viewing a single comment's thread. Show more comments above.

Comment author: FormallyknownasRoko 10 December 2010 05:06:28PM *  4 points [-]

Look, you have three people all of whom think it is a bad idea to spread this. All are smart. Two initially thought it was OK to spread it.

Furthermore, I would add that I wish I had never learned about any of these ideas. In fact, I wish I had never come across the initial link on the internet that caused me to think about transhumanism and thereby about the singularity; I wish very strongly that my mind had never come across the tools to inflict such large amounts of potential self-harm with such small durations of inattention, uncautiousness and/or stupidity, even if it is all premultiplied by a small probability. (not a very small one, mind you. More like 1/500 type numbers here)

If this is not enough warning to make you stop wanting to know more, then you deserve what you get.

Comment author: Desrtopa 10 December 2010 05:20:22PM 4 points [-]

Considering the extraordinary appeal that forbidden knowledge has even for the average person, let alone the exceptionally intellectually curious, I don't think this is a very effective way to warn a person off of seeking out the idea in question. Far from deserving what they get, such a person is behaving in a completely ordinary manner, to exceptionally severe consequence.

Personally, I don't want to know about the idea (at least not if it's impossible without causing myself significant psychological distress to no benefit,) but I've also put significant effort into training myself out of responses such as automatically clicking links to shock sites that say "Don't click this link!"

Comment author: Vaniver 10 December 2010 05:23:22PM 8 points [-]

Look, you have three people all of whom think it is a bad idea to spread this. All are smart. Two initially thought it was OK to spread it.

I see a lot more than three people here, most of whom are smart, and most of them think that Langford basilisks are fictional, and even if they aren't, censoring them is the wrong thing to do. You can't quarantine the internet, and so putting up warning signs makes more people fall into the pit.

Comment author: katydee 10 December 2010 06:09:43PM *  3 points [-]

I saw the original idea and the discussion around it, but I was (fortunately) under stress at the time and initially dismissed it as so implausible as to be unworthy of serious consideration. Given the reactions to it by Eliezer, Alicorn, and Roko, who seem very intelligent and know more about this topic than I do, I'm not so sure. I do know enough to say that, if the idea is something that should be taken seriously, it's really serious. I can tell you that I am quite happy that the original posts are no longer present, because if they were I am moderately confident that I would want to go back and see if I could make more sense out of the matter, and if Eliezer, Alicorn, and Roko are right about this, making sense out of the matter would be seriously detrimental to my health.

Thankfully, either it's a threat but I don't understand it fully, in which case I'm safe, or it's not a threat, in which case I'm also safe. But I am sufficiently concerned about the possibility that it's a threat that I don't understand fully but might be able to realize independently given enough thought that I'm consciously avoiding extended thought about this matter. I will respond to posts that directly relate to this one but am otherwise done with this topic-- rest assured that, if you missed this one, you're really quite all right for it!

Comment author: Vaniver 10 December 2010 06:21:41PM 5 points [-]

Given the reactions to it by Eliezer, Alicorn, and Roko, who seem very intelligent and know more about this topic than I do, I'm not so sure.

This line of argument really bothers me. What does it mean for E, A, and R to seem very intelligent? As far as I can tell, the necessary conclusion is "I will believe a controversial statement of theirs without considering it." When you word it like that, the standards are a lot higher than "seem very intelligent", or at least narrower- you need to know their track record on decisions like this.

(The controversial statement is "you don't want to know about X," not X itself, by the way.)

Comment author: katydee 10 December 2010 06:27:56PM 9 points [-]

I am willing to accept the idea that (intelligent) specialists in a field may know more about their field than nonspecialists and are therefore more qualified to evaluate matters related to their field than I.

Comment author: Vaniver 10 December 2010 06:37:09PM 5 points [-]

Good point, though I would point out that you need E, A, and R to be specialists when it comes to how people react to X, not just X, and I would say there's evidence that's not true.

Comment author: katydee 10 December 2010 06:44:08PM *  1 point [-]

I agree, but I know what conclusion I would draw from the belief in question if I actually believed it, so the issue of their knowledge of how people react is largely immaterial to me in particular. I was mostly posting to provide a data point in favor of keeping the material off LW, not to attempt to dissolve the issue completely or anything.

Comment author: Vladimir_Nesov 10 December 2010 06:27:07PM 0 points [-]

When you word it like that, the standards are a lot higher than "seem very intelligent", or at least narrower- you need to know their track record on decisions like this.

You don't need any specific kind of proof, you already have some state of knowledge about correctness of such statements. There is no "standard of evidence" for forming a state of knowledge, it just may be that without the evidence that meets that "standard" you don't expect to reach some level of certainty, or some level of stability of your state of knowledge (i.e. low expectation of changing your mind).

Comment author: FormallyknownasRoko 10 December 2010 05:34:09PM *  0 points [-]

Whatever man, go ahead and make your excuses, you have been warned.

Comment author: Vaniver 10 December 2010 05:41:37PM 8 points [-]

I have not only been warned, but I have stared the basilisk in the eyes, and I'm still here typing about it. In fact, I have only cared enough to do so because it was banned, and I wanted the information on how dangerous it was to judge the wisdom of the censorship.

On a more general note, being terrified of very unlikely terrible events is a known human failure mode. Perhaps it would be more effective at improving human rationality to expose people to ideas like this with the sole purpose of overcoming that sort of terror?

Comment author: Jack 10 December 2010 06:31:19PM *  5 points [-]

I'll just second that I also read it a while back (though after it was censored) and thought that it was quite interesting but wrong on multiple levels. Not 'probably wrong' but wrong like an invalid logic proof is wrong (though of course I am not 100% certain of anything). My main concern about the censorship is that not talking about what was wrong with the argument will allow the proliferation of the reasoning errors that left people thinking the conclusion was plausible. There is a kind of self-fulfilling prophesy involved in not recognizing these errors which is particularly worrying.

Comment author: JGWeissman 11 December 2010 01:58:40AM 7 points [-]

Consider this invalid proof that 1 = 2:

1. Let x = y
2. x^2 = x*y
3. x^2 - y^2 = x*y - y^2
4. (x - y)*(x + y) = y*(x - y)
5. x + y = y
6. y + y = y (substitute using 1)
7. 2y = y
8. 2 = 1

You could refute this by pointing out that step (5) involved division by (x - y) = (y - y) = 0, and you can't divide by 0.

But imagine if someone claimed that the proof is invalid because "you can't represent numbers with letters like 'x' and 'y'". You would think that they don't understand what is actually wrong with it, or why someone might mistakenly believe it. This is basically my reaction to everyone I have seen oppose the censorship because of some argument they present that the idea is wrong and no one would believe it.

Comment author: Jack 11 December 2010 03:11:11AM *  2 points [-]

I'm actually not sure if I understand your point. Either it is a round-about way of making it or I'm totally dense and the idea really is dangerous (or some third option).

It's not that the idea is wrong and no one would believe it, it's that the idea is wrong and when presented with with the explanation for why it's wrong no one should believe it. In addition, it's kind of important that people understand why it's wrong. I'm sympathetic to people with different minds that might have adverse reactions to things I don't but the solution to that is to warn them off, not censor the topics entirely.

Comment author: JGWeissman 11 December 2010 03:26:39AM 1 point [-]

Yes, the idea really is dangerous.

it's that the idea is wrong and when presented with with the explanation for why it's wrong no one should believe it.

And for those who understand the idea, but not why it is wrong, nor the explanation of why it is wrong?

the solution to that is to warn them off, not censor the topics entirely.

This is a politically reinforced heuristic that does not work for this problem.

Comment author: XiXiDu 11 December 2010 12:12:35PM *  6 points [-]

This is a politically reinforced heuristic that does not work for this problem.

Transparency is very important regarding people and organisations in powerful and unique positions. The way they act and what they claim in public is weak evidence in support of their honesty. To claim that they have to censor certain information in the name of the greater public good, and to fortify the decision based on their public reputation, does bear no evidence about their true objectives. The only way to solve this issue is by means of transparency.

Surely transparency might have negative consequences, but it mustn't and can outweigh the potential risks from just believing that certain people are telling the truth and do not engage in deception to follow through on their true objectives.

There is also nothing that Yudkowsky has ever achieved that would sufficiently prove his superior intellect that would in turn justify people to just believe him about some extraordinary claim.

Comment author: JGWeissman 11 December 2010 05:49:15PM 1 point [-]

When I say something is a misapplied politically reinforced heuristic, you only reinforce my point by making fully general political arguments that it is always right.

Censorship is not the most evil thing in the universe. The consequences of transparency are allowed to be worse than censorship. Deal with it.

Comment deleted 11 December 2010 06:04:46AM [-]
Comment author: Vaniver 11 December 2010 05:12:29PM 2 points [-]

For those curious: we do agree, but he went to quite a bit more effort in showing that than I did (and is similarly more convincing).

Comment author: Vladimir_Nesov 10 December 2010 05:53:19PM 2 points [-]

I have not only been warned, but I have stared the basilisk in the eyes, and I'm still here typing about it.

This isn't evidence about that hypothesis, it's expected that most certainly nothing happens. Yet you write for rhetorical purposes as if it's supposed to be evidence against the hypothesis. This constitutes either lying or confusion (I expect it's unintentional lying, with phrases produced without conscious reflection about their meaning, so a little of both lying and confusion).

Comment author: Jack 10 December 2010 06:05:56PM 5 points [-]

The sentence of Vaniver's you quote seems like a straight forward case of responding to hyperbole with hyperbole in kind.

Comment author: Vladimir_Nesov 10 December 2010 06:11:23PM 1 point [-]

That won't be as bad-intentioned, but still as wrong and deceptive.

Comment author: shokwave 10 December 2010 06:10:49PM 2 points [-]

I have not only been warned, but I have stared the basilisk in the eyes, and I'm still here typing about it.

The point we are trying to make is that we think the people who stared the basilisk in the eyes and metaphorically turned to stone are stronger evidence.

Comment author: Vaniver 10 December 2010 06:32:13PM 8 points [-]

The point we are trying to make is that we think the people who stared the basilisk in the eyes and metaphorically turned to stone are stronger evidence.

I get that. But I think it's important to consider both positive and negative evidence- if someone's testimony that they got turned to stone is important, so are the testimonies of people who didn't get turned to stone.

The question to me is whether the basilisk turns people to stone or people turn themselves into stone. I prefer the second because it requires no magic powers on the part of the basilisk. It might be that some people turn to stone when they see goatse for the first time, but that tells you more about humans and how they respond to shock than about goatse.

Indeed, that makes it somewhat useful to know what sort of things shock other people. Calling this idea 'dangerous' instead of 'dangerous to EY" strikes me as mind projection.

Comment author: shokwave 10 December 2010 07:14:37PM 1 point [-]

But I think it's important to consider both positive and negative evidence- if someone's testimony that they got turned to stone is important, so are the testimonies of people who didn't get turned to stone.

I am considering both.

It might be that some people turn to stone when they see goatse for the first time, but that tells you more about humans and how they respond to shock than about goatse.

I generally find myself in support of people who advocate a policy of keeping people from seeing Goatse.

Comment author: Vaniver 11 December 2010 12:35:39AM 3 points [-]

I generally find myself in support of people who advocate a policy of keeping people from seeing Goatse.

I'm not sure how to evaluate this statement. What do you mean by "keeping people from seeing Goatse"? Banning? Voluntarily choosing not to spread it? A filter like the one proposed in Australia that checks every request to the outside world?

Comment author: shokwave 11 December 2010 07:15:13AM 1 point [-]

Censoring posts that display Goatse on LessWrong.

Generally, censoring posts that display Goatse on non-Goatse websites.

Comment author: Vaniver 11 December 2010 04:28:26PM 7 points [-]

I am much more sympathetic to "keeping goatse off of site X" than "keeping people from seeing goatse," and so that's a reasonable policy. If your site is about posting pictures of cute kittens, then goatse is not a picture of a cute kitten.

However, it seems to me that suspected Langford basilisks are part of the material of LessWrong. Imagine someone posted in the discussion "hey guys, I really want to be an atheist but I can't stop worrying about whether or not the Rapture will happen, and if it does life will suck." It seems to me that we would have a lot to say to them about how they could approach the situation more rationally.

And, if Langford basilisks exist, religion has found them. Someone got a nightmare because of Roko's idea, but people fainted upon hearing Sinners in the Hands of an Angry God. Why are we not looking for the Perseus for this Medusa? If rationality is like an immune system, and we're interested in refining our rationality, we ought to be looking for antibodies.

Comment author: katydee 11 December 2010 08:02:22AM 2 points [-]

Is Goatse supposed to be a big deal? Someone showed it to me and I literally said "who cares?"

Comment author: Vladimir_Nesov 10 December 2010 06:14:13PM 1 point [-]

I don't understand this. (Play on conservation of expected evidence? In what way?)

Comment author: shokwave 10 December 2010 06:30:56PM 4 points [-]

Normal updating.

  • Original prior for basilik-danger.
  • Eliezer_Yudkowsky stares at basilisk, turns to stone (read: engages idea, decides to censor). Revise pr(basilisk-danger) upwards.
  • FormallyknownasRoko stares at basilisk, turns to stone (read: appears to truly wish e had never thought it). Revise pr(basilisk-danger) upwards.
  • Vladimir_Nesov stares at basilisk, turns to stone (read: engages idea, decides it is dangerous). Revise pr(basilisk-danger) upwards.
  • Vaniver stares at basilisk, is unharmed (read: engages idea, decides it is not dangerous). Revise pr(basilisk-danger) downwards.
  • Posterior is higher than original prior.

For the posterior to equal or lower than the prior, Vaniver would have to be more a rationalist than Eliezer, Roko, and you put together.

Comment author: Jack 10 December 2010 07:16:46PM 7 points [-]

Okay, but more than four people have engaged with the idea. Should we take a poll?

The problem of course is that majorities often believe stupid things. That is why a free marketplace of ideas free from censorship is a really good thing! The obvious thing to do is exchange information until agreement but we can't do that, at least not here.

Also, the people who think it should be censored all seem to disagree about how dangerous the idea really is, suggesting it isn't clear how it is dangerous. It also seems plausible that some people have influenced the thinking of other people- for example it looks like Roko regretted posting after talking to Eliezer. While Roko's regret is evidence that Eliezer is right, it isn't the same as independent/blind confirmation that the idea is dangerous.

Comment author: shokwave 10 December 2010 07:36:23PM -2 points [-]

The problem of course is that majorities often believe stupid things.

When you give all agents equal weight, sure. Without taking a poll of anything except my memory, Eliezer+Roko+VladNesov+Alicorn are against, DavidGerard+waitingforgodel+vaniver are for. Others are more sidelined than supporting a particular side.

The obvious thing to do is exchange information until agreement but we can't do that, at least not here.

Aumann agreement works in the case of hidden information - all you need are posteriors and common knowledge of the event alone.

While Roko's regret is evidence that Eliezer is right, it isn't the same as independent/blind confirmation that the idea is dangerous.

Roko increased his estimation and Eliezer decreased his estimation - and the amounts they did so are balanced according to the strength of their private signals. Looking at two Aumann-agreed conclusions gives you the same evidence as looking that the pre-Aumann (differing) conclusions - the same way that 10, 10 gives you the same average as 5, 15.

Comment author: TheOtherDave 10 December 2010 08:33:58PM 7 points [-]

Others are more sidelined than supporting a particular side.

I would prefer you not treat people avoiding a discussion as evidence that people don't differentially evaluate the assertions made in that discussion.

Doing so creates a perverse incentive whereby chiming in to say "me too!" starts to feel like a valuable service, which would likely chase me off the site altogether. (Similar concerns apply to upvoting comments I agree with but don't want to see more of.)

If you are seriously interested in data about how many people believe or disbelieve certain propositions, there exist techniques for gathering that data that are more reliable than speculating.

If you aren't interested, you could just not bring it up.

Comment author: Jack 10 December 2010 08:00:24PM *  3 points [-]

Without taking a poll of anything except my memory, Eliezer+Roko+VladNesov+Alicorn are against, DavidGerard+waitingforgodel+vaniver are for.

I'm for. I believe Tim Tyler is for.

Aumann agreement works in the case of hidden information - all you need are posteriors and common knowledge of the event alone.

Human's have this unfortunate feature of not being logically omniscient. In such cases where people don't see all the logical implications of an argument we can treat those implications as hidden information. If this wasn't the case then the censorship would be totally unnecessary as Roko's argument didn't actually include new information. We would have all turned to stone already.

Roko increased his estimation and Eliezer decreased his estimation - and the amounts they did so are balanced according to the strength of their private signals.

There is no way for you to have accurately assessed this. Roko and Eliezer aren't idealized Bayesian agents, it is extremely unlikely they performed a perfect Aumann agreement. If one is more persuasive than the other for reasons other than the evidence they share than their combined support for the proposition may not be worth the same as two people who independently came to support the proposition. Besides which, according to you, what information did they share exactly?

Comment author: Vaniver 10 December 2010 07:00:45PM 4 points [-]

For the posterior to equal or lower than the prior, Vaniver would have to be more a rationalist than Eliezer, Roko, and you put together.

How many of me would there have to be for that to work?

Also, why is rationalism the risk factor for this basilisk? Maybe the basilisk only turns to stone people with brown eyes (or the appropriate mental analog).

Comment author: shokwave 10 December 2010 07:25:11PM *  0 points [-]

How many of me would there have to be for that to work?

Only one; I meant 'you' in that line to refer to Vlad. It does raise the question "how many people disagree before I side with them instead of Eliezer/Roko/Vlad". And the answer to that is ... complicated. Each person's rationality, modified by how much it was applied in this particular case, is the weight I give to their evidence; then the full calculation of evidence for and against should bring my prior to within epsilon but preferably below my original prior for me to decide the idea is safe.

Also, why is rationalism the risk factor for this basilisk?

Rationalism is the ability to think well and this is a dangerous idea. If it were a dangerous bacterium then immune system would be the risk factor.

Comment author: Vaniver 11 December 2010 01:05:23AM 2 points [-]

Rationalism is the ability to think well and this is a dangerous idea. If it were a dangerous bacterium then immune system would be the risk factor.

Generally, if your immune system is fighting something, you're already sick. Most pathogens are benign or don't have the keys to your locks. This might be a similar situation- the idea is only troubling if your lock fits it- and it seems like then there would be rational methods to erode that fear (like the immune system mobs an infection).

Comment author: David_Gerard 10 December 2010 07:58:43PM 2 points [-]

Rationalism is the ability to think well and this is a dangerous idea. If it were a dangerous bacterium then immune system would be the risk factor.

Er, are you describing rationalism (I note you say that and not "rationality") as susceptible to autoimmune disorders? More so than in this post?

Comment deleted 10 December 2010 06:54:55PM [-]
Comment author: shokwave 10 December 2010 07:03:59PM -1 points [-]

Ensuring that is part of being a rationalist; if EY, Roko, and Vlad (apparently Alicorn as well?) were bad at error-checking and Vaniver was good at it, that would be sufficient to say that Vaniver is a better rationalist than E R V (A?) put together.

Comment author: David_Gerard 10 December 2010 07:36:55PM *  7 points [-]

Certainly. However, error-checking oneself is notoriously less effective than having outsiders do so.

"For the computer security community, the moral is obvious: if you are designing a system whose functions include providing evidence, it had better be able to withstand hostile review." - Ross Anderson, RISKS Digest vol 18 no 25

Until a clever new thing has had decent outside review, it just doesn't count as knowledge yet.

Comment author: Manfred 10 December 2010 07:30:36PM 3 points [-]

I haven't read fluffy (I have named it fluffy), but I'd guess it's an equivalent of a virus in a monoculture: every mode of thought has its blind spots, and so to trick respectable people on LW, you only need an idea that sits in the right blind spots. No need for general properties like "only infectious to stupid people."

Alicorn throws a bit of a wrench in this, as I don't think she shares as many blind spots with the others you mention, but it's still entirely possible. This also explains the apparent resistance of outsiders, without need for Eliezer to be lying when he says he thinks fluffy was wrong.

Comment author: Vladimir_Nesov 10 December 2010 06:58:25PM *  2 points [-]

Eliezer_Yudkowsky stares at basilisk, turns to stone (read: engages idea, decides to censor). Revise pr(basilisk-danger) upwards.

This equivocates the intended meaning of turning to stone in the original discussion you replied to. Fail. (But I understand what you meant now.)

Comment author: shokwave 10 December 2010 07:06:26PM 1 point [-]

Sorry, I should not have included censoring specifically. Change the "read:"s to 'engages, reacts negatively', 'engages, does not react negatively' and the argument still functions.

Comment author: Vladimir_Nesov 10 December 2010 07:09:29PM *  2 points [-]

The argument does seem to function, but you shouldn't have used the term in a sense conflicting with intended.

Comment author: TheOtherDave 10 December 2010 05:46:01PM 1 point [-]

Perhaps it would be more effective at improving human rationality to expose people to ideas like this with the sole purpose of overcoming that sort of terror?

You would need a mechanism for actually encouraging them to "overcome" the terror, rather than reinforce it. Otherwise you might find that your subjects are less rational after this process than they were before.

Comment author: Vaniver 10 December 2010 06:09:40PM 0 points [-]

Right- and current methodologies when it comes to that sort of therapy are better done in person than over the internet.

Comment author: FormallyknownasRoko 10 December 2010 05:58:02PM 0 points [-]

being terrified of very unlikely terrible events is a known human failure mode

one wonders how something like that might have evolved, doesn't one? What happened to all the humans who came with the mutation that made them want to find out whether the sabre-toothed tiger was friendly?

Comment author: Kingreaper 10 December 2010 06:29:21PM *  7 points [-]

one wonders how something like that might have evolved, doesn't one? What happened to all the humans who came with the mutation that made them want to find out whether the sabre-toothed tiger was friendly?

I don't see how very unlikely events that people knew the probability of would have been part of the evolutionary environment at all.

In fact, I would posit that the bias is most likely due to having a very high floor for probability. In the evolutionary environment things with probability you knew to be <1% would be unlikely to ever be brought to your attention. So not having any good method for intuitively handling probabilities between 1% and zero would be expected.

In fact, I don't think I have an innate handle on probability to any finer grain than ~10% increments. Anything more than that seems to require mathematical thought.

Comment author: FormallyknownasRoko 10 December 2010 06:32:22PM 0 points [-]

Probably less than 1% of cave-men died by actively seeking out the sabre-toothed tiger to see if it was friendly. But I digress.

Comment author: Kingreaper 10 December 2010 06:34:48PM *  8 points [-]

But probably far more than 1% of cave-men who chose to seek out a sabre-tooth tiger to see if they were friendly died due to doing so.

The relevant question on an issue of personal safety isn't "What % of the population die due to trying this?"

The relevant question is: "What % of the people who try this will die?"

In the first case, rollerskating downhill, while on fire, after having taken arsenic would seem safe (as I suspect no-one has ever done precisely that)

Comment author: Vaniver 10 December 2010 06:08:32PM 7 points [-]

one wonders how something like that might have evolved, doesn't one?

No, really, one doesn't wonder. It's pretty obvious. But if we've gotten to the point where "this bias paid off in the evolutionary environment!" is actually used as an argument, then we are off the rails of refining human rationality.

Comment author: FormallyknownasRoko 10 December 2010 06:17:43PM *  2 points [-]

What's wrong with using "this bias paid off in the evolutionary environment!" as an argument? I think people who paid more attention to this might make fewer mistakes, especially in domains where there isn't a systematic, exploitable difference between EEA and now.

The evolutionary environment contained enetities capable of dishing out severe punishments, unertainty, etc.

If anything, I think that the heuristic that an idea "obviously" can't be dangerous is the problem, not the heuristic that one should take care around possibilities of strong penalites.

Comment author: timtyler 10 December 2010 06:25:01PM *  4 points [-]

It is a fine argument for explaining the widespread occcurrence of fear. However, today humans are in an environment where their primitive paranoia is frequently triggered by inappropriate stimulii.

Dan Gardener goes into this in some detail in his book: Risk: The Science and Politics of Fear

Video of Dan discussing the topic: Author Daniel Gardner says Americans are the healthiest and safest humans in the world, but are irrationally plagued by fear. He talks with Maggie Rodriguez about his book 'The Science Of Fear.'

Comment author: Desrtopa 10 December 2010 06:34:41PM 0 points [-]

He says "we" are the healthiest and safest humans ever to live, but I'm very skeptical that this refers specifically to Americans rather than present day first world nation citizens in general.

Comment author: FormallyknownasRoko 10 December 2010 06:29:00PM *  0 points [-]

Yes, we are, in fact, safer than in the EEA, in contemporary USA.

But still, there are some real places where danger is real, like the Bronx or scientology or organized crime or a walking across a freeway. So, don't go rubbishing the heuristic of being frightened of potentially real danger.

I think it would only be legitimate to criticize fear itself on "outside view" grounds if we lived in a world with very little actual danger, which is not at all the case.

Comment author: Vaniver 10 December 2010 06:51:53PM 3 points [-]

But still, there are some real places where danger is real, like the Bronx or scientology or organized crime or a walking across a freeway.

So, this may be a good way to approach the issue: loss to individual humans is, roughly speaking, finite. Thus, the correct approach to fear is to gauge risks by their chance of loss, and then discount if it's not fatal.

So, we should be much less worried by a 1e-6 risk than a 1e-4 risk, and a 1e-4 risk than a 1e-2 risk. If you are more scared by a 1e-6 risk than a 1e-2 risk, you're reasoning fallaciously.

Now, one might respond- "but wait! This 1e-6 risk is 1e5 times worse than the 1e-2 risk!". But that seems to fall into the traps of visibility bias and privileging the hypothesis. If you're considering a 1e-6 risk, have you worked out not just all the higher order risks, but also all of the lower order risks that might have higher order impact? And so when you have an idea like the one in question, which I would give a risk of 1e-20 for discussion's sake, and you consider it without also bringing into your calculus essentially every other risk possible, you're not doing it rigorously. And, of course, humans can't do that computation.

Now, the kicker here is that we're talking about fear. I might fear the loss of every person I know just as strongly as I fear the loss of every person that exists, but be willing to do more to prevent the loss of everyone that exists (because that loss is actually larger). Fear has psychological ramifications, not decision-theoretic ones. If this idea has 1e-20 chances of coming to pass, you can ignore it on a fear level, and if you aren't, then I'm willing to consider that evidence you need help coping with fear.

Comment author: timtyler 10 December 2010 06:36:36PM *  1 point [-]

I have a healthy respect for the adaptive aspects of fear. However, we do need an explanation for the scale and prevalence of irrational paranoia.

The picture of an ancestral water hole surrounded by predators helps us to understand the origins of the phenomenon. The ancestral environment was a dangerous and nasty place where people led short, brutish lives. There, living in constant fear made sense.

Comment author: Emile 10 December 2010 07:08:18PM *  3 points [-]

Someone's been reading Terry Pratchett.

He always held that panic was the best means of survival. Back in the old days, his theory went, people faced with hungry sabre-toothed tigers could be divided into those who panicked and those who stood there saying, "What a magnificent brute!" or "Here pussy".

Comment author: XiXiDu 10 December 2010 05:59:50PM 14 points [-]

I wish I had never come across the initial link on the internet that caused me to think about transhumanism and thereby about the singularity;

I wish you'd talk to someone other than Yudkowsky about this. You don't need anyone to harm you, you already seem to harm yourself. You indulge yourself in self-inflicted psychological stress. As Seneca said, "there are more things that terrify us than there are that oppress us, and we suffer more often in opinion than in reality". You worry and pay interest for debt that will likely never be made.

Look, you have three people all of whom think it is a bad idea to spread this. All are smart.

I read about quite a few smart people who hold idiot beliefs, I only consider this to be marginal evidence.

Furthermore, I would add that I wish I had never learned about any of these ideas.

You'd rather be some ignorant pleasure maximizing device? For me truth is the most cherished good.

If this is not enough warning to make you stop wanting to know more, then you deserve what you get.

BS.

Comment author: FormallyknownasRoko 10 December 2010 06:04:07PM 5 points [-]

For me truth is the most cherished good.

More so than not opening yourself up to a small risk of severe consequences? E.g. if you found a diary that clearly belonged to some organized crime boss, would you open it up and read it? I see this situation as analogous.

Comment deleted 10 December 2010 06:44:15PM [-]
Comment author: FormallyknownasRoko 10 December 2010 07:23:36PM *  8 points [-]

Well I guess this is our true point of disagreement. I went to the effort of finding out a lot, went to SIAI and Oxford to learn even more, and in the end I am left seriously disappointed by all this knowedge. In the end it all boils down to:

"most people are irrational, hypocritical and selfish, if you try and tell them they shoot the messenger, and if you try and do anything you bear all the costs, internalize only tiny fractions of the value created if you succeed, and you almost certainly fail to have an effect anyway. And by the way the future is an impending train wreck"

I feel quite strongly that this knowledge is not a worthy thing to have sunk 5 years of my life into getting. I don't know, XiXiDu, you might prize such knowledge, including all the specifics of how that works out exactly.

If you really strongly value the specifics of this, then yes you probably would on net benefit from the censored knowledge, the knowledge that was never censored because I never posted it, and the knowledge that I never posted because I was never trusted with it anyway. But you still probably won't get it, because those who hold it correctly infer that the expected value of releasing it is strongly negative from an altruist's perspective.

Comment author: WrongBot 10 December 2010 08:30:22PM 29 points [-]

The future is probably an impending train wreck. But if we can save the train, then it'll grow wings and fly up into space while lightning flashes in the background and Dragonforce play a song about fiery battlefields or something. We're all stuck on the train anyway, so saving it is worth a shot.

I hate to see smart people who give a shit losing to despair. This is still the most important problem and you can still contribute to fixing it.

TL;DR: I want to give you a hug.

Comment author: Eliezer_Yudkowsky 10 December 2010 10:17:58PM 11 points [-]

most people are irrational, hypocritical and selfish, if you try and tell them they shoot the messenger, and if you try and do anything you bear all the costs, internalize only tiny fractions of the value created if you succeed,

So? They're just kids!

(or)

He glanced over toward his shoulder, and said, "That matter to you?"

Caw!

He looked back up and said, "Me neither."

Comment author: FormallyknownasRoko 10 December 2010 10:25:41PM 3 points [-]

I mean I guess I shouldn't complain that you don't find this bothers you, because you are, in fact, helping me by doing what you do and being very good at it, but that doesn't stop it being demotivating for me! I'll see what I can do regarding quant jobs.

Comment author: timtyler 10 December 2010 07:30:50PM *  3 points [-]

That doesn't sound right to me. Indeed, it sounds as though you are depressed :-(

Unsolicited advice over the public internet is rather unlikely to help - but maybe focus for a bit on what you want - and the specifics of how to get to there.

Comment author: Jack 10 December 2010 08:12:49PM 2 points [-]

Upvoted for the excellent summary!

"most people are irrational, hypocritical and selfish, if you try and tell them they shoot the messenger, and if you try and do anything you bear all the costs, internalize only tiny fractions of the value created if you succeed, and you almost certainly fail to have an effect anyway. And by the way the future is an impending train wreck"

Comment author: katydee 10 December 2010 08:14:28PM 3 points [-]

I'm curious about the "future is an impending train wreck" part. That doesn't seem particularly accurate to me.

Comment author: FormallyknownasRoko 10 December 2010 08:18:52PM *  1 point [-]

Maybe it will all be OK. Maybe the trains fly past each other on separate tracks. We don't know. There sure as hell isn't a driver though. All the inside-view evidence points to bad things,with the exception that Big Worlds could turn out nicely. Or horribly.

Comment author: timtyler 10 December 2010 08:56:57PM 0 points [-]
Comment author: katydee 10 December 2010 07:28:35PM 2 points [-]

This isn't meant as an insult, but why did it take you 5 years of dedicated effort to learn that?

Comment author: FormallyknownasRoko 10 December 2010 07:32:49PM *  3 points [-]

Specifics. Details. The lesson of science is that details can sometimes change the overall conclusion. Also some amount of nerdyness meaning that the statements about human nature weren't obvious to me.

Comment deleted 10 December 2010 08:03:24PM *  [-]
Comment author: steven0461 10 December 2010 09:50:05PM 6 points [-]

I would choose that knowledge if there was the chance that it wouldn't find out about it. As far as I understand your knowledge of the dangerous truth, it just increases the likelihood of suffering, it doesn't make it guaranteed.

I don't understand your reasoning here -- bad events don't get a "flawless victory" badness bonus for being guaranteed. A 100% chance of something bad isn't much worse than a 90% chance.

Comment author: XiXiDu 11 December 2010 09:07:07AM 0 points [-]

I said that I wouldn't want to know it if a bad outcome was guaranteed. But if it would make a bad outcome possible, but very-very-unlikely to actually occur, then the utility I assign to knowing the truth would outweigh the very unlikely possibility of something bad happening.

Comment author: FormallyknownasRoko 10 December 2010 10:34:52PM 0 points [-]

No, dude, you're wrong

Comment author: FormallyknownasRoko 10 December 2010 08:15:33PM *  4 points [-]

The compelling argument for me is that knowing about bad things is useful to the extent that you can do something about them, and it turns out that people who don't know anything (call them "non-cogniscenti") will probably free-ride their way to any benefits of action on the collective-action-problem that is the at issue here, whilst avoiding drawing any particular attention to themselves ==> avoiding the risks.

Vladimir Nesov doubts this prima facie, i.e. he asks "how do you know that the strategy of being a completely inert player is best?".

-- to which I answer, "if you want to be the first monkey shot into space, then good luck" ;D

Comment author: timtyler 10 December 2010 09:05:40PM *  0 points [-]

it turns out that people who don't know anything (call them "non-cogniscenti") will probably free-ride their way to any benefits of action on the collective-action-problem that is the at issue here, whilst avoiding drawing any particular attention to themselves ==> avoiding the risks.

This is the "collective-action-problem" - where the end of the world arrives - unless a select band of heroic messiahs arrive and transport everyone to heaven...?

That seems like a fantasy story designed to manipulate - I would council not getting sucked in.

Comment author: steven0461 10 December 2010 09:30:34PM 6 points [-]

I wonder what fraction of actual historical events a hostile observer taking similar liberties could summarize to also sound like some variety of "a fantasy story designed to manipulate".

Comment author: timtyler 10 December 2010 09:55:49PM *  0 points [-]

I don't know - but believing inaction is best is rather common - and there are pages all about it - e.g.:

http://en.wikipedia.org/wiki/Learned_helplessness

Comment author: FormallyknownasRoko 10 December 2010 10:28:14PM *  7 points [-]

No, this is the "collective-action-problem" - where the end of the world arrives - despite a select band of decidedly amateurish messiahs arriving and failing to accomplish anything significant.

You are looking at those amateurs now.

Comment author: timtyler 11 December 2010 12:18:07PM *  -2 points [-]

The END OF THE WORLD is probably the most frequently-repeated failed prediction of all time. Humans are doing spectacularly well - and the world is showing many signs of material and moral progress - all of which makes the apocalypse unlikely.

The reason for the interest here seems obvious - the Singularity Institute's funding is derived largely from donors who think it can help to SAVE THE WORLD. The world must first be at risk to enable heroic Messiahs to rescue everyone.

The most frequently-cited projected cause of the apocalypse: an engineering screw-up. Supposedly, future engineers are going to be so incompetent that they accidentally destroy the whole world. The main idea - as far as I can tell - is that a bug is going to destroy civilisation.

Also - as far as I can tell - this isn't the conclusion of analysis performed on previous engineering failures - or on the effects of previous bugs - but rather is wild extrapolation and guesswork.

Of course it is true that there may be a disaster, and END OF THE WORLD might arrive. However there is no credible evidence that this is likely to be a probable outcome. Instead, what we have appears to be mostly a bunch of fear mongering used for fundraising aimed at fighting the threat. That gets us into the whole area of the use and effects of fear mongering.

Fearmongering is a common means of psychological manipulation, used frequently by advertisers and marketers to produce irrational behaviour in their victims.

It has been particularly widely used in the IT industry - mainly in the form of fear, uncertainty and doubt.

Evidently, prolonged and widespread use is likely to help to produce a culture of fear. The long-term effects of that are not terribly clear - but it seems to be dubious territory.

I would council those using fear mongering for fund-raising purposes to be especially cautious of the harm this might do. It seems like a potentially dangerous form of meme warfare. Fear targets circuits in the human brain that evolved in an earlier, more dangerous era - where death was much more likely - so humans have an evolved vulnerability in the area. The modern super-stimulus of the END OF THE WORLD overloads those vulnerable circuits.

Maybe this is an effective way of extracting money from people - but also, maybe it is an unpleasant and unethical one. So, wannabe heroic Messiahs, please: take care. Starting out by screwing over your friends and associates by messing up their heads with a hostile and virulent meme complex may not be the greatest way to start out.

Comment author: katydee 10 December 2010 06:47:50PM *  1 point [-]

Ah, you remind me of me from a while back. When I was an elementary schooler, I once replied to someone asking "would you rather be happy or right" with "how can I be happy if I can't be right?" But these days I've moderated somewhat, and I feel that there is indeed knowledge that can be harmful.

Comment author: Manfred 10 December 2010 07:39:06PM 3 points [-]

Really thought you were going to go with Tom Riddle on this one. Perfect line break for it :)

Comment author: timtyler 10 December 2010 06:53:39PM *  1 point [-]

I would add that I wish I had never learned about any of these ideas. In fact, I wish I had never come across the initial link on the internet that caused me to think about transhumanism and thereby about the singularity;

Hmm. It is tricky to go back, I would imagine.

The material does come with some warnings, I believe. For instance, consider this one:

"Beware lest Friendliness eat your soul." - Eliezer Yudkowsky

Comment author: Vladimir_Nesov 10 December 2010 05:43:29PM 1 point [-]

In fact, I wish I had never come across the initial link on the internet that caused me to think about transhumanism and thereby about the singularity

As I understand, you donate (and plan to in the future) to existential risk charities, and that is one of the consequences of you having come across that link. How does this compute into net negative, in your estimation, or are you answering a different question?

Comment author: FormallyknownasRoko 10 December 2010 06:01:07PM -1 points [-]

Sure I want to donate. But if you express it as a hypothetical choice between being a person who didn't know about any of this and had no way of finding out, versus what I have now, I choose the former. Though since that is not an available choice, it is a somewhat academic question.

Comment author: XiXiDu 10 December 2010 06:48:51PM *  4 points [-]

But if you express it as a hypothetical choice between being a person who didn't know about any of this and had no way of finding out, versus what I have now, I choose the former.

I can't believe to hear this from a person who wrote about Ugh fields. I can't believe to read a plead for ignorance on a blog devoted to refining rationality. Ignorance is bliss, is that the new motto now?

Comment author: FormallyknownasRoko 10 December 2010 07:00:19PM 1 point [-]

Well look, one has to do cost/benefit calculations, not just blindly surge forward in some kind of post-enlightenment fervor. To me, it seems like there is only one positive term in the equation:: the altrustic value of giving money to some existential risk charity.

All the other terms are negative, at least for me. And unless I actually overcome excuses, akrasia, etc to donate a lot, I think it'll all have been a mutually detrimental waste of time.

Comment author: Vladimir_Nesov 10 December 2010 06:53:42PM *  0 points [-]

There is only one final criterion, the human decision problem. It trumps any other rule, however good or useful.

(You appeal to particular heuristics, using the feeling of indignation as a rhetoric weapon.)

Comment author: Vladimir_Nesov 10 December 2010 06:07:35PM 0 points [-]

Not helping. I was referring to the the moral value of donations as an argument for choosing to know, as opposed to not knowing. You don't seem to address that in your reply (did I miss something?).

Comment author: FormallyknownasRoko 10 December 2010 06:14:51PM 1 point [-]

Oh, I see. Well, I guess it depends upon how much I eventually donate and how much of an incremental difference that makes.

It would certainly be better to just donate, AND to also not know anything about anything dangerous. I'm not even sure that's possible, though. For all we know, just knowing about any of this is enough to land you in a lot of trouble either in the causal future or elsewhere.