All of G0W51's Comments + Replies

Panexistential risk is a good, intuitive, name.

True. Also, the Great Filter is more akin to an existential catastrophe than to existential risk, that is, the risk of an existential catastrophe.

Is there a term for a generalization of existential risk that includes the extinction of alien intelligences or the drastic decrease of their potential? Existential risk, that is, the extinction of Earth-originating intelligent life or the drastic decrease of its potential, does not sound nearly as harmful if there are alien civilizations that become sufficiently advanced in place of Earth-originating life. However, an existential risk sounds far more harmful if it compromises all intelligent life in the universe, or if there is no other intelligent life i... (read more)

That sounds about right.

It's later, but, unless I am mistaken, the arrival of the intelligence explosion isn't that much later than when most people will retire, so I don't think that fully explains it.

People could vote for government officials who have FAI research on their agenda, but currently, I think few if any politicians even know what FAI is. Why is that?

Why do people spend much, much more time worrying about their retirement plans than the intelligence explosion if they are a similar distance in the future? I understand that people spend less time worrying about the intelligence explosion than what would be socially optimal because the vast majority of its benefits will be in the very far future, which people care little about. However, it seems probable that the intelligence explosion will still have a substantial effect on many people in the near-ish future (within the next 100 years). Yet, hardly anyone worries about it. Why?

I would like to improve my instrumental rationality and improve my epistemic rationality as a means to do so. Currently, my main goal is to obtain useful knowledge (mainly in college) in order to obtain resources (mainly money). I'm not entirely sure what I want to do after that, but whatever it is, resources will probably be useful for it.

Improving my rationality. Are you looking for something more specific?

How much should you use LW, and how? Should you consistently read the articles on Main? What about discussion? What about the comments? Or should a more case-by-case system be used?

What exactly do you suggest using to invest, then?

Some parties may be more likely to accelerate scientific progress than others, and those who do could decrease existential risk by decreasing the time spent in high-risk states, for example the period when there are dangerous nano-technological weapons but other astronomical objects have not be colonized. This probably is not enough to justify voting, but I thought I would just let you know.

Presumably "employ the same strategy" should be interpreted loosely, as it seems problematic to give no consideration to agents who would use a slightly different allocation strategy.

Thanks for the idea. I will look into it.

What literature is available on who will be given moral consideration in a superintelligence's coherent extrapolated volition (CEV) and how much weight each agent will be given?

Nick Bostrom's Superintelligence mentions that it is an open problem as to whether AIs, non-human animals, currently deceased people, etc should be given moral consideration, and whether the values of those who aid in creating the superintelligence should be given more weight than that of others. However, Bostrom does not actually answer these questions, other than slightly advocati... (read more)

Is it okay to re-ask questions on open threads if they were not answered the last time the were ask on it? I had asked this question but received no answer, and I am concerened it would be spammy to re-ask.

My gut says that the performance in a vehicle collision will probably bring the head to a halt against a relatively immobile object, so the hat won't do much of anything as the crushable bits are crushed too fast to be effective.

I don't see how the latter clause follows from the former. You said that in the drop test, the impact reduction was roughly 25%. This isn't huge, but I can't say it "won't do much of anything." Were you thinking of something else to support your claim?

What literature is available on who will be given moral consideration in a superintelligence's coherent extrapolated volition (CEV), and how much weight each agent will be given?

Nick Bostrom's Superintelligence mentions that it is an open problem as to whether AIs, non-human animals, currently deceased people, etc should be given moral consideration, and whether the values of those who aid in creating the superintelligence should be given more weight than that of others. However, Bostrom does not actually answer these questions, other than slightly advoca... (read more)

Don't worry, I don't mind math. Alas, I mainly have difficulty understanding why people act how they do, so I doubt mathematics will help much with that. I think I'm going to take the suggestion someone gave of reading more textbooks. A psychology course should also help.

Severity Index (I assume this is based on the head injury criterion?)

Actually, in an email they said the head of NOCSAE did the test, so presumably the NOCSAE Severity Index was used. An NOCSAE article says, "There is no measurable difference in safety of helmets with scores below the 1200 SI threshold." So in other words, in the test the hats did not protect against any significant damage, because no significant damage was done even without the hat. Despite this, the webpage said said that, "The Crasche hat reduces the severity of blunt ... (read more)

I don't see how this would really help unless I am trying to do original research.

Where can one find information on the underlying causes of phenomena? I have noticed that most educational resources discuss superficial occurrences and trends but not their underlying causes. For example, this Wikipedia article discusses the happenings in the Somali Civil War but hardly discusses the underlying motivations of each side and why the war turned out how it did. Of course, such discussions are often opinionated and have no clear-cut answers, perhaps making Wikipedia a sub-optimal place for them.

I know LW might not be the best place to ask thi... (read more)

My gut says that the performance in a vehicle collision will probably bring the head to a halt...

Presumably, the impact would cause the pedestrian to fly back in roughly the same direction the car was moving during the impact, rather than come to a complete stop. That said, I don't really know enough about the tests to know if this would make a difference in efficacy. Could you link the exact data you received?

Perhaps the endowment effect evolved because placing high value on an object you own signals to others that the object is valuable, which signals that you are wealthy, which can increase social status, which can increase mating prospects. I have not seen this idea mentioned previously, but I only skimmed parts of the literature.

Yes, but I'm looking to see if it increases existential risk more than it decreases it, and if the increase is significant.

Where exactly is the logical prior being used in the decision procedure? It doesn't seem like it would be used for calculating U, as U was implied to be computable. I don't see why it would be needed for p, as p could be computed from the complexity of the program, perhaps via kolmogorov complexity. Also, what is the purpose of us? Can't the procedure just set us to be whatever U outputs?

Oh, I think I see. Confidence is a feeling, while credence is a belief.

I find it interesting that you both are underconfident and realize you are underconfident. Have you tried adjusting for underconfidence like you would any other cognitive bias? (At least you need not adjust for overconfidence!)

Alternatively, the site could let the users determine what it good. Users could "like" or "dislike" articles, and these likes and dislikes would affect the reputation of the publisher. The higher the publisher's reputation, the more likes, and the fewer dislikes and article has, the higher rank the article would get when being searched for, and articles with sufficiently low rankings would be hidden. Think Stack Exchange for science.

It could be expanded in many ways, for example by weighing likes and dislikes by high-status users more heavily than low-status ones, or by using numeric ratings instead.

Good job. Why hasn't this been published in a journal?

Oppression could cause an existential catastrophe if the oppressive regime is never ended.

I have heard (from the book Global Catastrophic Risks) that life extension could increase existential risk by giving oppressive regimes increased stability by decreasing how frequently they would need to select successors. However, I think it may also decrease existential risk by giving people a greater incentive to care about the far future (because they could be in it). What are your thoughts on the net effect of life extension?

The book Global Catastrophic Risks states that it does not appear plausible that molecular manufacturing will not come into existence before 2040 or 2050. I am not at all an expert on molecular manufacturing, but this seems hard to believe, given how little work seems to be going into it. I couldn't find any sources discussing when molecular manufacturing will come into existence. Thoughts?

But other than self-importance, why don't people take it seriously? Is it otherwise just due to the absurdity and availability heuristics?

I think something else is going on. The responses to this question about the feasibility of strong AI mostly stated that it was possible, though selection bias is probably largely at play, as knowledgable people would be more likely to answer than the ignorant would be.

Why don't people (outside small groups like LW) advocate the creation of superintelligence much? If it is Friendly, it would have tremendous benefits. If superintelligence's creation isn't being advocated out of fears of it being unFriendly, then why don't more people advocate FAI research? Is it just too long-term for people to really care about? Do people not think managing the risks is tractable?

Despite there being multiple posts on recommended reading, there does not seem to be any comprehensive and non-redundant list stating what one ought to read. The previous lists do not seem to cover much non-rationality-related but still useful material that LWers might not have otherwise learned about (e.g. material on productivity, happiness, health, and emotional intelligence). However, there still is good material on these topics, often in the form of LW blog posts.

So, what is the cause of the absence of a single, comprehensive list? Such a list sounds... (read more)

See the post in the sequence "37 Ways That Words Can Be Wrong" called "Sneaking in Connotations. It seems to be roughly what you're talking about.

Here's a potential existential risk. Suppose a chemical is used for some task or made as a byproduct of another task, especially one that is spread throughout the atmosphere. Additionally, suppose it causes sterility, but it takes a very long time to cause sterility. Perhaps such a chemical could attain widespread use before its deleterious effects are discovered, and by then, it would have already sterilized everyone, potentially causes an existential catastrophe. I know this scenario for causing an existential catastrophe seems very small compared to other risks, but is it worthy of consideration?

Perhaps it would be beneficial to introduce life to Mars in the hope that it could eventually evolve into intelligent life in the event that Earth becomes sterilized. There are some lifeforms on Earth that could survive on Mars. The outer space treaty would need to be amended to make this legal, though, as it currently prohibits placing life on Mars. That said, I find it doubtful that intelligent life ever would evolve from the microbes, given how extreme Mar's conditions are.

What are some recommended readings for those who want to decrease existential risk? I know Nick Bostrom's book Superintelligence, How can I reduce existential risk from AI?, and MIRI's article Reducing Long-Term Catastrophic Risks from Artificial Intelligence are useful, but what else? What about non-AI-related existential risks?

I've been reading the discussion between Holden et al on the utility of charities aimed at directly decreasing existential risk, but the discussion seems to have ended prematurely. It (basically) started with this post, then went to this post. Holden made a comment addressing the post, but I think it didn't fully address the post and I don't think Holden's comment was fully addressed either. Is there any place that continues the discussion?

That said, I don't accept any of the arguments given here for why it's unacceptable to assign a very low probability to a proposition. I think there is a general confusion here between "low subjective probability that a proposition is correct" and "high confidence that a proposition isn't correct"; I don't think those two things are equivalent.

I don't think you've really explained why you don't accept the arguments in the post. Could you please explain why and how the difference between assigning low probability to something and havi... (read more)

Karnofsky has, as far as I know, not endorsed measures of charitable effectiveness that discount the utility of potential people.

Actually, according to this transcript on page four, Holden finds that the claim that the value of creating a life is "some reasonable" ratio of the value of saving a current life is very questionable. More exactly, the transcript sad:

Holden: So there is this hypothesis that the far future is worth n lives and this causing this far future to exist is as good as saving n lives. That I meant to state as an accurate

I don't think you were being obtuse. Your post wasn't bad per se, it was just off-topic as, unless I am failing to interpret it, it doesn't really add anything new to rationality or applied rationality.

Also, in case you had trouble locating the open thread, just click "Discussion" at the top of the page, then click the link under "Latest Open Thread" on the right of the page.

I don't think Less Wrong discussion is the best place for poetry, though someone please correct me if I am mistaken.

You're absolutely right. I'm not sure how I missed or forgot about reading that.

The article said the leverage penalty "[penalizes] hypotheses that let you affect a large number of people, in proportion to the number of people affected." If this is all the leverage penalty does, then it doesn't matter if it takes 3^^^3 atoms or units of computation, because atoms and computations aren't people.

That said, the article doesn't precisely define what the leverage penalty is, so there could be something I'm missing. So, what exactly is the leverage penalty? Does it penalize how many units of computation, rather than people, you ca... (read more)

What if the mugger says he will give you a single moment of pleasure that is 3^^^3 times more intense than a standard good experience? Wouldn't the leverage penalty not apply and thus make the probability of the mugger telling the truth much higher?

I think the real reason the mugger shouldn't be given money is that people are more likely to be able to attain 3^^^3 utils by donating the five dollars to an existential risk-reducing charity. Even though the current universe presumably couldn't support 3^^^3 utils, there is a chance of being able to create or ... (read more)

The second problem can easily be explained by having your utility function not be linear in the number of non-destroyed universes.

Is Solomonoff induction a theorem for making optimal probability distributions or a definition of them? That is to say, has anyone proved that Solomonoff induction produces probability distributions that are "optimal," or was Solomonoff induction created to formalize what it means for a prediction to be optimal. In the former case, how could they define optimality?

(And another question: I posted this a couple days ago on the last open thread, but it was closed before I got a response. Is it okay to repost it?)