Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Comment author: Val 09 March 2017 07:59:48PM *  2 points [-]

Even with the limited AGI with very specific goals (build 1000 cars) the problem is not automatically solved.

The AI might deduce that if humans still exist, there is a higher than zero probability that a human will prevent it from finishing the task, so to be completely safe, all humans must be killed.

Comment author: username2 15 February 2017 12:01:43AM *  0 points [-]

Because there are plenty of all-seeing eye superpowers in this world. Not everyone is convinced that the very real, very powerful security regimes around the world would be suddenly left inept when the opponent is a computer instead of a human being.

My comment didn't contribute any less than yours to the discussion, which is rather the point. The validity of an allegory depends on the accuracy of the setup and rules, not the outcome. You seemed happy to engage until it was pointed out that the outcome was not what you expected.

Comment author: Val 15 February 2017 06:21:48PM 0 points [-]

Those "very real, very powerful security regimes around the world" are surprisingly inept at handling a few million people trying to migrate to other countries, and similarly inept at handling the crime waves and the political fallout generated by it.

And if you underestimate how much a threat could a mere "computer" be, read the "Friendship is Optimal" stories.

Comment author: Val 08 January 2017 11:44:42PM *  4 points [-]

This is a well-presented article, and even though most (or maybe all) of the information is easily available else-where, this is a well-written summary. It also includes aspects which are not talked about much, or which are often misunderstood. Especially the following one:

Debating the beliefs is a red herring. There could be two groups worshiping the same sacred scripture, and yet one of them would exhibit the dramatic changes in its members, white the other would be just another mainstream faith with boring compartmentalizing believers; so the difference is clearly not the scripture itself.

Indeed, the beliefs are not even close to be among the most important aspects of a cult. A cult is not merely a group which believes in something you personally find ridiculous. A cult can even have a stated core belief which is objectively true, or is a universally accepted good thing, like protecting the environment or world peace.

Comment author: Fluttershy 05 January 2017 12:20:00AM 5 points [-]

It helps that you shared the dialogue. I predict that Jane doesn't System-2-believe that Trump is trying to legalize rape; she's just offering the other conversation participants a chance to connect over how much they don't like Trump. This may sound dishonest to rationalists, but normal people don't frown upon this behavior as often, so I can't tell if it would be epistemically rational of Jane to expect to be rebuffed in the social environment you were in. Still, making claims like this about Trump may be an instrumentally rational thing for Jane to do in this situation, if she's looking to strengthen bonds with others.

Jane's System 1 is a good bayesian, and knows that Trump supporters are more likely to rebuff her, and that Trump supporters aren't social allies. She's testing the waters, albeit clumsily, to see who her social allies are.

Jane could have put more effort into her thoughts, and chosen a factually correct insult to throw at Trump. You could have said that even if he doesn't try to legalize rape, then he'll do some other specific thing that you don't approve of (and you'd have gotten bonus points for proactively thinking of a bad thing to say about him). The implementation of either of these changes would have had a roughly similar effect on the levels of nonviolence and agreeability of the conversation.

This generalizes to most conversations about social support. When looking for support, many people switch effortlessly between making low effort claims they don't believe, and making claims that they System-2-endorse. Agreeing with their sensible claims, and offering supportive alternative claims to their preposterous claims, can mark you as a social ally while letting you gently, nonviolently nudge them away from making preposterous claims.

Comment author: Val 05 January 2017 09:56:12PM *  1 point [-]

This comment was very insightful, and made me think that the young-earth creationist I talked about had a similar motivation. Despite this outrageous argument, she is a (relatively speaking) smart and educated person. Not academic-level, but neither grown up on the streets level.

Comment author: Val 03 January 2017 09:36:27PM *  2 points [-]

I always thought the talking snakes argument was very weak, but being confronted by a very weird argument from a young-earth creationist provided a great example for it:

If you believe in evolution, why don't you grow wings and fly away?

The point here is not about the appeal to ridicule (although it contains a hefty dose of that too). It's about a gross misrepresentation of a viewpoint. Compare the following flows of reasoning:

  • Christianity means that snakes can talk.
  • We can experimentally verify that snakes cannot talk.
  • Therefore, Christianity is false.


  • Evolution means people can spontaneously grow wings.
  • We can experimentally verify that people cannot spontaneously grow wings.
  • Therefore, evolution is false.

The big danger in this reasoning is that one can convince oneself of having used the experimental method, or of having been a rationalist. Because hey, we can scientifically verify the claim! - Without realizing that the verified claim is very different from the claims the discussed viewpoint actually holds.

I've even seen many self-proclaimed "rationalists" fall into this trap. Just as many religious people are reinforced by a "pat on the back" from their peers if they say something which is liked by the community they are in, so can people feel motivated to claim they are rationalists if that causes a pat on the back from people they interact with the most.

Comment author: Viliam 02 January 2017 03:45:23PM *  9 points [-]

A consequence of availability bias: the less you understand what other people do, the easier "in principle" it seems.

By "in principle" I mean that you wouldn't openly call it easy, because the work obviously requires specialized knowledge you don't have, and cannot quickly acquire. But it seems like for people who already have the specialized knowledge, it should be relatively straightforward.

"It's all just a big black box for me, but come on, it's only one black box, don't act like it's hundreds of boxes."

as opposed to:

"It's a transparent box with hundreds of tiny gadgets. Of course it takes a lot of time to get it right!"

Comment author: Val 03 January 2017 08:58:11PM 2 points [-]

Isn't this very closely related to the Dunning-Kruger effect?

Comment author: Val 29 November 2016 03:37:03PM *  2 points [-]

I'm not surprised Dawkins makes a cameo in it. The theist in the discussion is a very blunt strawman, just as Dawkins usually likes to invite the dumbest theists he can find, who say the stupidest things about evolution or global warming, thereby allegedly proving all theists wrong.

I'm sorry if I might have offended Dawkins, I know many readers here are a fan of him. However, I have to state that although I have no doubts about the values of his scientific work and his competence in his field, he does make a clown of himself with all those stawman attacks against theism.

Comment author: Val 06 November 2016 10:57:52AM 3 points [-]

For many people, religion helps a lot in replenishing willpower. Although, what I observed, it's less about stopping procrastination, and more about not despairing in a difficult or depressing situation. I might even safely guess that for a lot of believers this is among the primary causes of their beliefs.

I know that religious beliefs on this site are significantly below the offline average, I didn't want to convince anyone of anything, I just pointed out that for many people it helps. Maybe by acknowledging this fact we might understand why.

Comment author: ernestdezoe 25 October 2016 05:32:43PM *  0 points [-]

Unless the extraterrestrial species are the only macroscopic life-form on their planet, it's likely they evolved among other species and did not exterminate them all. This might lead to them having cultural values about preserving biodiversity and not exterminating species unless really necessary.

To me that's not a culture , but a bias (the hunter gatherer bias).....there are thousands of animal species serving no real purpose for our cause and still we slow down our growth because of concerns regarding their survival , not only that , but after having analyzed our daily values and necessities it becomes perfectly crystal clear how we'd only really need the 5 big crops + plants for photosynthesis , insects and impollinators in order to survive and thrive , plus we would be able to support much more people ! Imagine a planet where 15 billions humans live and each and everyone of them consumes 2700 kcal/day and contributes to the world's economy because nobody has to suffer hunger anymore.... that would be possible if we got rid of wastes and inefficiencies . So In my opinion if we ever find other forms of intelligent life and we can't trade with them , eat them , learn from them or acquire knowledge studying them , then yes I am all up for bombing them , just as I am all up for (and I know many will hate me for this :-D ) running a railway + HVDC line through the giant panda's territory , or finally get rid of domesticated animals like cows which convert calories and proteins from grains so poorly .

Also I agree with @woodchopper , we should stop sending messages literally "Across the Universe" in order to avoid perishing . An other approach we might use in the remote future could be only using old technologies to broadcast an "hello signal"..... stuff we've long moved from , so we could try to select for civilizations which are way behind us technologically so we could sort of be in control of their destiny like your usual anthill , but even then it could be a trap or they might catch up during the time necessary to make the trip or they could be monitored by some other advanced civilization which is not monitoring us , so we would just signal our presence to them as well...

Comment author: Val 25 October 2016 06:48:25PM *  1 point [-]

we'd only really need the 5 big crops + plants for photosynthesis , insects and impollinators in order to survive and thrive

Time and time it turned out that we underestimated the complexity of the biosphere. And time and time again our meddling backfired horribly.

Even if we were utterly selfish and had no moral objections, wiping out all but a handful of "useful" species would almost certainly lead to unforeseen consequences ending in the total destruction of the planet's biosphere. We did not yet manage to fully map the role each species plays in the natural balance, but it seems like it's very deeply entangled, everything depending on lots of other species. You cannot just remove a handful of them and expect them to thrive on their own.

Comment author: Lumifer 25 October 2016 06:06:22PM *  4 points [-]

You misunderstood my point.

The Europeans did not "proceed with a controlled extermination of the population". Yet, what happened to that population?

You don't need to start with a deliberate decision to exterminate in order to end up with almost none of the original population. Sometimes you just need to not care much.

Comment author: Val 25 October 2016 06:17:51PM 1 point [-]

True, the scenario is not implausible for a non-hostile alien civilization to arrive who are more efficient than us, and in the long term they will out-compete and out-breed us.

Such non-hostile assimilation is not unheard of in real life. It is happening now (or at least claimed by many to be happening) in Europe, both in the form of the migrant crisis and also in the form of smaller countries fearing that their cultural identities and values are being eroded by the larger, richer countries of the union.

View more: Next