I'm not sure if this is insightful enough to share here, but I'll try anyway.

A fair amount of wondering has been done about how FAI could figure out what humans actually want. A school of thought says persuasively that what we say we want is not what we actually want, so what we really want has to be extrapolated.

If we take the end-game promises of popular religions at face value, it occurs to me that Buddhism promises something between non-existence and wireheading (nirvana - "to blow out"), while Christianity promises wireheading (eternal bliss - heaven). I am not familiar enough with other religions to make statements about them.

In my anecdotal experience, it seems to me rationalists are quick to dismiss wireheading and non-existence as desirable possibilities. We experience this grasping desire to live, create, discover, and experience. We're not sure to what end, but we feel this indescribable zest, and we're convinced it's going to be great.

Look at people getting mental orgasms from Elon Musk launching a car into space. Whatever problems you have, space exploration is not going to solve them, and if your life is in harmony, you don't need space exploration. And yet there's this palpable zest about an unspoken implication... That perhaps an age of discovery is dawning, an age of adventure, an age of transcending our present problems and tackling larger ones. An age of being awesome.

There's a type of person that feels this zest, and this type is not a majority. The median person on Earth is confused by the world. They believe in things like Jesus Christ, and they press on in hope that adhering to divine guidance while they attempt to survive the trials and tribulations of life will be rewarded with not having to do this again. To such a person, the sight of two metal meteors descending from the sky with loud sonic booms, igniting engines and landing in synchrony does not necessarily inspire awe or enthusiasm as much as confusion and terror.

We, the few, are the seekers of something we cannot describe, and most of us find it hard to identify with the mindset of the median individual. The median individual does not want the excitement we seek, they just want an end... a release.

New to LessWrong?

New Comment
14 comments, sorted by Click to highlight new comments since: Today at 12:56 PM

There's a type of person that feels this zest, and this type is not a majority. The median person on Earth is confused by the world. They believe in things like Jesus Christ, and they press on in hope that adhering to divine guidance while they attempt to survive the trials and tribulations of life will be rewarded with not having to do this again. To such a person, the sight of two metal meteors descending from the sky with loud sonic booms, igniting engines and landing in synchrony does not necessarily inspire awe or enthusiasm as much as confusion and terror.

Were there really a lot of people in whom the SpaceX launch and the landing of the boosters inspired confusion and terror? I have not seen any of that. The reactions that I have observed have ranged all the way from disinterest to (as you put it) a palpable zest, but I have not observed anyone who felt terror or confusion.

The last paragraph, read without the context of the rest of the post, sounds like the monologue a movie villain would deliver to his girlfriend/accomplice immediately before savagely murdering a minor character he had just shown some affection.

A rationalist theology could include elements of most others, I think.

Eternal life away from your body? Working on the software, come back in a few decades. Mind melding with other human souls? Hardware is mostly there, but see above. Eternal bliss? We have it, just try to feed yourself. Extinguishment? Sure, go for it, that's old tech. All seeing entity that manipulates the world around you? Sup google. Hell if you don't convert? Basilisks don't exist.

As alway some interesting views and thinking get found here. Some of the statements I think I would push back on are: The median is confused. Well, I think it would be more accurate to say EVERYONE is confused if only because we're so limited in both our knowledge and any ability to observe so much of our reality on earth. Forget the metaphysical and philosophical/religous elements. Also, when suggesting confusion about somethings as complex as "the world" I'm not entirely sure there is a good common denominator to define not confused.

I think the characterization of most religous people as above -- and I'll cast it in the worst interprestation here -- as blindly hoping something will save them from bad shit and give them good things is just wrong. I've personally known a bunch of very religous people who are as rational or more rational than most athiests I've met. And, given that we simply don't know, strict atheism (as in a rejection of the monogod concept as reality) is as much a statementof faith and any belief in such an entity. But at least the religious will own their posistion as one of faith. Too many atheists will rebell against the acusation they, in the end, are makes statement based on the faith in their logic. Now, to be fair, more than a few "ateists" are really agnostics who simply say they don't find the arugements for a god convincing and use that as their day-to-day but acept they could be wrong. Why bring up this? It goes back to the assumption about who is and is not confused about the world.

What assumptions are loaded into the overal story here?

I'm not commenting to agree or disagree really, just saying something your post made me remember thinking about before...

I'm not convinced that the typical religious person has actually thought through the implications of the supposed end result of their religion. Even when confronted with the fact that wireheading or whatever is the end result, other tribal biases come into play.

A lot of optimism towards life and the future (vs old-world desire for a different, better heaven, or the Eastern desire for oblivion) comes from the fact that we are living in a golden age and, for many of those of us who have access to computers and higher learning, golden parts of the planet. For most of human history, life was proverbially nasty, brutish, and short: the shortness was a mercy. Even today, we age; we accumulate psychic injuries as we lose pets, friends, and family and as we accumulate injuries and indignities. It makes one tired, but in our golden lives we can stay proverbially awake longer because it just doesn't wear us out as fast.

We catch our breaths in nearly religious awe at those boosters because they represent a pinnacle in the things that make our lives golden: science represents real miracles to us. Our lives are objectively better than they were for our ancestors. A lot of people around the world, even now, don't have access to these benefits; if anything, the boosters and the beautiful car in space may represent to them exclusion and the bifurcation of the population. Why would they show joy? chances are high that neither they nor their children will see much of the benefit. It trickles down in first world countries to some extent, but less so the further removed one is from the epicenter of the scientific miracle.

I think my personal extrapolated volition is wireheading, and I'm definitely in the "Ooh science, what fun" camp.

But maybe I'm not very good at extrapolating, and an AI could come up with something better. It would, for instance, be great to be in a permanent state of bliss whilst actually acting in the world. Or in some private sandbox world occupied by myself and a few hundred friends.

I can't see how those possibilities would feel any different to me from a good wireheading though. And maybe the AI is trying to conserve resources.

This brings up an interesting ethical dilemma. If strong AI will ever be possible, it will be probably designed with the values of what you described as a small minority. Does this this small minority have the ethical right to enforce a new world upon the majority which will be against their values?

Wouldn't be the first time that a small minority were enacting the change they wanted. The universe is not ethical or anything.

That's true, but the change a strong AI would make would be probably completely irreversible and unmodifiable.

Is that different to how humans are the dominant planetary species?

The entire concept of CEV is meant to address this question.

https://arbital.com/p/cev/

Do you (or we) have the ethical right to enforce current world, the majority of which is against our values (as measured by the amount of complaining, at least).

I didn't say I had an answer. I only said it can be an interesting dilemma.

I'm not sure if this is insightful enough to share here, but I'll try anyway.

It sure is. Nice to see someone post something thought-provoking.