All of jlp's Comments + Replies

I'm not sure "status conflict" is the only possibility here; for example, the terminal value might be something like autonomy, or feeling genuinely listened to.

A few thoughts:

It sounds like he's being rebellious. Separate the rebelliousness from the question of profanity, and discuss them separately. You might say something like "Asking questions if you genuinely want to understand something better is great, but asking questions to try to frustrate or annoy me is not. I'm getting the sense that you're doing the latter."

If he persists, put your foot down -- but be really clear that it's for the intent to annoy you, rather than because he's asking questions in an attempt to honestly understand somethin... (read more)

2ChristianKl
Telling someone who tries to wage status conflicts with you that you want him to stop fighting for more status is pretty pointless.
0James_Miller
He is occasionally openly rebellious on a small scale. I suspect that being rebellious towards parents is a terminal value for many 10-year-old boys. Good idea.

Yeah, there are plenty of examples of dictators that go through great lengths to inflict tremendous amounts of pain on many people. It's terrifying to think of someone like that in control of an AGI.

Granted, people like that probably tend to be less likely than the average head-of-state to find themselves in control of an AGI, since brutal dictators often have unhealthy economies, and are therefore unlikely to win an AGI race. But it's not like they have a monopoly on revenge or psychopathy either.

3Houshalter
I think sociopaths are about 4% of the population, so your scenario isn't really that implausible. I just meant if all of societies' values change over time. Or just the FAI extracting out "true" utility function which includes all the negative stuff, like desire for revenge.

Practical problem #3: The agency successfully understands your intentions, and is willing to implement them, but not able to implement them.

For example, a fast intelligence explosion removes their capability of doing so before they can pull the plug. Or a change in their legal environment makes it illegal for them to pull the plug (and they aren't willing to put themselves at legal risk to do so).

Uh, plenty of born are born into worse-than-death situations already, at least by our standards, yet they generally make a go of their lives instead of committing suicide. We call many of them our "ancestors."

Can you elaborate? Your statement seems self-contradictory. By definition, situations "worse than death" would be the ones in which people prefer to kill themselves rather than continue living.

In the context of the original post, I take "worse-than-death" to mean (1) enough misery that a typical person would rather n... (read more)

3Matthew_Opitz
I'm guessing the author meant that the ancestral environment was one that many of us now would consider "worse than death" considering our higher standards of expectation for standard of living, whereas our ancestors were just perfectly happy to live in cold caves and die from unknown diseases and whatnot. I guess the question is, how much higher are our expectations now, really? And how much better do we really have it now, really? Some things, like material comfort and feelings of material security, have obviously gotten better, but others, such as positional social status anxiety and lack of warm social conviviality, have arguably gotten worse.

I don't put any stock in the scary scenarios where an evil Omega tortures a gazillion of my revived clones for eternity.

Could you elaborate on this? I'd be curious to hear your reasoning.

Does "don't put any stock" mean P(x) = 0? 0.01? 1e-10?

5Shmi
It means the noise level, down there with Pascal's Wager/Mugger and fairy tales coming true. Assigning a number to it would mean giving in to Pascal's Mugging.