LESSWRONG
LW

ThomasCederborg — LessWrong

Replying toA problem shared by many different alignment targets

A problem shared by many different alignment targets

It seems to me that we are going in circles and talking past each other to some degree in the discussion above. So I will just briefly summarise my position on the main topics that you raise (I've argued for these positions above. Here I'm just summarising). And then I will give a short outline of the argument for analysing Sovereign AI proposals now.

Regarding the relative priority of different research efforts:

The type of analysis that I am doing in the post is designed to reduce one of the serious AI risks that we face. This risk is due to a combination of the fact that (i): we might end up with a... (read 1155 more words →)

Replying toWhy I am not a successionist

ThomasCederborg9mo

Why I am not a successionist

If I understand your comment correctly, then I think that the claim about preference variation made in the second sentence is wrong. Specifically, I think that a significant fraction of people do have strong preferences, such that fulfilling those preferences would be very bad for a significant fraction of people. Some people would for example strongly prefer that earths ecosystems be destroyed (as a way of preventing animal suffering). Others would strongly prefer to protect earths ecosystems. The partitioning idea that you mention would be irrelevant to this value difference. There also exists no level of resources that would make this disagreement go away. (This seems to me to be in contradiction... (read 961 more words →)

Replying toA problem shared by many different alignment targets

ThomasCederborg10mo

A problem shared by many different alignment targets

The successful implementation of an instruction following AI would not remove the possibility that an AI Sovereign will be implemented later. For example: the path to an AI that implements the CEV of Humanity outlined on the CEV arbital page starts with an initial non-Sovereign AI (and this initial AI could be the type of instruction following AI that you mention). In other words: the successful implementation of an instruction following AI does not prevent the later implementation of a Group AI. It is in fact one step on the classical path to a Group AI. So even if we assume that the first clever AI will be an instruction following AI, then this... (read 3488 more words →)

Replying toA problem shared by many different alignment targets

ThomasCederborg1y

A problem shared by many different alignment targets

The proposals described in your points 1 and 2 are very different from any of the proposals that I am analysing in the post. I consider this to be a good thing. But I wanted to note explicitly that this discussion has now moved very far away from what was discussed in the post, and is best seen as a new discussion (a discussion that starts with the proposals described in your points 1 and 2). Making this clear is important, because it means that many points made in the post (and also earlier in this thread) do not apply to the class of proposals that we are now discussing.

In particular: all... (read 4287 more words →)

Replying toA problem shared by many different alignment targets

ThomasCederborg1y

A problem shared by many different alignment targets

There are no Pareto improvements relative to the new Pareto Baseline that you propose. Bob would indeed classify a scenario with an AI that takes no action as a Dark Future. However, consider Bob2, who takes another perfectly coherent position on how to classify an AI that never acts. If something literally never takes any action, then Bob2 simply does not classify it as a person. Bob2 therefore does not consider a scenario with an AI that literally never does anything to be a Dark Future (other than this difference, Bob2 agrees with Bob about morality). This is also a perfectly reasonable ontology. A single person like Bob2 is enough to make... (read 863 more words →)

Replying toA problem shared by many different alignment targets

ThomasCederborg1y

A problem shared by many different alignment targets

Given that you agreed with most of what I said in my reply, it seems like you should also agree that it is important to analyse these types of alignment targets. But in your original comment you said that you do not think that it is important to analyse these types of alignment targets.

Let's write Multi Person Sovereign AI Proposal (MPSAIP) for an alignment target proposal to build an AI Sovereign that gets its goal from the global population (in other words: the type of alignment target proposals that I was analysing in the post). I followed your links and can only find one argument against the urgency of analysing MPSAIPs now: that... (read 3186 more words →)

Replying toA problem shared by many different alignment targets

ThomasCederborg1y

A problem shared by many different alignment targets

I'm sorry if the list below looks like nitpicking. But I really do think that these distinctions are important.

Bob holds 1 as a value. Not as a belief.

Bob does not hold 2 as a belief or as a value. Bob thinks that someone as powerful as the AI has an obligation to punish someone like Dave. But that is not the same as 2.

Bob does not hold 3 as a belief or as a value. Bob thinks that for someone as powerful as the AI, the specific moral outrage in question renders the AI unethical. But that is not the same as 3.

Bob does hold 4 as a value. But it is... (read 2812 more words →)

Replying toA problem shared by many different alignment targets

ThomasCederborg1y

A problem shared by many different alignment targets

Bob really does not want the fate of the world to be determined by an unethical AI. There is no reason for such a position to be instrumental. For Bob, this would be worse than the scenario with no AI (in the Davidad proposal, this is the baseline that is used to determine whether or not something is a Pareto-improvement). Both scenarios contain non-punished heretics. But only one scenario contains an unethical AI. Bob prefers the scenario without an unethical AI (for non-instrumental reasons).

Regarding extrapolation:

The question is whether or not at least one person will continue to view a non-punishing AI as unethical after extrapolation. (When determining whether or not something is... (read 695 more words →)

Replying toA problem shared by many different alignment targets

ThomasCederborg1y

A problem shared by many different alignment targets

Consider Bob, who takes morality very seriously. Bob thinks that any scenario where the fate of the world is determined by an unethical AI, is worse than the scenario with no AI. Bob sticks with this moral position, regardless of how much stuff Bob would get in a scenario with an unethical AI. For a mind as powerful as an AI, Bob considers it to be a moral imperative to ensure that heretics do not escape punishment. If a group contains at least one person like Bob (and at least one person that would strongly object to being punished), then the set of Pareto-improvements is empty. In a population of billions, there will always exist at least some people with Bob's type of morality (and plenty of people that would strongly object to being punished). Which in turn means that for humanity, there exist no powerful AI, such that creating this AI would be a Pareto-improvement.

A problem shared by many different alignment targets

ThomasCederborg

The first section describes problems with a few different alignment targets. The second section argues that it is useful to view all of them as variations of a single alignment target: building an AI that does what a Group wants that AI to do. The post then goes on to argue that all of the individual problems described in the first section can be viewed as special cases of an underlying problem. A problem that is inherent in the core concept of a Group AI.

The implication of the first section is that the specific alignment targets analysed there should not be implemented. The rest of the post is more ambitious. The following... (read 10737 more words →)

Shutting down all competing AI projects might not buy a lot of time due to Internal Time Pressure

ThomasCederborg

Summary

A previous post argued that work on Alignment Target Analysis (ATA) needs to start now in order to reduce the probability of a bad alignment target getting successfully implemented. The present post is focused on one specific scenario where starting ATA work now would reduce the probability of disaster. Different types of time crunch can lead to situations where there will not be much time to do ATA later. We will make some optimistic assumptions in order to focus on one specific type of time crunch that remains despite these assumptions. We will assume a human augmentation project that leads to smarter humans. Then we assume the creation of a Limited AI... (read 3521 more words →)

Replying toThe case for more Alignment Target Analysis (ATA)

ThomasCederborg1y

The case for more Alignment Target Analysis (ATA)

Your comment makes me think that I might have been unclear regarding what I mean with ATA. The text below is an attempt to clarify.

Summary

Not all paths to powerful autonomous AI go through methods from the current paradigm. It seems difficult to rule out the possibility that a Sovereign AI will eventually be successfully aligned to some specific alignment target. At current levels of progress on ATA this would be very dangerous (because understanding an alignment target properly is difficult, and a seemingly-nice proposal can imply a very bad outcome). It is difficult to predict how long it would take to reach the level of understanding needed to prevent scenarios where a... (read 2347 more words →)

The case for more Alignment Target Analysis (ATA)

Chi Nguyen

Chi Nguyen, ThomasCederborg

Summary

We don’t have good proposals for alignment targets: The most recently published version of Coherent Extrapolated Volition (CEV), a fairly prominent alignment target, is Parliamentarian CEV (PCEV). PCEV gives a lot of extra influence to anyone who intrinsically values hurting other individuals (search the CEV arbital page for ADDED 2023 for Yudkowsky’s description of the issue). This feature went unnoticed for many years and would make a successfully implemented PCEV very dangerous.
Bad alignment target proposals are dangerous: There is no particular reason to think that discovery of this problem was inevitable. It went undetected for many years. There are also plausible paths along which PCEV (or a proposal with a similar issue) might have

... (read 5027 more words →)

A necessary Membrane formalism feature

ThomasCederborg

Summary

Consider a Membrane that is supposed to protect Steve. An AI wants to hurt Steve but does not want to pierce Steve's Membrane. The Membrane ensures that there is zero effect on predictions of things inside the membrane. The AI will never take any action that has any effect on what Steve does or experience. The Membrane also ensures that the AI will not have access to any information that is within Steve's Membrane. One does not have to be a clever AI to come up with a strategy that an AI could use to hurt Steve without piercing this type of Membrane. The AI could for example create and hurt minds... (read 3255 more words →)

Corrigibility could make things worse

ThomasCederborg

Summary: A Corrigibility method that works for a Pivotal Act AI (PAAI) but fails for a CEV style AI could make things worse. Any implemented Corrigibility method will necessarily be built on top of a set of unexamined implicit assumptions. One of those assumptions could be true for a PAAI, but false for a CEV style AI. The present post outlines one specific scenario where this happens. This scenario involves a Corrigibility method that only works for an AI design, if that design does not imply an identifiable outcome. The method fails when it is applied to an AI design, that does imply an identifiable outcome. When such an outcome does exist,... (read 1502 more words →)

The proposal to add a ``Last Judge'' to an AI, does not remove the urgency, of making progress on the ``what alignment target should be aimed at?'' question.

ThomasCederborg

Currently, there exist no viable proposed answer, to the ``what alignment target should be aimed at?'' question. There also exist no reliable way of distinguishing a good alignment target, from an alignment target, such that successfully hitting this alignment target, would result in an outcome, that would be far, far, worse than extinction. (see my previous post: A problem with the most recently published version of CEV ). This means that making progress on the ``what alignment target should be aimed at?'' question is urgent by default. There also exist no reliable way of preventing a scenario, where someone successfully hits an alignment target, without properly analysing it. For example because the... (read 5211 more words →)

Making progress on the ``what alignment target should be aimed at?'' question, is urgent

ThomasCederborg

There exists a class of AI proposals, where some AI is supposed to undergo an I. J. Good style intelligence explosion, and then defer the decision, of what alignment target, will eventually be hit, to someone else. One such proposal is the Pivotal Act AI (PAAI), proposed by ``person A'', in this post by Wei Dai Meta Questions about Metaphilosophy (this specific PAAI defers the decision to a group of uploads). Another example of this general type of AI, could be the result of some coalition of powerful governments, insisting on a Political Process AI (PPAI): an AI that defers the decision to a pre specified political process, designed to not ``disrupt... (read 5289 more words →)

A problem with the most recently published version of CEV

ThomasCederborg

This post outlines a thought experiment, designed to illustrate a problem with the Parliamentarian version of CEV (PCEV). It then shows that the problem is not inherent in the scenario of the thought experiment, by describing a large space of other designs, that does not suffer from this problem (based on modifying PCEV, in a way that gives each individual, meaningful influence, over the adoption of those preferences, that refer to her).

The most recently published version of CEV, uses Parliamentarianism to deal with disagreements, among individuals, that disagree on how to deal with disagreements. Let's refer to the AI, that implements this Parliamentarian definition, of the CEV of humanity, as PCEV. The... (read 2108 more words →)