All of AmberDawn's Comments + Replies

As far as current systems are concerned, the answer is that (as far as anyone knows) they don't find things rewarding or want things. But they can still run a search to optimize a training signal, and that gives you an agent.

At a very high level, the way reinforcement learning works is that the AI attempts to maximise a reward function. This reward function can be summed up as "The sum of all rewards you expect to get in the future". So using a bunch of maths, the AI looks at the rewards it's got in the past, the rewards it expects to get in the future, and selects the action that maximises the expected future rewards. The reward function can be defined within the algorithm itself, or come from the environment. For instance, if you want to train a four-legged robot to learn to walk, the reward might be the distance travelled in a certain direction. If you want to train it to play an Atari game, the reward is usually the score. None of this requires any sort of qualia, or for the agent to want things. It's a mathematical equation. AI behaves in the way it behaves as a result of the algorithm attempting to maximise it, and the AI can be said to "want" to maximise its reward function or "have the goal of" maximising its reward function because it reliably takes actions to move towards this outcome if it's a good enough AI.

Meta: There's an AI Governance tag and a Regulation and AI Risk tag   My own (very limited) understanding is: 1. Asking people not to build AI is like asking them to give up a money machine, almost 2. We need everyone to agree to stop 3. There is no clear line. With an atom bomb, it is pretty well defined if you sent it or not. It's much more vague with "did you do AI research?" 1. It's pretty easy to notice if someone sent an atom bomb. Not so easy to notice if they researched AI 4. AI research is getting cheaper. Today only a few actors can do it, but notice, there are already open source versions of gpt-like models. How long could we hold it back? 5. Still, people are trying to do things in this direction, and I'm pretty sure that the situation is "try any direction that seems at all plausible"

My comment-box got glitchy but just to add: this category of intervention might be a good thing to do for people who care about AI safety and don't have ML/programming skills, but do have people skills/comms skills/political skills/etc.  Maybe lots of people are indeed working on this sort of thing, I've just heard much less discussion of this kind of solution relative to technical solutions.

Is it intuitive to you why a calculator can sum numbers even though it doesn't want/feel anything? If so, and if an AGI still feels confusing, could you help me pin point the difference and I'll continue from there? ( +1 for the question!)

Assume you have a very simple reinforcement learning AI that does nothing but chooses between two actions, A and B. And it has a goal of "maximizing reward". "Reward", in this case, doesn't correspond to any qualia; rather "reward" is just a number that results from the AI choosing a particular action. So what "maximize reward" actually means in this context is "choose the action that results in the biggest numbers". Say that the AI is programmed to initially just try choosing A ten times in a row and B ten times in a row.  When the AI chooses A, it is shown the following numbers: 1, 2, 2, 1, 2, 2, 1, 1, 1, 2 (total 15). When the AI chooses B, it is shown the following numbers: 4, 3, 4, 5, 3, 4, 2, 4, 3, 2 (total 34). After the AI has tried both actions ten times, it is programmed to choose its remaining actions according to the rule "choose the action that has historically had the bigger total". Since action B has had the bigger total, it then proceeds to always choose B. To achieve this, we don't need to build the AI to have qualia, we just need to be able to build a system that implements a rule like "when the total for action A is greater than the total for action B, choose A, and vice versa; if they're both equal, pick one at random". When we say that an AI "is rewarded", we just mean "the AI is shown bigger numbers, and it has been programmed to act in ways that result in it being shown bigger numbers".  We talk about the AI having "goals" and "wanting" things by an application of the intentional stance. That's Daniel Dennett's term for the idea that, even if a chess-playing AI had a completely different motivational system than humans do (and chess-playing AIs do have that), we could talk about it having a "goal" of "wanting" to win at chess. If we assume that the AI "wants" to win the chess, then we can make more accurate predictions of its behavior - for instance, we can assume that it won't make moves that are obviously losing moves if it can avoid

Reinforcement Learning is easy to conceptualize. The key missing ingredient is that we explicitly specify algorithms to maximize the reward. So this is disanalogous to humans: to train your 5yo, you need only give the reward and the 5yo may adapt their behavior because they value the reward; in a reinforcement learning agent, the second step only occurs because we make it occur. You could just as well flip the algorithm to pursue minimal rewards instead.

Functionally. You can regard them all as form of behaviour. do they depend on qualia, or are they just accompanied by qualia?