People do not behave as if we have utilities given by a particular numerical function that collapses all of their hopes and goals into one number, and machines need not do it that way, either.
I think this point is well said, and completely correct.
..
Why not also think about making other kinds of systems?
An AGI could have a vast array of hedges, controls, limitations, conflicting tendencies and tropisms which frequently cancel each other out and prevent dangerous action.
The book does scratch the surface on these issues, but it is not all about fail-safe mind design and managed roll-out. We can develop a whole literature on those topics.
I agree. I find myself continually wanting to bring up issues in the latter class of issues... so copiously so, that frequently it feels like I am trying to redesign our forum topic. So, I have deleted numerous posts-in-progress that fall into that category. I guess those of us who have ideas about fail-safe mind design that are more subtle -- or to put it more neutrally -- do not fit the running paradigm in which the universe of discourse is that of transparent, low-dimensional (low dimensional function range space, not low dimensional function domain space) utility functions, need to start writing our own white papers.
When I hear the Bostrom claims only 7 people in the world are thinking full time and productively about (in essence) fail safe mind design, or that someone at MIRI wrote only FIVE people are doing so (though in the latter case, the author of that remark did say that there might be others doing this kind of work "on the margin", whatever that means), I am shocked.
It's hard to believe, for one thing. Though, the people making those statements must have good reasons for doing so.
But maybe the deriviation of such low numbers could be more understandable, if one stipulates that "work on the problem" is to be counted if and only if candidate people belong to the equivalence class of thinkers restricting their approach to this ONE, very narrow conceptual and computational vocabulary.
That kind of utility function-based discussion (remember when they were called 'heuristics' in the assigned projects, in our first AI courses?) has its value, but it's a tiny slice of the possible conceptual, logical and design pie ... about like looking at the night sky through a soda straw. If we restrict ourselves to such approaches, no wonder people think it will take 50 or 100 years to do AI of interest.
Ourside of the culture of collapsing utility functions and the like, I see lots of smart (often highly mathematical, so they count as serious) papers in whole brain chaotic resonant neurodynamics; new approachs to foundations of mental health issues and disorders of subjective empathy (even some application of deviant neurodynamics to deviant cohort value theory, and defective cohort "theory of mind" -- in the neuropsychiatric and mirror neuron sense) that are grounded in, say, pathologies with transient Default Node Network coupling... and distrubances of phase coupled equilibria across the brain.
If we run out of our own ideas to use from scratch (which I don't think is at all the case ... as your post might suggest, we have barely scratched the surface), then we can go have a look at current neurology and neurobiology, where people are not at all shy about looking for "information processing" mechanisms underlying complex personality traits, even underlying value and aesthetic judgements.
I saw a visual system neuroscientist's paper the other day offering a theory of why abstract (ie. non-representational) art is so intriguing to (not all, but some) human brains. It was a multi-layered paper, discussing some transiently coupled neurodynamical mechanisms of vision (the authors' specialties), some reward system neuromodulator concepts, and some traditional concepts expressed at a phenomenological, psychological level of description. An ambitious paper, yes!
But ambition is good. I keep saying, we can't expect to do real AI on the cheap.
A few hours or days reading such papers is good fertilizer, even if we do not seek to translate, in any direct way (like copying "algorithms" from natural brains) wetware brain research, into our goal, which presumably is to do dryware mind design --- and do it in a way where we choose our own functional limits, not have nature's 4.5 billion years of accidents choose boundary conditions on substrate platforms, for us.
Of course, not everyone is interested in doing this. I HAVE learned in this forum, that "AI" is a "big tent". Lots of uses exist for narrow AI, in thousands of indutries and fields. Thousands of narrow AI systems are already in play.
But, really... aren't most of us interested in this topic because we want the more ambitious result?
Bostrom says "we will not be concerned with the metaphysics of mind..." and "...not concern ourselves whether these entities have genuine self-awareness...."
Well, I guess we won't be BUILDING real minds anytime soon, then. One can hardly expect to create, that which one won't even openly discuss. Bostrom is wrting and speaking, using the language of "agency" and "goals" and "motivational sets", but he is only using those terms metaphorically.
Unless, that is, everyone else in here (other than me) actually is prepared to deny that we -- who spawned those concepts, to describe rich, conscious, intentionally entrained features of the lives of self-aware, genuine conscious creatures -- are different, i.e., that we are conscious and self-aware.
No one here needs a lesson in intellectual history. We all know that people did deny that , back in the behaviorism era. (I have studied the reasons -- philosophical and cultural -- and continue to uncover in great detail, mistaken assumptions out of which that intellectual fad grew.)
Only ff we do THAT again, will we NOT be using "agent" metaphorically, when we apply that to machines with no real consciousness, because ex hypothesi WE'd posess no minds either, in the sense we all know we do posess, as conscious humans.
We'd THEN be using it ('agent", "goal", "motive" ... the whole equivalence class of related nouns and predicates) in the same sense for both classes of entities (ourselves, and machines with no "awareness", where the latter is defined as anyting other than public, 3rd person observable behavior.)
Only in this case, would it not be a metaphor to use 'agent, motive', etc. in describing intelligent (but not conscious) machines, whcih evidently is the astringent conceptual model within which Bostrom wishes to frame HLAI --- proscribing considerations, as he does, of whether they are genuinely self-aware.
But, well, I always thought that that excessively positivistic attitude, had more than a little something to do with the "AI winter" (just like it is widely acknowledged to have been responsible for the neuroscience winter that paralleled it.)
Yet neuroscientists are not embarassed to now say, "That was a MISTAKE, and -- fortunately -- we are over it. We wasted some good years, but are no longer wasting time denying the existence of consciousness, the very thing that makes the brain interesting and so full of fundamental scientific interest. And now, the race is on to understand how the brain creates real mental states."
NEUROSCIENCE has gotten over that problem with discussing mental states qua mental states , clearly.
And this is one of the most striking about-faces in the modern intelllectual history of science.
So, back to us. What's wrong with computer science? Either AI-ers KNOW that real consciousness exists, just like neuroscientists do, and AI-ers just don't give a hoot about making machines that are actually conscious.
Or, AI-ers are afraid of tackling a problem that is a little more interesting, deeper, and harder (a challenge that gets thousands of neuroscientists and neurophilosophers up on the morning.)
I hope the latter is not true, because I think the depth and possibilities of the real thing -- AI with consciousnes -- are what gives it all the attraction (and holds, in the end, for reasons I won't attempt to desribe in a short post, the only possibility of making the things friendly, if not benificient.)
Isn't that what gives AI its real interest? Otherwise, why not just write business software?
Could it be that Bostrom is throwing out the baby with the bathwater, when he stipulates that the discussion, as he frames it, can be had (and meaningful progress made), without the interlocutors (us) being concerned about whether AIs have genuine self awareness, etc?
This is part of a weekly reading group on Nick Bostrom's book, Superintelligence. For more information about the group, and an index of posts so far see the announcement post. For the schedule of future topics, see MIRI's reading guide.
Welcome. This week we discuss the tenth section in the reading guide: Instrumentally convergent goals. This corresponds to the second part of Chapter 7.
This post summarizes the section, and offers a few relevant notes, and ideas for further investigation. Some of my own thoughts and questions for discussion are in the comments.
There is no need to proceed in order through this post, or to look at everything. Feel free to jump straight to the discussion. And if you are behind on the book, don't let it put you off discussing. Where applicable and I remember, page numbers indicate the rough part of the chapter that is most related (not necessarily that the chapter is being cited for the specific claim).
Reading: Instrumental convergence from Chapter 7 (p109-114)
Summary
Notes
1. Why do we care about CIVs?
CIVs to acquire resources and to preserve oneself and one's values play important roles in the argument for AI risk. The desired conclusions are that we can already predict that an AI would compete strongly with humans for resources, and also than an AI once turned on will go to great lengths to stay on and intact.
2. Related work
Steve Omohundro wrote the seminal paper on this topic. The LessWrong wiki links to all of the related papers I know of. Omohundro's list of CIVs (or as he calls them, 'basic AI drives') is a bit different from Bostrom's:
3. Convergence for values and situations
It seems potentially helpful to distinguish convergence over situations and convergence over values. That is, to think of instrumental goals on two axes - one of how universally agents with different values would want the thing, and one of how large a range of situations it is useful in. A warehouse full of corn is useful for almost any goals, but only in the narrow range of situations where you are a corn-eating organism who fears an apocalypse (or you can trade it). A world of resources converted into computing hardware is extremely valuable in a wide range of scenarios, but much more so if you don't especially value preserving the natural environment. Many things that are CIVs for humans don't make it onto Bostrom's list, I presume because he expects the scenario for AI to be different enough. For instance, procuring social status is useful for all kinds of human goals. For an AI in the situation of a human, it would appear to also be useful. For an AI more powerful than the rest of the world combined, social status is less helpful.
4. What sort of things are CIVs?
Arguably all CIVs mentioned above could be clustered under 'cause your goals to control more resources'. This implies causing more agents to have your values (e.g. protecting your values in yourself), causing those agents to have resources (e.g. getting resources and transforming them into better resources) and getting the agents to control the resources effectively as well as nominally (e.g. cognitive enhancement, rationality). It also suggests convergent values we haven't mentioned. To cause more agents to have one's values, one might create or protect other agents with your values, or spread your values to existing other agents. To improve the resources held by those with one's values, a very convergent goal in human society is to trade. This leads to a convergent goal of creating or acquiring resources which are highly valued by others, even if not by you. Money and social influence are particularly widely redeemable 'resources'. Trade also causes others to act like they have your values when they don't, which is a way of spreading one's values.
As I mentioned above, my guess is that these are left out of Superintelligence because they involve social interactions. I think Bostrom expects a powerful singleton, to whom other agents will be irrelevant. If you are not confident of the singleton scenario, these CIVs might be more interesting.
5. Another discussion
John Danaher discusses this section of Superintelligence, but not disagreeably enough to read as 'another view'.
Another view
I don't know of any strong criticism of the instrumental convergence thesis, so I will play devil's advocate.
In-depth investigations
If you are particularly interested in these topics, and want to do further research, these are a few plausible directions, some inspired by Luke Muehlhauser's list, which contains many suggestions related to parts of Superintelligence. These projects could be attempted at various levels of depth.
How to proceed
This has been a collection of notes on the chapter. The most important part of the reading group though is discussion, which is in the comments section. I pose some questions for you there, and I invite you to add your own. Please remember that this group contains a variety of levels of expertise: if a line of discussion seems too basic or too incomprehensible, look around for one that suits you better!
Next week, we will talk about the treacherous turn. To prepare, read “Existential catastrophe…” and “The treacherous turn” from Chapter 8. The discussion will go live at 6pm Pacific time next Monday 24th November. Sign up to be notified here.