Intelligence explosion in organizations, or why I'm not worried about the singularity

sbenthall

13 Intelligence explosion in organizations, or why I'm not worried about the singularity

27th Dec 2012

4 min read

13

If I understand the Singularitarian argument espoused by many members of this community (eg. Muehlhauser and Salamon), it goes something like this:

Machine intelligence is getting smarter.
Once an intelligence becomes sufficiently supra-human, its instrumental rationality will drive it towards cognitive self-enhancement (Bostrom), so making it a super-powerful, resource hungry superintelligence.
If a superintelligence isn't sufficiently human-like or 'friendly', that could be disastrous for humanity.
Machine intelligence is unlikely to be human-like or friendly unless we take precautions.

I am not particularly worried about the scenario envisioned in this argument. I think that my lack of concern is rational, so I'd like to try to convince you of it as well.*

It's not that I think the logic of this argument is incorrect so much as I think there is another related problem that we should be worrying about more. I think the world is already full of probably unfriendly supra-human intelligences that are scrambling for computational resources in a way that threatens humanity.

I'm in danger of getting into politics. Since I understand that political arguments are not welcome here, I will refer to these potentially unfriendly human intelligences broadly as organizations.

Smart organizations

By "organization" I mean something commonplace, with a twist. It's commonplace because I'm talking about a bunch of people coordinated somehow. The twist is that I want to include the information technology infrastructure used by that bunch of people within the extension of "organization".

Do organizations have intelligence? I think so. Here's some of the reasons why:

We can model human organizations as having preference functions. (Economists do this all the time)
Human organizations have a lot of optimization power.

I talked with Mr. Muehlhauser about this specifically. I gather that at least at the time he thought human organizations should not be counted as intelligences (or at least as intelligences with the potential to become superintelligences) because they are not as versatile as human beings.

So when I am talking about super-human intelligence, I specifically mean an agent that is as good or better at humans at just about every skill set that humans possess for achieving their goals. So that would include things like not just mathematical ability or theorem proving and playing chess, but also things like social manipulation and composing music and so on, which are all functions of the brain not the kidneys

...and then...

It would be a kind of weird [organization] that was better than the best human or even the median human at all the things that humans do. [Organizations] aren’t usually the best in music and AI research and theory proving and stock markets and composing novels. And so there certainly are [Organizations] that are better than median humans at certain things, like digging oil wells, but I don’t think there are [Organizations] as good or better than humans at all things. More to the point, there is an interesting difference here because [Organizations] are made of lots of humans and so they have the sorts of limitations on activities and intelligence that humans have. For example, they are not particularly rational in the sense defined by cognitive science. And the brains of the people that make up organizations are limited to the size of skulls, whereas you can have an AI that is the size of a warehouse.

I think that Muehlhauser is slightly mistaken on a few subtle but important points. I'm going to assert my position on them without much argument because I think they are fairly sensible, but if any reader disagrees I will try to defend them in the comments.

When judging whether an entity has intelligence, we should consider only the skills relevant to the entity's goals.
So, if organizations are not as good at a human being at composing music, that shouldn't disqualify them from being considered broadly intelligent if that has nothing to do with their goals.
Many organizations are quite good at AI research, or outsource their AI research to other organizations with which they are intertwined.
The cognitive power of an organization is not limited to the size of skulls. The computational power is of many organizations is comprised of both the skulls of its members and possibly "warehouses" of digital computers.
With the ubiquity of cloud computing, it's hard to say that a particular computational process has a static spatial bound at all.

In summary, organizations often have the kinds of skills necessary to achieve their goals, and can be vastly better at them than individual humans. Many have the skills necessary for their own cognitive enhancement, since if they are able to raise funding they can purchase computational resources and fund artificial intelligence research. More mundanely, organizations of all kinds hire analysts and use analytic software to make instrumentally rational decisions.

In sum, many organizations are of supra-human intelligence and strive actively to enhance their cognitive powers.

Mean organizations

Suppose the premise that there are organizations with supra-human intelligence that act to enhance their cognitive powers. And suppose the other premises of the Singularitarian argument outlined at the beginning of this post.

Then it follows that we should be concerned if one or more of these smart organizations are so unlike human beings in their motivational structure that they are 'mean'.

I believe the implications of this line of reasoning may be profound, but as this is my first post to LessWrong I would like to first see how this is received before going on.

* My preferred standard of rationality is communicative rationality, a Habermasian ideal of a rationality aimed at consensus through principled communication. As a consequence, when I believe a position to be rational, I believe that it is possible and desirable to convince other rational agents of it.

Intelligence explosion

Personal Blog

13

New Comment

Rendering 0/187 comments, sorted by

top scoring

(show more) Click to highlight new comments since: Today at 10:59 PM

Some comments are truncated due to high volume. (⌘F to expand all)Change truncation settings

Moderation Log

13 Intelligence explosion in organizations, or why I'm not worried about the singularity

by sbenthall

27th Dec 2012

4 min read

187

13

If I understand the Singularitarian argument espoused by many members of this community (eg. Muehlhauser and Salamon), it goes something like this:

Machine intelligence is getting smarter.
Once an intelligence becomes sufficiently supra-human, its instrumental rationality will drive it towards cognitive self-enhancement (Bostrom), so making it a super-powerful, resource hungry superintelligence.
If a superintelligence isn't sufficiently human-like or 'friendly', that could be disastrous for humanity.
Machine intelligence is unlikely to be human-like or friendly unless we take precautions.

I am not particularly worried about the scenario envisioned in this argument. I think that my lack of concern is rational, so I'd like to try to convince you of it as well.*

Smart organizations

Do organizations have intelligence? I think so. Here's some of the reasons why:

We can model human organizations as having preference functions. (Economists do this all the time)
Human organizations have a lot of optimization power.

So when I am talking about super-human intelligence, I specifically mean an agent that is as good or better at humans at just about every skill set that humans possess for achieving their goals. So that would include things like not just mathematical ability or theorem proving and playing chess, but also things like social manipulation and composing music and so on, which are all functions of the brain not the kidneys

...and then...

It would be a kind of weird [organization] that was better than the best human or even the median human at all the things that humans do. [Organizations] aren’t usually the best in music and AI research and theory proving and stock markets and composing novels. And so there certainly are [Organizations] that are better than median humans at certain things, like digging oil wells, but I don’t think there are [Organizations] as good or better than humans at all things. More to the point, there is an interesting difference here because [Organizations] are made of lots of humans and so they have the sorts of limitations on activities and intelligence that humans have. For example, they are not particularly rational in the sense defined by cognitive science. And the brains of the people that make up organizations are limited to the size of skulls, whereas you can have an AI that is the size of a warehouse.

When judging whether an entity has intelligence, we should consider only the skills relevant to the entity's goals.
So, if organizations are not as good at a human being at composing music, that shouldn't disqualify them from being considered broadly intelligent if that has nothing to do with their goals.
Many organizations are quite good at AI research, or outsource their AI research to other organizations with which they are intertwined.
The cognitive power of an organization is not limited to the size of skulls. The computational power is of many organizations is comprised of both the skulls of its members and possibly "warehouses" of digital computers.
With the ubiquity of cloud computing, it's hard to say that a particular computational process has a static spatial bound at all.

In sum, many organizations are of supra-human intelligence and strive actively to enhance their cognitive powers.

Mean organizations

Then it follows that we should be concerned if one or more of these smart organizations are so unlike human beings in their motivational structure that they are 'mean'.

I believe the implications of this line of reasoning may be profound, but as this is my first post to LessWrong I would like to first see how this is received before going on.

Intelligence explosion

Personal Blog

13

New Comment

Rendering 0/187 comments, sorted by

top scoring

(show more) Click to highlight new comments since: Today at 10:59 PM

Some comments are truncated due to high volume. (⌘F to expand all)Change truncation settings

Moderation Log

More from sbenthall

Curated and popular this week

187Comments

187

Comment Permalink

lukeprog14y60

I made it clear in our dialogue that I was stipulating a particular definition for intelligence:

SBENTHALL: Would you say that Google is a super-human intelligence?

ME: Well, yeah, so we have to be very careful about all the words that we are using of course. What I mean by intelligence is this notion of what sometimes is called optimization power, which is the ability to achieve one's goals in a wide range of environments and a wide range of constraints. And so for example, humans have a lot more optimization power than chimpanzees. That's why even though we are slower than many animals and not as strong as many animals, we have this thing called intelligence that allows us to commence farming and science and build cities and put footprints on the moon. And so it is humans that are steering the future of the globe and not chimpanzees or stronger things like blue whales. So that's kind of the intuitive notion. There are lots of technical papers that would be more precise.

So, I'm not going to argue about the definition of intelligence. Likewise, I won't argue about the definition of rationality. What I mean by rationality is the concept of rationality from economics and cognitive science, though if we want to get philosophical then it gets more complicated than simple Bayesianism. (Aside: Is the theory of "communicative rationality" specified well enough that we can measure degrees of it, as we can with Bayesian rationality?)

As for this general line of argument about organizations and intelligence explosion, I refer to the earlier Hanson-Yudkowsky debate, especially (as Emile noted) the UberTool discussion. A summary of the Hanson-Yudkowsky debate is here.

I also refer interested parties to the comments here by gwern and by anonymous1.

sbenthall14y20

I've realized I didn't address your direct query:

(Aside: Is the theory of "communicative rationality" specified well enough that we can measure degrees of it, as we can with Bayesian rationality?)

Not yet. It's a qualitatively described theory. I think it's probably possible to render it into quantitative terms, but as far as I know it has not yet been done.

2sbenthall14y

Thanks for this response, Luke. I don't want to argue about definitions either. I believe I'm familiar with how you use the term rationality. I believe it's compatible with (mutually reinforcing with) communicative rationality for the most part, though I believe there are some differences between Habermas' and Yudkowsky's epistemologies. I brought up communicative rationality because (a) I think it's an important concept that is in some ways an advance in how to think about rationality and, (b) I wanted to disclose some of my own predispositions and values for the sake of establishing expectations. Thanks for the link to the Hanson-Yudkowsky debate. From perusing the summary and a few of the posts by the debaters, I guess I'd say I find Hanson's counterarguments largely compelling. I'd also respond with two other points (mostly hoping you will direct me to where they've already been discussed): Since the computational complexity of so many kinds of problems has been proven to be within certain complexity classes, recursive improvement in algorithms alone is likely to hit asymptotic walls for a lot of interesting domains. So, self-modifying AI alone, without taking resources into account, seems unlikely (maybe provably impossible) to be a big threat. That said, since there already are self-modifying intelligent organizations that are taking over the world (or trying to, facing competition from each other), what's gone into Singularity research definitely isn't useless. Rather, it's directly applicable to what's happening right now. I agree very strongly with the thrust of what IlyaShpitser's been saying.

7IlyaShpitser14y

Of course Google is a super-human intelligence (in a sense of optimizing for goals). I agree with gwern et al that probably a company's productivity scaling is sublinear wrt number of components in it, but that should make it an easier special case to consider. We can still comprehend its goals and mostly what it's doing. Why not deal with a special case first?

See in context