= 27d567bc2d48d9f40e9ccc20ea370b15)
Nornagest comments on International cooperation vs. AI arms race - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (143)
Oh, that makes more sense. I'd assumed, since this thread was rooted under the OP, that you were responding to that.
After reading James's post, though, I don't think it's meant to be treated as comprehensive, much less prescriptive. He seems to be giving some (fictional) outlines of outcomes that could arise in the absence of early and aggressive cooperation on AI development; the stakes at that point are high, so the consequences are rather precipitous, but this is still something to avoid rather than something to pursue. Reading between the lines, in fact, I'd say the policy implications he's gesturing towards are much the same as those you've been talking about upthread.
On the other hand, it's very early to be hashing out scenarios like this, and doing so doesn't say anything particularly good about us from a PR perspective. It's hard enough getting people to take AI seriously as a risk, full stop; we don't need to exacerbate that with wild apocalyptic fantasies just yet.
This bears investigating. I mean, come on, the popular view of AI among the masses is that All AI Is A Crapshoot, that every single time it will end in the Robot Wars. So how on Earth can it be difficult to convince people that UFAI is an issue?
I mean, hell, if I wanted to scare someone, I'd just point out that no currently-known model of AGI includes a way to explicitly specify goals desirable to humans. That oughtta scare folks.
I've talked to a number of folks who conclude that AIs will be superintelligent and therefore will naturally derive and follow the true morality (you know, the same one we do), and dismiss all that Robot Wars stuff as television crap (not unreasonably, as far as it goes).
Which one's that, eh ;-)?
Are these religious people? I mean, come on, where do you get moral realism if not from some kind of moral metaphysics?
Certainly it's not unreasonable. One UFAI versus humans with no FAI to fight back, I wouldn't call anything so one-sided a war.
(And I'm sooo not making the Dalek reference that I really want to. Someone else should do it.)
Pedantic complaint about language: moral realism simply says that moral claims do state facts, and at least some of them are true. It takes further assumptions ("internalism") to claim that these moral facts are universally compelling in the sense of moving any intelligent being to action. (I personally believe the latter assumption to be nonsense, hence AGI is a really bad idea.)
Granted, I don't know of any nice precise term for that position that all intelligent beings must necessarily do the right thing, possibly because it's so ridiculous no philosopher would profess it publicly in such words. On the other hand, motivational internalism would seem to be very intuitive, judging by the pervasiveness of the view that AI doesn't pose any risk.
Isn't it called Convergence?
Are you under the impression that CEV advocates around here believe that all intelligent beings must necessarily do the right thing?
On the whole, confusion reigns, but there is a fairly consistent tendency to reject Intrinsic Motivation without argument.
What's "Intrinsic Motivation"? The only hits for it on LW are about akrasia.
As in intrinsically motivating states and concpets
Does "Intrinsic Motivation" in this context entail that all intelligent beings must necessarily do the right thing?
If so, then I agree that we tend to reject it. As for "without argument"... do you mean you've read the local discussions of the topic and find them unconvincing? Or do you mean you believe it hasn't been discussed at all?
If not, then I don't know what you're saying.
If you prefer to continue expressing yourself in gnomic utterances, that's of course your choice, but I find it an unhelpful way to communicate and will tap out here if so.
If not, I'm
Little argument and none convincing.
Eh, maybe? I've seen "convergence thesis" thrown about on LW, but it's hardly established terminology. Not sure it would be fair to use a phrase so easily confused with Bostrom's much more reasonable Instrumental Convergence Thesis either. (Also, it has nothing to do with CEV so I don't see the point of that link.)
I've never had that conversation with explicitly religious people, and moral realism at the "some things are just wrong and any sufficiently intelligent system will know it" level is hardly unheard of among atheists.
Really? I mean, sorry for blathering, but I find this extremely surprising. I always considered it a simple fact that if you don't have some kind of religious/faith-based metaphysics operating, you can't be a moral realist. What experiment could you possibly perform to test moral-realist hypotheses, particularly when dealing with nonhumans? It simply doesn't make any sense.
Oh well.
Moral realism makes no more sense with religion. As CS Lewis said: "Nonsense does not cease to be nonsense when we put the words 'God can' before it."
Disagreed, depending on your definition of "morality". A sufficiently totalitarian God can easily not only decide what is moral but force us to find the proper morality morally compelling.
(There is at least one religion that actually believes something along these lines, though I don't follow it.)
Ok, that definition is not nonsense. But in that case, it could happen without God too. Maybe the universe's laws cause people to converge on some morality, either due to the logic of evolutionary cooperation or another principle. It could even be an extra feature of physics that forces this convergence.
Perhaps Eli and you are talking past each other a bit. A certain kind of god would be strong evidence for moral realism, but moral realism wouldn't be strong evidence for a god of any kind.
Well sure, but if you're claiming physics enforces a moral order, you've reinvented non-theistic religion.
Why? Beliefs that make no sense are very common. Atheists are no exception.
Actually, if anything, I'd call it the reverse. Religious people know where we're making unevidenced assumptions.
That would be epistemology...
There are rationally acceptable subjects that don't use empiricism, such as maths, and there are subjects such as economics which have a mixed epistemology.
However, if this epistemological-sounding complaint is actually about metaphysics, ie "what experiment could you perform to detect a non-natural moral property", the answer is that moral realists have to suppose the existence of special psychological faculty.
Might I suggest you take a look at the metaethics sequence? This position is explained very well.
Which position? The metaethics sequence isn't clearly re4alist, or anything else.
Well no, not really. The meta-ethics sequence takes a cognitivist position: there is some cognitive algorithm called ethics, which actual people implement imperfectly but which you could somehow generalize to obtain a "perfect" reification.
That's not moral realism ("morality is a part of the universe itself, external to human beings"), that's objective moral-cognitivism ("morality is a measurable part of us but has no other grounding in external reality").
You talk as though religion were something that appeared in people's minds fully formed and without causes, and that the logical fallacies associated with it were then caused by religion.
Hmm. Fair point. "We imagine the universe as we are."
From abstract reason or psychological facts, or physical facts, or a mixture.
There is a subject called economics. It tells you how to achieve certain goals, such as maximising GDP. It doesn't do that by corresponding to a metaphysical Economics Object, it does that with a mixture of theoretical reasoning and examination of evidence.
There is a subject called ethics. It tells you how to achieve certain goals, such as maximising happiness....
Well there's the problem: ethics does not automatically start out with a happiness-utilitarian goal. Lots of extent ethical systems use other terminal goals. For instance...
"Such as"
Sufficient rationality will tell you how to maximize any goal, once you can clearly define the goal.
Rationality is quite helpful for clarifying goals too.
Problem is, economics is not a science:
http://www.theatlantic.com/business/archive/2013/04/the-laws-of-economics-dont-exist/274901/
Of course economics doesn't have the well-established laws of physical science: it wouldn't be much of an analogy for ethics if it did.But having an epistemology that doens't work very well is not the same as having an epistemology that requires non-natural entities.
The main problem with economics is not its descriptive, but its predictive power. Too many of economics' calculations need to suppose that everyone will behave rationally, which regular people can't be trusted to do. Same problem with politics.
Well, there's a couple prongs to that. For one thing, it's tagged as fiction in most people's minds, as might be suggested by the fact that it's easily described in trope. That's bad enough by itself.
Probably more importantly, though, there's a ferocious tendency to anthropomorphize this sort of thing, and you can't really grok UFAI without burning a good bit of that tendency out of your head. Sure, we ourselves aren't capital-F Friendly, but we're a far cry yet from a paperclip maximizer or even most of the subtler failures of machine ethics; a jealous or capricious machine god is bad, but we're talking Screwtape here, not Azathoth. HAL and Agent Smith are the villains of their stories, but they're human in most of the ways that count.
You may also notice that we tend to win fictional robot wars.
Also, note that the tropes tend to work against people who say "we have a systematic proof that our design of AI will be Friendly". In fact, in general the only way a fictional AI will turn out 'friendly' is if it is created entirely by accident - ANY fictional attempt to intentionally create a Friendly AI will result in an abomination, usually through some kind of "dick Genie" interpretation of its Friendliness rules.
Yeah. I think I'd consider that a form of backdoor anthropomorphization by way of vitalism, though. Since we tend to think of physically nonhuman intelligences as cognitively human, and since we tend to think of human ethics and cognition as something sacred and ineffable, fictional attempts to eff them tend to be written as crude morality plays.
Intelligence arising organically from a telephone exchange or an educational game or something doesn't trigger the same taboos.