Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Comment author: skeptical_lurker 04 October 2016 05:23:48AM *  3 points [-]

I've been thinking about what seems to be the standard LW pitch on AI risk. It goes like this: "Consider an AI that is given a goal by humans. Since 'convert the planet into computronium' is a subgoal of most goals, it does this and kills humanity."

The problem, which various people have pointed out, is that this implies an intelligence capable of taking over the world, but not capable of working out that when a human says pursue a certain goal, they would not want this goal to be pursued in a way that leads to the destruction of the world.

Worse, the argument can then be made that this idea that an AI will interpret goals so literally without modelling a human mind constitutes an "autistic AI" and that only autistic people would assume that AI would be similarly autistic. I do not endorse this argument in any way, but I guess its still better to avoid arguments that signal low social skills, all other things being equal.

Is there any consensus on what the best 'elevator pitch' argument for AI risk is? Instead of focusing on any one failure mode, I would go with something like this:

"Most philosophers agree that there is no reason why superintelligence is not possible. Anything which is possible will eventually be achieved, and so will superintelligence, perhaps in the far future, perhaps in the next few decades. At some point, superintelligences will be as far above humans as we are above ants. I do not know what will happen at this point, but the only reference case we have is humans and ants, and if superintelligences decide that humans are an infestation, we will be exterminated."

Incidentally, this is the sort of thing I mean by painting LW style ideas as autistic (via David Pierce)

As far as we can tell, digital computers are still zombies. Our machines are becoming autistically intelligent, but not supersentient - nor even conscious. [...] Full-Spectrum Superintelligence entails: [...] social intelligence [...] a metric to distinguish the important from the trivial [...] a capacity to navigate, reason logically about, and solve problems in multiple state-spaces of consciousness [e.g. dreaming states (cf. lucid dreaming), waking consciousness, echolocatory competence, visual discrimination, synaesthesia in all its existing and potential guises, humour, introspection, the different realms of psychedelia [...] and finally "Autistic", pattern-matching, rule-following, mathematico-linguistic intelligence, i.e. the standard, mind-blind cognitive tool-kit scored by existing IQ tests. High-functioning "autistic" intelligence is indispensable to higher mathematics, computer science and the natural sciences. High-functioning autistic intelligence is necessary - but not sufficient - for a civilisation capable of advanced technology that can cure ageing and disease, systematically phase out the biology of suffering, and take us to the stars. And for programming artificial intelligence.

Sometimes David Pierce seems very smart. And sometimes he seems to imply that the ability to think logically while on psychedelic drugs is as important as 'autistic intelligence'. I don't think he thinks that autistic people are zombies that do not experience subjective experience, but that also does seem implied.

Comment author: twanvl 05 October 2016 01:41:18PM 1 point [-]

The problem, which various people have pointed out, is that this implies an intelligence capable of taking over the world, but not capable of working out that when a human says pursue a certain goal, they would not want this goal to be pursued in a way that leads to the destruction of the world.

The entity providing the goals for the AI wouldn't have to be a human, it might instead be a corporation. A reasonable goal for such an AI might be to 'maximize shareholder value'. The shareholders are not humans either, and what they value is only money.

Comment author: gjm 17 November 2015 04:34:02PM 25 points [-]

Nice.

To fight back against terrible terminology from the other side (i.e., producing rather than consuming) I suggest a commitment to refuse to say "Type I error" or "Type II error" and always say "false positive" or "false negative" instead.

Comment author: twanvl 18 November 2015 10:33:27AM 1 point [-]

I find "false positive" and "false negative" also a bit confusing, albeit less so than "type I" and "type II" errors. Perhaps because of a programming background, I usually interpret 'false' and 'negative' (and '0') as the same thing. So is a 'false positive' something that is false but is mistaken as positive, or something that is positive (true), but that is mistaken as false (negative)? In other words, does 'false' apply to the postiveness (it is actually negative, but classified as positive), to being classified as positive (it is actually positive, but classified as positive)?

Perhaps we should call false positives "spurious" and false negatives "missed".

Comment author: twanvl 13 November 2015 03:26:18PM 4 points [-]

The link you provided contains absolutely no physics, as far as I can tell. Nor is there any math aside from some basic logic. So I am skeptical on whether this theory is correct (or even falsifiable).

Comment author: philh 13 October 2015 08:05:50PM 12 points [-]

I have an intuition that if we implemented universal basic income, the prices of necessities would rise to the point where people without other sources of income would still be in poverty. I assume there are UBI supporters who've spent more time thinking about that question than I have, and I'm interested in their responses.

(I have some thoughts myself on the general directions responses might take, but I haven't fleshed them out, and I might not care enough to do so.)

Comment author: twanvl 13 October 2015 08:28:57PM 7 points [-]

Why would the price of necessities rise?

There are three reasons why the price might go up: 1. demand increases 2. supply decreases 3. inflation

Right now, everyone is already consuming these necessities, so if UBI is introduced, demand will not go up. So 1 would not be true.

Supply could go down if enough people stop working. But if this reduces supply of the necessities, there is a strong incentive for people on just UBI to start working again. There is also increasing automation. So I find 2 unlikely.

That leaves 3, inflation. I am not an economist, but as far as I understand this shouldn't be a significant factor.

Comment author: Thomas 12 October 2015 04:03:26PM -2 points [-]

What can one expect after this super-task is done to see?

Nothing?

At a meta level, if there were this basic a problem, don't you think it would have already been noticed?

It has been noticed, but never resolved properly. A consensus among top mathematicians, that everything is/must be okay prevails.

One dissident.

https://www.youtube.com/watch?t=27&v=4DNlEq0ZrTo

Comment author: twanvl 12 October 2015 04:13:10PM 4 points [-]

What can one expect after this super-task is done to see?

This question presupposes that the task will ever be done. Since, if I understand correctly, you are doing an infinite number of swaps, you will never be done.

You could similarly define a super-task (whatever that is) of adding 1 to a number. Start with 0, at time 0 add 1, add one more at time 0.5, and again at 0.75. What is the value when you are done? Clearly you are counting to infinity, so even though you started with a natural number, you don't end up with one. That is because you don't "end up" at all.

Comment author: Thomas 12 October 2015 01:11:03PM 1 point [-]

Not that it counts much, but I do believe that the ZFC is inconsistent.

Comment author: twanvl 12 October 2015 02:12:23PM 2 points [-]

Why do you believe that? And do you also believe that ZF is inconsistent?

Comment author: twanvl 23 September 2015 02:49:04PM 0 points [-]

Not about the game itself, but the wording of the questions is a bit confusing to me:

In the above network, suppose that we were to observe the variable labeled "A". Which other variables would this influence?

The act of observing a variable doesn't influence any of the variables, it would only change your beliefs about the variables. The only things influencing a variable are its parents in the Bayesian network.

Comment author: Gunnar_Zarncke 10 September 2015 07:57:23PM 1 point [-]

Just because it is obvious to you doesn't mean that everybody immediately jumps to it. Me included and I like the statistics classes back then. Could you please point to the wheels?

Comment author: twanvl 11 September 2015 11:53:21AM 1 point [-]

The wheels in this case come from robust statistics.

One example of a good robust estimator for the center is the [truncated mean]https://en.wikipedia.org/wiki/Truncated_mean). To put it simply: throw away the lowest x% and the highest x% of the samples, and take the mean of the rest. If x=0 you get the regular mean, if x=50% then you get the median.

Comment author: Manfred 04 September 2015 06:25:44AM *  1 point [-]

Man, that sleeping beauty digression at the end. I'm so easily aggroed.

I wonder if I should write up a rant or a better attempt at exposition, against this "clearly the problem is underdetermined" position. The correct answer is somewhat difficult and all expositions I've seen (or written!) so far have had big shortcomings. But even if one didn't know the right answer, one should readily conclude that a principled answer exists. We assign things probabilities for fairly deep reasons, reasons undisturbed by trifles like the existence of an amnesia pill.

That aside, good talk :)

Comment author: twanvl 04 September 2015 04:28:23PM 2 points [-]

I wonder if I should write up a rant or a better attempt at exposition, against this "clearly the problem is underdetermined" position.

You should.

Comment author: [deleted] 30 June 2015 08:32:03AM *  0 points [-]

Argument for a powerful AI being unlikely - was this considered before?

One problem I see here is the "lone hero inventor" implicit assumption, namely that there are people optimizing things for their goals on their own and an AI could be extremely powerful at this. I would like to propose a different model.

This model would be that intelligence is primarily a social, communication skill, it is the skill to disassemble (understand, lat. intelligo), play with and reassemble ideas acquired from other people. Like literally what we are doing on this forum. It is conversational. The whole standing on the shoulder of giants thing, not the lone hero thing.

In this model, inventions are made by the whole of humankind, a network, where each brain is a node communicating slightly modified ideas to each other.

In such a network one 10000 IQ node does not get very powerful, it doesn't even make the network very powerful i.e. a friendly AI does not quickly solve mortality even with human help.

The primary reason I think such a model is correct that intelligence means thinking, we think in concepts, and concepts are not really nailed down but they are constantly modified through a social communication process. Atoms used to mean indivisible units, then they became divisible into little ping-pong balls, and then the model was updated into something entirely different by quantum physics, but is quantum physics based atom theory about the same atoms that were once thought to be indivisible or is this a different thing now? Is modern atomic theory still about atoms? What are we even mapping here and where does the map end and the territory begin?

So the point is human knowledge is increased by a social communication process where we keep throwing bleggs at each other, and keep redefining what bleggs and rubes mean now, keep juggling these concept, keep asking what you really mean under bleggs, and so on. Intelligence is this communication ability, it is to disassemble Joe's concept of bleggs and understand how it differs from Jane's concept of bleggs and maybe assemble a new concept that describes both bleggs.

Without this communication, what would be even intelligence? What would lone intelligence be? It is almost a contradictory term in itself. What would a brain alone in a universe intelligere i.e. understand if nothing would talk to it? Just tinker with matter somehow without any communication whatsoever? But even if we imagine such an "idiot inventor genius", some kind of a mega-plumber on steroids instead of an intellectual or academic it needs goals for that kind of tinkering with that material stuff, for that it needs concepts, and concepts come and evolve from a constant social ping-pong.

An AI would be yet another node in our network, and participate in this process of throwing blegg-concepts at each other probably far better than any human can, but still just a node.

In response to comment by [deleted] on Top 9+2 myths about AI risk
Comment author: twanvl 30 June 2015 02:20:38PM 2 points [-]

An AI would be yet another node in our network, and participate in this process of throwing blegg-concepts at each other probably far better than any human can, but still just a node.

Why would an AI be a single node? I can run two programs in parallel right now on my computer, and they can talk to each other just fine. So if communication is necessary for intelligence, why couldn't an AI be split up into many communicating sub-AIs?

View more: Next