Comment author: skeptical_lurker 04 October 2016 05:23:48AM *  3 points [-]

I've been thinking about what seems to be the standard LW pitch on AI risk. It goes like this: "Consider an AI that is given a goal by humans. Since 'convert the planet into computronium' is a subgoal of most goals, it does this and kills humanity."

The problem, which various people have pointed out, is that this implies an intelligence capable of taking over the world, but not capable of working out that when a human says pursue a certain goal, they would not want this goal to be pursued in a way that leads to the destruction of the world.

Worse, the argument can then be made that this idea that an AI will interpret goals so literally without modelling a human mind constitutes an "autistic AI" and that only autistic people would assume that AI would be similarly autistic. I do not endorse this argument in any way, but I guess its still better to avoid arguments that signal low social skills, all other things being equal.

Is there any consensus on what the best 'elevator pitch' argument for AI risk is? Instead of focusing on any one failure mode, I would go with something like this:

"Most philosophers agree that there is no reason why superintelligence is not possible. Anything which is possible will eventually be achieved, and so will superintelligence, perhaps in the far future, perhaps in the next few decades. At some point, superintelligences will be as far above humans as we are above ants. I do not know what will happen at this point, but the only reference case we have is humans and ants, and if superintelligences decide that humans are an infestation, we will be exterminated."

Incidentally, this is the sort of thing I mean by painting LW style ideas as autistic (via David Pierce)

As far as we can tell, digital computers are still zombies. Our machines are becoming autistically intelligent, but not supersentient - nor even conscious. [...] Full-Spectrum Superintelligence entails: [...] social intelligence [...] a metric to distinguish the important from the trivial [...] a capacity to navigate, reason logically about, and solve problems in multiple state-spaces of consciousness [e.g. dreaming states (cf. lucid dreaming), waking consciousness, echolocatory competence, visual discrimination, synaesthesia in all its existing and potential guises, humour, introspection, the different realms of psychedelia [...] and finally "Autistic", pattern-matching, rule-following, mathematico-linguistic intelligence, i.e. the standard, mind-blind cognitive tool-kit scored by existing IQ tests. High-functioning "autistic" intelligence is indispensable to higher mathematics, computer science and the natural sciences. High-functioning autistic intelligence is necessary - but not sufficient - for a civilisation capable of advanced technology that can cure ageing and disease, systematically phase out the biology of suffering, and take us to the stars. And for programming artificial intelligence.

Sometimes David Pierce seems very smart. And sometimes he seems to imply that the ability to think logically while on psychedelic drugs is as important as 'autistic intelligence'. I don't think he thinks that autistic people are zombies that do not experience subjective experience, but that also does seem implied.

Comment author: twanvl 05 October 2016 01:41:18PM 1 point [-]

The problem, which various people have pointed out, is that this implies an intelligence capable of taking over the world, but not capable of working out that when a human says pursue a certain goal, they would not want this goal to be pursued in a way that leads to the destruction of the world.

The entity providing the goals for the AI wouldn't have to be a human, it might instead be a corporation. A reasonable goal for such an AI might be to 'maximize shareholder value'. The shareholders are not humans either, and what they value is only money.

Comment author: gjm 17 November 2015 04:34:02PM 25 points [-]

Nice.

To fight back against terrible terminology from the other side (i.e., producing rather than consuming) I suggest a commitment to refuse to say "Type I error" or "Type II error" and always say "false positive" or "false negative" instead.

Comment author: twanvl 18 November 2015 10:33:27AM 1 point [-]

I find "false positive" and "false negative" also a bit confusing, albeit less so than "type I" and "type II" errors. Perhaps because of a programming background, I usually interpret 'false' and 'negative' (and '0') as the same thing. So is a 'false positive' something that is false but is mistaken as positive, or something that is positive (true), but that is mistaken as false (negative)? In other words, does 'false' apply to the postiveness (it is actually negative, but classified as positive), to being classified as positive (it is actually positive, but classified as positive)?

Perhaps we should call false positives "spurious" and false negatives "missed".

Comment author: twanvl 13 November 2015 03:26:18PM 4 points [-]

The link you provided contains absolutely no physics, as far as I can tell. Nor is there any math aside from some basic logic. So I am skeptical on whether this theory is correct (or even falsifiable).

Comment author: Thomas 13 October 2015 06:00:02PM -3 points [-]

So you say, ZFC has nothing to do with time? Time in physics is uncovered in ZFC?

Comment author: twanvl 13 October 2015 08:34:17PM 1 point [-]

Physics is built on top of mathematics, and almost all of mathematics can be built on top of ZFC (there are other choices). But there is as much time in ZFC as there are words in a single pixel on your screen.

Comment author: philh 13 October 2015 08:05:50PM 12 points [-]

I have an intuition that if we implemented universal basic income, the prices of necessities would rise to the point where people without other sources of income would still be in poverty. I assume there are UBI supporters who've spent more time thinking about that question than I have, and I'm interested in their responses.

(I have some thoughts myself on the general directions responses might take, but I haven't fleshed them out, and I might not care enough to do so.)

Comment author: twanvl 13 October 2015 08:28:57PM 7 points [-]

Why would the price of necessities rise?

There are three reasons why the price might go up: 1. demand increases 2. supply decreases 3. inflation

Right now, everyone is already consuming these necessities, so if UBI is introduced, demand will not go up. So 1 would not be true.

Supply could go down if enough people stop working. But if this reduces supply of the necessities, there is a strong incentive for people on just UBI to start working again. There is also increasing automation. So I find 2 unlikely.

That leaves 3, inflation. I am not an economist, but as far as I understand this shouldn't be a significant factor.

Comment author: Thomas 12 October 2015 04:46:19PM 0 points [-]

So the sequence of distributions that you gave, or my example of the sequence 0,1,2,3,4,... both don't converge. Calling them a supertask doesn't change that fact.

I don't understand you.

Comment author: twanvl 12 October 2015 05:47:00PM *  3 points [-]

Define the sequence S by

S(0) = 0
S(n+1) = 1 + S(n)

This is a sequence of natural numbers. This sequence does not converge, which means that the limit as n goes to infinite of S(n) is not a natural number (nor a real number for that matter).

You could try to write it as a function of time, S'(t) such that S'(1-0.5^n) = S(n). That is, S'(0)=0, S'(0.5)=1, S'(0.75)=2, etc. A possible formula is S'(t) = -log_2(1-t). You could then ask what is S'(1). The answer is that this is the same as the limit S(infinity), or as log(0), which are both not defined. So in fact S' is not a function from numbers between 0 and 1 inclusive to natural or real numbers, since the domain excludes 1.

You can similarly define a sequence of distributions over the natural numbers by

T(0) = {i -> 0.5 * 0.5^i}
T(n+1) = the same as T(n) except two values swapped

This is the example that you gave above. The sequence T(n) doesn't converge (I haven't checked, but the discussion above suggests that it doesn't), meaning that the limit "lim_{n->inf} T(n)" is not defined.

Comment author: Thomas 12 October 2015 04:29:03PM -3 points [-]

This question presupposes that the task will ever be done

Sure. It's called super-tasks.

https://en.wikipedia.org/wiki/Supertask

"a supertask is a countably infinite sequence of operations that occur sequentially within a finite interval of time."

You can't avoid supertasks, when you endorse infinity.

Therefore, I don't.

Comment author: twanvl 12 October 2015 04:43:06PM 3 points [-]

This question presupposes that the task will ever be done Sure. It's called super-tasks.

From mathematics we know that not all sequences converge. So the sequence of distributions that you gave, or my example of the sequence 0,1,2,3,4,... both don't converge. Calling them a supertask doesn't change that fact.

What mathematicians often do in such cases is to define a new object to denote the hypothetical value at the end of sequence. This is how you end up with real numbers, distributions (generalized functions), etc. To be fully formal you would have to keep track of the sequence itself, which for real numbers gives you Cauchy sequences for instance. In most cases these objects behave a lot like the elements of the sequence, so real numbers are a lot like rational numbers. But not always, and sometimes there is some weirdness.

From the wikipedia link:

In philosophy, a supertask is a countably infinite sequence of operations that occur sequentially within a finite interval of time.

This refers to something called "time". Most of mathematics, ZFC included, has no notion of time. Now, you could take a variable, and call it time. And you can say that a given countably infinite sequences "takes place" in finite "time". But that is just you putting semantics on this sequence and this variable.

Comment author: Thomas 12 October 2015 04:03:26PM -2 points [-]

What can one expect after this super-task is done to see?

Nothing?

At a meta level, if there were this basic a problem, don't you think it would have already been noticed?

It has been noticed, but never resolved properly. A consensus among top mathematicians, that everything is/must be okay prevails.

One dissident.

https://www.youtube.com/watch?t=27&v=4DNlEq0ZrTo

Comment author: twanvl 12 October 2015 04:13:10PM 4 points [-]

What can one expect after this super-task is done to see?

This question presupposes that the task will ever be done. Since, if I understand correctly, you are doing an infinite number of swaps, you will never be done.

You could similarly define a super-task (whatever that is) of adding 1 to a number. Start with 0, at time 0 add 1, add one more at time 0.5, and again at 0.75. What is the value when you are done? Clearly you are counting to infinity, so even though you started with a natural number, you don't end up with one. That is because you don't "end up" at all.

Comment author: Thomas 12 October 2015 01:11:03PM 1 point [-]

Not that it counts much, but I do believe that the ZFC is inconsistent.

Comment author: twanvl 12 October 2015 02:12:23PM 2 points [-]

Why do you believe that? And do you also believe that ZF is inconsistent?

Comment author: twanvl 23 September 2015 02:49:04PM 0 points [-]

Not about the game itself, but the wording of the questions is a bit confusing to me:

In the above network, suppose that we were to observe the variable labeled "A". Which other variables would this influence?

The act of observing a variable doesn't influence any of the variables, it would only change your beliefs about the variables. The only things influencing a variable are its parents in the Bayesian network.

View more: Next