Comment author: skeptical_lurker 04 October 2016 05:23:48AM *  3 points [-]

I've been thinking about what seems to be the standard LW pitch on AI risk. It goes like this: "Consider an AI that is given a goal by humans. Since 'convert the planet into computronium' is a subgoal of most goals, it does this and kills humanity."

The problem, which various people have pointed out, is that this implies an intelligence capable of taking over the world, but not capable of working out that when a human says pursue a certain goal, they would not want this goal to be pursued in a way that leads to the destruction of the world.

Worse, the argument can then be made that this idea that an AI will interpret goals so literally without modelling a human mind constitutes an "autistic AI" and that only autistic people would assume that AI would be similarly autistic. I do not endorse this argument in any way, but I guess its still better to avoid arguments that signal low social skills, all other things being equal.

Is there any consensus on what the best 'elevator pitch' argument for AI risk is? Instead of focusing on any one failure mode, I would go with something like this:

"Most philosophers agree that there is no reason why superintelligence is not possible. Anything which is possible will eventually be achieved, and so will superintelligence, perhaps in the far future, perhaps in the next few decades. At some point, superintelligences will be as far above humans as we are above ants. I do not know what will happen at this point, but the only reference case we have is humans and ants, and if superintelligences decide that humans are an infestation, we will be exterminated."

Incidentally, this is the sort of thing I mean by painting LW style ideas as autistic (via David Pierce)

As far as we can tell, digital computers are still zombies. Our machines are becoming autistically intelligent, but not supersentient - nor even conscious. [...] Full-Spectrum Superintelligence entails: [...] social intelligence [...] a metric to distinguish the important from the trivial [...] a capacity to navigate, reason logically about, and solve problems in multiple state-spaces of consciousness [e.g. dreaming states (cf. lucid dreaming), waking consciousness, echolocatory competence, visual discrimination, synaesthesia in all its existing and potential guises, humour, introspection, the different realms of psychedelia [...] and finally "Autistic", pattern-matching, rule-following, mathematico-linguistic intelligence, i.e. the standard, mind-blind cognitive tool-kit scored by existing IQ tests. High-functioning "autistic" intelligence is indispensable to higher mathematics, computer science and the natural sciences. High-functioning autistic intelligence is necessary - but not sufficient - for a civilisation capable of advanced technology that can cure ageing and disease, systematically phase out the biology of suffering, and take us to the stars. And for programming artificial intelligence.

Sometimes David Pierce seems very smart. And sometimes he seems to imply that the ability to think logically while on psychedelic drugs is as important as 'autistic intelligence'. I don't think he thinks that autistic people are zombies that do not experience subjective experience, but that also does seem implied.

Comment author: twanvl 05 October 2016 01:41:18PM 1 point [-]

The problem, which various people have pointed out, is that this implies an intelligence capable of taking over the world, but not capable of working out that when a human says pursue a certain goal, they would not want this goal to be pursued in a way that leads to the destruction of the world.

The entity providing the goals for the AI wouldn't have to be a human, it might instead be a corporation. A reasonable goal for such an AI might be to 'maximize shareholder value'. The shareholders are not humans either, and what they value is only money.

Comment author: gjm 17 November 2015 04:34:02PM 25 points [-]

Nice.

To fight back against terrible terminology from the other side (i.e., producing rather than consuming) I suggest a commitment to refuse to say "Type I error" or "Type II error" and always say "false positive" or "false negative" instead.

Comment author: twanvl 18 November 2015 10:33:27AM 1 point [-]

I find "false positive" and "false negative" also a bit confusing, albeit less so than "type I" and "type II" errors. Perhaps because of a programming background, I usually interpret 'false' and 'negative' (and '0') as the same thing. So is a 'false positive' something that is false but is mistaken as positive, or something that is positive (true), but that is mistaken as false (negative)? In other words, does 'false' apply to the postiveness (it is actually negative, but classified as positive), to being classified as positive (it is actually positive, but classified as positive)?

Perhaps we should call false positives "spurious" and false negatives "missed".

Comment author: twanvl 13 November 2015 03:26:18PM 4 points [-]

The link you provided contains absolutely no physics, as far as I can tell. Nor is there any math aside from some basic logic. So I am skeptical on whether this theory is correct (or even falsifiable).

Comment author: philh 13 October 2015 08:05:50PM 12 points [-]

I have an intuition that if we implemented universal basic income, the prices of necessities would rise to the point where people without other sources of income would still be in poverty. I assume there are UBI supporters who've spent more time thinking about that question than I have, and I'm interested in their responses.

(I have some thoughts myself on the general directions responses might take, but I haven't fleshed them out, and I might not care enough to do so.)

Comment author: twanvl 13 October 2015 08:28:57PM 7 points [-]

Why would the price of necessities rise?

There are three reasons why the price might go up: 1. demand increases 2. supply decreases 3. inflation

Right now, everyone is already consuming these necessities, so if UBI is introduced, demand will not go up. So 1 would not be true.

Supply could go down if enough people stop working. But if this reduces supply of the necessities, there is a strong incentive for people on just UBI to start working again. There is also increasing automation. So I find 2 unlikely.

That leaves 3, inflation. I am not an economist, but as far as I understand this shouldn't be a significant factor.

Comment author: Thomas 12 October 2015 04:46:19PM 0 points [-]

So the sequence of distributions that you gave, or my example of the sequence 0,1,2,3,4,... both don't converge. Calling them a supertask doesn't change that fact.

I don't understand you.

Comment author: twanvl 12 October 2015 05:47:00PM *  3 points [-]

Define the sequence S by

S(0) = 0
S(n+1) = 1 + S(n)

This is a sequence of natural numbers. This sequence does not converge, which means that the limit as n goes to infinite of S(n) is not a natural number (nor a real number for that matter).

You could try to write it as a function of time, S'(t) such that S'(1-0.5^n) = S(n). That is, S'(0)=0, S'(0.5)=1, S'(0.75)=2, etc. A possible formula is S'(t) = -log_2(1-t). You could then ask what is S'(1). The answer is that this is the same as the limit S(infinity), or as log(0), which are both not defined. So in fact S' is not a function from numbers between 0 and 1 inclusive to natural or real numbers, since the domain excludes 1.

You can similarly define a sequence of distributions over the natural numbers by

T(0) = {i -> 0.5 * 0.5^i}
T(n+1) = the same as T(n) except two values swapped

This is the example that you gave above. The sequence T(n) doesn't converge (I haven't checked, but the discussion above suggests that it doesn't), meaning that the limit "lim_{n->inf} T(n)" is not defined.

Comment author: Thomas 12 October 2015 04:03:26PM -2 points [-]

What can one expect after this super-task is done to see?

Nothing?

At a meta level, if there were this basic a problem, don't you think it would have already been noticed?

It has been noticed, but never resolved properly. A consensus among top mathematicians, that everything is/must be okay prevails.

One dissident.

https://www.youtube.com/watch?t=27&v=4DNlEq0ZrTo

Comment author: twanvl 12 October 2015 04:13:10PM 4 points [-]

What can one expect after this super-task is done to see?

This question presupposes that the task will ever be done. Since, if I understand correctly, you are doing an infinite number of swaps, you will never be done.

You could similarly define a super-task (whatever that is) of adding 1 to a number. Start with 0, at time 0 add 1, add one more at time 0.5, and again at 0.75. What is the value when you are done? Clearly you are counting to infinity, so even though you started with a natural number, you don't end up with one. That is because you don't "end up" at all.

Comment author: Thomas 12 October 2015 01:11:03PM 1 point [-]

Not that it counts much, but I do believe that the ZFC is inconsistent.

Comment author: twanvl 12 October 2015 02:12:23PM 2 points [-]

Why do you believe that? And do you also believe that ZF is inconsistent?

Comment author: twanvl 23 September 2015 02:49:04PM 0 points [-]

Not about the game itself, but the wording of the questions is a bit confusing to me:

In the above network, suppose that we were to observe the variable labeled "A". Which other variables would this influence?

The act of observing a variable doesn't influence any of the variables, it would only change your beliefs about the variables. The only things influencing a variable are its parents in the Bayesian network.

Comment author: Gunnar_Zarncke 10 September 2015 07:57:23PM 1 point [-]

Just because it is obvious to you doesn't mean that everybody immediately jumps to it. Me included and I like the statistics classes back then. Could you please point to the wheels?

Comment author: twanvl 11 September 2015 11:53:21AM 1 point [-]

The wheels in this case come from robust statistics.

One example of a good robust estimator for the center is the [truncated mean]https://en.wikipedia.org/wiki/Truncated_mean). To put it simply: throw away the lowest x% and the highest x% of the samples, and take the mean of the rest. If x=0 you get the regular mean, if x=50% then you get the median.

Comment author: Manfred 04 September 2015 06:25:44AM *  1 point [-]

Man, that sleeping beauty digression at the end. I'm so easily aggroed.

I wonder if I should write up a rant or a better attempt at exposition, against this "clearly the problem is underdetermined" position. The correct answer is somewhat difficult and all expositions I've seen (or written!) so far have had big shortcomings. But even if one didn't know the right answer, one should readily conclude that a principled answer exists. We assign things probabilities for fairly deep reasons, reasons undisturbed by trifles like the existence of an amnesia pill.

That aside, good talk :)

Comment author: twanvl 04 September 2015 04:28:23PM 2 points [-]

I wonder if I should write up a rant or a better attempt at exposition, against this "clearly the problem is underdetermined" position.

You should.

View more: Next