You're looking at Less Wrong's discussion board. This includes all posts, including those that haven't been promoted to the front page yet. For more information, see About Less Wrong.

gwern comments on Risks from AI and Charitable Giving - Less Wrong Discussion

2 Post author: XiXiDu 13 March 2012 01:54PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Article Navigation

Comments (126)

Sort By: Old

You are viewing a single comment's thread.

Comment author: gwern 13 March 2012 07:46:32PM * 28 points [-]

P1 Fast, and therefore dangerous, recursive self-improvement is logically possible.

All your counter-arguments are enthymematic; as far as I can tell, you are actually arguing against a proposition which looks more like

P1 Recursive self-improvement of arbitrary programs towards unalterable goals is possible with very small constant factors and P or better general asymptotic complexity

I would find your enthymematic far more convincing if you explained why things like Goedel machines are either fallacious or irrelevant.

P1.b The fast computation of a simple algorithm is sufficient to outsmart and overpower humanity.

Your argument is basically an argument from fiction; it's funny that you chose that example of the Roman Empire when recently Reddit spawned a novel arguing that a Marine Corps (surely less dangerous than your 100) could do just that. I will note in passing that black powder's formulation is so simple and famous that even I, who prefers archery, knows it: saltpeter, charcoal, and sulfur. I know for certain that the latter two are available in the Roman empire and suspect the former would not be hard to get. EDIT: and this same day, a Mafia-related paper I was reading for entertainment mentioned that Sicily - one of the oldest Roman possessions - was one of the largest global exporters of sulfur in the 18th/19th centuries. So that ingredient is covered, in spades!

Consider that it takes a whole technological civilization to produce a modern smartphone.

A civilization which exists and is there for the taking.

If you were going to speed up a chimp brain a million times, would it quickly reach human-level intelligence? If not, why then would it be different for a human-level intelligence trying to reach transhuman intelligence?

Chimp brains have not improved at all, even to the point of building computers. There is an obvious disanalogy here...

And to do so efficiently it takes random mutation, a whole society of minds

All of which are available to a 'simple algorithm'. Artificial life was first explored by von Neumann himself!

An AI with simple values will simply lack the creativity, due to a lack of drives, to pursue the huge spectrum of research that a society of humans does pursue.

Are you serious? Are you seriously claiming this? Dead-simple chess and Go algorithms routinely turn out fascinating moves. Genetic algorithms are renowned for producing results which are bizarre and inhuman and creative. Have you never read about the famous circuit which has disconnected parts but won't function without them?

What is this bullshit 'computers can't exhibit creativity' doing here? Searle, why did you steal XiXiDu's account and post this?

Yet even if we assume that there is one complete theory of general intelligence, once discovered, one just has to throw more resources at it. It might be able to incorporate all human knowledge, adapt it and find new patterns. But would it really be vastly superior to human society and their expert systems?

'I may be completely wrong, but hey, I can still ask rhetorically whether I'm not actually right!'

P3 Fast, and therefore dangerous, recursive self-improvement is economically feasible.

This implies P2.

So if the AI can do that, why wouldn't humans be able to use the same algorithms to predict what the initial AI is going to do? And if the AI can't do that, how is it going to maximize expected utility if it is unable to predict what it is going to do?

Why can't I predict the next move of my chess algorithm? Why is there no algorithm to predict the AI algorithm simpler and faster than the original AI algorithm?

A plan for world domination seems like something that can't be concealed from its creators. Lying is no option if your algorithms are open to inspection.

This is just naive. Source code can be available and either the maliciousness not obvious (see the Underhanded C Contest) or not prove what you think it proves (see Reflections on Trusting Trust, just for starters). Assuming you are even inspecting all the existing code rather than a stub left behind to look like an AI.

Therefore the probability of an AI to undergo explosive recursive self-improvement (P(FOOM)) is the probability of the conjunction (P#∧P#) of its premises:

No. Not all the premises are necessary, so a conjunction is inappropriate and establishes a lower bound, at best.

I'm going to stop here. This might have been a useful exercise if you were trying to establish solely necessary premises, in the same vein as Chalmer's paper or Drake equation-style examination of cryonics, but you're not doing that.

Comment author: XiXiDu comment score below threshold [+] (11 children)

Comment author: gwern 13 March 2012 09:21:25PM * 24 points [-]

The point is that P2 does not imply P3, yet P2 has to be true in the first place.

By covering premises which are subsumed or implied by other premises, you are engaged in one of the more effective ways to bias a conjunctive or necessary analysis: you are increasing the number of premises and double-counting probabilities. By the conjunction rule, this usually decreases the final probability.

I've pointed out before that use of the conjunction approach can yield arbitrarily small probabilities based on how many conjuncts one wishes to include.

For example, if I were to argue that based on methodological considerations one could never have greater than 99% confidence in a theory and say that none of the premises could therefore be more than 0.99, I can take a theory of 2 conjuncts with 0.99^2 =98% maximum confidence and knock it down to 94% solely by splitting each conjunct into 3 premises ('this premise conceals a great deal of complexity; let us estimate it by taking a closer look at 3 equivalent but finer-grained propositions...') and claiming each is max 99%, since 0.99^6=0.941.

With your 5 premises, that'd be starting with 95%, and then I can knock it down to 90% by splitting each premise which I could do very easily and have already implied in my first criticism about the self-improvement premise.

You can do this quite easily with cryonics as well - one attempt I saw included transportation to the hospital and used no probability >99%! Needless to say, the person concluded cryonics was a ludicrously bad idea.

It's a strange kind of analysis that only allows the final probability to get smaller and smaller and smaller...

(Obviously as a violation of Cox's theorems - by putting upper bounds on probabilities - this lets us get Dutchbooked.)

Comment author: XiXiDu 14 March 2012 09:48:15AM * 1 point [-]

It wasn't my intention to double-count probabilities. As far as what I wrote suggested that, I am simply wrong. My intention was to show that risks from AI are not as likely as its logical possibility and not as likely as its physical possibility. My intention was to show that there are various premises that need to be true, each of which introduces an additional probability penalty.

Comment author: Gabriel 14 March 2012 11:19:27AM * 3 points [-]

Then you should decompose it like this:

P(FOOM) = P(premise_n | premises_1..n-1) * P(premise_n-1 | premises_1..n-2) * ... * P(premise_2 | premise_1) * P(premise_1)

Then you're precisely measuring the additional probability penalty introduced. And if premise PX implies premise PY, you throw out PY for simplicity. If you can give an upper bound on P(PX) then you gave exactly the same upper bound P(PX & PY). You can't make it stronger by reordering and writing P(PX & PY) = PY * (PX | PY) and then saying 'but PY doesn't imply PX so there'.

Talking about the conjunctive fallacy looks disingenuous when the conjuncts have strong dependencies.

Comment author: XiXiDu comment score below threshold [+] (7 children)

Comment author: wedrifid 14 March 2012 02:56:58PM 2 points [-]

I am sick of being accused of being disingenuous, using dark arts and countless other things like asking "rhetorical questions".

Using somewhat different language this is exactly what you declare about yourself. Those things which you describe so casually as your own preferred behaviors are seen by those with a lesswrong mindset as disengenuity and the abuse of the dark arts. That isn't necessarily a bad thing - you'd fit right in at MENSA for example, aside from the entry requirement I suppose - it just isn't received well on lesswrong.

Comment author: XiXiDu comment score below threshold [+] (2 children)

Comment author: Bugmaster 14 March 2012 08:05:42PM 0 points [-]

That isn't necessarily a bad thing - you'd fit right in at MENSA for example, aside from the entry requirement I suppose - it just isn't received well on lesswrong.

Was that some sort of a dig at Mensa, or XiXiDu, or both ? I know next to nothing about Mensa, so I feel like I'm missing the context here... Aren't they just a bunch of guys who solve IQ tests as a hobby ?

Comment author: wedrifid 14 March 2012 09:26:27PM 1 point [-]

Was that some sort of a dig at Mensa, or XiXiDu, or both ?

Neither, more of a mild compliment combined with an acknowledgement that the lesswrong way is not the only way - or even particularly common.

Comment author: Gabriel 14 March 2012 09:55:52PM * 4 points [-]

All those accusations rather look incredible sad. As if those people are just pissed off that someone tried to criticize their most cherished ideas but they don't know what to say other than ridiculing the opponent based on his inexperience.

Sorry, I didn't mean to ridicule you. I'm not annoyed be the fact that you're criticizing -- if I'm annoyed at all (does 'someone is WRONG on the Internet!' syndrome count as annoyance?). I wasn't bothered by your criticisms of SI when you started posting them. But since then you've been going at it, repeating the same arguments over and over again.

You're trying to create something out of nothing here. Currently available arguments about intelligence explosion are simple. There's no deep math in them (and that's a problem for sure but it cuts both ways — the SIAI don't have a mathy model of intelligence explosion, you don't have mathy arguments that recursive self-improvement will run into fundamental limitations). People are moved by those arguments to various extents. And that's it. We're done. Someone has to come up with a novel insight that will shed additional light on the issue. Until then, people won't change their minds by being exposed to the same arguments even if they come with a brand new rhetorical packaging, heretofore unseen decomposition into bullet points, and a sprinkling of yet-unseen cool quotations.

People will change their minds by being exposed to new background knowledge that isn't a directly about intelligence explosion but causes them to see existing arguments in new light. The sequences are a likely example of that. They will also change their minds for epistemologically insane reasons like social pressure. Both those factors are hard to affect and writing posts on LessWrong seems like one of the worst ways to go about it.

No one likes being told the same thing over and over again in an insistent tone of voice. If you do that, people will get frustrated and want to criticize you. If you give in to your intuitive feeling that you need to rephrase just a little bit and this time they will surely see the light, then you will eventually rephrase your way to bullshit and give those frustrated people ample opportunity to poke holes in your arguments.

Comment author: asr 13 March 2012 09:14:20PM 2 points [-]

A plan for world domination seems like something that can't be concealed from its creators. Lying is no option if your algorithms are open to inspection.

This is just naive. Source code can be available and either the maliciousness not obvious (see the Underhanded C Contest) or not prove what you think it proves (see Reflections on Trusting Trust, just for starters). Assuming you are even inspecting all the existing code rather than a stub left behind to look like an AI.

You are arguing past each-other. XiXiDu is saying that a programmer can create software that can be inspected reliably. We are very close to having provably-correct kernels and compilers, which would make it practical to build reliably sandboxed software, such that we can look inside the sandbox and see that the software data structures are what they ought to be.

It is separately true that not all software can be reliably understood by static inspection, which is all that the underhanded C contest is demonstrating. I would stipulate that the same is true at run-time. But that's not the case here. Presumably developers of a large complicated AI will design it to be easy to debug -- I don't think they have much chance of a working program otherwise.

Comment author: gwern 13 March 2012 09:24:30PM 2 points [-]

No, you are ignoring Xi's context. The claim is not about what a programmer on the team might do, it is about what the AI might write. Notice that the section starts 'The goals of an AI will be under scrutiny at any time...'

Comment author: asr 14 March 2012 03:25:22AM * 0 points [-]

Yes. I thought Xi's claim was that if you have an AI and put it to work writing software, the programmers supervising the AI can look at the internal "motivations", "goals", and "planning" data structures and see what the AI is really doing. Obfuscation is beside the point.

Comment author: Bugmaster 14 March 2012 07:29:28PM 2 points [-]

I agree with you and XiXiDu that such observation should be possible in principle, but I also sort of agree with the detractors. You say,

Presumably developers of a large complicated AI will design it to be easy to debug...

Oh, I'm sure they'd try. But have you ever seen a large software project ? There's usually mountains and mountains of code that runs in parallel on multiple nodes all over the place. Pieces of it are usually written with good intentions in mind; other pieces are written in a caffeine-fueled fog two days before the deadline, and peppered with years-old comments to the extent of, "TODO: fix this when I have more time". When the code breaks in some significant way, it's usually easier to write it from scratch than to debug the fault.

And that's just enterprise software, which is orders of magnitude less complex than an AGI would be. So yes, it should be possible to write transparent and easily debuggable code in theory, but in practice, I predict that people would write code the usual way, instead.

Comment author: XiXiDu comment score below threshold [+] (0 children)

Comment author: XiXiDu comment score below threshold [+] (2 children)

Comment author: Bugmaster 14 March 2012 08:56:40AM 5 points [-]

Searle, why did you steal XiXiDu's account and post this?

I disagree with the gist of your comment, but I upvoted it because this quote made me LOL.

That said, I don't think that XiXiDu is claiming that computers can't exhibit creativity, period. Rather, he's saying that the kind of computers that SIAI is envisioning can't exhibit creativity, because they are implicitly (and inadvertently) designed not to.

Comment author: XiXiDu comment score below threshold [+] (13 children)

Comment author: XiXiDu 14 March 2012 09:45:09AM * -3 points [-]

All your counter-arguments are enthymematic...

Yes, a thorough analysis would take a long time and I am not the right person to do that. I only have the capability to improve it incrementally.

The reason for why I post something like this anyway is that SIAI and people like you are exclusively stating that which speaks in favor of your worldview, without any critical analysis of your beliefs. And if someone else does it for you then you make accusations in the most hypocritical manner possible.

...why things like Goedel machines are either fallacious or irrelevant.

I can't review the work of Jürgen Schmidhuber because I lack the mathematical background. But you knew that.

If his work was relevant in estimating risks from AI then it is up to people like you to write about it and show how his work does constitute evidence for your claims.

I did the best I can do. I even interviewed a bunch of AI researchers and asked others about his work in private.

Have you taken the effort to ask actual experts? You people mainly rely on surveys that ask for time-frames and then interpret that to mean that risks from AI are nigh. Even though most AI researchers who answered that AGI will happen soon would deny the implications that you assume. Which is just another proof of your general dishonesty and conformation bias.

Your argument is basically an argument from fiction;

So? What's your point?

...passing that black powder's formulation is so simple and famous that even I, who prefers archery, knows it: saltpeter, charcoal, and sulfur.

This wouldn't be nearly enough.

A civilization which exists and is there for the taking.

Magical thinking.

And to do so efficiently it takes random mutation, a whole society of minds

All of which are available to a 'simple algorithm'. Artificial life was first explored by von Neumann himself!

Yes.

Chimp brains have not improved at all, even to the point of building computers. There is an obvious disanalogy here...

Yes, what I said.

Dead-simple chess and Go algorithms routinely turn out fascinating moves. Genetic algorithms are renowned for producing results which are bizarre and inhuman and creative.

I am aware of that. Not sure what's your point though.

What is this bullshit 'computers can't exhibit creativity' doing here?

I never argued that.

Why can't I predict the next move of my chess algorithm? Why is there no algorithm to predict the AI algorithm simpler and faster than the original AI algorithm?

The point was that humans can use the same technique that the AI does. I never claimed that it would be possible to predict the next move.

Source code can be available and either the maliciousness not obvious (see the Underhanded C Contest) or not prove what you think it proves (see Reflections on Trusting Trust, just for starters). Assuming you are even inspecting all the existing code rather than a stub left behind to look like an AI.

This is just naive. We're talking about a plan for world domination, which doesn't just include massive amounts of code that would have to be hidden from inspection but also massive amount of actions.

Comment author: gwern 14 March 2012 03:25:18PM * 3 points [-]

I can't review the work of Jürgen Schmidhuber because I lack the mathematical background. But you knew that.

I know nothing about you.

If his work was relevant in estimating risks from AI then it is up to people like you to write about it and show how his work does constitute evidence for your claims.

His papers and those of Legg or Hutter are all online. I've hosted some of them myself eg. the recent Journal of Consciousness Studies one. The abstracts are pretty clear. They've been mentioned and discussed constantly on LW. You yourself have posted material on them, and material designed as an introduction for relative beginners so hopefully you read & learned from it.

Have you taken the effort to ask actual experts? You people mainly rely on surveys that ask for time-frames and then interpret that to mean that risks from AI are nigh. Even though most AI researchers who answered that AGI will happen soon would deny the implications that you assume. Which is just another proof of your general dishonesty and conformation bias.

So unless they agree in every detail, their forecasts are useless?

So? What's your point?

That story is an intuition pump (not one of my favorites, incidentally) - and your story is a pump with a broken-off handle.

This wouldn't be nearly enough.

Gee, I don't suppose you would care to enlarge on that.

Magical thinking.

Which means what, exactly? It's magical thinking to point out that our current civilization exists and is available to any AI we might make?

I am aware of that. Not sure what's your point though.

You said they necessarily lack creativity.

The point was that humans can use the same technique that the AI does. I never claimed that it would be possible to predict the next move.

I'll reiterate the quote:

So if the AI can do that, why wouldn't humans be able to use the same algorithms to predict what the initial AI is going to do?

We're talking about a plan for world domination, which doesn't just include massive amounts of code that would have to be hidden from inspection but also massive amount of actions.

So in other words, we would be able to detect the AI had gone bad while it was in the process of executing the massive amount of actions of taking over the world. I agree! Unfortunately, that's may not be a useful time to detect it...

Comment author: XiXiDu 14 March 2012 04:42:10PM 2 points [-]

I know nothing about you.

Okay, you are right. I was wrong to expect you to have read my comments where I explained how I lack the most basic education (currently trying to change that).

You yourself have posted material on them, and material designed as an introduction for relative beginners so hopefully you read & learned from it.

Yeah, I post a lot of stuff that I sense to be important and that I would love to be able to read. I hope to be able to do so in future.

So unless they agree in every detail, their forecasts are useless?

No. But if you are interested in risks from AI you should ask them about risks from AI and not just about when human-level AI will likely be invented.

That story is an intuition pump (not one of my favorites, incidentally) - and your story is a pump with a broken-off handle.

My story isn't a story but a quickly written discussion post, as a reply to post where an argument in favor of risks from AI has been outlined that was much too vague to be useful.

The problem I have is that it can be very misleading to just state that it is likely physically possible to invent smarter than human intelligence that could then be applied to its own improvement. It misses a lot of details.

Show me how that is going to work out. Or at least outline how a smarter-than-human AI is supposed to take over the world. Why is nobody doing that?

Just saying that there will be "a positive feedback loop in which an intelligence is making itself smarter" makes it look like something that couldn't possible fail.

Gee, I don't suppose you would care to enlarge on that.

100 people are not enough to produce and employ any toxic gas or bombs in a way that would defeat a wide-stretched empire with many thousands of people.

It's magical thinking to point out that our current civilization exists and is available to any AI we might make?

It is magically thinking because you don't know how that could possible work out in practice.

You said they necessarily lack creativity.

I said that there is nothing but evolution, a simple algorithm, when it comes to creativity and the discovery of unknown unknowns. I said that the full potential of evolution can only be tapped by a society of minds and its culture. I said that it is highly speculative that there exists a simple algorithm that would constitute a consequentialist AI with simple values that could achieve the same as aforementioned society of minds and therefore work better than evolution.

You just turned that into "XiXiDu believes that simple algorithms can't exhibit creativity."

So in other words, we would be able to detect the AI had gone bad while it was in the process of executing the massive amount of actions of taking over the world. I agree! Unfortunately, that's may not be a useful time to detect it...

Well, then we agree.

Comment author: gwern 14 March 2012 07:16:24PM * 4 points [-]

Show me how that is going to work out. Or at least outline how a smarter-than-human AI is supposed to take over the world. Why is nobody doing that?

People have suggested dozens of scenarios, from taking over the Internet to hacking militaries to producing nanoassemblers & eating everything. The scenarios will never be enough for critics because until they are actually executed there will always be some doubt that they would work - at which point there would be no need to discuss them any more. Just like in cryonics (if you already had the technology to revive someone, there would be no need to discuss whether it would work). This is intrinsic to any discussion of threats that have not already struck or technologies which don't already exist.

I am reminded of the quote, "'Should we trust models or observations?' In reply we note that if we had observations of the future, we obviously would trust them more than models, but unfortunately observations of the future are not available at this time."

100 people are not enough to produce and employ any toxic gas or bombs in a way that would defeat a wide-stretched empire with many thousands of people.

Because that's the best way to take over...

I said that it is highly speculative that there exists a simple algorithm that would constitute a consequentialist AI with simple values that could achieve the same as aforementioned society of minds and therefore work better than evolution. You just turned that into "XiXiDu believes that simple algorithms can't exhibit creativity."

That is not what you said. I'll requote it:

Complex values are the cornerstone of diversity, which in turn enables creativity and drives the exploration of various conflicting routes. A singleton with a stable utility-function lacks the feedback provided by a society of minds and its cultural evolution...An AI with simple values will simply lack the creativity, due to a lack of drives, to pursue the huge spectrum of research that a society of humans does pursue. Which will allow an AI to solve some well-defined narrow problems, but it will be unable to make use of the broad range of synergetic effects of cultural evolution. Cultural evolution is a result of the interaction of a wide range of utility-functions.

If a singleton lacks feedback from diversity and something which is the 'cornerstone' of diversity is something a singleton cannot have... This is actually even stronger a claim than simple algorithms, because a singleton could be a very complex algorithm. (You see how charitable I'm being towards your claims? Yet no one appreciates it.)

And that's not even getting into your claim about spectrum of research, which seems to impute stupidity to even ultraintelligent agents.

('Let's see, I'm too dumb to see that I am systematically underinvesting in research despite the high returns when I do investigate something other than X, and apparently I'm also too dumb to notice that I am underperforming compared to those oh-so-diverse humans' research programs. Gosh, no wonder I'm failing! I wonder why I am so stupid like this, I can't seem to find any proofs of it.')

Comment author: Bugmaster 14 March 2012 07:56:12PM 2 points [-]

People have suggested dozens of scenarios, from taking over the Internet to hacking militaries to producing nanoassemblers & eating everything. The scenarios will never be enough for critics because until they are actually executed there will always be some doubt that they would work...

Speaking as one of the critics, I've got to say that these scenarios are "not enough" for me not because there's "some doubt that they would work", but because there's massive doubt that they would work. To use an analogy, I look both ways before crossing the street because I'm afraid of being hit by a car; but I don't look up all the time, despite the fact that a meteorite could, theoretically, drop out of the sky and squash me flat. Cars are likely; meteorites are not.

I could elaborate regarding the reasons why I doubt some of these world takeover scenarios (including "hacking the Internet" and "eating everything"), if you're interested.

Comment author: gwern 14 March 2012 09:01:30PM * 3 points [-]

I could elaborate regarding the reasons why I doubt some of these world takeover scenarios (including "hacking the Internet" and "eating everything"), if you're interested.

Not really. Any scenario presented in any level of detail can be faulted with elaborate scenarios why it would not work. I've seen this at work with cryonics: no matter how detailed a future scenario is presented or how many options are presented in a disjunctive argument, no matter how many humans recovered from death or how many organs preserved and brought back, there are people who just never seem to think it has a non-zero chance of working because it has not yet worked.

For example, if I wanted to elaborate on the hacking the Internet scenario, I could ask you your probability on the possibility and then present information on Warhol worm simulations, prevalence of existing worms, number of root vulnerabilities a year, vulnerabilities exposed by static analysis tools like Coverity, the early results from fuzz testers, the size of the computer crime blackmarket, etc. until I was blue in the face, check whether you had changed your opinion and even if you said you changed it a little, you still would not do a single thing in your life differently.

Because, after all, disagreements are not about information. There's a lot of evidence reasoning is only about arguing and disproving other people's theories, and it's increasingly clear to me that politics and theism are strongly heritable or determined by underlying cognitive properties like performance on the CRT or personality traits; why would cryonics or AI be any different?

The point of writing is to assemble useful information for those receptive, and use those not receptive to clean up errors or omissions. If someone reads my modafinil or nicotine essays and is a puritan with regard to supplements, I don't expect them to change their minds; at most, I hope they'll have a good citation for a negative point or mention a broken hyperlink.

Comment author: Bugmaster 14 March 2012 09:32:54PM 2 points [-]

Any scenario presented in any level of detail can be faulted with elaborate scenarios why it would not work.

That is a rather uncharitable interpretation of my words. I am fully willing to grant that your scenarios are possible, but are they likely ? If you showed me a highly detailed plan for building a new kind of skyscraper out of steel and concrete, I might try and poke some holes in it, but I'd agree that it would probably work. On the other hand, if you showed me a highly detailed plan for building a space elevator out of candy-canes, I would conclude that it would probably fail to work. I would conclude this not merely because I've never seen a space elevator before, but also because I know that candy-canes make a poor construction material. Sure, you could postulate super-strong diamondoid candy-canes of some sort, but then you'd need to explain where you're going to get them from.

there are people who just never seem to think it has a non-zero chance of working because it has not yet worked.

For the record, I believe that cryonics has a non-zero chance of working.

...until I was blue in the face, check whether you had changed your opinion and even if you said you changed it a little, you still would not do a single thing in your life differently

I think this would depend on how much my opinion had, in fact, changed. If you're going to simply go ahead and assume that I'm a disingenuous liar, then sure, there's no point in talking to me. Is there anything I can say or do (short of agreeing with you unconditionally) to prove my sincerity, or is the mere fact of my disagreement with you evidence enough of my dishonesty and/or stupidity ?

The point of writing is to assemble useful information for those receptive, and use those not receptive to clean up errors or omissions.

And yet, de-converted atheists as well as converted theists do exist. Perhaps more importantly, the above sentence makes you sound as though you'd made up your mind on the topic, and thus nothing and no one could persuade you to change it in any way -- which is kind of like what you're accusing me of doing.

Comment author: XiXiDu 14 March 2012 08:24:36PM 2 points [-]

People have suggested dozens of scenarios, from taking over the Internet to hacking militaries to producing nanoassemblers & eating everything.

But none of them make any sense to me, see below.

That is not what you said. I'll requote it:

Wait, your quote said what I said I said you said I didn't say.

Because that's the best way to take over...

I have no idea. You don't have any idea either or you'd have told me by now. You are just saying that magic will happen and the world will be ours. That's the problem with risks from AI.

Let's see, I'm too dumb to see that I am systematically underinvesting in research despite the high returns when I do investigate something other than X, and apparently I'm also too dumb to notice that I am underperforming compared to those oh-so-diverse humans' research programs.

See, that's the problem. The AI can't acquire the resources that are necessary to acquire resources in the first place. It might figure out that it will need to pursue various strategies or build nanoassemblers, but how does it do that?

Taking over the Internet is no answer, because the question is how. Building nanoassemblers is no answer, because the question is how.

Comment author: gwern 14 March 2012 09:09:39PM 3 points [-]

I have no idea. You don't have any idea either or you'd have told me by now. You are just saying that magic will happen and the world will be ours. That's the problem with risks from AI.

We have plenty of ideas. Yvain posted a Discussion thread filled with ideas how. "Alternate history" is an old sub-genre dating back at least to Mark Twain (who makes many concrete suggestions about how his Connecticut yankee would do something similar).

But what's the point? See my reply to Bugmaster - it's impossible or would defeat the point of the discussion to actually execute the strategies, and anything short of execution is vulnerable to 'that's magic!11!!1'

The AI can't acquire the resources that are necessary to acquire resources in the first place. It might figure out that it will need to pursue various strategies or build nanoassemblers, but how does it do that?

By reading the many discussions of what could go wrong and implementing whatever is easiest, like hacking computers. Oh the irony!

Comment author: Bugmaster 14 March 2012 07:44:24PM * 4 points [-]

Which means what, exactly? It's magical thinking to point out that our current civilization exists and is available to any AI we might make?

I can't speak for XiXiDu, but I myself have noticed a bit of magical thinking that is sometimes employed by proponents of AGI/FAI. It goes something like this (exaggerated for effect):

1). It's possible to create an AI that would recursively make itself smarter
2). Therefore the AI would make itself very nearly infinitely smart
3). The AI would then use its intelligence to acquire godlike powers

As I see it, though, #2 does not necessarily follow from #1, unless one makes an implicit assumption that Moore's Law (or something like it) is a universal and unstoppable law of nature (like the speed of light or something). And #3 does not follow from #2, for reasons that XiXiDu articulated -- even if we assume that godlike powers can exist at all, which I personally doubt.

If you took the ten smartest scientists alive in the world today, and transported them to Ancient Rome, they wouldn't be able to build an iPhone from scratch no matter how smart they were. In addition, assuming that what we know of science today is more or less correct, we could predict with a high degree of certainty that no future scientist, no matter how superhumanly smart, would be able to build a perpetual motion device.

Edited to add: I was in the process of outlining a discussion post on this very subject, but then XiXiDu scooped me. Bah, I say !

Comment author: amcknight 14 March 2012 08:05:13PM 1 point [-]

I'd still like to see you write it, if it's concise.

Comment author: gwern 14 March 2012 08:51:00PM * 0 points [-]

As I see it, though, #2 does not necessarily follow from #1, unless one makes an implicit assumption that Moore's Law (or something like it) is a universal and unstoppable law of nature (like the speed of light or something). And #3 does not follow from #2, for reasons that XiXiDu articulated -- even if we assume that godlike powers can exist at all, which I personally doubt.

#2 does not need to follow since we already know it's false - infinite intelligence is not on offer by the basic laws of physics aside from Tipler's dubious theories. If it is replaced by 'will make itself much smarter than us', that is enough. (Have you read Chalmer's paper?)

And #3 does not follow from #2, for reasons that XiXiDu articulated -- even if we assume thAnd #3 does not follow from #2, for reasons that XiXiDu articulated -- even if we assume that godlike powers can exist at all, which I personally doubt.at godlike powers can exist at all, which I personally doubt.

Which reasons would those be? And as I've pointed out, the only way to cure your doubt if the prior history of humanity is not enough would be to actually demonstrate the powers, with the obvious issue that is.

Comment author: Bugmaster 14 March 2012 09:14:39PM 1 point [-]

If it is replaced by 'will make itself much smarter than us', that is enough.

Ok, but how much smarter ? Stephen Hawking is much smarter than me, for example, but I'm not worried about his existence, and in fact see it as a very good thing, though I'm not expecting him to invent "gray goo" anytime soon (or, in fact, ever).

I realize that quantifying intelligence is a tricky proposition, so let me put it this way: can you list some feats of intelligence, currently inaccessible to us, which you would expect a dangerously smart AI to be able to achieve ? And, segueing into #3, how do these feats of intelligence translate into operational capabilities ?

(Have you read Chalmer's paper?)

Probably not; which paper are you referring to ?

Which reasons would those be?

The ones I alluded to in my next paragraph:

If you took the ten smartest scientists alive in the world today, and transported them to Ancient Rome, they wouldn't be able to build an iPhone from scratch no matter how smart they were.

The problem here is that raw intelligence is not enough to achieve a tangible effect on the world. If your goal is to develop and deploy a specific technology, such as an iPhone, you need the infrastructure that would supply your raw materials and labor. This means that your technology can't be too far ahead of what everyone else in the world is already using.

Even if you were ten times smarter than any human, you still wouldn't be able to conjure a modern CPU (such as the one used in iPhones) out of thin air. You'd need (among other things) a factory, and a power supply to run it, and mines to extract the raw ores, and refineries to produce plastics, and the people to run them full-time, and the infrastructure to feed those people, and a government (or some other hegemony) to organize them, and so on and so forth... None of which existed in Ancient Rome (with the possible exception of the hegemony, and even that's a stretch). Sure, you could build all of that stuff from scratch, but then you wouldn't be going "FOOM", you'd be going "are we there yet" for a century or so (optimistically speaking).

the only way to cure your doubt if the prior history of humanity is not enough

Are you referring to some specific historical events ? If so, which ones ?

Comment author: amcknight 14 March 2012 08:02:50PM 7 points [-]

people like you are exclusively stating that which speaks in favor of your worldview, without any critical analysis of your beliefs. And if someone else does it for you then you make accusations in the most hypocritical manner possible.

Which is just another proof of your general dishonesty

Come on. Maybe you disagree with gwern's response and think he missed a bunch of your points, but this is just name-calling. I like your posts, but a comment like this make me lose respect for you.

gwern comments on Risks from AI and Charitable Giving - Less Wrong Discussion

Article Navigation

Comments (126)

Nearest Meetups

Virtual Study Room

Recent Comments

Recent Posts

Recent Open Threads

Recent Rationality Diaries

Recent Wiki Edits

Top Contributors, 30 Days

Recent Karma Awards

Comments (126)

Top Contributors, 30 Days

You'll need to login or register to do that

(Don't worry, it only takes a few seconds)

Create

Login