wedrifid comments on What I would like the SIAI to publish - Less Wrong

27 Post author: XiXiDu 01 November 2010 02:07PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Article Navigation

Comments (218)
Tags: siai artificial intelligence existential risks

Comments (218)

You are viewing a single comment's thread. Show more comments above.

Comment author: wedrifid 02 November 2010 11:14:35AM 5 points [-]

So why isn't for example nanotechnology a more likely and therefore bigger existential risk than AGI?

That is a good question and I have no idea. The degree of existential threat there is most significantly determined by relative ease of creation. I don't know enough to be able to predict which would be produced first - self replicating nano-technology or an AGI. SIAI believes the former is likely to be produced first and I do not know whether or not they have supported that claim.

Other factors contributing to the risk are:

  • Complexity - the number of ways the engineer could screw up while creating it in a way that would be catastrophic. The 'grey goo' risk is concentrated more specifically to the self replication mechanism of the nanotech while just about any mistake in an AI could kill us.
  • Awareness of the risks. It is not too difficult to understand the risks when creating a self replicating nano-bot. It is hard to imagine an engineer creating one not seeing the problem and being damn careful. Unfortunately it is not hard to imagine Ben.
Comment author: David_Gerard 02 November 2010 11:31:09AM *  4 points [-]

I find myself confused at the fact that Drexlerian nanotechnology of any sort is advocated as possible by people who think physics and chemistry work. Materials scientists - i.e. the chemists who actually work with nanotechnology in real life - have documented at length why his ideas would need to violate both.

This is the sort of claim that makes me ask advocates to document their Bayesian network. Do their priors include the expert opinions of materials scientists, who (pretty much universally as far as I can tell) consider Drexler and fans to be clueless?

(The RW article on nanotechnology is mostly written by a very annoyed materials scientist who works at nanoscale for a living. It talks about what real-life nanotechnology is and includes lots of references that advocates can go argue with. He was inspired to write it by arguing with cryonics advocates who would literally answer almost any objection to its feasibility with "But, nanobots!")

Comment author: timtyler 02 November 2010 09:05:37PM *  2 points [-]

That RationalWiki article is a farce. The central "argument" seems to be:

imagine a car production line with its hi-tech robotic arms that work fine at our macroscopic scale. To get a glimpse of what it would be like to operate a production line on the microscopic scale, imagine filling the factory completely with gravel and trying to watch the mechanical arms move through it - and then imagine if the gravel was also sticky.

So: they don't even know that Drexler-style nanofactories operate in a vacuum!

They also need to look up "Kinesin Transport Protein".

Comment author: David_Gerard 03 November 2010 09:05:24AM *  1 point [-]

Drexler-style nanofactories don't operate in a vacuum, because they don't exist and no-one has any idea whatsoever how to make such a thing exist, at all. They are presently a purely hypothetical concept with no actual scientific or technological grounding.

The gravel analogy is not so much an argument as a very simple example for the beginner that a nanotechnology fantasist might be able to get their head around; the implicit actual argument would be "please, learn some chemistry and physics so you have some idea what you're talking about." Which is not an argument that people will tend to accept (in general people don't take any sort of advice on any topic, ever), but when experts tell you you're verging on not even wrong and there remains absolutely nothing to show for the concept after 25 years, it might be worth allowing for the possibility that Drexlerian nanotechnology is, even if the requisite hypothetical technology and hypothetical scientific breakthroughs happen, ridiculously far ahead of anything we have the slightest understanding of.

Comment author: timtyler 03 November 2010 09:09:56AM *  1 point [-]

"The proposal for Drexler-style nanofactories has them operating in a vacuum", then.

If these wannabe-critics don't understand that then they have a very superficial understanding of Drexler's proposals - but are sufficiently unaware of that to parade their ignorance in public.

Comment author: David_Gerard 03 November 2010 09:18:18AM -2 points [-]

The "wannabe-critics" are actual chemists and physicists who actually work at nanoscale - Drexler advocates tend to fit neither qualification - and who have written long lists of reasons why this stuff can't possibly work and why Drexler is to engineering what Ayn Rand is to philosophy.

I'm sure they'll change their tune when there's the slightest visible progress on any of Drexler's proposals; the existence proof would be pretty convincing.

Comment author: timtyler 03 November 2010 09:25:17AM *  2 points [-]

Hah! A lot of the edits on that article seem to have been made by you!

Comment author: David_Gerard 03 November 2010 09:30:34AM *  -1 points [-]

Yep. Mostly written by Armondikov, who is said annoyed material scientist. I am not, but spent some effort asking other material scientists who work or have worked at nanoscale their expert opinions.

Thankfully, the article on the wiki has references, as I noted in my original comment.

So what were the priors that went into your considered opinion?

Powered by Reddit Powered by Reddit