timtyler comments on Safety Culture and the Marginal Effect of a Dollar - Less Wrong

23 Post author: jimrandomh 09 June 2011 03:59AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Article Navigation

Comments (105)
Tags: ai singularity safety philanthropy

Comments (105)

You are viewing a single comment's thread. Show more comments above.

Comment author: timtyler 09 June 2011 05:30:38PM *  3 points [-]

The model is this: assume that if an AI is created, it's because one researcher, chosen at random from the pool of all researchers, has the key insight; and humanity survives if and only if that researcher is careful and takes safety seriously.

The "key insight" model seems deeply flawed. We know that the technical side of the problem involves performing inductive inference - which is a close cousin of stream compression. So, progress is very likely to look like progress with stream compression. Some low-hanging fruit - and then gradually diminishing returns. Rather like digging a big hole in the ground.

Comment author: timtyler 11 June 2011 12:01:44PM *  1 point [-]

Here's Bob Mottram making much the same point as I just made:

If you define an improvement of intelligence as being like optimizing a bunch of algorithms, such that you can do more, or the same, with fewer computes (approaching maximum compression) then the chances of a hard takeoff appear grim indeed.

Experience with optimizing systems such as genetics algorithms/programming suggests that the rate of improvement in performance decreases over time and becomes more and more tortuous with small improvements occurring less frequently. There may be occasional discoveries of new vistas on the fitness landscape, but these are not common as we march towards optimum compression.

Thus far nobody seems to have really been able to address this problem of a general optimizer which doesn't run out of steam over time. To show that a hard takeoff is possible, at least in principle, it's going to be necessary to demonstrate that you can devise an optimizer which doesn't run out of steam, and in fact does the opposite.

Powered by Reddit Powered by Reddit