timtyler comments on Safety Culture and the Marginal Effect of a Dollar - Less Wrong

23 Post author: jimrandomh 09 June 2011 03:59AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Article Navigation

Comments (105)
Tags: ai singularity safety philanthropy

Comments (105)

You are viewing a single comment's thread.

Comment author: timtyler 09 June 2011 05:30:38PM *  3 points [-]

The model is this: assume that if an AI is created, it's because one researcher, chosen at random from the pool of all researchers, has the key insight; and humanity survives if and only if that researcher is careful and takes safety seriously.

The "key insight" model seems deeply flawed. We know that the technical side of the problem involves performing inductive inference - which is a close cousin of stream compression. So, progress is very likely to look like progress with stream compression. Some low-hanging fruit - and then gradually diminishing returns. Rather like digging a big hole in the ground.

Comment author: timtyler 11 June 2011 12:01:44PM *  1 point [-]

Here's Bob Mottram making much the same point as I just made:

If you define an improvement of intelligence as being like optimizing a bunch of algorithms, such that you can do more, or the same, with fewer computes (approaching maximum compression) then the chances of a hard takeoff appear grim indeed.

Experience with optimizing systems such as genetics algorithms/programming suggests that the rate of improvement in performance decreases over time and becomes more and more tortuous with small improvements occurring less frequently. There may be occasional discoveries of new vistas on the fitness landscape, but these are not common as we march towards optimum compression.

Thus far nobody seems to have really been able to address this problem of a general optimizer which doesn't run out of steam over time. To show that a hard takeoff is possible, at least in principle, it's going to be necessary to demonstrate that you can devise an optimizer which doesn't run out of steam, and in fact does the opposite.

Comment author: GuySrinivasan 09 June 2011 05:45:19PM 0 points [-]

How confident should we be that general AI involves solely hard work on existing problems like performing inductive inference? I agree that if there are no more Key Insights, and instead just a bunch of insights that some researcher will eventually have, then most of the gains from the proposal can't be realized. Next steps: somehow estimate the probability that there are 0, 1, or several Key Insights remaining before general AI is "just" a matter of tons of hard research/experimentation, and estimate the gains from the 100-paper-strategy for the scenarios in which there are 0 or several Key Insights remaining.

Comment author: timtyler 09 June 2011 06:03:02PM *  0 points [-]

How confident should we be that general AI involves solely hard work on existing problems like performing inductive inference?

I didn't really claim that. There's also the whole issue of what utility function to use - and some other things as well - tree pruning strategies, for instance. Just that inductive inference is the key technology for the technical side of the problem - the part not to do with values.

Much has been written about the link between induction and intelligence: Hutter. Mahoney. Me.

Powered by Reddit Powered by Reddit