D_Malik comments on Open Thread for February 18-24 2014 - Less Wrong

4 Post author: eggman 19 February 2014 12:57PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (454)

You are viewing a single comment's thread. Show more comments above.

Comment author: D_Malik 20 February 2014 10:42:31PM 3 points [-]

Omohundro has a paper on instrumental goals that many/most intelligences would converge on. For instance, they would strive to model themselves, to expand their capabilities, to represent their goals in terms of utility functions, to protect their utility functions against change, etc. None of these are universally true because we can just posit a pathological intelligence whose terminal goal is to not do these things. (And to some extent e.g. humans do in fact behave pathologically like that.)

We can say very little "optimizers over possible futures" in full generality, because that concept can be very broad if you define "optimizer" sufficiently broadly. Is a thermostat an intelligence, with the goal of achieving some temperature? Or consider a rock - we can see it as a mind with the goal of continuing its existence, and thus decides to be hard.

Comment author: shminux 20 February 2014 11:34:58PM 0 points [-]

It seems that the paper discusses the inside view of intelligence, not the ways to detect one by its non-human-like artifacts.

I agree that it is hard to tell an intelligence without pattern-matching it to humans, that's why I asked the question in the first place. But there hopefully should be at least some way to be convinced that a rock is not very intelligent, even if you can't put yourself in its crystalline shoes.