= 27d567bc2d48d9f40e9ccc20ea370b15)
D_Malik comments on Open Thread for February 18-24 2014 - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (454)
Can one detect intelligence in retrospect?
Let me explain. Let's take the definition of an intelligent agent as an optimizer over possible futures, steering the world toward the preferred one. Now, suppose we look at the world after the optimizer is done. Only one of the many possible worlds, the one steered by the optimizer, is accessible to retrospection. Let's further assume that we have no access to the internals of the optimizer, only to the recorded history. In particular, we cannot rely on it having human-like goals and use pattern-matching to whatever a human would do.
Is there still enough data left to tell with high probability that an intelligent optimizer is at work, and not just a random process? If so, how would one determine that? If not, what hope do we have of detecting an alien intelligence?
Omohundro has a paper on instrumental goals that many/most intelligences would converge on. For instance, they would strive to model themselves, to expand their capabilities, to represent their goals in terms of utility functions, to protect their utility functions against change, etc. None of these are universally true because we can just posit a pathological intelligence whose terminal goal is to not do these things. (And to some extent e.g. humans do in fact behave pathologically like that.)
We can say very little "optimizers over possible futures" in full generality, because that concept can be very broad if you define "optimizer" sufficiently broadly. Is a thermostat an intelligence, with the goal of achieving some temperature? Or consider a rock - we can see it as a mind with the goal of continuing its existence, and thus decides to be hard.
It seems that the paper discusses the inside view of intelligence, not the ways to detect one by its non-human-like artifacts.
I agree that it is hard to tell an intelligence without pattern-matching it to humans, that's why I asked the question in the first place. But there hopefully should be at least some way to be convinced that a rock is not very intelligent, even if you can't put yourself in its crystalline shoes.