roll
1
1
21
roll has not written any posts yet.
roll has not written any posts yet.
That depends what your initial probability is and why. If it already low due to updates on predictions about the system, then updating on "unpredictable" will increase the probability by lowering the strength of those predictions. Since destruction of humanity is rather important, even if the existential AI risk scenario is of low probability it matters exactly how low.
The importance should not weight upon our estimation, unless you proclaim that I should succumb to a bias. Furthermore, it is the destruction of the mankind that is the prediction being made here. Via multitude of assumptions, the most dubious one being that the system will have real-world, physical goal. Number of paperclips is... (read more)
Well, that's the Luke's aspirations; I was referring to the work done so far. The whole enterprise has the feeling of over optimistic startup with ill defined extremely ambitious goals; those don't have any success rate even for much much simpler goals.
A suggestion: it may be a bad idea to use word 'artificial intelligence' in the name without qualifiers, as to serious people in the field
the 'artificial intelligence' has much, much broader meaning than what SI is concerning itself with
there is very significant disdain for the commonplace/'science fiction' use of 'artificial intelligence'
Center for Helpful Artificial Optimizer Safety
What concerns me is lack of research into artificial optimizers in general... Artificial optimizers are commonplace already, they are algorithms to find optimal solutions to mathematical models, not to optimize the real world in the manner that SI is concerned with (correct me if I am wrong). Furthermore the premise is that such optimizers would 'foom', and i fail to see how foom is not a type of singularity.
Hmm what do you think would have happened with that someone if the name was more attractive and that person spent more time looking into SI? Do you think that person wouldn't ultimately dismiss it? Many of the premises here seem more far fetched than singularity. I know that from our perspective it'd be great to have feedback from such people, but it wastes their time and it is unclear if that is globally beneficial.
From what I gathered SI's relevance rests upon an enormous conjunction of implied and a very narrow approach as solution, both of which were decided upon significant time in the past. Subsequently, truly microscopic probability of relevance is easily attained; I estimate at most 10^-20 due to multiple use of narrow guesses into a huge space of possibilities.
I certainly agree, and I am not even sure what the official SI position is on the probability of such failure. I know that Eliezer in hist writing does give the impression that any mistake will mean certain doom, which I believe to be an exaggeration. But failure of this kind is fundamentally unpredictable, and if a low probability even kills you, you are still dead, and I think that it is high enough that the Friendly AI type effort would not be wasted.
Unpredictable is a subjective quality. It'd look much better if the people speaking of unpredictability had demonstrable accomplishment. If there is a trillion equally probable unpredictable outcomes, out of... (read 441 more words →)
There's probably a lot of low hanging fruit, for example use of correct priors, e.g. given Gaussian prior distribution, a quite strong proof should be needed before you should believe someone (including yourself) has very high intelligence or expertise on a task.
Furthermore, many ways of evaluating people are to some extent self reinforcing as the people being evaluated are aware of evaluation. A smart person or expert can relatively cheaply demonstrate intelligence and/or expertise, in some way that provides very strong evidence, and will do so even for relatively moderate payoffs. Likewise, very selfless people can cheaply (in terms of their utilons) help strangers, and will do so for lower payoffs than people with more selfish utilities.
Other issue is gradual updates on huge amounts of weak, possibly non-independent evidence.
I think what may be confusing about expected outcome is the name. You don't actually expect to get 5 dollars out of this :) . You don't even expect to get, say, 5 million dollars after 1 million games, such would be rather unlikely. You do expect to get 5$ per game if you played infinitely many games, though, and if you are playing such games on small amounts of money you can choose the game to play based on expected outcome.
Any references? I haven't seen anything that is in any way relevant to the type of optimization that we currently know how to implement. The SI is concerned with notion of some 'utility function', which appears very fuzzy and incoherent - what it is, a mathematical function? What does it have at input and what it has at output? The number of paperclips in the universe is given as example of 'utility function', but you can't have 'universe' as the input domain to a mathematical function. In the AI the 'utility function' is defined on the model rather... (read more)