What is your take on the strategy-swallowing point: if humans can do it, then not very superintelligent AIs can.
It's easy to imagine AIXI-like Bayesian EU maximizers that are powerful optimizers but incapable of solving philosophical problems like consciousness, decision theory, and foundations of mathematics, which seem to be necessary in order to build an FAI. It's possible that that's wrong, that one can't actually get to "not very superintelligent AIs" unless they possessed the same level of philosophical ability that humans have, but it certainly doesn't seem safe to assume this.
BTW, what does "strategy-swallowing" mean? Just "strategically relevant", or more than that?
But, e.g. Friendliness just sounds silly. I use "safe" too, but safety can be achieved just by limiting capabilities, which doesn't reflect the desire to realize the benefits.
I suggested "optimal AI" to Luke earlier, but he didn't like that. Here are some more options to replace "Friendly AI" with: human-optimal AI, normative AI (rename what I called "normative AI" in this post to something else), AI normativity. It would be interesting and useful to know what options Eliezer considered and discarded before settling on "Friendly AI", and what options Nick considered and discarded before settling on "AI control".
(I wonder why Nick doesn't like to blog. It seems like he'd want to run at least some of the more novel or potentially controversial ideas in his book by a wider audience, before committing them permanently to print.)
It's easy to imagine AIXI-like Bayesian EU maximizers that are powerful optimizers but incapable of solving philosophical problems like consciousness, decision theory, and foundations of mathematics, which seem to be necessary in order to build an FAI. It's possible that that's wrong, that one can't actually get to "not very superintelligent AIs" unless they possessed the same level of philosophical ability that humans have, but it certainly doesn't seem safe to assume this.
Such systems, hemmed in and restrained, could certainly work on better...
In the past, people like Eliezer Yudkowsky (see 1, 2, 3, 4, and 5) have argued that MIRI has a medium probability of success. What is this probability estimate based on and how is success defined?
I've read standard MIRI literature (like "Evidence and Import" and "Five Theses"), but I may have missed something.
-
(Meta: I don't think this deserves a discussion thread, but I posted this on the open thread and no-one responded, and I think it's important enough to merit a response.)