If it's that capable, it's probably also that dangerous.
No. We already have computers that help design better airplanes etc., and they are not dangerous at all. Sewing-Machine's question is right on.
Building machines that help us solve intelligence-bound problems (even if these problems are related to the real world, like building better airplanes) seems to be massively easier than building machines that will "understand" the existence of the real world and try to take it over for whatever reason. Evidence: we have had much success with the former task, but practically no progress on the latter. Moreover, the latter task looks very dangerous, kinda like nuclear weaponry.
Why do some people become so enamored with the singleton scenario that they can't settle for anything less? What's wrong with humans using "smart enough" machines to solve world hunger and such, working out any ethical issues along the way, instead of delegating the whole task to one big AI? If you think you need the singleton to protect you from some danger, what can be more dangerous than a singleton?
Why do some people become so enamored with the singleton scenario that they can't settle for anything less? What's wrong with humans using "smart enough" machines to solve world hunger and such, working out any ethical issues along the way, instead of delegating the whole task to one big AI?
It's potentially dangerous, given the uncertainty about what exactly you are talking about. If it's not dangerous, go for it.
Settling for something less than a singleton won't solve the problem of human-indifferent intelligence explosion.
...If you think you
According to Eliezer, making AI safe requires solving two problems:
1) Formalize a utility function whose fulfillment would constitute "good" to us. CEV is intended as a step toward that.
2) Invent a way to code an AI so that it's mathematically guaranteed not to change its goals after many cycles of self-improvement, negotiations etc. TDT is intended as a step toward that.
It is obvious to me that (2) must be solved, but I'm not sure about (1). The problem in (1) is that we're asked to formalize a whole lot of things that don't look like they should be necessary. If the AI is tasked with building a faster and more efficient airplane, does it really need to understand that humans don't like to be bored?
To put the question sharply, which of the following looks easier to formalize:
a) Please output a proof of the Riemann hypothesis, and please don't get out of your box along the way.
b) Please do whatever the CEV of humanity wants.
Note that I'm not asking if (a) is easy in absolute terms, only if it's easier than (b). If you disagree that (a) looks easier than (b), why?