What happens after a FAI is built? There's a lot of discussion on how to build one, and what traits it needs to have, but little on what happens afterward. How does the world/humanity transition from the current systems of government to a better one? Do we just assume that the FAI is capable of handling a peaceful and voluntary global transition, or are there some risks involved? How do you go about convincing the entirety of humanity that the AI that has been created is "safe" and to put our trust in it?
Local thinking about FAI is predicated on the assumption that an AI is probably capable of (and will initiate) extremely rapid self-improvement (the local jargon is "FOOMing," which doesn't stand for anything as far as I know, it just sounds evocative), such that it rapidly becomes a significantly superhuman intelligence, and thereafter all such decisions can profitably left up to the FAI itself.
Relatedly, local thinking about why FAI is important is largely predicated on the same assumption... if AIs will probably FOOM, then UFAI will probably i...
This is a thread where people can ask questions that they would ordinarily feel embarrassed for not knowing the answer to. The previous thread is at close to 500 comments.