= 27d567bc2d48d9f40e9ccc20ea370b15)
Wei_Dai comments on What if AI doesn't quite go FOOM? - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (186)
I am disagreeing on the question of fact. What we can do without an FAI is by far superior to any scraps we can expect smiley-face maximiser to contribute due to exchanges.The greatest of the existential risks that not having an FAI entails is the threat of an uFAI. anti-AI removes that. We do have some potential for survival based on other technologies within our grasp. SIAI would have to devote itself to solving other hard problems.
Wei mentioned a combinatorial explosion. He may have been applying it somewhat differently than I am but I am claiming that an overwhelming number of the possible mind designs that Smiley is bargaining with are also bad for me. He is bargaining with a whole lot of Clippy's brothers and sisters. Bargaining with a whole lot of GAIs that are released that care primarily about their own propagation. Even more importantly that small proportion of FAIs that do exist are not friendly to things I care about. Almost none of them will result in me personally being alive.
This all assumes that the bargaining does in fact go ahead. I'm not certain either way either and nor am I certain that in the specific case of Smiley one of his optimal trading partners will be an FAI which I happen to like.
All this means that I am comfortable with the assertion you quote. If you or Rolf did try to stop me from pressing that no-AI button then you would just be obstacles that needed to be eliminated, even if your motives are pure. My life and all that I hold dear is at stake!
I think that makes some sense. It's not clear to me that building a smiley-face maximizer that trades with AIs in other possible worlds would be better than having a no-AI future.
There is another possibility to consider though. Both we and the smiley-face maximizer would be better off if we did allow it to be built, and then it gives our preferences some control (enough for us to be better off than the no-AI future). It's not clear that this opportunity for trade can be realized, but we should spend some time thinking about it before ruling it out.
It seems like we really need a theory of games that tells us (human beings) how to play games with superintelligences. We can't depend on our FAIs to play the games for us, because we have to decide now what to do, including the above example, and also what kind of FAI to build.
Sounds like Drescher's bounded Newcomb. This perspective suddenly painted it FAI-complete.
Can you please elaborate? I looked up "FAI-complete", and found this but I still don't get your point.
See the DT list. (Copy of the post here.) FAI-complete problem = solving it means that FAI gets solved as well.