the dichotomy "paper clip maximizer vs. Friendly AI" seems like a false dichotomy - I imagine that the sort of AI that people would actually build would be somewhere in the middle. Any recommended reading on this point appreciated.
Mainly Complexity of value. There is no way for human values to magically jump inside the AI, so if it's not specifically created to reflect them, it won't have them, and whatever the AI ends up with won't come close to human values, because human values are too complex to be resembled by any given structure that happens to be formed in the AI.
The more AI's preference diverges from ours, the more we lose, and this loss is on astronomic scale (even if preference diverges relatively little). The falloff with imperfect reflection of values might be so sharp that any ad-hoc solution turns the future worthless. Or maybe not, with certain classes of values that contain a component of sympathy that reflects values perfectly while giving them smaller weight in the overall game, but then we'd want to technically understand this "sympathy" to have any confidence in the outcome.
The more AI's preference diverges from ours, the more we lose, and this loss is on astronomic scale (even if preference diverges relatively little).
This depends on something like aggregative utilitarianism. If additional resources have diminishing marginal value in fulfilling human aims, that getting a little slice of the universe (in the course of negotiating terms of surrender with the inhuman AI, if it can make credible commitments, or because we serve as acausal bargaining chips with other civilizations elsewhere in the universe) may be enough. Is getting 100% of the lightcone a hundred times better than 1%?
This thread is for the discussion of Less Wrong topics that have not appeared in recent posts. If a discussion gets unwieldy, celebrate by turning it into a top-level post.