RobbBB comments on Magical Categories - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (89)
You're confusing 'smart enough to solve FAI' with 'actually solved FAI', and you're confusing 'actually solved FAI' with 'self-modified to become Friendly'. Most possible artificial superintelligences have no desire to invest much time into figuring out human value, and most possible ones that do figure out human value have no desire to replace their own desires with the desires of humans. If the genie knows how to build a Friendly AI, that doesn't imply that the genie is Friendly; so superintelligence doesn't in any way imply Friendliness even if it implies the ability to become Friendly.
Why does that comment make his point irrelevant? Are you claiming that it's easy to program superintelligences to be 'rational'', where 'rationality' doesn't mean instrumental or epistemic rationality but instead means something that involves being a moral paragon? It just looks to me like black-boxing human morality to make it look simpler or more universal.
And how do you code that? If the programmers don't know what 'be moral' means, then how do they code the AI to want to 'be moral'? See Truly Part Of You.
A human with superintelligence-level superpowers would be an existential threat. An artificial intelligence with superintelligence-level superpowers would therefore also be an existential threat, if it were merely as ethical as a human. If your bar is set low enough to cause an extinction event, you should probably raise your bar a bit.
No. Read Haidt's paper, and beware of goalpost drift.
No. Human law isn't built for superintelligences, so it doesn't put special effort into blocking loopholes that would be available to an ASI. E.g., there's no law against disassembling the Sun, because no lawmaker anticipated that anyone would have that capability.
... Which isn't computable, and provides no particular method for figuring out what the variables are. 'Preferences' isn't operationalized.
Values in general are what matters for Friendly AI, not moral values. Moral values are a proper subset of what's important and worth protecting in humanity.