David_Gerard comments on Less Wrong: Open Thread, December 2010 - Less Wrong Discussion
You are viewing a comment permalink. View the original post to see all comments and the full post content.
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (84)
Not just the output, the input and means of computation are also potential minefields of moral politics. After all this touches on what amounts to the ultimate moral question: "If I had ultimate power how would I decide how to use it?" When you are answering that question in public you must use extreme caution, at least you must if you have any real intent to gain power.
There are some things that are safe to say about CEV, particularly things on a technical side. But for most part it is best to avoid giving too many straight answers. I said something on the subject of what can be considered the subproblem ("Do you confess to being consequentialist, even when it sounds nasty?"). Eliezer's responses took a similar position:
When describing CEV mechanisms in detail from the position of someone with more than detached academic interest you are stuck between a rock and a hard place.
On one hand you must signal idealistic egalitarian thinking such that you do not trigger in the average reader those aversive instincts we have for avoiding human tyrants.
On the other hand you must also be aware that other important members (ie. many of those likely to fund you) of your audience will have a deeper understanding of the practical issues and will see the same description as naive to the point of being outright dangerous and destructive.
My application is so that an organisation can work out not only what people want from it but what they would want from it. This assumes some general intelligences on hand to do the working out, but we have those.