Regina Spektor wrote a song about CEV called "The Calculation". She seems to agree with Shane Legg that it may well be impossible, but she appears to think there's hope. YouTube video here.
Excerpt:
You went into the kitchen cupboard
Got yourself another hour
And you gave
Half of it to me
We sat there looking at the faces
Of these strangers in the pages
'Til we knew 'em mathematicallyThey were in our minds
Until forever
But we didn't mind
We didn't know betterSo we made our own computer out of macaroni pieces
And it did our thinking while we lived our lives
It counted up our feelings
And divided them up even
And it called that calculation perfect loveDidn't even know that love was bigger
Didn't even know
That love was so, so
Hey Hey HeyHey this fire it's burnin'
Burnin' us up
So I suppose she's critical of naive aggregation methods for preference fulfillment.
So we made our own computer out of macaroni pieces
So that's the secret of achieving true AI!
This thread is for the discussion of Less Wrong topics that have not appeared in recent posts. If a discussion gets unwieldy, celebrate by turning it into a top-level post.