I read Shalizi's post on the difficulties of central planning, where he noted that even using something as simple as linear optimization to organize things becomes impossible if you need to do it on the scale of a national economy. This made me significantly reduce my belief in the proposition that something like CEV would be anywhere near computationally tractable, at least in the form that it's usually discussed.
That made me consider something like Goertzel & Pitt's human-assisted CBV approach, where much of the necessary computation gets outsourced to humans, as an approach that's more likely to work. Of course, their approach pretty much requires a slow takeoff in order to work, and I consider a hard takeoff pretty likely. Logically I should then have updated more towards considering that we'll end of losing our complexity of value during the Singularity but I didn't, possibly because I was already giving that a very high probability anyway and I can't perceive my intuitive probability estimates in sufficiently high precision for the difference to register. However I did update considerably towards thinking that Goertzel's ideas on Friendliness have more merit than I'd previously presumed, and that people should be looking in a direction like the one Goerzel & Pitt propose.
The principal challenge of CEV is not the part where you take into account all the specific, contingent individuals who exist on the earth, in the course of following some recipe for extrapolating, aggregating, etc. The important step is figuring out the "recipe" itself. If we had a recipe which we knew to perfectly reflect true human ideals, but it proved to be computationally infeasible, we (or a FAI) could then think about feasible approximations to this ideal.
The important questions are: What sort of decision system is a typical human being?...
Admitting to being wrong isn't easy, but it's something we want to encourage.
So ... were you convinced by someone's arguments lately? Did you realize a heated disagreement was actually a misunderstanding? Here's the place to talk about it!