You're looking at Less Wrong's discussion board. This includes all posts, including those that haven't been promoted to the front page yet. For more information, see About Less Wrong.

Mitchell_Porter comments on Have you changed your mind lately? On what? - Less Wrong Discussion

25 Post author: Emile 04 June 2012 07:54PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (105)

You are viewing a single comment's thread. Show more comments above.

Comment author: Kaj_Sotala 05 June 2012 08:28:30AM 15 points [-]

I read Shalizi's post on the difficulties of central planning, where he noted that even using something as simple as linear optimization to organize things becomes impossible if you need to do it on the scale of a national economy. This made me significantly reduce my belief in the proposition that something like CEV would be anywhere near computationally tractable, at least in the form that it's usually discussed.

That made me consider something like Goertzel & Pitt's human-assisted CBV approach, where much of the necessary computation gets outsourced to humans, as an approach that's more likely to work. Of course, their approach pretty much requires a slow takeoff in order to work, and I consider a hard takeoff pretty likely. Logically I should then have updated more towards considering that we'll end of losing our complexity of value during the Singularity but I didn't, possibly because I was already giving that a very high probability anyway and I can't perceive my intuitive probability estimates in sufficiently high precision for the difference to register. However I did update considerably towards thinking that Goertzel's ideas on Friendliness have more merit than I'd previously presumed, and that people should be looking in a direction like the one Goerzel & Pitt propose.

Comment author: Mitchell_Porter 06 June 2012 02:39:06PM *  4 points [-]

The principal challenge of CEV is not the part where you take into account all the specific, contingent individuals who exist on the earth, in the course of following some recipe for extrapolating, aggregating, etc. The important step is figuring out the "recipe" itself. If we had a recipe which we knew to perfectly reflect true human ideals, but it proved to be computationally infeasible, we (or a FAI) could then think about feasible approximations to this ideal.

The important questions are: What sort of decision system is a typical human being? What are the values and value templates with which it comes equipped? These questions should have exact answers just as objective as the facts about how many limbs or how many bones a typical human has, and those answers should in turn imply that "friendliness" and "CEV" ought to work in some specific way.