loqi comments on Applying utility functions to humans considered harmful - Less Wrong

26 Post author: Kaj_Sotala 03 February 2010 07:22PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (114)

You are viewing a single comment's thread. Show more comments above.

Comment author: loqi 05 February 2010 04:12:00AM *  1 point [-]

Raising awareness about the problem isn't just about getting would-be uFAI'ers to mend their sinful ways, you know. It's absolutely necessary if you're convinced you need help with it. As you said, inaction is untenable. If you're certain that a goal of this magnitude is basically impossible given the status quo, taking some initial risks is a trivial decision. It doesn't follow that additional risks share the same justification.

I'm also not convinced we understand the boundaries between "intelligent" and "self-improving" well enough to assume we can experiment with one and not the other. What sort of "practical tasks" do you have in mind that don't involve potentially intelligent information-processing systems, and why do you think they'll be at all relevant to the "real" work ahead?