Kaj_Sotala comments on Three Approaches to "Friendliness" - Less Wrong

14 Post author: Wei_Dai 17 July 2013 07:46AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Article Navigation

Comments (84)
Tags: strategy philosophy fai metaphilosophy

Comments (84)

You are viewing a single comment's thread. Show more comments above.

Comment author: Kaj_Sotala 17 July 2013 11:03:41AM 3 points [-]

Yes, you could probably broaden the concept to cover negative utilitarianism as well, though Bostrom's original article specifically defined astronomical waste as being

an opportunity cost: a potential good, lives worth living, is not being realized.

That said, even if you did redefine the concept in the way that you mentioned, the term "astronomical waste" still implies an emphasis on taking over the universe - which is compatible with negative utilitarianism, but not necessarily every ethical theory. I would suspect that most people's "folk morality" would say something like "it's important to fix our current problems, but expanding into space is morally relevant only as far as it affects the primary issues" (with different people differing on what counts as a "primary issue").

I'm not sure whether you intended the emphasis on space expansion to be there, but if it was incidental, maybe you rather meant something like

I put "Friendliness" in quotes in the title, because I think what we really want, and what MIRI seems to be working towards, is closer to "optimality": create an AI that makes our world into what we'd consider the best possible one

?

(I hope to also post a more substantial comment soon, but I need to think about your post a bit more first.)

Comment author: Wei_Dai 17 July 2013 10:12:11PM *  1 point [-]

That said, even if you did redefine the concept in the way that you mentioned, the term "astronomical waste" still implies an emphasis on taking over the universe - which is compatible with negative utilitarianism, but not necessarily every ethical theory. I would suspect that most people's "folk morality" would say something like "it's important to fix our current problems, but expanding into space is morally relevant only as far as it affects the primary issues" (with different people differing on what counts as a "primary issue").

This is getting a bit far from the original topic, but my personal approach to handling moral uncertainty is inspired by Bostrom and Ord, and works by giving each moral faction a share of my resources and letting them trade to make Pareto improvements. So the "unbounded utility" faction in me was responsible for writing the OP (using its share of my time), and the intended audience is the "unbounded utility" factions in others. That's why it seems to be assuming unbounded utility and has an emphasis on space expansion, even though I'm far from certain that it represents "correct morality" or my "actual values".

Powered by Reddit Powered by Reddit