You're looking at Less Wrong's discussion board. This includes all posts, including those that haven't been promoted to the front page yet. For more information, see About Less Wrong.

Kaj_Sotala comments on Three kinds of moral uncertainty - Less Wrong Discussion

32 Post author: Kaj_Sotala 30 December 2012 10:43AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Article Navigation

Comments (15)
Tags: moral_uncertainty ethics philosophy

Comments (15)

You are viewing a single comment's thread. Show more comments above.

Comment author: Kaj_Sotala 30 December 2012 06:34:28PM *  5 points [-]

Thanks!

Kind of, though "intrinsic uncertainty" also suggests the possibility that the subsystems might be generating moral intuitions which simply cannot be reconciled and that the conflict might be unresolvable unless one is willing to completely cut away or rewrite parts of their own mind. (Though this does not presuppose that the conflict really is unresolvable, merely that it might be.) That makes "not finished computing" somewhat ill-fitting of a name, since that seems to imply that the conflict could be eventually resolved. Not sure if "mere internal conflict" really is it, either. "Intrinsic" was meant to refer to this kind of conflict emerging from an agent holding mutually incompatible intrinsic values, and it being impossible to resolve the conflict via appeal to instrumental considerations.

Comment author: Gavin 30 December 2012 06:51:26PM *  2 points [-]

Is your purpose to describe the underlying moral issue or what different issues feel like?

For instance:

  • I feel morally certain but in reality would change my view if strong evidence were presented.
  • I feel morally certain and won't change my mind.
  • I feel descriptive uncertainty, and if I just read the right moral formulation I would agree that it described me perfectly
  • I feel descriptive uncertainty, but actually have a deep internal conflict
  • I feel deep internal conflict, and am right about it
  • I feel deep internal conflict, but more epistemic information would resolve the issue

I think the issue may be that you're trying to categorize "how it feels now" with "my underlying morality, to which I have limited access" in the same system. Maybe two sets of categories are needed? For instance, the top level system can experience descriptive uncertainty, but the underlying reality cannot.

ETA: Here's my attempt at extending the categories to cover both conscious feelings and the underlying reality.

Conscious moral states:

  • Moral certainty -- I feel like I know the answer with no serious reservations
  • Basic moral uncertainty -- I feel like I don't know how to tackle the problem at all
  • Descriptive moral uncertainty -- I feel like I know this, but can't come up with a good description
  • Epistemic moral uncertainty -- I feel like I need more information to figure it out
  • Conflicted moral uncertainty -- I feel like there are two values competing

Subconscious (real? territory?) moral states:

  • Moral certainty -- I have a clear algorithm for this problem
  • Basic moral uncertainty -- I just don't have an answer at all
  • Conflicted moral uncertainty -- I have two or more systems which give different answers
  • Epistemic moral uncertainty -- I need more information to give a confident answer
Comment author: Kaj_Sotala 31 December 2012 07:46:32AM 1 point [-]

Interesting. Your extended categorization seems like it could very possibly be useful - I'll have to think about it some more.

Comment author: Academian 30 December 2012 09:20:51PM *  0 points [-]

Kind of, though "intrinsic uncertainty" also suggests the possibility that the subsystems might be generating moral intuitions which simply cannot be reconciled and that the conflict might be unresolvable unless one is willing to completely cut away or rewrite parts of their own mind.

Don't you think that things being perfectly balanced in a way such that there is no resolution is sort of a measure zero set of outcomes? In drift-diffusion models of neural groups in human and animal brains arrive at decisions/actions (explained pretty well here), even if the drift term (tendency to eventually favor one outcome) is zero, the diffusion term (tendency to randomly select some outcome) would eventually result in a decision being made, with probability 1, where more subtle conflicts tend to take more time to resolve.

This is why I prefer to think of those situations as "not finished computing" rather than "intrinsically unresolvable".

Do you maybe have a different notion of resolving a conflict, that makes unresolvedness a sustainable situation?

Comment author: Kaj_Sotala 31 December 2012 07:39:10AM *  1 point [-]

Don't you think that things being perfectly balanced in a way such that there is no resolution is sort of a measure zero set of outcomes?

I don't really have any good data on this: my preliminary notion that some such conflicts might be unresolvable is mostly just based on introspection, but we all know how reliable that is. And even if it was reliable, I'm still young and it could turn out that my conflicts will eventually be resolved as well. So if there are theoretical reasons to presume that there will eventually be a resolution, I will update in that direction.

That said, based on a brief skim of the page you linked, the drift-diffusion model seems to mostly just predict that a person will eventually take some action - I'm not sure whether it excludes the possibility of a person taking an action, but regardless remaining conflicted of whether it was the right one. This seems to often be the case with moral uncertainty.

For example, my personal conflict gets rather complicated, but basically it's over the fact that I work in the x-risk field, which part of my brain considers the Right Thing To Do due to all the usual reasons that you'd expect. But I also have strong negative utilitarian intuitions which "argue" that life going extinct would in the long run be the right thing as it would eliminate suffering. I don't assign a very high probability on humanity actually surviving the Singularity regardless of what we do, so I don't exactly feel that my work is actively unethical, but I do feel that it might be a waste of time and that my efforts might be better spent on something that actually did reduce suffering while life on Earth still existed. This conflict keeps eating into my motivation and making me accomplish less, and I don't see it getting resolved anytime soon. Even if I did switch to another line of work, I expect that I would just end up conflicted and guilty over not working on AI risk.

(I also have other personal conflicts, but that's the biggest one.)

Comment author: atucker 31 December 2012 06:20:59PM -1 points [-]

I think that if you tried to examine my different moral subsystems absent a scenario, they might be irreconcilable, but in practice they normally wind up deciding on something just because reality almost never throws me a moral problem that's a perfect toss-up. If I constrain my moral opinions to the domains of actions that I can take, I'll feel pulled in different directions, but there's almost always a winning faction eventually.

Powered by Reddit Powered by Reddit