If we try to translate sentences involving "should" into descriptive sentences about the world, they will probably sound like "action A increases the value of utility function U". If I was a consistent utility maximizer, and U was my utility function, then believing such a statement would make me take action A. No further verbal convincing would be necessary.

Since we are not consistent utility maximizers, we run an approximate implementation of that mechanism which is vulnerable to verbal manipulation, often by sentences involving "should". So the murkiness in the meaning of "should" is proportional to the difference between us and utility maximizers. Does that make sense?

(It may or may not be productive to describe a person as a utility maximizer plus error. But I'm going with that because we have no better theory yet.)

I'm not convinced there are things that are objectively valuable. I'm of the belief that if there are no agents to value something, then that something has effectively no value.

The second sentence doesn't follow from the first. If rational agents converge on their values, that is objective enough. Analogy: one can accept that mathematical truth is objective (mathematicians will converge) without being a Platonists (mathematical truths have an existence separate from humans)

Without objective values, it might just be a matter of testing different sets of terminal subjective values, until we find the optimum (an hopefully don't get trapped in a local maximum).

I fin d that hard to follow. If the test i rationally justifiable, and leads to the uniform results, how is that not objective? You seem to be using "objective" (having a truth value independent of individual humans) to mean what I would mean by "real" (having existence independent of humans).

Hi everyone,

I'm a PhD student in artificial intelligence/robotics, though my work is related to computational neuroscience, and I have strong interests in philosophy of mind, meta-ethics and the "meaning of life". Though I feel that I should treat finishing my PhD as a personal priority, I like to think about these things. As such, I've been working on an explanation for consciousness and a blueprint for artificial general intelligence, and trying to conceive of a set of weighted values that can be applied to scientifically observable/measurable/calculable quantities, both of which have some implications for an explanation of the "meaning" of life.

At the center of the value system I'm working on is a broad notion of "information". Though still at preliminary stages, I'm considering a hierarchy of weights for the value of different types of information, and trying to determine how bad this is as a utility function. At the moment, I consider the preservation and creation of all information valuable; at an everyday level I try to translate this into learning and creating new knowledge and searching for unique, meaningful experiences.

I've been aware of Less Wrong for years, though haven't quite mustered the motivation to read all of the sequences. Nevertheless, I've lurked here on and off over that time, and read lots of interesting discussions. I consider the ability to make rational decisions, and not be fooled by illogical arguments, important. Though without a definite set of values and goals, any action is simply shooting in the dark.

I think this community vastly over-estimates its grip on meta-ethical concepts like moral realism or moral anti-realism. (E.g. the hopelessly confused discussion in this thread). I don't think the meta-ethics sequence resolves these sorts of basic issues.

I'm assuming a lot of background in this post that you don't seem to have. Have you read the sequences, specifically the metaethics stuff?

Moral philosophy on LW is decades (at the usual philosophical pace) ahead of what you would learn elsewhere and a lot of the stuff you mentioned is considered solved or obsolete.

I hadn't come across the von Neumann-Morgenstern utility theorem before reading this post, thanks for drawing it to my attention.

Looking at Moral Philosophy through the lens of agents working with utility/value functions is an interesting exercise; it's something I'm still working on. In the long run, I think some deep thinking needs to be done about what we end up selecting as terminal values, and how we incorporate them into a utility function. (I hope that isn't stating something that is blindingly obvious.)

I guess where you might be headed is into Meta-Ethics. As I understand it, meta-ethics includes debates on moral relativism that are closely related to the existence terminal/intrinsic values. Moral relativism asserts that all values are subjective (i.e., only the beliefs of individuals), rather than objective (i.e., universally true). So no practice or activity is inherently right or wrong, it is just the perception of people that makes it so. As you might imagine, this can be used as a defense of violent cultural practices (it could even be used in defense of baby-eating).

I tend to agree with the position of moral relativism; unfortunate though it may be, I'm not convinced there are things that are objectively valuable. I'm of the belief that if there are no agents to value something, then that something has effectively no value. That holds for people and their values too. That said, we do exist, and I think subjective values count for something.

Humanity has come to some degree of consensus over what should be valued. Probably largely as a result of evolution and social conditioning. So from here, I think it mightn't be wasted effort to explore the selection of different intrinsic values.

Luke Muehlhauser has called morality an engineering problem. While Sam Harris has described morality as a landscape, i.e., the surface is our terminal value we are trying to maximize (Harris picked the well-being of conscious creatures) and the societal practices as the variables. Though I don't know that well-being is the best terminal value, I like the idea of treating morality as an optimization problem. I think this is a reasonable way to view ethics. Without objective values, it might just be a matter of testing different sets of terminal subjective values, until we find the optimum (an hopefully don't get trapped in a local maximum).

Nevertheless, I think it's interesting to suppose that something is objectively valuable. It doesn't seem like a stretch to me to say that the knowledge of what is objectively valuable, would, itself, be objectively valuable. And that the search for that knowledge would probably also be objectively valuable. After all that, it would be somewhat ironic if it turned out the universal objective values don't include the survival of life on Earth.