= 27d567bc2d48d9f40e9ccc20ea370b15)
orthonormal comments on Post Your Utility Function - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (273)
Given the length of the thread I branched from, it looks like you and P.J. Eby ended up talking past each other to some extent, and I think that you both failed to distinguish explicitly between the current map (which is what you calculate the territory to be) and a hypothetical future map.
P.J. Eby was (correctly) insisting that your utility function is only in contact with your current map, not the territory directly. You were (correctly) insisting that your utility function cares about (what it calculates to be) the future territory, and not just the future map.
Is that a fair statement of the key points?
Utility function is no more "in contact" with your current map than the actual truth of 2+2=4 is in contact with display of a calculator that displays the statement. Utility function may care about past territory (and even counterfactual territory) as well as future territory, with map being its part. Keeping a map in good health is instrumentally a very strong move: just by injecting an agent with your preferences somewhere in the territory you improve it immensely.
While there might exist some abstracted idealized dynamic that is a mathematical object independent of your map, any feasible heuristic for calculating your utility function (including, of course, any calculation you actually do) will depend on your map.
If Omega came through tomorrow and made all pigs conscious with human-like thoughts and emotions, my moral views on pig farming wouldn't be instantly changed; only when information about this development gets to me and my map gets altered will I start assigning a much higher disutility to factory farming of pigs.
Or, to put it another way, a decision algorithm refers directly to the possible worlds in the territory (and their probabilities, etc), but it evaluates these referents by looking at the corresponding objects in its current map. I think that, since we're talking about practical purposes, this is a relevant point.
Agree completely. Of the worlds where my future map looks to diverge from the territory, though, I'm generally more repulsed by the ones in which my map says it's fine where it's not than by the opposite.
This something of a nitpick, but this isn't strictly true. If others are trying to calculate your utility function (in order to help you), this will depend on their maps rather than yours (though probably including their map of your map). The difference becomes important if their maps are more accurate than yours in some respect (or if they can affect how accurate your map is).
For example, if you know that I value not being deceived (and not merely the subjective experience of not being deceived), and you care about my welfare, then I think that you should not deceive me, even if you know that I might perceive my welfare to be higher if you did.
Oh, good point. I should have restricted it to "any calculation you personally do", in which case I believe it holds.
At which point it becomes trivial: any calculation that is done on your map is done using your map, just Markovity of computation...
A related point is that you can create tools that make decisions themselves, in situations only of possibility of which you are aware.
Right. It's trivial, but relevant when discussing in what sense our decision algorithms refer to territory versus map.
I can't parse this. What do you mean?
If you install an alarm system that uses a video camera to recognize movement and calls the police if it's armed, you are delegating some of the map-making and decision-making to the alarm system. You are neither aware of the exact nature of possible intruders, nor making a decision regarding calling the police before any intrusion actually occurs. The system decides what to do by itself, according to the aspect of your values it implements. You map is not involved.
Yes, but your decision to install it (as well as your decision to arm it) comes from your map. You would not install it if you thought you had virtually no chance of being burglarized, or if you thought that it would have a false alarm every five minutes when the train went past.
We can make choices that cause other (human, mechanical, etc) agents to act in particular ways, as one of the manners in which we affect possible futures. But these sorts of choices are evaluated by us in the same way as others.
I fear we've resorted to arguing about the semantics of "map" versus "territory", as I don't see a scenario where we'd predict or decide differently from each other on account of this disagreement. As such, I'm willing to drop it for now unless you see such a scenario.
(My disagreement with Mr. Eby, on the other hand, appears to be more substantive.)
It appears to lead nowhere: your comments are clear, while his are all smoke and mirrors, in many many words.
And does this alarm system have "preferences" that are "about" reality? Or does it merely generate outputs in response to inputs, according to the "values it implements"?
My argument is simply that humans are no different than this hypothetical alarm system; the things we call preferences are no different than variables in the alarm system's controller - an implementation of values that are not our own.
If there are any "preferences about reality" in the system, they belong to the maker of the alarm system, as it is merely an implementation of the maker's values.
By analogy, if our preferences are the implementation of any values, they are the "values" of natural selection, not our own.
If now you say that natural selection doesn't have any preferences or values, then we are left with no preferences anywhere -- merely isomorphism between control systems and their environments. Saying this isomorphism is "about" something is saying that a mental entity (the "about" relationship) exists in the real world, i.e., supernaturalism.
In short, what I'm saying is that anybody who argues human preferences are "about" reality is anthropomorphizing the alarm system.
However, if you say that the alarm system does have preferences by some reductionistic definition of "preference", and you assert that human preference is exactly the same, then we are still left to determine the manner in which these preferences are "about" reality.
If nobody made the alarm system, but it just happened to be formed by a spontaneous jumbling of parts, can it still be said to have preferences? Are its "preferences" still "about" reality in that case?
Both. You are now trying to explain away the rainbow, by insisting that it consists of atoms, which can't in themselves possess the properties of a rainbow.