Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

SilasBarta comments on Decision theory: Why we need to reduce “could”, “would”, “should” - Less Wrong

19 Post author: AnnaSalamon 02 September 2009 09:23AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (46)

You are viewing a single comment's thread.

Comment author: SilasBarta 02 September 2009 03:40:15PM 4 points [-]

Warning: grouchiness follows.

A draft-reader suggested to me that this question is poorly motivated: what other kinds of agents could there be, besides “could”/“would”/“should” agents?

Actually, I made the same criticism of that category, except in more detail. Was that acausal, or am I just more worthy of reviewing your drafts?

And your response in the footnote looks like little more than, "don't worry, you'll get it some day, like schoolkids and fractions". Not helpful.

Humans ... have CSA-like structure. That is, we consider “alternatives” and act out the alternative from which we “expect” the highest payoff

Excuse me, isn't this just the classical "rational agent" model that research has long since refuted? For one thing, many actions people perform are trivially impossible to interpret this way (in the sense of your diagram), given reaction times and known computational properties of the brain. That is, the brain doesn't have enough time to form enough distinct substates isomorphic to several human-like responses, then evaluate them, then compare the evaluations.

For another, the whole heuristics and biases literature repeated ad infinitum on OB/LW.

Finally, even when humans do believe they're evaluating several choices looking for the best payoff (per some multivariable utility function), what really happens is that they pick one quickly based on "gut instinct" -- meaning some heuristic, good or bad -- and then bend all conscious evaluation to favor it. In at least some laboratory settings, this is shown explicitly: the researchers can predict what the subject will do, and then the subject gives some plausible-sounding rationalization for why they did it.

(And if you say, "using heuristics is a kind of evaluation of alternatives", then you're again stretching the boundaries of the concept of a CSA wide enough to be unhelpful.)

There are indeed cases where people do truly consider the alternatives and make sure they are computing the actual consequences and the actual congruence with their actual values, but this is an art people have to genuinely work towards; it is not characteristic of general human action.


In any case, all of the above assumes a distinction I'm not convinced you've made. To count as a CSA, is it necessary that you be physically able to extract the alternatives under consideration ("Silas considered making his post polite, but assigned it low utility")? Because the technology certainly doesn't exist to do that on humans. Or is it only necessary that it be possible in principle? If the latter, you run back into the problem of the laws of physics being embedded in all parts of the universe:

I observe a pebble. Therefore, I know the laws of the universe. Therefore, I can compute arbitrary counterfactuals. Therefore, I compute a zero pebble-utility for everything the pebble "pebble-could" do, except follow the laws of physics.

Therefore, there is no "not-CSA" option.

Comment author: conchis 03 September 2009 06:25:19PM *  0 points [-]

all of the above assumes a distinction I'm not convinced you've made

If it is possible in principle, to physically extract the alternatives/utility assignments etc., wouldn't that be sufficient to ground the CSA--non-CSA distinction, without running afoul of either current technological limitations, or the pebble-as-CSA problem? (Granted, we might not always know whether a given agent is really a CSA or not, but that doesn't seem to obviate the distinction itself.)

Comment author: SilasBarta 03 September 2009 10:32:29PM *  2 points [-]

Thanks for your reply.

For the purposes of the argument I was making, "possible in principle to physically extract" is the same as "possible in principle to extract". For once you know the laws of physics, which supposedly you can learn from a pebble, you can physically extract data that is functionally equivalent to alternatives/utility assignments.

For example, our knowledge of thermodynamics and chemistry tells us that a chemical would go to a lower energy state (and perhaps release heat) if it could observe certain other chemicals (which we call "catalysts"). It is our knowledge of science that justifies saying that there is this lower energy state that it "has a tendency" to want to go to, which is an "alternative" lacking "couldness" in the same sense of the proposed CSAs.

Laying down rules for what counts as evidence that a body is considering alternatives, is messier than AnnaSalamon thinks.

Comment author: conchis 03 September 2009 10:50:12PM *  1 point [-]

Laying down rules for what counts as evidence that a body is considering alternatives, is mess[y]

Agreed. But I don't think that means that it's not possible to do so, or that there aren't clear cases on either side of the line. My previous formulation probably wasn't as clear as it should have been, but would the distinction seem more tenable to you if I said "possible in principle to observe physical representations of" instead of "possible in principle to physically extract"? I think the former better captures my intended meaning.

If there were a (potentially) observable physical process going on inside the pebble that contained representations of alternative paths available to it, and the utility assigned to them, then I think you could argue that the pebble is a CSA. But we have no evidence of that whatsoever. Those representations might exist in our minds once we decide to model the pebble in that way, but that isn't the same thing at all.

On the other hand, we do seem to have such evidence for e.g. chess-playing computers, and (while claims about what neuroimaging studies have identified are frequently overstated) we also seem to be gathering it for the human brain.

Comment author: SilasBarta 03 September 2009 11:37:30PM *  1 point [-]

but would the distinction ... seem more tenable to you if I said "possible in principle to observe physical representations of" instead of "possible in principle to physically extract"?

Heh, I actually had a response half-written up to this position, until I decided that something like the comment I did make would be more relevant. So, let's port that over...

The answer to your question is: yes, as long as you can specify what observations of the system (and you may of course include any physically-possible mode of entanglement) count as evidence for it having considered multiple alternatives.

This criterion, I think, is what AnnaSalamon should be focusing on: what does it mean for "alternative-consideration" to be embedded in a physical system? In such a limited world as chess, it's easy to see the embedding. [Now begins what I hadn't written before.] I think that's a great example of what I'm wondering about: what is this possible class of intelligent algorithms that stands in contrast to CSAs? If there were a good chess computer that was not a CSA, what would it be doing instead?

You could imagine one, perhaps, that computes moves purely as a function of the current board configuration. If bishop here, more than three pawns between here, move knight there, etc.

The first thing to notice is that for the program to actually be good, it would require that some other process was able to find a lot of regularity to the search space, and compactly express it. And to find that regularity, it would have to interact with it. So, such a good "insta-evaluator" implicitly contains the result of previous simulations.

Arguably, this, rather than(?) a CSA is what humans (mostly) are. Throughout our evolutionary history, a self-replicating process iterated through a lot of experiences that told it what "does work" and "doesn't work". The way we exist today, just the same as in the case of chess above, implicitly contains a compression of previous evaluations of "does work" and "doesn't work", known as heuristics, which together guide our behavior.

Is a machine that acts purely this way, and without humans' ability to consciously consider alternatives, what AnnaSalamon means by a non-CSA algorithm? Or would it include that too?