Vladimir_Nesov comments on Open Thread: April 2010 - Less Wrong

4 Post author: Unnamed 01 April 2010 03:21PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (524)

You are viewing a single comment's thread. Show more comments above.

Comment author: Vladimir_Nesov 07 April 2010 08:13:37AM *  3 points [-]

I now understand you as talking only about what kind of object preference is (an I/O map) and about how this kind of object can contain certain preferences that we worry might be lost (like considerations of faulty hardware).

Correct. Note that "strategy" is a pretty standard term, while "I/O map" sounds ambiguous, though it emphasizes that everything except the behavior at I/O is disregarded.

You have not said anything about what kind of static analysis would take you from an agent's strategy to an agent's preference.

An agent is more than its strategy: strategy is only external behavior, normal form of the algorithm implemented in the agent. The same strategy can be implemented by many different programs. I strongly suspect that it takes more than a strategy to define preference, that introspective properties are important (how the behavior is computed, as opposed to just what the resulting behavior is). It is sufficient for preference, when it is defined, to talk about strategies, and disregard how they could be computed; but to define (extract) a preference, a single strategy may be insufficient, it may be necessary to look at how the reference agent (e.g. a human) works on the inside. Besides, the agent is never given as its strategy, it is given as its source code that normalizes to that strategy, and computing the strategy may be tough (and pointless).