FWIW - here (finally) is the related post I mentioned, which motivated this observation: Natural Abstraction: Convergent Preferences Over Information Structures The context is a power-seeking-style analysis of the naturality of abstractions, where I was determined to have transitive preferences.
It had quite a bit of scope creep already, so I ended up not including a general treatment of the (transitive) 'sum over orbits' version of retargetability (and some parts I considered only optimality - sorry! still think it makes sense to start there first and then generalize in this case). The full translation also isn't necessarily as easy as I thought - it turns out that is transitive specifically for binary functions, so... (read more)
Not sure this is exactly what you meant by the full preference ordering, but might be of interest: I give the preorder of universally-shared-preferences between "models" here (in section 4).
Basically, it is the Blackwell order, if you extend the Blackwell setting to include a system.
The natural abstraction hypothesis claims (in part) that a wide variety of agents will learn to use the same abstractions and concepts to reason about the world.
What's the simplest possible setting where we can state something like this formally? Under what conditions is it true?
One necessary condition to say that agents use 'the same abstractions' is that their decisions depend on the same coarse-grained information about the environment. Motivated by this, we consider a setting where an agent needs to decide which information to pass though an information bottleneck, and ask: when do different utility functions prefer to preserve the same information?
As we develop this setting, we:
Thanks for the reply. I'll clean this up into a standalone post and/or cover this in a related larger post I'm working on, depending on how some details turn out.
What are here?
Variables I forgot to rename, when I changed how I was labelling the arguments of in my example. This should be , , retargetable (as arguments to ).
I appreciate this generalization of the results - I think it's a good step towards showing the underlying structure involved here.
One point I want to comment on is transitivity of , as a relation on induced functions . Namely, it isn't, and can even contain cycles of non-equivalent elements. (This came up when I was trying to apply a version of these results, and hoping that would be the preference relation I was looking for out of the box.) Quite possibly you noticed this since you give 'limited transitivity' in Lemma B.1 rather than full transitivity, but to give a concrete example:
Let and . The permutations are with the usual... (read 415 more words →)
I think this line of research is interesting. I really like the core concept of abstraction as summarizing the information that's relevant 'far away'.
A few thoughts:
- For a common human abstraction to be mostly recoverable as a 'natural' abstraction, it must depend mostly on the thing it is trying to abstract, and not e.g. evolutionary or cultural history, or biological implementation. This seems more plausible for 'trees' than it does for 'justice'. There may be natural game-theoretic abstractions related to justice, but I'd expect human concepts and behaviors around justice to depend also in important ways on e.g. our innate social instincts. Innate instincts and drives seem likely to a) be complex... (read more)
A while back I was looking for toy examples of environments with different amounts of 'naturalness' to their abstractions, and along the way noticed a connection between this version of Gooder Regulator and the Blackwell order.
Inspired by this, I expanded on this perspective of preferences-over-models / abstraction a fair bit here.
It includes among other things:
Personally I think these results are pretty neat; I hope they might be of interest.
I also make one... (read more)