"Intelligence measures an agent's ability to achieve goals in a wide range of environments." (Shane Legg) [1] A little while ago I tried to equip Hutter's universal agent, AIXI, with a utility function, so instead of taking its clues about its goals from the environment, the agent is equipped with...
Wireheading has been debated on Less Wrong over and over and over again, and people's opinions seem to be grounded in strong intuitions. I could not find any consistent definition around, so I wonder how much of the debate is over the sound of falling trees. This article is an...
I'm Anja Heinisch, the new visiting fellow at SI. I've been researching replacing AIXI's reward system with a proper utility function. Here I will describe my AIXI+utility function model, address concerns about restricting the model to bounded or finite utility, and analyze some of the implications of modifiable utility functions,...
So I would only consider the formulation in terms of semimeasures to be satisfactory if the semimeasures are specific enough that the correct semimeasure plus the observation sequence is enough information to determine everything that's happening in the environment.
Can you make an example of a situation in which that would not be the case? I think the semimeasure AIXI and deterministic programs AIXI are pretty much equivalent, am I overlooking something here?
If we're going to allow infinite episodic utilities, we'll need some way of comparing how big different nonconvergent series are.
I think we need that even without infinite episodic utilities. I still think there might be possibilities involving surreal numbers, but I... (read more)
I think you are proposing to have some hypotheses privileged in the beginning of Solomonoff induction, but not too much because the uncertainty helps fight wireheading by means of providing knowledge about the existence of an idealized, "true" utility function and world model. I that a correct summary? (Just trying to test whether I understand what you mean.)
In particular they can make positive use of wire-heading to reprogram themselves even if the basic architecture M doesn't allow it
Can you explain this more?
"Intelligence measures an agent's ability to achieve goals in a wide range of environments." (Shane Legg) [1]
A little while ago I tried to equip Hutter's universal agent, AIXI, with a utility function, so instead of taking its clues about its goals from the environment, the agent is equipped with intrinsic preferences over possible future observations.
The universal AIXI agent is defined to receive reward from the environment through its perception channel. This idea originates from the field of reinforcement learning, where an algorithm is observed and then rewarded by a person if this person approves of the outputs. It is less appropriate as a model of AGI capable of autonomy, with no clear master watching over it... (read 1593 more words →)
They just do interpersonal comparisons; lots of their ideas generalize to intrapersonal comparisons though.
I recommend the book "Fair Division and Collective Welfare" by H. J. Moulin, it discusses some of these problems and several related others.
True. :)
I get that now, thanks.
you forgot to multiply by 2^-l(q)
I think then you would count that twice, wouldn't you? Because my original formula already contains the Solomonoff probability...
Let's stick with delusion boxes for now, because assuming that we can read off from the environment whether the agent has wireheaded breaks dualism. So even if we specify utility directly over environments, we still need to master the task of specifying which action/environment combinations contain delusion boxes to evaluate them correctly. It is still the same problem, just phrased differently.
I think there is something off with the formulas that use policies: If you already choose the policy
=y_{%3Ck}y_k)
then you cannot choose an y_k in the argmax.
Also for the Solomonoff prior you must sum over all programs
=x_{1:m_k}) .
Could you maybe expand on the proof of Lemma 1 a little bit? I am not sure I get what you mean yet.
Wireheading has been debated on Less Wrong over and over and over again, and people's opinions seem to be grounded in strong intuitions. I could not find any consistent definition around, so I wonder how much of the debate is over the sound of falling trees. This article is an attempt to get closer to a definition that captures people's intuitions and eliminates confusion.
Let's start with describing the typical exemplars of the category "Wireheading" that come to mind.
I'm Anja Heinisch, the new visiting fellow at SI. I've been researching replacing AIXI's reward system with a proper utility function. Here I will describe my AIXI+utility function model, address concerns about restricting the model to bounded or finite utility, and analyze some of the implications of modifiable utility functions, e.g. wireheading and dynamic consistency. Comments, questions and advice (especially about related research and material) will be highly appreciated.
Marcus Hutter's (2003) universal agent AIXI addresses the problem of rational action in a (partially) unknown computable universe, given infinite computing power and a halting oracle. The agent interacts with its environment in discrete time cycles, producing an action-perception sequence with actions (agent outputs)
and... (read 1723 more words →)
Super hard to say without further specification of the approximation method used for the physical implementation.