I agree with the first point, and I don't have solid solutions to this. There's also the fact that some games are easier to optimize than others (name a number game I described at the end vs. chess), and this complexity is impossible to capture while staying computation-agnostic. Maybe one can use the length of the shortest proof that taking action a leads to utility u(a) to account for these issues..
The second point is more controversial, my intuition is that first agent is an equally good optimizer, even if it did better in terms of payoffs. Also, at least in the setting of deterministic games, utility functions are arbitrary up to encoding the same preference orderings (once randomness is introduced this stops being true)
[Epistemic status: the idea is simple and intuitive, so I wouldn't be surprised if it has appeared before. If so, I would appreciate pointers.]
Eliezer defines optimization power as the ability to produce solutions higher on the preference ordering. This feels right, but it would be nice to convert this intuition into a score that can be measured, a universal metric that allows one to meaningfully state that AlphaZero is better at Go than Deep Blue was at chess. In this post I try accomplishing just that. I start by giving a candidate metric and some of its properties. I then discuss how it can be approximately computed in complex settings, and... (read 741 more words →)
I agree with the point that we should be investing more into research on direct reduction of suffering(as a phenomenon that happens in brains), rather than reducing the proxies for it.
This is true for humans as well as for animals: e.g. investing into discovering direct stimulation/surgery approaches to reducing or even turning off pain (or just the painfullness of pain, see pain asymbolia) might have greater impact on life satisfaction than it's opportunity cost for say cancer research.
I am not at all knowledgeble on the subject (and would love to be corrected), but I suspect that ever since lobotomy was declared unethical no interventions for pain other than chemical ones have been seriously investigated.
Thanks for the pointers, I should have done more research before writing this up. After a quick glance my initial impression is that there still isn't a good solution (even an uncomputable one) to the problems such as the one I describe at the end, or the one AlexMennen mentions.