LESSWRONG
LW

All of Benjamin's Comments + Replies

All AGI safety questions welcome (especially basic ones) [Sept 2022]
Benjamin40

Is there a way to measure agenticness? Or at least a relative measure. Like this: https://xkcd.com/2307/ but more objective.

Reply
3the gears to ascension
Not yet one that fits in the universe but we've finally at least got one that doesn't, which is a big improvement over "uh no idea lol": https://arxiv.org/abs/2208.08345
4plex
Not yet. There will soon be a $200k prize on Superlinear for people to try and define agency in a formal way, then write a program to detect it.
AGI Safety FAQ / all-dumb-questions-allowed thread
Benjamin90

It seems like instrumental convergence is restricted to agent AI's, is that true? 

Also what is going on with mesa-optimizers? Why is it expected that they will will be more likely to become agentic than the base optimizer when they are more resource constrained?

Reply
5plex
The more agentic a system is the more it is likely to adopt convergent instrumental goals, yes. Why agents are powerful explores why agentic mesa optimizers might arise accidentally during training. In particular, agents are an efficient way to solve many challenges, so the mesa optimizer being resource constrained would lean in the direction of more agency under some circumstances.