Below I summarize the thoughts of other people on what's the Theory of Impact for Activation Engineering. I mostly base it on the "discussion" parts of the papers and the answers under @Chris_Leong's post What's the theory of impact for activation vectors?
Alex Turner's posts on controlling a maze-solving policy network, and a paper on steering GPT-2-XL by adding an activation vector, introduced activation engineering, a set of "techniques which steer models by modifying their activations".
As a complement to prompt engineering and finetuning, activation engineering is a low-overhead way to steer models at runtime.
Over a year later and there's a Slack server to coordinate research projects and propose new ideas or open problems... (read 572 more words →)
This is another reply in this vein, I'm quite new to this so don't feel obliged to read through. I just told myself I will publish this.
I agree (90-99% agreement) with almost all of the points Eliezer made. And the rest is where I probably didn't understand enough or where there's no need for a comment, e.g.:
1. - 8. agree
9. Not sure if I understand it right - if the AGI has been successfully designed not to kill everyone then why need oversight? If it is capable to do so and the design fails then on the other hand what would our oversight do? I don't think this is like the nuclear... (read more)
is there any society where people are ostracised for not possessing difficult skills?
Depending on what you call "difficult". I think you will try to fit in if 80%+ of your peers know the skill, but OTOH if 80%+ people know have the skill then is it "difficult"?
In gen pop I feel this way about driving cars - some people hate it and have to deal with a lot of stress/anxiety to learn this skills and although they could live without it. But living without it means they can't do some of the things - they have less power.
I'd say then the power motivation (increase the space of your action) could be lumped... (read more)
I came here for this tip specifically to read Kaj's sequence as well! Used this plugin for now but I hope it becomes a feature.
Is the Slack still active? The link to join is not.