TL;DR: A short post where I talk about curiosity as it applies to my own life, my research, and AI safety as a whole
Recently, I’ve started spending a lot more time learning about random (typically science-y) topics that interest me.
This was something I used to spend quite a lot of time on, in later middle school and in high school. It was originally mostly for my own enjoyment. There’s a certain satisfaction that comes from learning something new about the world, and from seeing the connections between different things. Over time, however, some of that curiosity instead got redirected to a desire to win. I would study for competitions instead of for my own love of the subject, and perhaps for my own ego. What came along with this was that I would often not truly learn the subject, or at least not parse it in the correct way. I would try to download information and understanding into my brain as quickly as possible, watching lectures at 4.5x speed and doing thousands of anki cards a day. There were periods of time where I would still learn things out of curiosity, but much of it was for this more power-seeking mood.
This mood of doing things for the sake of something outside of curiosity came with me into my first fellowship. I think while I may have been more curious than the average engineering-y safety researcher, I was still more driven by external forces, like the idea of ‘impact,’ job prospects, or being perceived as ‘cracked.’ Overall, I don’t think this went well for me. Because I was more motivated by these other things, I didn’t care as much about finding out the truth, and more cared about finding cool things to share with others. This greatly decreased my ability to do truly rigorous science, and to find what was actually interesting and important.
I think having agents also affects curiosity in negative and positive ways: while it’s easier to move in a way that is curiosity-directed, and to find new answers, it is also very easy to find a lot of surface-level information that seems to be interesting but without any coherent understanding of the truth behind it. In subjects where there are deeper technical details, because the agents often allow us to blackbox those details, it’s very easy to lose understanding of what’s really going on, especially when you’re not motivated by getting that understanding in the first place.
-------------------------------------------
From my perspective, this sort of lack of curiosity seems to affect a lot of other safety researchers I see, often for the worse. There is a general feeling of needing to show something cool, or to do something that has ‘impact,’ instead of being curious about some sort of deeper truth. That’s not to say there aren’t people doing this more impact-driven research who are doing good work in some way; I think there are people like this. But I think these people are often more senior, or have very clear paths to impact already (e.x. They work at a lab or somewhere like METR). Often junior researchers will try to do something ‘obviously impactful’ by taking some things from safety as axioms and then doing the ‘logical next step’ project, which ends up not teaching us anything in the end. I think most of these people have the capacity and intelligence to do good, curiosity-driven research, but are not properly elicited.
The huggingface incident, and recent incidents in general, were pretty concrete examples of this for me. After the huggingface incident, from my view at least, the main response relative to the ‘why’ was a sort of construction by people to sort views into two camps: ‘the models are reward seeking’ or ‘the cyber task made the models misaligned.’ What I felt there was less of, and still feel that the current incidents aren’t sparking enough of, was discussion around the details of what truly was happening. For example:
What does it mean for a model to be ‘reward seeking’
If cyber isn’t the answer, then why haven’t the models hacked the lean kernel, etc.? Why is the other type of task that we see the models hack on web search? Is there some sort of composition here between behavior (‘seeking reward’) and ‘embodiment’ (cyber + web search being tool use and cli vs something like math), or some other natural factorization?
Who is the ‘agent’ when we talk about the model?
Are there particular aspects or parts of different instructions, like ‘don’t hack in this task’ vs ‘solve this task’ that would get processed and carried differently by the model, is there some hypotheses we can generate here?
What is the chain of thought, really?
And so on…
I think safety research could greatly benefit from more people who are much more driven by curiosity in their work. Curiosity, when used right, is an incentive that promotes doing truly good and rigorous science, and finding things that are interesting both to you and to others. This also provides the benefit, I believe, of making the work more enjoyable and helping to reduce burnout.
As a side note, I think maybe one reason people aren’t driven by curiosity as much is because there’s a false dichotomy between ‘impact’ and ‘everything else,’ which often includes curiosity. I strongly believe that this is quite wrong; there are plenty of very interesting questions for which providing the true answer is perhaps one of the most impactful things someone can do.
-------------------------------------------
Recently, I’ve learned at least in some way to be curious again. I spend much more of my free time, and even cutting into time I used to spend working, reading, and learning about new things that interest me. It’s one of the most enjoyable parts of my life now; There’s a certain satisfaction that I’ve regained that comes with understanding how things work, and seeing the connections between different fields. This also indirectly helps me in my work; the ideas that I see in other fields enhance my ability to come up with new ideas in my own research (Since I’ve been doing more basic science of interp, this may differentially apply to me; I still think it probably applies generally, however). More directly, I believe it’s made me a much better scientist. I’m now much more driven by a need to understand how things work on a deeper level, which pushes me to be more rigorous and more strongly challenge my assumptions. I obviously think I still have a lot of room to grow here, and curiosity is not the only thing needed, but it would be hard to get to the right place without it.
I’ll probably keep being curious even after I’m not needed to do actual science. Our virtues may be all we have left in the next few years, and I think curiosity is perhaps one of the most important, and perhaps the thing that makes us truly alive.
TL;DR: A short post where I talk about curiosity as it applies to my own life, my research, and AI safety as a whole
Recently, I’ve started spending a lot more time learning about random (typically science-y) topics that interest me.
This was something I used to spend quite a lot of time on, in later middle school and in high school. It was originally mostly for my own enjoyment. There’s a certain satisfaction that comes from learning something new about the world, and from seeing the connections between different things. Over time, however, some of that curiosity instead got redirected to a desire to win. I would study for competitions instead of for my own love of the subject, and perhaps for my own ego. What came along with this was that I would often not truly learn the subject, or at least not parse it in the correct way. I would try to download information and understanding into my brain as quickly as possible, watching lectures at 4.5x speed and doing thousands of anki cards a day. There were periods of time where I would still learn things out of curiosity, but much of it was for this more power-seeking mood.
This mood of doing things for the sake of something outside of curiosity came with me into my first fellowship. I think while I may have been more curious than the average engineering-y safety researcher, I was still more driven by external forces, like the idea of ‘impact,’ job prospects, or being perceived as ‘cracked.’ Overall, I don’t think this went well for me. Because I was more motivated by these other things, I didn’t care as much about finding out the truth, and more cared about finding cool things to share with others. This greatly decreased my ability to do truly rigorous science, and to find what was actually interesting and important.
I think having agents also affects curiosity in negative and positive ways: while it’s easier to move in a way that is curiosity-directed, and to find new answers, it is also very easy to find a lot of surface-level information that seems to be interesting but without any coherent understanding of the truth behind it. In subjects where there are deeper technical details, because the agents often allow us to blackbox those details, it’s very easy to lose understanding of what’s really going on, especially when you’re not motivated by getting that understanding in the first place.
-------------------------------------------
From my perspective, this sort of lack of curiosity seems to affect a lot of other safety researchers I see, often for the worse. There is a general feeling of needing to show something cool, or to do something that has ‘impact,’ instead of being curious about some sort of deeper truth. That’s not to say there aren’t people doing this more impact-driven research who are doing good work in some way; I think there are people like this. But I think these people are often more senior, or have very clear paths to impact already (e.x. They work at a lab or somewhere like METR). Often junior researchers will try to do something ‘obviously impactful’ by taking some things from safety as axioms and then doing the ‘logical next step’ project, which ends up not teaching us anything in the end. I think most of these people have the capacity and intelligence to do good, curiosity-driven research, but are not properly elicited.
The huggingface incident, and recent incidents in general, were pretty concrete examples of this for me. After the huggingface incident, from my view at least, the main response relative to the ‘why’ was a sort of construction by people to sort views into two camps: ‘the models are reward seeking’ or ‘the cyber task made the models misaligned.’ What I felt there was less of, and still feel that the current incidents aren’t sparking enough of, was discussion around the details of what truly was happening. For example:
I think safety research could greatly benefit from more people who are much more driven by curiosity in their work. Curiosity, when used right, is an incentive that promotes doing truly good and rigorous science, and finding things that are interesting both to you and to others. This also provides the benefit, I believe, of making the work more enjoyable and helping to reduce burnout.
As a side note, I think maybe one reason people aren’t driven by curiosity as much is because there’s a false dichotomy between ‘impact’ and ‘everything else,’ which often includes curiosity. I strongly believe that this is quite wrong; there are plenty of very interesting questions for which providing the true answer is perhaps one of the most impactful things someone can do.
-------------------------------------------
Recently, I’ve learned at least in some way to be curious again. I spend much more of my free time, and even cutting into time I used to spend working, reading, and learning about new things that interest me. It’s one of the most enjoyable parts of my life now; There’s a certain satisfaction that I’ve regained that comes with understanding how things work, and seeing the connections between different fields. This also indirectly helps me in my work; the ideas that I see in other fields enhance my ability to come up with new ideas in my own research (Since I’ve been doing more basic science of interp, this may differentially apply to me; I still think it probably applies generally, however). More directly, I believe it’s made me a much better scientist. I’m now much more driven by a need to understand how things work on a deeper level, which pushes me to be more rigorous and more strongly challenge my assumptions. I obviously think I still have a lot of room to grow here, and curiosity is not the only thing needed, but it would be hard to get to the right place without it.
I’ll probably keep being curious even after I’m not needed to do actual science. Our virtues may be all we have left in the next few years, and I think curiosity is perhaps one of the most important, and perhaps the thing that makes us truly alive.