Scott Aaronson recently wrote something relevant to these issues:
...Max Ra: What would change your mind to explore research on the AI alignment problem? For a week? A month? A semester?
Scott: The central thing would be finding an actual potentially-answerable technical question around AI alignment, even just a small one, that piqued my interest and that I felt like I had an unusual angle on. In general, I have an absolutely terrible track record at working on topics because I abstractly feel like I “should” work on them. My entire scientific career has basica
It was quite hard to get to this forecast, and in the end I don't think it will be that useful. I think it's just generally really hard. I don't have a clear sense for why Eliezer and I weren't able to get to more bets, but I'm not that surprised.
I do think that this kind of betting has a lot of hazards and there's a good chance that we are both going t come out behind in social EV. For example: (i) if you try to do it reasonably quickly then you basically just know there are reasons that your position is dumb that you just haven't noticed, (ii) it's easier to throw something in someone's face as a mistake than to gloat about it as a win, (iii) there is all kinds of adverse selection...
Would it be helpful to think about something like "what Brier score will a person in the reference class of "people-similar-to-Eliezer_2022-in-all-relevant-ways" have after making a bunch of predictions on Metaculus?" Perhaps we should set up this sort of question on Metaculus or Manifold? Though I would probably refrain from explicitly mentioning Eliezer in it.
It's funny that this post has probably made me feel more doomy about AI risk than any other LW post published this year. Perhaps for no particularly good reason. There's just something really disturbing to me about seeing a vivid case where folks like Jacob, Eli and Samotsvety, apparently along with many others, predict a tiny chance that a certain thing in AI progress will happen (by a certain time), and then it just... happens.