All of Verden's Comments + Replies

It's funny that this post has probably made me feel more doomy about AI risk than any other LW post published this year. Perhaps for no particularly good reason. There's just something really disturbing to me about seeing a vivid case where folks like Jacob, Eli and Samotsvety, apparently along with many others, predict a tiny chance that a certain thing in AI progress will happen (by a certain time), and then it just... happens.

I'm not totally sure what you're referring to, but if you're talking about Paul's guess of "~15% on singularity by 2030 and ~40% on singularity by 2040", then I want to point out that looking at these two questions, his prediction seems in line with the Metaculus community prediction

2Jotto999
1. I disagree with the community on that.  Knocking out silver turing, Montezuma (in the way described), 90% equivalent on Winogrande, and 75th percentile on maths SAT will either take longer to be actually demonstrated in a unified ML system, OR it will happen way sooner than 39 months before "an AI which can perform any task humans can perform in 2021, as well or superior to the best humans in their domain.", which is incredibly broad.  If the questions mean what they are written to mean, as I read them, it's a hell of a lot more than 39 months (median community estimate). 2. The thing I said is about some important scenarios described by people giving significant probability to a hostile hard takeoff scenario.  I included the comment here in this subthread because I don't think it contributed much to the discussion.
5paulfchristiano
I don't think it will ever seem plausible for an accident to turn everyone's atoms into ML hardware though, because we will probably remain closer to an equilibrium with no free energy for powerful AI to harvest.
Answer by Verden240

Scott Aaronson recently wrote something relevant to these issues:

Max Ra: What would change your mind to explore research on the AI alignment problem? For a week? A month? A semester?

Scott: The central thing would be finding an actual potentially-answerable technical question around AI alignment, even just a small one, that piqued my interest and that I felt like I had an unusual angle on. In general, I have an absolutely terrible track record at working on topics because I abstractly feel like I “should” work on them. My entire scientific career has basica

... (read more)
8Lucius Bushnaq
Extremely valuable I'd guess, but the whole problem is that alignment is still preparadigmatic. We don't actually know yet what the well-defined nerd snipe questions we should be asking are. I think that preparadigmatic research and paradigmatic research are two different skill sets, and most Highly Impressive People in mainstream STEM are masters at the later, not the former. I do think we're more paradigmatic than we were a year ago, and that we might transition fully some time soon. I've got a list of concrete experiments on modularity in ML systems I'd like run for example, and I think any ML savvy person could probably do those, no skill at thinking about fuzzy far mode things required. So I'm not sure a sequence like this could be written today, but maybe in six months?
1[comment deleted]

Can someone explain to me why we don't see people with differing complex views on something placing bets in a similar fashion more often? 

It was quite hard to get to this forecast, and in the end I don't think it will be that useful. I think it's just generally really hard. I don't have a clear sense for why Eliezer and I weren't able to get to more bets, but I'm not that surprised.

I do think that this kind of betting has a lot of hazards and there's a good chance that we are both going t come out behind in social EV. For example: (i) if you try to do it reasonably quickly then you basically just know there are reasons that your position is dumb that you just haven't noticed, (ii) it's easier to throw something in someone's face as a mistake than to gloat about it as a win, (iii) there is all kinds of adverse selection...

3Jotto999
Several different tough hurdles have to be passed, and usually aren't.  For one, they would have to both even agree on criteria that they both think are relevant enough, and that they can define well-enough for it to be resolvable. They also have to agree to an offer with whatever odds, and the amount of money. They then also have to be risk-tolerant enough to go through knowing they may lose money, or may be humiliated somewhat (though with really good betting etiquette, IMO it need not humiliating if they're good sports about it).  And also the obvious counterparty risk, as people may simply not pay up.

Would it be helpful to think about something like "what Brier score will a person in the reference class of "people-similar-to-Eliezer_2022-in-all-relevant-ways" have after making a bunch of predictions on Metaculus?" Perhaps we should set up this sort of question on Metaculus or Manifold? Though I would probably refrain from explicitly mentioning Eliezer in it.

2Jotto999
That might be possible, but it would take a lot of effort to make resolvable.  Who is "similar to Eliezer", and how do we define that in advance? Which forecasting questions are we going to check their Brier score on? Which not-like-Eliezer forecasters are we going to compare them to (since scores are much more informative for ranking forecasters, than in isolation)? Etc.  I'd rather people like Eliezer just placed their forecasts properly!