Firstly, it would seem to me to be much more difficult to FOOM with an LLM, it would seem much more difficult to create a superintelligence in the first place, and it seems like getting them to act creatively and be reliable are going to be much harder problems than making sure they aren't too creative.
Au contraire, for me at least. I am no expert on AI, but prior to the LLM blowup and seeing AutoGPT emerge almost immediately, I thought that endowing AI with the agency would take an elaborate engineering effort that went somehow beyond imitation of human outputs, such as language or imagery. I was somewhat skeptical of the orthogonality thesis. I also thought that it would take massive centralized computing resources not only to train but also to operate trained models (as I said, no expert). Obviously that is not true, and in a utopian outcome, access to LLMs will probably be a commodity good, with lots of roughly comparable models from many vendors to choose from and widely available open-source or hacked models as well.
Now, I see the creation of increasingly capable autonomous agents as just a matter of time, and ChaosGPT is overwhelming empirical evidence of orthogonality as far as I'm concerned. Clearly morality has to be enforced on the fundamentally amoral intelligence that is the LLM.
For me, my p(doom) increased due to the orthogonality thesis being conclusively proved correct and realizing just how cheap and widely available advanced AI models would be to the general public.
Edit: One other factor I forgot to mention is how instantaneously we'd shift from "AI doom is sci-fi, don't worry about it" to "AI doom is unrealistic because it just won't happen, don't worry about it" as LLMs became an instant sensation. I have been deeply disappointed on this issue by Tyler Cowen, who I really did not expect to shift from his usual thoughtful, balanced engagement with advanced ideas to just utter punditry on the issue. I think I understand where he's coming from - the huge importance of growth, the desire not to see AI killed by overregulation in the manner of nuclear power, etc - but still.
It has reinforced my belief that a fair fraction of the wealthy segment of the boomer generation will see AI as a way to cheat death (a goal I'm a big fan of), and will rush full-steam ahead to extract longevity tech out of it because they personally do not have time to wait to align AI, and they're dead either way. I expect approximately zero of them to admit this is a motivation, and only a few more to be crisply conscious of it.
Your link went to a post which we had previously argued about... Your wonderful post goes into all sorts of details about the efficiency of the brain, most centrally energy efficiency, but doesn't talk about the kinds of efficiency that matter most. The kind of efficiency that matters most is something like "Performance on various world-takeover and R&D tasks, as a function of total $, compute, etc. initially controlled." Here are the kinds of efficiency you talk about in that post (direct quote):
Yeah, those are all interesting and worth thinking about but not what matters at the end of the day. (To be clear, I am not yet convinced by your arguments in the post, but that's a separate discussion) Consider the birds vs. planes analogy. My guess is that planes still aren't as efficient as birds in a bunch of metrics (energy expended per kg per mile travelled, dollar cost to manufacture per kg, energy cost to manufacture per kg...) but that hasn't stopped planes from being enormously useful militarily and economically, much more so than birds. (We used to use birds to carry messages; occasionally people have experimented with them for military purposes also e.g. anti-drone warfare).
Funnily enough, I think these assumptions are approximately correct* & yet I think once we get human-level AGI, we'll be weeks rather than years from superintelligence. If you agree with me on this, then it seems a bit unfair to dunk on EY so much, even if he was wrong about various kinds of brain efficiency. Basically, if he's wrong about these kinds of brain efficiency, then the maximum limits of intelligence reachable by FOOM are lower than Yud thought, and also the slope of the intelligence explosion will probably be a bit less steep. And I'm grateful that your post exists carefully working through the issues there. But quantitatively if it still takes only a few weeks to reach superintelligence -- by which I mean AGI which is significantly more competent than the best-ever humans at task X, for all relevant intellectual tasks -- then the bottom line conclusions Yudkowsky drew appear to be correct, no?
(I'd like to give more concrete details about what I expect singularity to look like, but I'm hesitant because I'm a bit constrained in what I can and should say. I'd be curious though to hear your thoughts on Gwern's fictional takeover story, which I think is unrealistic in a bunch of ways but am curious to hear whether it's violating any of the efficiency limits you've argued for in that brain efficiency post--and then how the story would need to be changed in order to respect those limits.)
*To elaborate: I'm anticipating a mostly-software singularity, not hardware-based, so while I do think there's probably significant room for improvement in hardware I don't think it matters to my bottom line. I also expect that the first AGIs will be trained on more than 1e24 FLOP, and while I no longer think that 1e13 parameters will be required, I think it's quite plausible that 1e13 parameters will be required. I guess the main way in which I substantively disagree with your assumptions is the "no significant further room for any software improvement at all" bit. If we interpret it narrowly as no significant further room for improvement in the list of efficiency dimensions you gave earlier, then sure, I'm happy to take that assumption on board. If we interpret it broadly as no significant further room for capabilities-per-dollar, or capabilities-per-flop, then I don't accept that assumption, and claim that you haven't done nearly enough to establish it, and instead seem to be making a similar mistake to a hypothetical bird enthusiast in 1900 who declared that planes would never outcompete pigeons because it wasn't possible to be substantially more efficient than birds.