they don't seem to have made a case for the AI killing literally everyone which addresses the decision theory counterargument effectively.

I think that's the crux here. I don't think the decision theory counterargument alone would move me from 99% to 75% - there are quite a few other reasons my probability is lower than that, but not purely on the merits of the argument in focus here. I would be surprised if that weren't the case for many others as well, and very surprised if they didn't put >75% probably on AI killing literally everyone.

I guess my position comes down to: There are many places where I and presumably you disagree with Nate and Eliezer's view and think their credences are quite different from ours, and I'm confused by the framing of this particular one as something like "this seems like a piece missing from your comms strategy". Unless you have better reasons than I for thinking they don't put >75% probability on this - which is definitely plausible and may have happened in IRL conversations I wasn't a part of, in which case I'm wrong.

Further, as far as I can tell, central thought leaders of MIRI (Eliezer, Nate Soares) don't actually believe that misaligned AI takeover will lead to the deaths of literally all humans:

This is confusing to me; those quotes are compatible with Eliezer and Nate believing that it's very likely that misaligned AI takeover leads to the deaths of literally all humans.

Perhaps you're making some point about how if they think it's at all plausible that it doesn't lead to everyone dying, they shouldn't say "building misaligned smarter-than-human systems will kill everyone". But that doesn't seem quite right to me: if someone believed event X will happen with 99.99% probability and they wanted to be succinct, I don't think it's very unreasonable to say "X will happen" instead of "X is very likely to happen" (as long as when it comes up at all, they're honest with their estimates).

I agree, and one of the reasons I endorse this strongly: there's a difference between performance at a task in a completion format and a chat format^[1].

Chess, for instance, is most commonly represented in the training data in PGN format - and prompting it with something that's similar should elicit whatever capabilities it possesses more strongly than transferring them over to another format.

I think this should generalize pretty well across more capabilities - if you expect the bulk of the capability to come from pre-training, then I think you should also expect these capabilities to be most salient in completion formats.

1. ^{^}

   Everything in a language model is technically in completion format all the time, but the distinction I want to draw here is related to tasks that are closest to the training data (and therefore closest to what powerful models are superhuman at). Models are superhuman at next-token prediction, and inasmuch as many examples of capabilities in the training data are given in prose as opposed to a chat format (for example, chess), you should expect those capabilities to be most salient in the former.

Yeah sorry I should have been more precise. I think it's so non-trivial that it plausibly contains most of the difficulty in the overall problem - which is a statement I think many people working on mechanistic interpretability would disagree with.

When I think of how I approach solving this problem in practice, I think of interfacing with structures within ML systems that satisfy an increasing list of desiderata for values, covering the rest with standard mech interp techniques, and then steering them with human preferences. I certainly think it's probable that there are valuable insights to this process from neuroscience, but I don't think a good solution to this problem (under the constraints I mention above) requires that it be general to the human brain as well. We steer the system with our preferences (and interfacing with internal objectives seems to avoid the usual problems with preferences) - while something that allows us to actually directly translate our values from our system to theirs would be great, I expect that constraint of generality to make it harder than necessary.

I think that it's a technical problem too - maybe you mean we don't have to answer weird philosophical problems? But then I would say that we still have to answer the question of what we want to interface with / how to do top-down neuroscience, which I would call a technical problem but which I think routes through things that I think you may consider philosophical problems (of the nature "what do we want to interface with / what do we mean by values in this system?") 

Yeah, that's fair. Two minor points then:

First is that I don't really expect us to come up with a fully general answer to this problem in time. I wouldn't be surprised if we had to trade off some generality for indexing on the system in front of us - this gets us some degree of non-robustness, but hopefully enough to buy us a lot more time before stuff like the problem behind deep deception breaks a lack of True Names. Hopefully then we can get the AI systems to solve the harder problem for us in the time we've bought, with systems more powerful than us. The relevance here is that if this is the case, then trying to generalize our findings to an entirely non-ML setting, while definitely something we want, might not be something we get, and maybe it makes sense to index lightly on a particular paradigm if the general problem seems really hard.

Second is that this claim as stated in your reply seems like a softer one (that I agree with more) than the one made in the overall post.

Yeah, I suppose then for me that class of research ends up looking pretty similar to solving the hard conceptual problem directly? Like, in that the problem in both cases is "understand what you want to interface with in this system in front of you that has property X you want to interface with". I can see an argument for this being easier with ML systems than brains because of the operability.

I am hopeful there are in fact clean values or generators of values in our brains such that we could just understand those mechanisms, and not other mechanisms. In worlds where this is not the case, I get more pessimistic about our chances of ever aligning AIs, because in those worlds all computations in the brain are necessary to do a "human mortality", which means that if you try to do, say, RLHF or DPO to your model and hope that it ends up aligned afterwards, it will not be aligned, because it is not literally simulating an entire human brain. It's doing less than that, and so it must be some necessary computation its missing.

There are two different hypotheses I feel like this is equivocating between: there being values in our brain that are represented in a clean enough fashion that you can identify them by looking at the brain bottom-up; and that there are values in our brain that can be separated out from other things that aren't necessary to "human morality", but that requires answering some questions of how to separate them. Descriptiveness vs prescriptiveness - or from a mech interp standpoint, one of my core disagreements with the mech interp people where they think we can identify values or other high-level concepts like deception simply by looking at the model's linear representations bottom-up, where I think that'll be a highly non-trivial problem.

Thanks for the comment, I'm glad it helped!

I'm not sure if I know exactly what parts you feel fuzzy on, but some scattered thoughts:

Abstracting over a lot of nuance and complexity, one could model internal optimization as being a ~general-purpose search process / module that the model can make use of. A general-purpose search process requires a goal to evaluate the consequences of different plans that you're searching over. This goal is fed into the search module as an input.

This input is probably described in the model's internal language; i.e., it's described in terms of concepts that the model learns corresponding to things in the environment. This seems true even if the model uses some very direct pointer to things in the environment - it still has to be represented as information that makes sense to the search process, which is written in the model's ontology.

So the inputs to the search process are part of the system itself. Which is to say that the "property" of the optimization that corresponds to what it's targeted at, is in the complexity class of the system that the optimization is internal to. I think this generalizes to the case where the optimization isn't as cleanly represented as a general-purpose search module.