All of Kenzi Amodei's Comments + Replies

Examples of 'strong, general optimization pressures'? Maybe the sorts of things in that table from Superintelligence. ?Optimization pressure = something like a selective filter, where "strong" means that it was strongly selected for? And maybe the reason to say 'optimization' is to imply that there was a trait that was selected for, strongly, in the same direction (or towards the same narrow target, more like?) for many "generations". Mm, or that all the many different elements of the agent were built towards that trait, with nothing else being a stron... (read more)

I didn't know that about Bayesian inference-ish updating baking in an Occam-ish prior. Does it need to be complexity penalizing, or would any consistent prior-choosing rule work? I assume the former from the phrasing.

Why is that? "does not much constrain the end results" could just mean that unless we assume the agent is Occam ish, then we can't tell from its posteriors whether it did Bayesian inference or something else. But I don't see why that couldn't be true of some non-Occam-ish prior picking rule, as long as we knew what that was.

I think this... (read more)

Had a very visceral experience of feeling surrounded by a bunch of epistemically efficient (wrt me) agents in a markets game tonight. Just like "yup, I can choose to bet, or not bet, and if I do bet, I may even make money, because the market may well be wrong, but I will definitely, definitely lose money in expectation if I bet at all"

I seem to have found max comment length? Here's the rest:

I can't tell if I should also be trying to think about whether there's a reasonable definition of "the goals of google maps" wherein it actually is maximizing its goals right now in a way we can't advance. I don't think there is one?

I don't know why this hasn't happened to corporations - you'd think someone would try it, at some point, and that if it actually worked pretty well it would eventually allow them to outcompete, even if it was the sort of innovation that meant you had to climb uphill for a bit you'd expect people to keep periodically trying and for one of them eventually to overcome the activation energy barrier?

Boundedly rational ?means rational even when you don't have infinite computing power? Naturalistic ?refers to naturalized induction, where you're not a cartesian dualist who thinks your processes can't be messed with by stuff in the world and also you're not just thinking of yourself as a little black dot in the middle of Conway's game of life? Google says economic agent means one who has an impact on the economy by buying, selling or trading; I assign 65% to that being roughly the meaning in use here?

Somehow the epistemic efficiency thing reminds me of ... (read more)

Does it have to be (1) and (2)? My impression is that either one should be sufficient to count - I guess unless they turn out to be isomorphic, but naively I'd expect there to be edge cases with just one or the other.

Gosh this is just like reading the sequences, in the sense that I'm quite confused about what order to read things in. Currently defaulting to reading in the order on the VA list page

My guess why not to use a mathy definition at this point: because we don't want to undershoot when these protocols should be in effect. If that were the only c... (read more)

I'm surprised you want to use the word "advanced" to for this concept; implies to me this is the main kind of high-level safety missing from standard "safety" models? I guess the list of bullet points does cover a whole lot of scenarios. It does make it sound sexy, and not like something you'd want to ignore. Obvious alternative usage for the word advanced relative to safety would be for "actually" safe (over just claimed safe). Maybe that has other words available to it like provably.

I have the intuition that many proposals fail against advanced agent... (read more)