I think this post shifts the burden to risk-concerned folks and only justifies that risk with its own poor analogies. 7/10 of the analogy-makers you cite here have pdooms <=50%, so they are only aiming for plausibility. You admit that the analogies point to the "logical possibility" of stark misalignment, but one person's logical possibility is another person's plausibility.
To give an example, Golden Retrievers are much more cherry-picked than Cotra's lion/chimpanzee examples. Of all the species on Earth, the ones we've successfully domesticated are a tiny, tiny minority. Maybe you'd say we have a high success rate when we try to domesticate a species, but that took a long time in... (read more)
I think this post shifts the burden to risk-concerned folks and only justifies that risk with its own poor analogies. 7/10 of the analogy-makers you cite here have pdooms <=50%, so they are only aiming for plausibility. You admit that the analogies point to the "logical possibility" of stark misalignment, but one person's logical possibility is another person's plausibility.
To give an example, Golden Retrievers are much more cherry-picked than Cotra's lion/chimpanzee examples. Of all the species on Earth, the ones we've successfully domesticated are a tiny, tiny minority. Maybe you'd say we have a high success rate when we try to domesticate a species, but that took a long time in... (read more)