Ooh, Value Learning sounds cool – I'll check that out. 
And yup, explicitly noting Goodhart's Law would have been nice.

Thanks for the comment!

I found The Flinch (Julien Smith) to be a good read! It’s less a book, and more an extended self-help essay. It was also useful to approach it as learning a soft skill, rather than explicitly gaining novel information.

The Action-Waiting-Reference framework clicked for me – thank you!

Ooh that makes sense – thank you!

I’m not sure I understand indexical uncertainty! To clarify – if we lived in a classical world, would this uncertainty not be present?

Ooh, this sounds like a neat follow-up – thank you for sharing!