Content feedback:
The Preface to the Sequence on Value Learning contains the following advice on research directions for that sequence:
If you try to disprove the arguments in the posts, or to create formalisms that sidestep the issues brought up, you may very well generate a new interesting direction of work that has not been considered before.
This provides specific direction on what to look at and what work needs done. If such a statement for this sequence is possible, I think it would be valuable to include.
Thoughts on counterfactual reasoning
These examples of counterfactuals are presented as equivalent, but they seem meaningfully distinct:
Specifically, they don't seem equally difficult for me to evaluate. I can easily imagine the sun going out, but I'm not even sure what it would mean if 2+2=3. It confuses me that these two different examples are presented as equivalent, because they seem to be instances of meaningfully distinct classes of something. I spent some time trying to characterize why the su... (read more)