Do we want to promote a theory that says "the very best thing is right, everything else is wrong",
No. That just means the better your imagination gets, the less you do.
Consequentialism solves all of this:
If something is impossible, it won't be reachable from the action set and therefore won't come into it. If something is bad, but nothing you can do will change it, it will cancel out. If some outcome is not actually preferable to some other outcome, you will have marked it as such in your utility assignment. If something good also comes with something worse, the utility of that possibility should reflect that. Etcetera.
In practice, you don't actually compute this, because it is uncomputable. Instead you follow simple rules that get you good results, like "don't throw away money" and "don't kill people" and "feed yourself" (Notice how the rules are justified by appealing to their expected consequences, though).
Thank you. As I understand it, "Consequentialism" means the idea that you should optimize outcomes.... It is a theory of right action. It requires a theory of "goodness" to go along with it. So, you're saying that "awesomeness" or "utility" is what is to be measured or approximated. Is that utilitarianism?
One of the criteria moral philosophers use to asses the credibility and power of a moral theory is "applicability". That is, how easy is it for humans to implement a moral rule? For example, a rule exists like "donate 23 hours a day to charity" it would be impossible for humans to fulfill the goal.
This lead me to start thinking about whether we want to be able to to pursue "the moral theoretical truth" should such a truth exist, or if we want to find the most applicable and practical set of rules, such that reasonably intramentaly rational (human) agents could figure out what is best in any given situation.
I feel like this is sort of like a map-territory distinction in a loose way. For example, the best thing to do in situation X might be A. A may be so difficult or require so much sacrifice, that B might be preferable, even if the overall outcome is not as good. This reminds me of how Eliezer says that the map is not the territory, but you can't fold the territory and put it in your pocket.
I'd love to be able to understand this issue a little better. If anyone has any thoughts, ideas or evidence, I'd appreciate hearing them.
Thanks,
Jeremy