What most people don't realize is that there is a mathematically optimal way to update your beliefs in response to evidence, and a mathematically optimal way to figure out which decision is most likely to bring you the most of what you want

You've said something similar in a recent video interview posted on LW, and it cringed me then, as it does now. We don't know of such optimal ways in the generality the context of your statement suggests, and any such optimal methods would be impractical even if known, which again is in conflict with the context. Similarly, turning to the interview, SingInst's standard positions on many issues don't follow from formal considerations such as logic and decision theory, there is no formal theory that represents them to any significant extent. If there is strength to the main arguments that support these positions, it doesn't currently take that form.