If you feel that you're being led by abstract reasoning in directions that feel consistently feel wrong then there's probably something wrong with the reasoning.

Yup, my confidence in the reasoning here on LW and my own ability to judge it is very low. The main reason for this is described in your post above, taken to its logical extreme you end up doing seemingly crazy stuff like trying to stop people from creating baby universes rather than solving friendly AI.

I don't know how to deal with this. Where do I draw the line? What are the upper and lower bounds? Are risks from AI above or below the line of uncertainty that I better ignore, given my own uncertainty and the uncertainty in the meta-level reasoning involved?

I am too uneducated and probably not smart enough to figure this out, yet I face the problems that people who are much more educated and intelligent than me devised.