"Even after we know, we'll be left with a remaining question - is that how we should calculate should-ness? So it's not just a matter of sheer factual reductionism, it's a moral question."

I think that fits a different pattern. Specifcally the whole epiphenomenalism/pzombie thing.

If I actually fully understood everything about how the brain generates that sense of shouldness, not just some qualitative evolutionary history of why it might be there... ie, if I knew how to build that feeling out of toothpicks and rubber bands and fully understood why what I did did what it did, then I'd actually genuinely understand something I really don't understand now, and that understanding may, itself, tell me something about why I, ahem, should or shouldn't accept that particular computation of shouldness.