People could (and some of them probably do) value not-taking-over-the-world very highly.

This is a statement about valuation of states of the world, a valuation that is best satisfied by some form of taking over the world (probably much more subtle than what gets classified so by the valuation itself).

I think they are silly, but it is their values that matter to them, not my evaluation thereof.

It's still your evaluation of their situation that says whether you should consider their opinion on the matter of their values, or know what they value better than they do. What is the epistemic content of your thinking they are silly?