You don't value other people's lives because they value their own lives. Paperclip maximizers value paperclips, but you won't take that into account. It's not so much contribution of other people's utility functions that drives your decisions (or morality). You just want mostly the same things, and care about others' well-being (which you should to an unknown extent, but which you obviously do at least somewhat).