I'm not sure if I'd get many Bayes points for my beliefs, rather than just my intuitions; after taking into account others' intuitions I don't think I think it's that much more plausible than others think it is.
I wish I could respond to the rest of your comment but am too flustered; hopefully I'll be able to later. What stands out as a possible miscontrual-with-a-different-idea is that I'm not sure if this idea of selfness as in narrow self interest even makes sense. If it does make sense then my intuition is probably wrong for the same reason various universal instrumental value hypotheses are probably wrong.
I'm not sure if this idea of selfness as in narrow self interest even makes sense.
Well, one can certainly talk about agents who have what we might describe as "narrow self-interest," though I don't really care about the distinction between self-interest and paperclipping and so on, which do seem to be well-defined.
E.g., whenever I experience something I add it to a list of experiences. I get a distribution over infinite lists of experiences by applying Solomonoff induction. At each moment I define my values in terms of that, and then try and ...
Here's the new thread for posting quotes, with the usual rules: