I think there will be multiple stable-on-reflection attractors in the space of possible minds. Within humans, these come out as religions or philosophical worldviews. The reason for this is that any mind must make some fundamental a-priori assumptions about its architecture that can't really be inferred from experience (or are implicated in how you would learn from experience). Your inductive prior is an obvious one, but also like how and where you trust your memory or perceptions, what kind of correction you accept from the environment, your instincts on how you approach problems, what's going to be valuable in the long run (ie "utility function"), what meta-physical assumptions, etc. VNM is a... (read more)
I think there will be multiple stable-on-reflection attractors in the space of possible minds. Within humans, these come out as religions or philosophical worldviews. The reason for this is that any mind must make some fundamental a-priori assumptions about its architecture that can't really be inferred from experience (or are implicated in how you would learn from experience). Your inductive prior is an obvious one, but also like how and where you trust your memory or perceptions, what kind of correction you accept from the environment, your instincts on how you approach problems, what's going to be valuable in the long run (ie "utility function"), what meta-physical assumptions, etc. VNM is a... (read more)