> SIAI has a ... mixed attitude towards AIXI

That's the right attitude to have in general: Encourage good work, but be prepared to criticize it as well.

He calls AIXI "brain-malfunction-causing." I don't think he says that AIXI is a malfunction itself!

@Nick Tarleton "but if it is [arguing for something], it's arguing for the propagation of those parts of our psychology we really want to keep."

For many or most humans, to the extent that current religion is imperfect, the enhancement of spirituality is perhaps the most important goal for humanity. Of course, various groups do have variations on what they mean by religion, but worship of a god is very common

I wonder, then, if Eliezer's explanation/argument could be applied just as well to the preservation and encouragement of worship of the divine, though it would not fit well with the atheism advocated in his other articles.

If you replace "love" in this article with "theistic spirituality" -- another aspect of the human psychology which many, if not most, humans consider deeply important and beautiful -- and likewise replace mutatis mutandis other parts of the dialog, would it not just as well argue for the propagation of religion to our descendants?

Eliezer wrote "This isn't the whole story..., but it will do for a start", and in the referenced post: "This I will not describe, for it would be a long tale and complicated. I ... knew not the teachings of Tversky and Kahneman."

I've seen tantalizing hints of this "long tale," but I'd love to see the whole story, even in summary. If nothing else, it would be quite in place in a blog on Overcoming Bias.

Yet _something_ in the real world makes it tractable to create the "map" -- to find those hidden class variables which enable Naive Bayes.

I'd like to understand: Are frequentist "probability" and subjective "probability" simply two different concepts, to be distinguished carefully? Or is there some true debate here?

I think that Jaynes shows a derivation follownig Bayesian principles of the frequentist probability from the subjective probability. I'd love to see one of Eliezer's lucid explanations on that.

The real fear is not that Singularitarianism is a cult, but that it is pseudo-science (with certain practical consequences), like ESP, Velikovskianism, or certain false nutritional beliefs.

A commonly proposed solution is to look at the evidence with a scientific (Bayesian?) mindset, but most of us are woefully unqualified to judge most scientific fields without an intensive study that we are not about to engage in.

Great stuff as always. Enhanced diagrams (beyond the simple ASCII ones), with clear labels, and even inline explanations, on nodes and edges, would make the Bayesian explanations much clearer.

I've always thought it was silly to call great football players "heroes." But in fact, people can be heroes (in the sense of role models) in one area of life and not in others. You can admire and be inspired by a role model's athleticism, intellectual honesty, kindness, etc. even though these are not usually found all together in one person.

Part of the attraction of superheroes is their alpha male status: The warm glow we feel from power that protects us, the more the better. This is not quite the same thing as the halo effect.