I feel morally obligated to restate a potentially relevant observation:

I think that an important underlying difference of perspective here is that the Less Wrong memes tend to automatically think of all AGIs as essentially computer programs whereas Goertzel-like memes tend to automatically think of at least some AGIs as non-negligibly essentially person-like. I think this is at least partially because the Less Wrong memes want to write an FAI that is essentially some machine learning algorithms plus a universal prior on top of sound decision theory whereas the Goertzel-like memes want to write an FAI that is essentially roughly half progam-like and half person-like. Less Wrong memes think that person AIs won't be sufficiently person-like but they sort of tend to assume that conclusion rather than argue for it, which causes memes that aren't familiar with Less Wrong memes to wonder why Less Wrong memes are so incredibly confident that all AIs will necessarily act like autistic OCD people without any possibility at all of acting like normal reasonable people. From that perspective the Goertzel-like memes look justified in being rather skeptical of Less Wrong memes. After all, it is easy to imagine a gradation between AIXI and whole brain emulations. Goertzel-like memes wish to create an AI somewhere between those two points, Less Wrong memes wish to create an AI that's even more AIXI-like than AIXI is (in the sense of being more formally and theoretically well-founded than AIXI is). It's important that each look at the specific kinds of AI that the other has in mind and start the exchange from there.

We don't know if AIXI-approximating AIs would even be intelligent; how then can we be so confident that AIXI is a normative model and a definition of intelligence? This and other intuitions are likely underlying Goertzel's cautious epistemic state, and LessWrong/SingInst truly hasn't addressed issues like this. We don't know what it takes to build AGI, we don't know if intelligence runs on Bayes structure. Modern decision theory indicates that Eliezer was wrong, that Bayes structure isn't fundamental to agentic optimization, that it only applies in certain cases, that Bayesian information theoretic models of cognition might not capture the special sauce of intelligence. What is fundamental? We don't know! In the meantime we should be careful about drawing conclusions based on the assumed fundamental-ness of mathematical models which may or may not ultimately be accurate models, may or may not actually let you build literalistic self-improving AIs of the sort that LessWrong likes to speculate about.