Actually, I was responding to your "AI theorising was already (one of) his main interest/s", not your larger point.
I consider the possibility that Eliezer has intentionally deceived his donors all along as so unlikely as to not be worth discussing.
ADDED. Re-reading parent for the second time, I notice your "whether he was honest, consciously deceiving, or anywhere inbetween" (emphasis mine). So, since you (I now realize) probably were entertaining the possibility that he is "unconsciously deceiving" (i.e., has conveniently fooled himself), let me extend my reply.
People can be scrupulously honest in almost all matters, NihilCredo, and still deceive themselves about their motivations for doing something, so I humbly suggest that even though Eliezer has shown himself willing to issue an image-damaging public recantation when he discovers that something he has published is wrong that is not nearly enough evidence to trust his public statements about his motivations.
What one does instead is look at his decisions. And even more you look at what he is able to stay motivated to do over a long period of time. Consider for example the two years he spent blogging about rationality. This is educational writing or communication and it is extremely good educational communication. No matter how smart the person is, he cannot communicate or teach that effectively without doing a heck of a lot of hard work. And IMO no human being can work that hard for two whole years voluntarily (i.e., without fear of losing something he needs or loves and already has) unless the person is deriving some sort of real human satisfaction from the work. (Even with a very strong "negative" motivation like fear, it is hard to work that hard for 2 years without making yourself sick, and E sure did not look or act sick when I chatted with him at a Sep 2009 meetup.) And this is where the explanation gets complicated, and I want to cut it short.
There are only so many kinds of real human motivation. Scientists of course are usually motivated by the pleasure of discovery, of extending their understanding of the world. Many, perhaps most, scientists are motivated by reputation, for the good opinion of other scientists or the public at large. I find it unlikely however that any combination of those 2 motivations would have been enough for any human being to perform the way E did during his 2 years of "educating through blogging".
So, to summarize, I have some strong or firm reasons to believe that while he was writing those excellent blog posts, E regularly found pleasure and consequently found motivation in the idea of producing understanding in his readers, and this pleasure is an example of a "friendly impulse" or "altruistic desire" in E (part of the implementation in the human mind of the human capacity for what the evolutionary psychologists call reciprocal altruism).
And I know enough psychology to know that if E is capable of being motivated to extremely hard work by "the friendly impulse" when he started his blogging at age 27, then he was also capable of being motivated in his daydreams and in his career planning by "the friendly impulse" when he was a teenager (which is when he says he saw that AI research is the best way to help people and when he began his interest in AI theorizing). (It is rare for a person to be able to learn (even if they really want to) how to find pleasure (and consequently long-term motivation) from altruism / friendliness if they lacked the capacity in their teens like I did.)
Now I am not saying that E does not derive a lot of pleasure from scientific theorizing (most scientists of his caliber do), but I am saying that I believe his statements that the reason that most of his theorizing is about AI rather than string theory or population genetics is what he says it is.
This is all very condensed and it relies on beliefs of mine that are definitely not settled science, e.g., the belief that the only way a person every voluntarily works as hard as E must have for 2 years is if they find pleasure in the work) but it does explain just a little of the basis for the probability assignment I made in grandparent.
Definitely an interesting comment. Thanks.
I don't think I find your psychological argument very relevant here. The conspiracy allows - indeed, it makes a cardinal assumption - that Eliezer loves doing what he does, i.e. discussing and spreading ideas about rationality and theorising about AI and futurology; the only proposed dissonance between his statements and his findings would be that he is (whether intentionally or not, see below) overblowing the danger of a near-omnipotent unfriendly AI. And of course, people can be untruthful in one field and still ...
Major update here.
The state of affairs regarding the SIAI and its underlying rationale and rules of operation are insufficiently clear.
Most of the arguments involve a few propositions and the use of probability and utility calculations to legitimate action. Here much is uncertain to an extent that I'm not able to judge any nested probability estimations. Even if you tell me, where is the data on which you base those estimations?
There seems to be an highly complicated framework of estimations to support and reinforce each other. I'm not sure how you call this in English, but in German I'd call that a castle in the air.
I know that what I'm saying may simply be due to a lack of knowledge and education, that is why I am inquiring about it. How many of you, who currently support the SIAI, are able to analyse the reasoning that led you to support the SIAI in the first place, or at least substantiate your estimations with other kinds of evidence than a coherent internal logic?
I can follow much of the reasoning and arguments on this site. But I'm currently unable to judge their overall credence. Are the conclusions justified? Is the coherent framework build around the SIAI based on firm ground? There seems to be no critical inspection or examination by a third party. There is no peer review. Yet people are willing to donate considerable amounts of money.
I'm concerned that, although consistently so, the SIAI and its supporters are updating on fictional evidence. This post is meant to inquire about the foundations of your basic premises. Are you creating models to treat subsequent models or are your propositions based on fact?
An example here is the use of the Many-worlds interpretation. Itself a logical implication, can it be used to make further inferences and estimations without additional evidence? MWI might be the only consistent non-magic interpretation of quantum mechanics. The problem here is that such conclusions are, I believe, widely considered not to be enough to base further speculations and estimations on. Isn't that similar to what you are doing when speculating about the possibility of superhuman AI and its consequences? What I'm trying to say here is that if the cornerstone of your argumentation, if one of your basic tenets is the likelihood of superhuman AI, although a valid speculation given what we know about reality, you are already in over your head with debt. Debt in the form of other kinds of evidence. Not to say that it is a false hypothesis, that it is not even wrong, but that you cannot base a whole movement and a huge framework of further inference and supportive argumentation on such premises, on ideas that are themselves not based on firm ground.
The gist of the matter is that a coherent and consistent framework of sound argumentation based on unsupported inference is nothing more than its description implies. It is fiction. Imagination allows for endless possibilities while scientific evidence provides hints of what might be possible and what impossible. Science does provide the ability to assess your data. Any hint that empirical criticism provides gives you new information on which you can build on. Not because it bears truth value but because it gives you an idea of what might be possible. An opportunity to try something. There’s that which seemingly fails or contradicts itself and that which seems to work and is consistent.
And that is my problem. Given my current educational background and knowledge I cannot differentiate LW between a consistent internal logic, i.e. imagination or fiction, and something which is sufficiently based on empirical criticism to provide a firm substantiation of the strong arguments for action that are proclaimed by the SIAI.
Further, do you have an explanation for the circumstance that Eliezer Yudkowsky is the only semi-popular person who's aware of something that might shatter the universe? Why is it that people like Vernor Vinge, Robin Hanson or Ray Kurzweil are not running amok using all their influence to convince people of the risks ahead, or at least give all they have to the SIAI? Why aren't Eric Drexler, Gary Drescher or AI researches like Marvin Minsky worried to the extent that they signal their support?
I'm talking to quite a few educated people outside this community. They do not doubt all those claims for no particular reason. Rather they tell me that there are too many open questions to focus on the possibilities depicted by the SIAI and to neglect other near-term risks that might wipe us out as well.
I believe that many people out there know a lot more than I do, so far, about related topics and yet they seem not to be nearly as concerned about the relevant issues than the average Less Wrong member. I could have named other people. That's besides the point though, it's not just Hanson or Vinge but everyone versus Eliezer Yudkowsky and some unknown followers. What about the other Bayesians out there? Are they simply not as literate as Eliezer Yudkowsky in the maths or maybe somehow teach but not use their own methods of reasoning and decision making?
What do you expect me to do, just believe Eliezer Yudkowsky? Like I believed so much in the past which made sense but turned out to be wrong? Maybe after a few years of study I'll know more.
...
2011-01-06: As this post received over 500 comments I am reluctant to delete it. But I feel that it is outdated and that I could do much better today. This post has however been slightly improved to account for some shortcomings but has not been completely rewritten, neither have its conclusions been changed. Please account for this when reading comments that were written before this update.
2012-08-04: A list of some of my critical posts can be found here: SIAI/lesswrong Critiques: Index