I was dismayed that Pei has such a poor opinion of the Singularity Institute's arguments, and that he thinks we are not making a constructive contribution. If we want the support of the AGI community, it seems we'll have to improve our communication.

Well, my prior for someone on the internet who's asking for money being scam is no less than 99% (and I do avoid pascal mugging by not taking strings from such sources as proper hypotheses), and I think that is a very common prior, so there better be good evidence that it isn't scam - a panel of accomplished scientists and engineers, working to save the world, etc etc. think something on the scale of IPCC. rather than some weak evidence that it is scam, and something even less convincing than e.g. Steorn's perpetual motion device.

Scamming works best by self deceit though, so even though you are almost certainly just a bunch of fraudsters, you still feel genuinely wronged and insulted by suggestion that you are, because the first people that you would have defrauded would have been yourselves. You'd also feel wronged that there is nothing you could of done to look better. There isn't; if your cause was genuine it would of been started decades ago by more qualified people.

I don't think it matters how it is defined... One still shouldn't double count the evidence.

I thought about it some more and the relevant question is - how do we guess what are his abilities? And what is his aptitude at those abilities? Is there statistical methods we can use? (e.g. SPR) What would the outcome be? How can we deduce his utility function?

Normally, when one has e.g. high mathematical aptitude, or programming aptitude, or the like, as a teenager one still has to work on it and train (the brain undergoes significant synaptic pruning at about 20 years of age, limiting your opportunity to improve afterwards), and regardless of the final goal, the intelligent people tend to have a lot of things to show from back when they were practising. I think most people see absence of such stuff as a very strong indicator of lack of ability, especially as seeing it so provides incentive to demonstrate the ability.

I don't think you understood DanielVarga's point. He's saying that the numbers available for some of those features already have an unknown amount of the other features factored in. In other words, if you update on each feature separately, you'll end up double-counting an unknown amount of the data. (Hopefully this explanation is reasonably accurate.)

http://en.wikipedia.org/wiki/Conditional_independence

To explain the issue here in intuitive terms: let's say we have the hypothesis that Alice owns a cat, and we start with the prior probability of a person owning a cat (let's say 1 in 20), and then update on the evidence: she recently moved from an apartment building that doesn't allow cats to one that does (3 times more likely if she has a cat than if she doesn't), she regularly goes to a pet store now (7 times more likely if she has a cat than if she doesn't), and when she goes out there's white hair on her jacket sleeves (5 times more likely if she has a cat than if she doesn't). Putting all of these together by Bayes' Rule, we end up 85% confident she has a cat, but in fact we're wrong: she has a dog. And thinking about it in retrospect, we shouldn't have gotten 85% certainty of cat ownership. How did we get so confident in a wrong conclusion?

It's because, while each of those likelihoods is valid in isolation, they're not independent: there are a big chunk of people who move to pet-friendly apartments and go to pet stores regularly and have pet hair on their sleeves, and not all of them are cat owners. Those people are called pet owners in general, but even if we didn't know that, a good Bayesian would have kept tabs on the cross-correlations and noted that the straightforward estimate would be thereby invalid.

EDITED TO ADD: So the difference between that and the IQ test example is that you don't expect there to be an exceptional number of people who get the first two questions right and then do poorly on the rest of the test. The analogue there would be that, even though ability to solve mathematical problems correlates with ability to solve language problems, you should only count that correlation once. If a person does well on a slate of math problems, that's evidence they'll do well on language problems, but doing well on a second math test doesn't count as more strong evidence they'll do well on word problems. (That is, there are sharply diminishing returns.)

It seems you are talking about high-functioning psychopaths, rather than psychopaths according to the diagnostic DSM-IV criteria. Thus the prior should be different from 0.03. Assuming a high-functioning psychopath is necessarily a psychopath then it seems it should be far lower than 0.03, at least from looking at the criteria:

A) There is a pervasive pattern of disregard for and violation of the rights of others occurring since age 15 years, as >indicated by three or more of the following: failure to conform to social norms with respect to lawful behaviors as indicated by repeatedly performing acts that are >grounds for arrest; deception, as indicated by repeatedly lying, use of aliases, or conning others for personal profit or pleasure; impulsiveness or failure to plan ahead; irritability and aggressiveness, as indicated by repeated physical fights or assaults; reckless disregard for safety of self or others; consistent irresponsibility, as indicated by repeated failure to sustain consistent work behavior or honor financial >obligations; lack of remorse, as indicated by being indifferent to or rationalizing having hurt, mistreated, or stolen from another; B) The individual is at least age 18 years. C) There is evidence of conduct disorder with onset before age 15 years. D) The occurrence of antisocial behavior is not exclusively during the course of schizophrenia or a manic episode."

Eliezer either picked a much less lucrative career than he could have gotten with the same hours and enjoyment because he wanted to be altruistic, or I'm mistaken about career prospects for good programmers, or he's a dirty rotten conscious liar about his ability to program.

Note that Melinda Gates corresponds to the same criteria about as well.

I think that's an overreaction... It's not that you can't do the math, it's that you have to be very clear on what numbers go where and understand which you have to estimate and which can be objectively measured.