I should also add:

6) Where do you place the odds of you/your institute creating an unfriendly AI in an attempt to create a friendly one?

7) Do you have any external validation (ie, unassociated with your institute and not currently worshiping you) for this estimate, or does it come exclusively from calculations *you* made?

Eliezer, I have a few practical questions for you. If you don't want to answer them in this tread, that's fine, but I am curious:

1) Do you believe humans have a chance of achieving uploading without the use of a strong AI? If so, where do you place the odds?

2) Do you believe that uploaded human minds might be capable of improving themselves/increasing their own intelligence within the framework of human preference? If so, where do you place the odds?

3) Do you believe that increased-intelligence-uploaded humans might be able to create an fAI with more success than us meat-men? If so, where do you place the odds?

4) Where do you place the odds of you/your institute creating an fAI faster than 1-3 occurring?

5) Where do you place the odds of someone else creating an unfriendly AI faster than 1-3 occurring?

Thank you!!!

Eliezer- Have you written anything fictional or otherwise about how you envision an ideal post-fAI or post-singularity world? Care to share?

Michael- ah yes, that makes a lot of sense. Of course if the worm's only got 213 neurons, it's not going to have hundreds of neurotransmitters. That being said, it might have quite a few different receptor sub-types and synaptic modification mechanisms. Even so... It would seem theoretically feasible to me for someone to hook up electrodes to each neuron at a time and catalog not only the location and connections of each neuron, but also what the output of each synapse is and what the resulting PSPs are during normal C. elegans behaviors... Now *that's* something I should tell Brenner about, given his penchant for megalomaniacal information gathering projects (he did the C. Elegans genome, a catalog of each cell in its body throughout its embryonic development, and its neural connections).

Doug- too much stuff to put into an actual calculation, but I doubt we have complete knowledge, given how little we understand epigentics (iRNA, 22URNAs, and other micro RNAs), synaptic transcription, cytoskeletal transport, microglial roles, the enteric nervous system, native neuro-regeneration, and low and behold, neurotransmitters themselves. The 3rd edition of Kandel I was taught out of as an undergrad said nothing of orexins, histamine, the other roles of meletonin beyond the pineal gland, or the functions of the multifarious set of cannibinoid receptors, yet we now know (a short 2 years later) that all of these transmitters seem to play critical roles. Now, not being an elegans gal, I don't know if it has much simpler neurotransmission than we do. I would venture to guess it is simpler, but not extraordinarily so, and probably much more affected by simple epigenetic mechanisms like RNA interference. In C. elegans, iRNA messages are rapidly amplified, quickly shutting off the target gene in all of its cells (mammals have not been observed to amplify). Now, here's the kicker- it gets into the germ cells too! So offspring will also produce iRNAs and shut off genes! Now, due to amplification error, iRNAs are eventually lost if the worms are bred long enough, but Craig Mellow is now exploring the hypothesis that amplified iRNAs can be eventually permenantly disable (viral) DNA that's been incorporated into the C. elegans genome, either by more permenant epigenetic modification (methylation or coiling), or by splicing it out... Sorry for the MoBio lecture, but DUDE! This stuff is supercool!!!

Kennaway- I meant why can't we make something that C. elegans does in the same way that C. elegans does it using it's neural information. Clearly our knowledge must be incomplete in some respect. If we could do that, then imitating not only the size, but the programming of the caterpillar would be much more feasible. At least three complex programs are obvious: 1) crawl -coordinated and changeable sinusoidal motion seems a great way to move, yet the MIT 'caterpillar' is quite laughable in comparison to the dexterity of the real thing, 2)Seek- this involves a circular motion of the head, sensing some chemical signal, and changing directions accordingly, 3) Navigate- caterpillar is skillfully able to go over, under, and around objects, correcting its path to its original without doing the weird head-twirling thing, indicating that aside from chemeoattraction, it has some sense of directional orientation, which it must have or else its motion would be a random walk with correction and not a direct march. I wonder how much of these behaviors operate independently of the brain.

All of this reminds me of something I read in Robert Sapolski's book "Monkey Luv" (a really fluffy pop-sci book about baboon society, though Sapolski himself in person is quite amazing), about how human populations under different living conditions had almost predictable (at least in hindsight)explicative religions. People living in rainforests with many different creatures struggling at cross-purposes to survive developed polytheistic religions in which gods routinely fought and destroyed each other for their own goals. Desert dwellers (semites) saw only one great expanse of land, one horizon, one sky, one ecosystem, and so invented monotheism.

I wonder what god(s) we 21st century American rationalists will invent...

I am pleased that you mention that (at present) the human brain is still the best predictor of other humans' behavior, even if we don't understand why (yet). I've always known my intuitions to be very good predictors of what people will do and feel, though it's always been a struggle trying to formalize what I already *know* into some useful model that could be applied by anyone...

However, I was once told my greatest strength in understanding human behavior was *not* my intuitions, but my ability to evaluate intuitions as one piece of evidence among others, not assuming they are tyrranically correct (which they are certainly not), and thus improving accuracy... Maybe instead of throwing the baby out with the bathwater on human intuitions of empathy, we should practice some sort of semi-statistical evaluation of how certain we feel about a conclusion and update it for other factors. Do you do this already, Eliezer? How?

You can also replace, "Because you enjoy looking" with "Because you have to look" for many high-power jobs and positions. Dominance-Submission relationships in business and politics are very important to outcome. I would guess that a lot of bad decisions are made because of the necessity of this dance at high levels... how to crush it out? Not easy. Human nature, it would seem.

Why not, I can't help myself: Caledonian = Thersites, Eliezer = Agamemnon

Thersites only clamour’d in the throng,
Loquacious, loud, and turbulent of tongue:
Awed by no shame, by no respect controll’d,
In scandal busy, in reproaches bold:
With witty malice studious to defame,
Scorn all his joy, and laughter all his aim:—
But chief he gloried with licentious style
To lash the great, and monarchs to revile.
...
Sharp was his voice; which in the shrillest tone,
Thus with injurious taunts attack’d the throne.
Whate’er our master craves submit we must,
Plagued with his pride, or punish’d for his lust.
Oh women of Achaia; men no more!
Hence let us fly, and let him waste his store
In loves and pleasures on the Phrygian shore.
We may be wanted on some busy day,
When Hector comes: so great Achilles may:
From him he forced the prize we jointly gave,
From him, the fierce, the fearless, and the brave:
And durst he, as he ought, resent that wrong,
This mighty tyrant were no tyrant long.”
...
“Peace, factious monster, born to vex the state,
With wrangling talents form’d for foul debate:
Curb that impetuous tongue, nor rashly vain,
And singly mad, asperse the sovereign reign.
Have we not known thee, slave! of all our host,
The man who acts the least, upbraids the most?
...
Expel the council where our princes meet,
And send thee scourged and howling through the fleet.”