Suppose that inventing a recursively self improving AI is tantamount to solving a grand mathematical problem, similar in difficulty to the Riemann hypothesis, etc. Let's call it the RSI theorem.

This theorem would then constitute the primary obstacle in the development of a "true" strong AI. Other AI systems could be developed, for example, by simulating a human brain at 10,000x speed, but these sorts of systems would not capture the spirit (or capability) of a truly recursively self-improving super intelligence.

Do you disagree? Or, how likely is this scenario, and what are the consequences? How hard would the "RSI theorem" be?

I thought I'd pose an informal poll, possibly to become a top-level, in preparation for my article about How to Explain.

The question: on all the topics you consider yourself an "expert" or "very knowledgeable about", do you believe you understand them at least at Level 2? That is, do you believe you are aware of the inferential connections between your expertise and layperson-level knowledge?

Or, to put it another way, do you think that, given enough time, but using only your present knowledge, you could teach a reasonably-intelligent layperson, one-on-one, to understand complex topics in your expertise, teaching them every intermediate topic necessary for grounding the hardest level?

Edit: Per DanArmak's query, anything you can re-derive or infer from your present knowledge counts as part of your present knowledge for purposes of answering this question.

I'll save my answer for later -- though I suspect many of you already know it!

pjeby wrote about something like that in this post:

So you are just a consultant, called on the spot to voice an opinion, which may or may not be promptly taken out of context and used to other ends than your own. Are you beginning to see why changing some things can be so hard?

You are not running the show. You didn't even write the script, and you are certainly not the star. You are, at best, the star's agent.

Cool... that's really close to where I work. I'll probably make it. Thanks for taking the initiative guys.

I'm not sure if I buy that the "frequentist" explanations (as in the disease testing example) are best characterized by being frequentist -- it seems to me that they are just stating the problem and the data in a more relevant way to the question that's being asked. Without those extra statements, you have to decode the information down from a more abstract level.

Is there any philosophy worth reading?

As far as I can tell, a great deal of "philosophy" (basically the intellectuals' wastebasket taxon) consists of wordplay, apologetics, or outright nonsense. Consequently, for any given philosophical work, my prior strongly favors not reading it because the expected benefit won't outweigh the cost. It takes a great deal of evidence to tip the balance.

For example: I've heard vague rumors that GWF Hegel concludes that the Prussian State (under which, coincidentally, he lived) was the best form of human existence. I've also heard that Descartes "proves" that God exists. Now, whether or not Hegel or Descartes may have had any valid insights, this is enough to tell me that it's not worth my time to go looking for them.

However, at the same time I'm concerned that this leads me to read things that only reinforce the beliefs I already have. And there's little point in seeking information if it doesn't change your beliefs.

It's a complicated question what purpose philosophy serves, but I wouldn't be posting here if I thought it served none. So my question is: What philosophical works and authors have you found especially valuable, for whatever reason? Perhaps the recommendations of such esteemed individuals as yourselves will carry enough evidentiary weight that I'll actually read the darned things.

Could you break down your objection?

EDIT to look at it from another angle: it's clear that the first serve in this discussion has to come from the cryonicists, since we're the ones trying to change people's minds. But cryonicists have served and served and served; there's a massive literature arguing in favour, of which I'd pick out Ben Best's "Scientific Justification of Cryonics Practice". If you don't feel that anything in that literature is enough to show that cryonics might be a good idea, you're going to have to make some sort of actual return to those serves, at least to be more specific about what you feel is missing from them that could and should be there to justify signing up. Because as I've shown, no-one is even trying to return any of those serves.

There is a total lack of evidence in support of resurrecting a frozen human because its never been done and as of now nobody knows if it is even possible.

Others have already addressed this claim but I'd like to address it another way briefly. In particular, just because a specific technological goal has not yet been achieved does not mean there is no evidence for that goal. If one said in 1968 that there was no evidence that humans could go to the Moon that would be regarded as likely incorrect. Here's a brief list of technologies we don't have today. I'd be deeply surprised if you don't consider it likely that we'll have at least some of these at some point in the future: 1) practical fusion power, 2) A human mission to Mars 3) Substantial life extension 4) direct brain-computer interfaces.

All of these examples fit your model of being technologies which we don't have yet. The third example, life extension seems particularly relevant. Based on your comment above I'm pretty sure you would not be willing to say "There is a total lack of evidence in support of substantial life extension of humans because its never been done and as of now nobody knows if it is even possible."

I have been thinking about "holding off on proposing solutions." Can anyone comment on whether this is more about the social friction involved in rejecting someone's solution without injuring their pride, or more about the difficulty of getting an idea out of your head once it's there?

If it's mostly social, then I would expect the method to not be useful when used by a single person; and conversely. My anecdote is that I feel it's helped me when thinking solo, but this may be wishful thinking.

This is a mostly-shameless plug for the small donation matching scheme I proposed in May:

I'm still looking for three people to cross the "membrane that separates procrastinators and helpers" by donating $60 to the Singularity Institute. If you're interested, see my original comment. I will match your donation.