A lot of it depends on what sort of system an AI implementation ends up having to be like. These are examples only of things you might need to do.
1) Prove that the messy AI do not have the same sort of security holes that modern computers do. A true AI botnet would be a very scary thing.
2) If has a component that evolves, prove that the it is an evolutionarily stable strategy for whatever is evolving to optimize what you want optimized.
There might also be work done to try and computationally categorize how real neural nets are different from computers. And human from other animal brains. Anything to help us push back the "Here be dragons signs" on portions of Messy AI so we know what we can and can't use. And if we can't figure out how to use human level bits safely, don't use them until we have no other choice.
There are also questions of how best to deploy AI (close sourced/design giving you more control, open source more minds to check the code).
If this is a bit jumbled it is because Messy AI has huge numbers of possibilities and we are pretty ignorant about it really.
By the way, what do you mean by "messy AI"?
Inspired by the talk by Anna Salamon I decided to do my own calculations about the future. This post is a place for discussion about mine and others calculations.
To me there are two possible paths for the likely development of intelligence, that I can identify.
World 1) Fast and conceptually clean. Intelligence is a concrete value like the number of neutrons in a reactor. I assign a 20% chance of this.
World 2) Slow and messy. Intelligence is contextual, much like say fitness in evolutionary biology. Proofs of intelligence of a system are only doable by a much higher intelligence entity, as it will involve discussing the complex environment. I'd assign about an 60% chance to this.
Worlds 3) Other. The other 20% chance is the rest of the scenarios that are not either of these two.
Both types of AI have the potential to change the world, both possibly destroying humanity if we don't use them correctly. So they both have the same rewards.
So for world 1, I'll go with the same figures as Anna Salamon, because I can't find strong arguments against them (and it will serve as a refresher )
Probability of an eventual AI (before humanity dies otherwise) = 80%
Probability that AI will kill us = 80%
Probability that we manage safeguards = 40%
Probability that current work will save us = 30%
So we get 7%*20%. Gives us 1.4%
So for world 2. Assume we have an SIAI that is working on the problem of how to make messy AI Friendly or at least as Friendly as possible. It seems less likely we would make AI and harder to create safeguards as they have to act over longer time.
Probability of an eventual AI (before humanity dies otherwise) = 70%
Probability that AI will kill us (and/or we will have to give up humanity due to hard scrapple evolution) = 80%
Probability that we manage safeguards = 30%
Probability that current work will save us = 20%
So we get a factor of 3% times 60% give a 1.8%.
Both have the factor of 7billion lives times n, so that can be discounted. They pretty much weigh the same. Or as near as dammit for a back of the envelope calcs, considering my meta-uncertainty is high as well.
They do however interfere. The right action in world 1 is not the same as the right action in world 2. Working on Friendliness of conceptually clean AI and suppressing all work and discussion on messy AI hurts world 2 as it increases the chance we might end up with messy UFAI. There is no Singularity Institute for messy AI in this world, and I doubt there will be if SIAI becomes somewhat mainstream in AI communities, so giving money to SIAI hurts world 2, it might have a small negative expected life cost. Working on Friendliness for Messy AI wouldn't intefere with the Clean AI world, as long as it didn't do stupid tests until the messy/clean divide became solved. This tips the scales somewhat towards working on messy FAI and how it is deployed. World 3 is so varied I can't really say much about.
So for me the best information I should seek is getting more information on the messy/clean divide. Which is why I always go on about whether SIAI has a way of making sure it is on the right track with the Decision Theory/conceptually clean path.
So how do the rest of you run the numbers on the singularity?