I'm not sure autonomy is the right word in reference to an algorithm, as the laws for the algorithm, so to speak, have been set down by another agent, the programmer. Which means the real question from a policy perspective may be what level of autonomy should you grant to the programmers.
The idea that there is a human to balance software has a weird opposite; I frequently see programs given "autonomous" access to data that the programmer isn't permitted to touch. If you wouldn't want programmers making life-and-death decisions, you don't want them codifying those decision-making processes, either.
I suspect the fact that programmers are making life and death decisions might be more alarming to many people than the idea that programs are making life and death decisions. For those who aren't alarmed by that possibility yet, a simple phrase may instill alarm: "These are the same kinds of people who programmed Outlook."
... I frequently see programs given "autonomous" access to data that the programmer isn't permitted to touch. If you wouldn't want programmers making life-and-death decisions, you don't want them codifying those decision-making processes, either.
In the context of the first quoted sentence, the second is false. An independent code review can make the program far more secure than any one of the reviewers would be.
If you mean something like, 'we don't have a good decision procedure for this case', then that's not a programming problem, it's a domain knowledge problem.
I would like to raise a discussion topic in the spirit of trying to quantify risk from uncontrolled / unsupervised software.
What is the maximum autonomy that has been granted to an algorithm according to your best estimates? What is the likely trend in the future?
The estimates could be in terms of money, human lives, processes, etc.
Another estimate could be on the time it takes for a human to come in the process and say "This isn't right".
A high speed trading algorithm has a lot of money on the line, but a drone might have lives on the line.
A lot of business processes might get affected by data coming in via an API from a system that might have had slightly different assumptions resulting in catastrophic events. eg. http://en.wikipedia.org/wiki/2010_Flash_Crash
The reason this topic might be worth researching is that it is a relatively easy to communicate risk of AGI. There might be many people who have an implicit assumption that whatever software is being deployed in the real world, there are humans to counter balance it. For them, empirical evidence that they are mistaken about the autonomy given to present day software may shift beliefs.
EDIT : formatting