TheAncientGeek comments on No Universally Compelling Arguments in Math or Science - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (227)
There is no problem of strict formulation, because that is not what I am aiming at, it's your assumption.
I am aware that the variation isn't random. I don't think that is significant.
I don't think sudden catastrophic failure is likely in incremental/evolutionary progress.
I don't think mathematical "proof" is going to be as reliable as you think, given the complexity.
One of the key disanalogies between your "ecosystem" formulation and human development of technology is that natural selection isn't an actor subject to feedback within the system.
If an organism develops a mutation which is sufficiently unfavorable to the Blind Idiot God, the worst case scenario is that it's stillborn, or under exceptional circumstances, triggers an evolution to extinction. There is no possible failure state where an organism develops such an unfavorable mutation that evolution itself keels over dead.
However, in an ecosystem where multiple species interrelate and impose selection effects on each other, then a sudden change in circumstances for one species can result in rapid extinction for others.
We impose selection effects on technology, but a sudden change in technology which kills us all would not be a novel occurrence by the standards of ecosystem operation.
ETA: It seems that your argument all along has boiled down to "We'll just deliberately not do that" when it comes to cases of catastrophic failure. But the argument of Eliezer and MIRI all along has been that such catastrophic failure is much, much harder to avoid than it intuitively appears.