My current position is I don't know what the correct action to take to nudge the world the way I want. The world seems to be working somewhat at this point and any nudge may send it into a path towards something that doesn't work (even sub-human AI might change the order of the world so much it stops working).
So my strategy is to try and prepare a nudge that could be used in case of emergencies. While trying to live a semi-normal life as well and cope with akrasia etc, it is not going quickly.
Could you define sub-human AI, please?
It seems to me that we already have all manner of sub-human AI. The AIs that deal with telephone traffic, data mining, air-traffic control, the Gov't and Intelligence services, the Military, Universities that have AI programs, Zoos that have breeding programs (and sequence the genomes of endangered animals to find the best mate for the animal), etc.
Are these types of AI far too primitive to even be considered sub-human, in your opinion?
An uplifting message as we enter the new year, quoted from Edge.org:
A few thoughts: when considering the heavy skepticism that the singularity hypothesis receives, it is important to remember that there is a much weaker hypothesis, highlighted here by Tegmark, that still has extremely counter-intuitive implications about our place in spacetime; one might call it the bottleneck hypothesis - the hypothesis that 21st century humanity occupies a pivotal place in the evolution of the universe, simply because we may well be a part of the small space/time window during which it is decided whether earth-originating life will colonize the universe or not.
The bottleneck hypothesis is weaker than the singularity hypothesis - we can be at the bottleneck even if smarter-than-human AI is impossible or extremely impractical, but if smarter-than-human AI is possible and reasonably practical, then we are surely at the bottleneck of the universe. The bottleneck hypothesis is based upon less controversial science than the singularity hypothesis, and is robust to different assumptions about what is feasible in an engineering sense (AI/no AI, ems/no ems, nuclear rockets/generation ships/cryonics advances, etc) so might be accepted by a larger number of people.
Related is Hanson's "Dream Time" idea.