Indeed, however this is dependent upon utility function - many people value the people who are alive now to an extent that cannot be compensated by future lives, even if there may be many orders of magnitude more people in the future.
Sure. One thing I might mention to someone with that utility function is that if humanity gets destroyed by an enhanced psychopath, that will probably happen right around the same time that enhanced scientists would be working to speed technological progress. So even someone with a relatively myopic utility function will in many cases favor caution.
So I see what you mean in principle, but in practice I think the co-ordination problem is too hard.
Clearly there are a lot of people very interested in the ethics of genetic enhancement. The current consensus among the scientific community in the West seems to be that enhancing kids is totally unethical, and gene modification techs should only be used to fix genetic diseases. In other words, currently in the West at least, there is a very strong (and effective, within the West) attempt being made to enforce coordination on this problem.
I think the current coordination strategy is a fairly hopeless one, for reasons I outlined in my post. All I'm trying to do is improve on it. Do you think I've succeeded there? Can you think of an even better coordination strategy than mine? The thing I like about my idea is that it doesn't require total coordination. It just requires that some things get discovered before other things, which is something that individual scientists can affect.
I agree that affecting the future is hard. But from my perspective (and the perspective of many other people who do think future lives are very important), it's worth attempting even if it's hard. If you're the kind of person who gives up when faced with hard challenges, that's fine; I guess we're just different in that way. "Shut up and do the impossible" and all that--the logic is similar to that of FAI. (Challenges can be exciting; easy video games aren't always very fun.) And in some cases things can be surprisingly possible (for example, it's surprisingly easy to find the email addresses of prominent scientists online, and they also have office hours).
I appreciate specific criticisms but if you're just going to be generically demoralizing, I don't usually find myself getting a lot out of that.
Sure. One thing I might mention to someone with that utility function is that if humanity gets destroyed by an enhanced psychopath, that will probably happen right around the same time that enhanced scientists would be working to speed technological progress. So even someone with a relatively myopic utility function will in many cases favor caution.
I get the idea that FAI takes more intelligence than AGI, as AGI might be able to be brute-forced by reverse-engineering the brain or evolutionary approaches, whereas de novo AI is far harder, let alone AGI. ...
I previously wrote a post hypothesizing that inter-group conflict is more common when most humans belong to readily identifiable, discrete factions.
This seems relevant to the recent human gene editing advance. Full human gene editing capability probably won't come soon, but this got me thinking anyway. Consider the following two scenarios:
1. Designer babies become socially acceptable and widespread some time in the near future. Because our knowledge of the human genome is still maturing, they initially aren't that much different than regular humans. As our knowledge matures, they get better and better. Fortunately, there's a large population of "semi-enhanced" humans from the early days of designer babies to keep the peace between the "fully enhanced" and "not at all enhanced" factions.
2. Designer babies are considered socially unacceptable in many parts of the world. Meanwhile, the technology needed to produce them continues to advance. At a certain point people start having them anyway. By this point the technology has advanced to the point where designer babies clearly outclass regular babies at everything, and there's a schism between "fully enhanced" and "not at all enhanced" humans.
Of course, there's another scenario where designer babies just never become widespread. But that seems like an unstable equilibrium given the 100+ sovereign countries in the world, each with their own set of laws, and the desire of parents everywhere to give birth to the best kids possible.
We already see tons of drama related to the current inequalities between individuals, especially inequality that's allegedly genetic in origin. Designer babies might shape up to be the greatest internet flame war of this century. This flame war could spill over in to real world violence. But since one of the parties has not arrived to the flame war yet, maybe we can prepare.
One way to prepare might be differential technological development. In particular, maybe it's possible to decrease the cost of gene editing/selection technologies while retarding advances in our knowledge of which genes contribute to intelligence. This could allow designer baby technology to become socially acceptable and widespread before "fully enhanced" humans were possible. Just as with emulations, a slow societal transition seems preferable to a fast one.
Other ideas (edit: speculative!): extend the benefits of designer babies to everyone for free regardless of their social class. Push for mandatory birth control technology so unwanted and therefore unenhanced babies are no longer a thing. (Imagine how lousy it would be to be born as an unwanted child in a world where everyone was enhanced except you.) Require designer babies to possess genes for compassion, benevolence, and reflectiveness by law, and try to discover those genes before we discover genes for intelligence. (Edit: leaning towards reflectiveness being the most important of these.) (Researching the genetic basis of psychopathy to prevent enhanced psychopaths also seems like a good idea... although I guess this would also create the knowledge necessary to deliberately create psychopaths?) Regulate the modification of genes like height if game theory suggests allowing arbitrary modifications to them would be a bad idea.
I don't know very much about the details of these technologies, and I'm open to radically revising my views if I'm missing something important. Please tell me if there's anything I got wrong in the comments.