One question/concern I have been monitoring for a while now is the response from conservative Christianity. It's not looking good. Google "Singularity image of the beast" to get an idea.

What kind of problems do you think this will lead to, down the line?

You already live in a world filled with a huge sea of agents which have values different than your own. We create new generations of agents all the time, and eventually infuse them with power and responsibility.

This is true, but:

1. I'm not comparing ANN-based AGI to the status quo, but to a future with some sort of near-optimal FAI.
2. The new agents we currently create aren't much more powerful than ourselves, and cannot take over the universe and foreclose the possibility of a better outcome.
3. Humans or humanity as a whole seem capable of making moral and philosophical progress, and this capability is likely to persist in future generations. I'm not sure the same will be true of ANN-based AGIs.

That being said, I do believe that the AGI we create will be far more aligned with our values than our children are.

I look forward to your post explaining this, but again my fear is that since to a large extent I don't know what my own values are (especially when it comes to post-Singularity problems like how to reorganize the universe on a large scale, i.e., whether we should run it according to Eliezer's Fun Theory, or convert it to hedonium, or what sort of hedonium exactly, or to spend most of the resources available to me on some sort of attempt to break out of any potential simulations we might be in, or run simulations of my own), straightforward approaches at value learning won't work when it comes to people like me, and there won't be time to work out how to teach the AGI to solve these and other philosophical problems.

The real fear is perhaps that of being left behind. The only solution to that really is to use AGI to accelerate the development of uploading.

Because we care about preserving our personal identities whereas many AGIs probably won't, AGIs will be faced with fewer constraints when it comes to improving themselves or designing new generations of AGIs, and along with a time advantage that is likely quite large in subjective time, this probably means that AGIs will always have a large advantage in intelligence until they reach the maximum feasible level in this universe and human uploads slowly catch up. Are you not worried that during this time, the AGIs will take over the universe and reorganize it according to their imperfect understanding of our values, which will look disastrous when we become superintelligences ourselves and figure out what we really want?