Why not?
Different configurations of artificial neurons (e.g., RNNs vs CNNs) are better at learning different things. If you build an AGI and don't test whether it can learn to do philosophy, it may not be able to learn to do philosophy very well. In the rush to build AGIs in order to reap the economic benefits, people probably won't have time to test for this.
The memetic software of humanity is the same software that we will instill into AGI.
I'm guessing that AGIs will have a very different distribution of capabilities from humans (e.g., they'll have much more working memory, and be able to do complex calculations instantaneously and with very low error, but bad at certain things that we neglect to optimize for when building them) so they'll probably develop a different set of memetic software that's more optimal for them.
As long as you continue to exist into the future, and as long as you have a significant share in the future decision structure (ie wealth or voting rights), then that should suffice - you will have time to figure out your long term values.
I guess that could potentially work while AGIs are maxed out at human level or slightly beyond and costing $1000/year, but if I'm not very optimistic that any social structure we come up with could preserve our share of the universe as the AGIs improve themselves and become more powerful. For example, if an AGI or a group of AGIs figures out a way to colonize the universe using resources under their sole control, why would they give the rest of us a share?
Today the foundry industry produces about 10 million mid-high end GPUs per year.
Surely there are lots of foundries (Intel's for example) that could be retooled to build GPUs if it became profitable to do so?
This puts some constraints on the likely transition time, and it is likely measured in years.
The hope is that we use this time to develop the necessary social structures to prevent AGIs from taking over the universe (without giving us a significant share of it)?
If you build an AGI and don't test whether it can learn to do philosophy, it may not be able to learn to do philosophy very well.
AGI to me is synonymous with a universal learning machine, and in particular with a ULM that learns at human capability. Philosophy is highly unlikely to require any specialized structures - because humans do philosophy with the same general cortical circuitry that's used for everything else.
In the rush to build AGIs in order to reap the economic benefits, people probably won't have time to test for this.
This is a potent...
If it's worth saying, but not worth its own post (even in Discussion), then it goes here.
Notes for future OT posters:
1. Please add the 'open_thread' tag.
2. Check if there is an active Open Thread before posting a new one. (Immediately before; refresh the list-of-threads page before posting.)
3. Open Threads should be posted in Discussion, and not Main.
4. Open Threads should start on Monday, and end on Sunday.