What are your thoughts on the following Great Filter hypothesis: (1) Reward-based learning is the only efficient way to create AI. (2) AGI is easy, but FAI is hard to invent because the universe is so unpredictable (intelligent systems themselves being the most unpredictable structures) and nearly all reward functions will diverge once the AI starts to self-improve and create copies of itself. (3) The reward functions needed for a friendly reinforcement learner reflect reality in complex ways. In the case of humans they are learned by trial and error during evolution. (4) Because of this, the invention of FAI requires a simulation in which it can safely learn complex reward functions via evolution or narrow AI, which is time-consuming. (5) However, once AGI is widely regarded as feasible, people will realize that whoever invents it first will have nearly unlimited power. An AI arms race will ensue in which unfriendly AGIs are much more likely to arise.
I don't see why an unfriendly AGI would be significantly less likely to leave a trail of astronomical evidence of its existence than a friendly AI or an interstellar civilisation in general.
If it's worth saying, but not worth its own post (even in Discussion), then it goes here.
Notes for future OT posters:
1. Please add the 'open_thread' tag.
2. Check if there is an active Open Thread before posting a new one. (Immediately before; refresh the list-of-threads page before posting.)
3. Open Threads should be posted in Discussion, and not Main.
4. Open Threads should start on Monday, and end on Sunday.