Most planning around AI risk seems to start from the premise that superintelligence will come from de novo AGI before whole brain emulation becomes possible. I haven't seen any analysis that assumes both uploads-first and the AI FOOM thesis (Edit: apparently I fail at literature searching), a deficiency that I'll try to get a start on correcting in this post.
It is likely possible to use evolutionary algorithms to efficiently modify uploaded brains. If so, uploads would likely be able to set off an intelligence explosion by running evolutionary algorithms on themselves, selecting for something like higher general intelligence.
Since brains are poorly understood, it would likely be very difficult to select for higher intelligence without causing significant value drift. Thus, setting off an intelligence explosion in that way would probably produce unfriendly AI if done carelessly. On the other hand, at some point, the modified upload would reach a point where it is capable of figuring out how to improve itself without causing a significant amount of further value drift, and it may be possible to reach that point before too much value drift had already taken place. The expected amount of value drift can be decreased by having long generations between iterations of the evolutionary algorithm, to give the improved brains more time to figure out how to modify the evolutionary algorithm to minimize further value drift.
Another possibility is that such an evolutionary algorithm could be used to create brains that are smarter than humans but not by very much, and hopefully with values not too divergent from ours, who would then stop using the evolutionary algorithm and start using their intellects to research de novo Friendly AI, if that ends up looking easier than continuing to run the evolutionary algorithm without too much further value drift.
The strategies of using slow iterations of the evolutionary algorithm, or stopping it after not too long, require coordination among everyone capable of making such modifications to uploads. Thus, it seems safer for whole brain emulation technology to be either heavily regulated or owned by a monopoly, rather than being widely available and unregulated. This closely parallels the AI openness debate, and I'd expect people more concerned with bad actors relative to accidents to disagree.
With de novo artificial superintelligence, the overwhelmingly most likely outcomes are the optimal achievable outcome (if we manage to align its goals with ours) and extinction (if we don't). But uploads start out with human values, and when creating a superintelligence by modifying uploads, the goal would be to not corrupt them too much in the process. Since its values could get partially corrupted, an intelligence explosion that starts with an upload seems much more likely to result in outcomes that are both significantly worse than optimal and significantly better than extinction. Since human brains also already have a capacity for malice, this process also seems slightly more likely to result in outcomes worse than extinction.
The early ways to upload brains will probably be destructive, and may be very risky. Thus the first uploads may be selected for high risk-tolerance. Running an evolutionary algorithm on an uploaded brain would probably involve creating a large number of psychologically broken copies, since the average change to a brain will be negative. Thus the uploads that run evolutionary algorithms on themselves will be selected for not being horrified by this. Both of these selection effects seem like they would select against people who would take caution and goal stability seriously (uploads that run evolutionary algorithms on themselves would also be selected for being okay with creating and deleting spur copies, but this doesn't obviously correlate in either direction with caution). This could be partially mitigated by a monopoly on brain emulation technology. A possible (but probably smaller) source of positive selection is that currently, people who are enthusiastic about uploading their brains correlate strongly with people who are concerned about AI safety, and this correlation may continue once whole brain emulation technology is actually available.
Assuming that hardware speed is not close to being a limiting factor for whole brain emulation, emulations will be able to run at much faster than human speed. This should make emulations better able to monitor the behavior of AIs. Unless we develop ways of evaluating the capabilities of human brains that are much faster than giving them time to attempt difficult tasks, running evolutionary algorithms on brain emulations could only be done very slowly in subjective time (even though it may be quite fast in objective time), which would give emulations a significant advantage in monitoring such a process.
Although there are effects going in both directions, it seems like the uploads-first scenario is probably safer than de novo AI. If this is the case, then it might make sense to accelerate technologies that are needed for whole brain emulation if there are tractable ways of doing so. On the other hand, it is possible that technologies that are useful for whole brain emulation would also be useful for neuromorphic AI, which is probably very unsafe, since it is not amenable to formal verification or being given explicit goals (and unlike emulations, they don't start off already having human goals). Thus, it is probably important to be careful about not accelerating non-WBE neuromorphic AI while attempting to accelerate whole brain emulation. For instance, it seems plausible to me that getting better models of neurons would be useful for creating neuromorphic AIs while better brain scanning would not, and both technologies are necessary for brain uploading, so if that is true, it may make sense to work on improving brain scanning but not on improving neural models.
I realize that Luke considered economic growth a crucial consideration, but I was really relying on Keith Stanovich's proposed distinction between intelligence and rationality. It seems possible that you can increase someone's raw capability without making them reflective enough to get all of the important answers right. This would mean that rationality is not just a bonus. On the other hand, these things might go hand in hand. It seems worth investigating to me and relevant to comparing 'AI-first' and 'IA-first' risk mitigation strategies.
Also, I think the language of 'critical levels' is probably better than the language of 'acceleration' in this context. It seems safe to assume that FAI is a more difficult problem than AGI at this point, but I don't think it follows only from that that IA will accelerate FAI more than it accelerates AGI. That depends on many more facts, of which problem difficulty is just one. I have no problem with ceteris paribus clauses, but it's not clear what we're holding equal here. The identity and size of the party in control of the IA technology intuitively seems to me like the biggest consideration besides problem difficulty. A large part of why I consider IA-first an alternative worth thinking about is not because I think it's likely to differentially affect technological development in the obvious way, but because we may be below some critical threshold of intelligence necessary to build FAI and thus IA-first would be preferable to AI-first because AI-first would almost certainly fail. This also is not a sure thing and I think also warrants investigation.
Forgive me if I'm starting to ramble but, something I find interesting about this is, unless you have other reasons to reject the relevance of this point, it seems to me you have also implied that, if there is no hardware overhang, and WBE begins as a monopoly in the way that nuclear weapons began, then the monopolist may have to choose between uploading a lone human, psychological effects be damned, and delaying the use of an IA technique that is actually available to them. It seems that you could allow the lone emulation to interact with biological humans, and perhaps even 'pause' itself so that it experiences a natural amount of subjective time during social interaction, but if you abuse this too much for the sake of maintaining the emulation's mental health, then you sacrifice the gains in subjective time. Sacrificing subjective time is perhaps not so bad as it might seem because speed intelligence can be useful for other reasons, some of which you outlined in the article. Nonetheless, this seems like a related problem where you have to ask yourself what you would actually do with an AI that is only good for proving theorems. There often seems to be a negative correlation between safety and usefulness. Still, I don't know what I would choose if I could choose between uploading exactly one extraordinary human right now and not doing so. My default is to not do it and subsequently think very hard, because that's the reversible decision, but that can't be done forever.
Even if intelligence doesn't help at all for advancing FAI relative to AGI except via rationality, it still seems pretty unlikely that intelligence amplification would hurt, even if it doesn't lead to improvements in rationality. It's not like intelligence amplification would decrease rationality.
... (read more)