I can "scale up" a threaded program by giving more processors for the threads to run on, but this doesn't actually improve the program output (apart from rounding error and nondeterministic effects), it just makes the output faster.
Sure. Computers don't always behave like brains do. Indeed, they are mostly designed to compensate for brain weakenesses - to be strong where we are weak.
I can "scale up" an approximation algorithm that has a variable discretization size N, and that actually improves the output... but how do you adjust "N" in a worm brain?
We know that nature can easily scale brains up - because it did so with chimpanzees. Scaling brains up may not be trivial - but it's probably much easier than building one in the first place. Once we can build a brain, we will probablly be able to make another one that is bigger.
Once we can build a brain, we will certainly be able to make another one that is bigger, but that won't make it better. "Given this arrangement of neurons, what firing patterns will develop" is almost a completely different task than "given this problem to solve, what arrangement of neurons will best solve it", which itself is merely a footnote to the task of "wait, what are our definitions of 'problem' and 'best' again?"
Nature scaled chimpanzee brains up by creating billions of them and running them through millions of years...
If you were a utilitarian, then why would you want to risk creating an AGI that had the potential to be an existential risk, when you could eliminate all suffering with the advent of WBE (whole brain emulation) and hence virtual reality (or digital alteration of your source code) and hence utopia? Wouldn't you want to try to prevent AI research and just promote WBE research? Or is it that AGI is more likely to come before WBE and so we should focus our efforts on making sure that the AGI is friendly? Or maybe uploading isn't possible for technological or philosophical reasons (substrate dependence)?
Is there a link to a discussion on this that I'm missing out on?