This cybernetic replacement fully emulates all interactions that it can have with any neighboring cells including any changes in those interactions based on inputs received and time passed, but is not biological.

Why would that be possible? Neurons have to process biochemicals. A full replacement would have to as well. How could it do that without being at least partly biological?

It might be the case that an adequate replacement -- not a full replacment -- could be non-biological. But it might not.

Very sure. The biological view just seems to be a tacked on requirement to reject emulations by definition.

Note that I specifically said in the OP that I'm not much concerned about the biological view being right, but about some third possibility nobody's thought about yet.

Anyone who would hold the biological view should answer the questions in this though experiment.

A new technology is created to extend the life of the human brain. If any brain cell dies it is immediately replaced with a cybernetic replacement. This cybernetic replacement fully emulates all interactions that it can have with any neighboring cells including any changes in those interactions based on inputs received and time passed, but is not biological. Over time the subject's whole brain is replaced, cell by cell. Consider the resulting brain. Either it perfectly emulates a human mind or it doesn't. If it doesn't, then what is there to the human mind besides the interactions of brain cells? Either it is conscious or it isn't. If it isn't then how was consciousness lost and at what point in the process?

This is similar to an argument Charlmers gives. My worry here is that it seems like brain damage can do weird, non-intuitive things to a person's state of consciousness, so one-by-one replacement of neurons might to similar weird things, perhaps slowly causing you to lose consciousness without realizing what was happening.

Very sure. The biological view just seems to be a tacked on requirement to reject emulations by definition. Anyone who would hold the biological view should answer the questions in this though experiment.

A new technology is created to extend the life of the human brain. If any brain cell dies it is immediately replaced with a cybernetic replacement. This cybernetic replacement fully emulates all interactions that it can have with any neighboring cells including any changes in those interactions based on inputs received and time passed, but is not biological. Over time the subject's whole brain is replaced, cell by cell. Consider the resulting brain. Either it perfectly emulates a human mind or it doesn't. If it doesn't, then what is there to the human mind besides the interactions of brain cells? Either it is conscious or it isn't. If it isn't then how was consciousness lost and at what point in the process?

Why are we talking about jobs rather than man-hours worked? Automation reduced man-hours worked. We went from much longer work weeks to 40 hour work weeks as well as raising standards of living.

AI will reduce work time further. If someone can use AI to produce as much in 30 hours as they did in 40, they could chose to work anywhere from 30 - 40 hours and be better off. Many people would chose to work less as they compare the marginal values of free time and extra pay.

Why are we seeing long term unemployment instead of shorter work weeks now? Is this inevitable or is there some structural or institutional problem causing it?

Contrast fishing (an international "commons") with forestry (a series of national commons). Many countries have successful forestry programs that preserve quite decently; but the devastation caused by overfishing is extreme. These two industries are not fundamentally different, but the fact that one requires international cooperation and the other doesn't seem to make all the difference. You could also glance at the successful international pollution reductions (eg CFCs and acid rain). A singleton should be able to do better than the painstakingly negotiated treaties of today!

On specifics, the WHO seems to have a pretty decent track record on pandemics (not nearly as good as it should me, much better than it could be). I'm not all that knowledgeable on the various rules governing fissile materials, but they seem to be working acceptably, in any place that they can be enforced. And, of course, regulations of synthetic biology and AI are essentially impossible without a singleton or extreme global coordination.

Why is it legitimate to assume that a singleton would be effective at solving existential risks? A one world government would have all the same internal problems as current governments. The only problems that scaling up would automatically eliminate are those of conflicts between different states, and these would likely be transformed into conflicts between interest groups in one state. This is not a reduction to a solved problem.

There are wars of secession and revolution now. There are also violent conflicts among ethnic and religious groups within one state. There is terrorism. Why would a one world government ruling over a more diverse populace than any current government not have these problems? People won't automatically accept the singleton any more than they accept the current governments.

Even with unified powers, governments regularly mismanage crises. Current governments (even democratic first world governments) have problems dealing with such things as predictable weather and earthquakes along known fault lines. Why would a one world government be better able to handle much less predictable crises, like a pandemic?

I think you're conflating "rationalist" and "intellectual." I agree that there is a stereotype that intellectuals only listen to Great Works like Bach or Mozart, but I'm curious where the OP picked up that this stereotype also ought to apply to LW-style rationalists. I mean, Eliezer takes pains in the Sequences to make anime references specifically to avoid this kind of thing.

I thought rationalists just sort of listened only to Great Works like Bach or Mozart

Why?

Write down your own Fermi estimation attempts here. One Fermi estimate per comment, please!