Reminds me of the motto "Strong Opinions, Weakly Held". There's no point having a blurry opinion, or not expressing what you believe to be the most likely candidate for a good way forward, even if it's more likely by only a small margin. By expressing (and/or acting on) a clearly expressed, falsifiable opinion, you expose it to criticism, refutation, improvement, etc. And if you hold it weakly, then you will be open to reconsidering. Refusing to make up your mind, and kindof oscilating between a few options, perhaps waiting to see where the wind blows, has its advantages, but especially when it comes to getitng things done, is most often a clear loser. Despite this, our brains seem to prefer it instinctively, maybe due to some ancestral environment echoes about being proven wrong in the eyes of the tribe?

The route to AI that you're suggesting is a plausible one; people like Nick Bostrom have talked about scenarios like this at length. Scenarios where we gradually shift our 'computational substrate' to non-biological hardware over several generations. But that's not necessarily what uploading is! As I mentioned, uploading is the transferring of a consciousness from some specific piece of hardware to another piece of hardware. The title and wording of your post implies that you are talking about uploading, but our discussion indicates you are actually talking about building an AI, which is an entirely different concept, and everyone who is confused about this distinction would do well to clearly understand it before talking about it.

The whole idea of uploading concerns human consciousness. Specifically, transferring a human consciousness to a non-biological context. If you're not talking about human consciousness, then you're just talking about building an AI.

There are still many intermediate steps. What does it mean "to be conscious of a sensory input"? Are we talking system 1 or system 2?

The system 1/system 2 distinction is only tangentially related here.

If the brain is composed of modules, which it likely is, what if some of them are digital and able to move to where the information is and others are not?

It's irrelevant whether the brain is 'composed of modules' or not. If what you mean is whether it is possible for consciousness to be distributed, well that's a good question. If it's possible for consciousness to be distributed then you could imagine being 'spread out' over a very large computer network (possibly many light-years in length). But the situation becomes tricky because if, say, your 'leg' was in one star system and your 'eye' was in another star system, stimulus from your eye could not cause a reaction from your leg in time shorter than several years, otherwise you violate the speed of light limit and causality. So either you cannot be 'spread out', or your perception of time slows down so extremely that several years seems instantaneous (just like the fraction of a second required for you to move your human leg seems instantaneous now).

what does it mean to "travel" when you can receive sensory inputs from any point in the network?

To be able to shorten the time which it takes to be conscious of a sensory input. If the sensor is at point A and you are at distance x from that sensor, you require at least x/c time to be aware of an input from that sensor.

The whole point of travel is to have low-latency, high-bandwidth access to information that exists at some point in the universe.

but that each step is small enough to make sense for a non-adventurous person

It still seems to me that the step from 'being tied to a specific piece of hardware' - whether that hardware is an entirely biological brain or an enhanced biological brain - to being pure information capable of moving from hardware to hardware is a pretty big step, regardless of how it is performed. It's the very essence of digitizing something. A physical book is information tied to hardware; uploading consists of scanning the book.

It seems like the problem of "the disruptive nature of the process from biological to digital being" still exists, except in your scenario it's only being pushed farther down the road.

You could imagine a radical outside-in modification of the human body, going from people with only their limbs replaced, to others with everything but their brain replaced, to others still with most of their brain except their cerebral cortex replaced, and even their cerebral cortex vastly modified and extended. But the seat of consciousness would still be their brain, and a person in such a state would not be considered an upload. They are not 50% uploaded or 5% uploaded or even 1% uploaded. They are zero percent uploaded, because their consciousness is still tied to some particular piece of hardware, exactly the way it was before. A true upload would not have their consciousness tied to any particular hardware - they would exist as information and could travel around at will.

A good 'test' for uploading might be the speed-of-light test. If your consciousness is capable of travelling at the speed of light, you are uploaded. Otherwise you aren't uploaded. The great thing about this test is that nature provides a very clear boundary between pure information and matter. Matter can contain information but is necessarily always travels slower than light, whereas pure information can and does travel at the speed of light.

Sure, it might be possible to have a 'half-uploaded' state where some of your consciousness is still tied to some hardware and some of it can move freely. But we don't yet know if such a state is possible. Actually, we don't even know if uploading is possible. It could be that your consciousness is always doomed to being tied to some hardware and it could never move at the speed of light.

This whole conversation sounds to me like people arguing whether width or height is a more important factor to the area of a rectangle. Or perhaps what percentage of the total each is responsible for.

It seems we humans are desperate to associate everything with a single cause, or if it has multiple causes, allocate causality to x% of multiple factors. However, success quite often has multiple contributing factors and exhibits "the chain is as strong as its weakest link" type behaviour. When phrased in terms of the contribution width and height make to the area of a rectangle, a lot of the conversation sounds like a category error. A lot of the metaphors we try and apply quite simply do not make sense.

The truly insidious effects are when the content of the stories changes the reward but not by going through the standard quality-evaluation function.

For instance, maybe the AI figures out that the order of the stories affects the rewards. Or perhaps it finds how stories that create a climate of joy/fear on campus lead to overall higher/lower evaluations for that period. Then the AI may be motivated to "take a hit" to push through some fear mongering so as to raise its evaluations for the following period. Perhaps it finds that causing strife in the student union, or perhaps causing racial conflict, or causing trouble with the university faculty affects its rewards one way or another. Perhaps if it's unhappy with a certain editor, it can slip through bad enough errors to get the editor fired, hopefully replaced with a more rewarding editor.

etc etc.

Got a project we worked on for my startup covered on hackaday.com!

Thanks for the encouragement! Would you mind offering your opinion on a few things though?

1. How many people would a complete overhaul take, and how long would it take (roughly)?
2. Why are the site owners reluctant to change?
3. What do you think of my rough cost-benefit argument? The things I said are my intuition, but I could easily be overlooking certain things, and I don't understand it well enough to be too confident in the intuition. So what do you think? (you seem to share the belief in the value of the benefits, but don't seem to think they outweigh the costs)

Also, I don't want to get anyone's hopes up about my contributions. I'm still learning to code and I don't know how good I'll be in 13 weeks when I finish my bootcamp and I can't tell how long it'll be before I'm capable enough to contribute to something like this.