Anonymous, that reminds me of some anecdote by Feynman where he has complex mathematical ideas described to him by young students. He wouldn't fully understand them, but he would imagine a shape, and for each new concept he'd add an extra bit, like a squiggly tail or other appendage. When something didn't fit in right, it would be instantly obvious to him, even if he couldn't explain exactly why.

Improvised sensory modality for maths?

And note that Eliezer never answered your question, namely, if you can modify yourself so that you never get bored, do you care about or need to have fun?

Richard, probably you wouldn't care or need to have fun. But why would you do that? Modifying yourself that way would just demonstrate that you value the means of fun more than the ends. Even if you could make that modification, would you?

How odd, I just finished reading The State of the Art yesterday. And even stranger, I thought 'Theory of Fun' while reading it. Also, nowhere near the first time that something I've been reading has come up here in a short timeframe. Need to spend less time on this blog!

Trying to anticipate the next few posts without reading:

Any Theory of Fun will have to focus on that elusive magical barrier that distinguishes what we do from what Orgasmium does. Why should it be that we place a different on earning fun from simply mainlining it? The intuitive answer is that 'fun' is the synthesis of endeavour and payoff. Fun is what our brains do when we are rewarded for effort. The more economical and elegant the effort we put in for higher rewards the better. It's more fun to play Guitar Hero when you're good at it, right?

But it can't just be about ratio of effort to reward, since orgasmium has an infinite ratio in this sense. So we want to put in a quantity of elegant, efficient effort, and get back a requisite reward. Still lots of taboo-able terms in there, but I'll think further on this.

Phil, what Vlad and Nick said. I've no doubt we won't look much like this in 100 years, but it's still humanity and its heritage shaping the future. Go extinct and you ain't shaping nothing. This isn't a magical boundary, it's a pretty well-defined one.

'Precise steering' in your sense has never existed historically, yet we exist in a non-null state.

Aron, Robin, we're only just entering the phase during which we can steer things to either a really bad or really good place. Only thinking in the short term, even if you're not confident in your predictions, is pretty irresponsible when you consider what our relative capabilities might be in 25, 50, 100 years.

There's absolutely no guarantee that humanity won't go the way of the neanderthal in the grand scheme of things. They probably 'thought' of themselves as doing just fine, extrapolating a nice stable future of hunting, gathering, procreating etc.

Marcello, have a go at writing a post for this site, I'd be really interested to read some of your extended thoughts on this sort of thing.

in a Big World, I don't have to worry as much about creating diversity or giving possibilities a chance to exist, relative to how much I worry about average quality of life for sentients.

Can't say fairer than that.

Eliezer, given the proportion of your selves that get run over every day, have you stopped crossing the road? Leaving the house?

Or do you just make sure that you improve the standard of living for everyone in your Hubble Sphere by a certain number of utilons and call it a good day on average?

design cycles have stayed about the same length while chips have gotten hundreds of times more complex, and also much faster, both of which soak up computing power.

So...if you use chip x to simulate its successor chip y, and chip y to simulate its successor, chip z, the complexity and speed progressions both scale at exactly the right ratio to keep simulation times roughly constant? Interesting stuff.

Sounds as though the introduction of black-box 2015 chips would lead to a small bump and level off quite quickly, short of a few huge insights, which Jed seems to suggest are quite rare. Eliezer, is this another veiled suggestion that hardware is not what we need to be working on if we're looking to FOOM?

Changes to software that involve revising pervasive assumptions have always been difficult, of course.

Welcome to Overcoming Bias.

Also, while economists have many abstractions for modeling details of labor teams and labor markets, our standard is that the simplest versions should be of just a single aggregate quantity of labor.

Granted, but as long as we can assume that things like numbers of workers, hours worked and level of training won't drop through the floor, then brain emulation or uploading should naturally lead to productivity going through the roof shouldn't it?

Or is that just a wild abstraction with no corroborating features whatsoever?

because our computing hardware has run so far ahead of AI theory, we have incredibly fast computers we don't know how to use for thinking; getting AI right could produce a huge, discontinuous jolt, as the speed of high-grade thought on this planet suddenly dropped into computer time.

Now there's a scary thought.

Right, that's it, I'm gonna start cooking up some nitroglycerin and book my Eurostar ticket tonight. Who's with me?

I dread to think of the proportion of my selves that have already suffered horrible gravitational death.