Eliezer: part of the AIXI sequence, which I don't think I'll end up writing.

Ahh, that's a shame, though fully understood. Don't suppose you (or anyone) can link to some literature about AIXI? Haven't been able to find anything comprehensive yet comprehensible to an amateur.

Tim Tyler: Brainpower went into making new brains historically - via sexual selection. Feedback from the previous generation of brains into the next generation has taken place historically.

Tim, Dawkins has a nice sequence in The Blind Watchmaker about a species of bird in which the female began selecting for big, lustrous tails. This led to birds with tails so big they could barely fly to escape predators. While selecting for intelligence in a partner is obviously plausible, I'd have to see very compelling evidence that it's leading to continuously smarter people, or even that it ever could. Possibly a loop of some description there, but K definitely < 1.

However!

Owing to our tremendous lack of insight into how genes affect brains

So what happens when we start to figure out what those genes do? and then start switching them on and off, and gaining more knowledge and insight into how the brain attacks problems? As we've read recently, natural selection increased brainpower (insight) massively through blind stumbling onto low-hanging fruit in a relatively small amount of time. Why would we suppose it reached any sort of limit - or at least a limit we couldn't surmount? The 18 years to maturity thing is pretty irrelevant here, as long as, say, 5% compound insight can be gained per generation. You're still looking at exponential increases, and you might only need a handful of generations before the FOOM itself switches medium.

Andrew, we're not talking about the equivalent of a human studying neuroscience by groping in the dark. If an AI truly, truly groks every line of its own code, it can pretty much do what it wants with it. No need for trial and error when you have full visibility and understanding of every neuron in your head.

How, you ask? What do such recursive algorithms look like? Mere details; the code monkeys can worry about all that stuff!

Know the feeling. I'm a fully qualified ex-Catholic atheist, but when my girlfriend told me that her family generally just has a pasta dish on Christmas day I was shocked. Anything but turkey makes baby Jesus cry!

Those childhood priors sure get burnt in deeply.

Silas - yeah, that's about the size of it.

Eliezer, when you come to edit this for popular publication, lose the maths, or at least put it in an endnote. You're good enough at explaining concepts that if someone's going to get it, they're going to get it without the equations. However, a number of those people will switch off there and then. I skipped it and did fine, but algebra is a stop sign for a number of very intelligent people I know.

So...noise can be useful in decision theory as long as you don't expect it to do any work. And the mistake gets easier to make the more complex your system. Sounds right enough to me.

[nitpick]

Your 'by definition' link needs a look, Eliezer.

Or imagine that the combination changes every second. In this case, 0-0-0-0, 0-0-0-0 is just as good as the randomized algorithm - no better and no worse.

If it changes every second, trying the same set of four over and over is marginally better than random.

If you've just entered 0-0-0-0 and got it wrong, then on the next try every sequence except 0-0-0-0 has a small chance of being the correct sequence from the previous, and hence is incorrect this try.

Anyone care to work out exactly how much better off 0-0-0-0 is than a random set in this case?

[/nitpick]

Here is the ultimate work of Modern Art, that truly defies all rules: It isn't mine, it isn't real, and no one knows it exists...

It's...it's beautiful.

Great post for the most part, though I do have to agree with Tim's straw man alert.

Something I learnt while studying postmodern fiction (yeah Eliezer, that's right): Art can be referential, or memetic, or both, or neither. Most is both, in that it (very roughly) is 'like' reality (i.e. it's memetic) and 'seeks to tell us something about' reality (i.e. it's referential). However, there's some really interesting stuff that is neither - defying ideas like logic, causation and induction (let alone plot, character etc) and blatantly having no regard for what Eliezer would call terminal values. (Except, in some cases, at a meta-level outside the text. But not in all cases.) Read up on Alain Robbe-Grillet's fiction and Sam Beckett's 'Trilogy' (and later poetry) for a start. Oh, and John Cage - yes, even 4'33.

Randomness, noise and so on can be astonishingly beautiful, in art or in nature, even to the novice. Or do you think that there are two parts of your brain, one which finds a painting beautiful, and one that finds the night sky beautiful? Yes, there's some high-minded bullshit out there, but as Tim says, please don't draw false boundaries simply to justify your profound bottom line.

And beware of putting a nuts-and-bolts heuristic in place of a sense of aesthetic beauty. You may then find yourself conflicted between finding something beautiful and being unable to understand why. And that truly would be a tragedy.

/asks self if he should beat his wife

/realises self is not married

I think we should rename it "Robin and Eliezers' Varied Thoughts".

John, Alex; Meh. Long as it's interesting who cares? Nobody promised anyone posts on any particular topic, and nodoby's forcing your mouse clicks. If it makes you feel better, rename your bookmark 'Metaphysical Singularity Sci-Fi'.

the strong whiff of sci-fi geekdom that pervades most of Eliezer's posts.

You say it like it's a bad thing.

What would effective cryo policy look like? Or conversely, what in current policy is inhibiting the proper development of cryogenics?

Ruling parties come and go in waves. Work out when you reckon you'll be unfrozen and vote with that year's election in mind.

Question: if you're on your deathbed and about to have your head frozen, should you be allowed to pre-register your votes for the next few elections? "Palin's counting on a low turnout amongst the dead for 2016, as they tend to vote primarily for the Democratic candidate."

Oh, by the way, well done America.

"If your dreams and aspirations interfere with your rationality and impartiality then you're no longer following the Way. The Way doesn't drive one off course; one rather loses sight of the Way and hence goes off course."

[The Book of Eliezer 4:24 18-20]