A world without complex novelty would be lacking. But so would a world without some simple pleasures. There are people who really do enjoy woodworking. I can't picture a utopia where no one ever whittles. And a few of them will fancy it enough to get really, really good at it, for pretty much the same reason that there are a handful of devoted enthusiasts. Even without Olympic competitions and marathons, I'd bet there would still be plenty of runners, who did so purely for it's own sake, rather than to get better or to compete, or for novelty. Given an infinite amount of time, everyone is likely to spend a great deal of time on such non-novel things. So, what's most disturbing about carving 162,329 table legs is that he altered his utility function to want to do it.

(As best I can grasp the Law, there are insights you can't understand at all without having a brain of sufficient size and sufficient design. Humans are not maximal in this sense, and I don't think there should be any maximum—but that's a rather deep topic, which I shall not explore further in this blog post. Note that Greg Egan seems to explicitly believe the reverse—that humans can understand anything understandable—which explains a lot.)

Perhaps I'm missing something, but it seems to me that any mind capable of designing a turning-complete computer can, in principle, understand any class of problem. I say "class of problem", because I doubt we can even wrap our brains around a 10x10x10x10 Rubik's Cube. But we are aware of simpler puzzles of that class. (And honestly, I'm just using an operational definition of "classes of problem", and haven't fleshed out the notion.) There will always be harder logic puzzles, riddles, and games. But I'm not sure there exist entirely new classes of problems, waiting to be discovered. So we may well start running out of novelty of that type after a couple million years, or even just a couple thousand years.

This seems like an obvious choice for a death-related poem...

Do not go gentle into that good night
Dylan Thomas, 1914 - 1953

Do not go gentle into that good night,
Old age should burn and rave at close of day;
Rage, rage against the dying of the light.

Though wise men at their end know dark is right,
Because their words had forked no lightning they
Do not go gentle into that good night.

Good men, the last wave by, crying how bright
Their frail deeds might have danced in a green bay,
Rage, rage against the dying of the light.

Wild men who caught and sang the sun in flight,
And learn, too late, they grieved it on its way,
Do not go gentle into that good night.

Grave men, near death, who see with blinding sight
Blind eyes could blaze like meteors and be gay,
Rage, rage against the dying of the light.

And you, my father, there on the sad height,
Curse, bless, me now with your fierce tears, I pray.
Do not go gentle into that good night.
Rage, rage against the dying of the light.

It's still under copyright, but I don't think that really matters...

I'm a fan of both versions of Ozymandias, but here's Shelley's version:

I met a traveller from an antique land
Who said: "Two vast and trunkless legs of stone
Stand in the desert. Near them, on the sand,
Half sunk, a shattered visage lies, whose frown,
And wrinkled lip, and sneer of cold command,
Tell that its sculptor well those passions read
Which yet survive, stamped on these lifeless things,
The hand that mocked them and the heart that fed:
And on the pedestal these words appear:
'My name is Ozymandias, king of kings:
Look on my works, ye Mighty, and despair!'
Nothing beside remains. Round the decay
Of that colossal wreck, boundless and bare
The lone and level sands stretch far away."

The world is indeed often surprising but frequently surprising in a way that scientific experiments open new questions that weren't in the mind of the scientists beforehand.

When looking at something like Reiki, Bill Nye and friends argue don't consider it an open question whether or not Reiki works just because we don't have well controlled studies investigating it. They consider it not to work because they don't believe that there's ki.

You might argue that they are wrong to do so, but that's still how they operate.

I've seen people like Bill Nye repeat this, and seen plenty of science-themed reminders that test results are often surprising

The phrase science-themed sounds to me more like it refers to science mythology than serious history of science. To the extend that you want to sensible talk about what scientists do, you have to listen to people who study what scientists do and that's not the speciality of a rocket scientist like Wernher Von Braun.

Yeah, but if you steel-man it,

The act of steelmanning means to argue against a different position then the one the person is holding. It very worthwhile to criticize people for holding positions for the wrong reasons.

I think he was trying to make something similar to a map-territory distinction. It's often useful to make a distinction between the data and our best interpretation of the data. Some conclusions don't require much extrapolation, but others require a great deal.

To me what you are saying doesn't seem like a description of the map-territory distinction. A map is not an extrapolation of the territory but an abstraction of it.

If things are "open questions" until they are above a confidence interval of, say, 0.99

That sentence doesn't look to me like it's inspired by looking at what scientists do. I'm not aware of a scientific community having the standard of question being closed when they are over a confidence interval of 0.99.

You might argue that scientists should do things that way, but that doesn't have much to do with the question of how scientists act in the real world.

It's often useful to make a distinction between the data and our best interpretation of the data.

Statements about what's useful are different than statements that describe what scientists do in reality.

Using a historical example which happens to be false just complicates things. If I recall, philosophers first hypothesized a round earth around 600 BCE, but didn't prove it experimentally until 300 BCE.

I think you missed a point. At 300 BCE they were not centrally concerned with proving via experiment that the earths is round. They instead cared about things making sense intuitively. The idea that it's important to prove claims via experiment came with Descartes into the scientific mosaic which happened much later.

Nobody at the time between 600 BCE and 300 BCE said: “The part of the Earth that I can see at any given time appears to be flat, which would be the case when looking at a small piece of many differently shaped objects up close, so I don’t have enough information to know what the shape of the Earth is. One reasonable hypothesis is that the Earth is flat, but until we have tools and techniques that can be used to prove or disprove that hypothesis, it is an open question.”

If things are "open questions" until they are above a confidence interval of, say, 0.99, then just about everything we discuss here is an open question, as the quote suggests.

"How does action at a distance work?" wasn't an open question shortly after Descartes. It became again an open question when Newton was shown to be right by the expedition that measured the shape of the earth.

In biology the central dogma of molecular biology was considered a close question for a long time. Biologists where confident about the fact that a lot of the DNA is junk DNA that doesn't do anything.

One of the main reasons why we don't consider the question of whether homeopathy works an open question isn't just that we lack empiric evidence for it working but that we based on our theories of chemistry we don't believe that it could work.

Whether or not chiropratics interventions work was a question that scientists considered not to be open for a long time.

A scientist doesn’t say, “Well we know the Earth is flat because that’s the way it looks, that’s what’s intuitive, and that’s what everyone agrees is true,” a scientist says, “The part of the Earth that I can see at any given time appears to be flat, which would be the case when looking at a small piece of many differently shaped objects up close, so I don’t have enough information to know what the shape of the Earth is. One reasonable hypothesis is that the Earth is flat, but until we have tools and techniques that can be used to prove or disprove that hypothesis, it is an open question.”

That seems to be a flawed conception of science that doesn't have much to do with the reality of how scientists operate.

The author does not seem to understanding survivorship bias. He never approaches the question of whether the things he proposes are the reason for Musk's success actually work, or whether they happen to work for Musk in a context-dependent way. In other words, if you give this as advice to someone random, will they end up successful or an outcast. I'd guess the latter in most cases. This is in general the problem of evaluating the reasons behind success.

Also, unnecessary evolutionary psychology, done badly, even to the point of suggesting group selection. Ick.

The idea that using technical language (which isn't actually any more precise in meaning in the examples cited) in regular life is beneficial in being more scientific is also pretty suspect.

Rather, the idea is that by appealing to both the "head" and the "heart" we can convey a fuller message about EA, and that this will amplify our reach among people who otherwise might not know about it or take it seriously.

Coddletrop. This post is talking about dark arts, about bypassing the head entirely. "Superdonor" indeed.

Most people aren't trying and failing to achieve their dreams. We aren’t even trying. We don’t have well-articulated dreams

Evidence or typical mind fallacy..? X-)

Once you start trying to appeal to "emotionally-oriented" thinkers (what a euphemism), you start bringing them into your group. Once you bring them into your group, they start participating in creating your group policy. Once they start participating in creating your group policy - you stop being effective, because they don't care about effective, and they outnumber you.

Don't court the Iron Law of Oligarchy so directly. Keep your focus on your organization's purpose, rather than your organization. It will last slightly longer that way.