“…Solomonoff’s malignness…”

I was friends with Ray Solomonoff; he was a lovely guy and definitely not malign.

Epistemic status: true but not useful.

I was all set to disagree with this when I reread it more carefully and noticed it said “superhuman reasoning” and not “superintelligence”.  Your definition of “reasoning” can make this obviously true or probably false.  

The Antarctic Treaty (and subsequent treaties) forbid colonization.  They also forbid extraction of useful resources from Antarctica, thereby eliminating one of the main motivations for colonization.  They further forbid any profitable capitalist activity on the continent.  So you can’t even do activities that would tend toward permanent settlement, like surveying to find mining opportunities, or opening a tourist hotel.  Basically, the treaty system is set up so that not only can’t you colonize, but you can’t even get close to colonizing.

Northern Greenland is inhabited, and it’s at a similar latitude.

(Begin semi-joke paragraph) I think the US should pull out of the treaty, and then announce that Antarctica is now part of the US, all countries are welcome to continue their purely scientific activity provided they get a visa, and announce the continent is now open to productive activity.  What’s the point of having the world’s most powerful navy if you can’t do a fait accompli once in a while?  Trump would love it, since it’s simultaneously unprecedented, arrogant and profitable.  Biggest real estate development deal ever!  It’s huuuge!

A fascinating recent paper on the topic of human bandwidth  is https://arxiv.org/abs/2408.10234.  Title and abstract:

The Unbearable Slowness of Being

Jieyu Zheng, Markus Meister

This article is about the neural conundrum behind the slowness of human behavior. The information throughput of a human being is about 10 bits/s. In comparison, our sensory systems gather data at an enormous rate, no less than 1 gigabits/s. The stark contrast between these numbers remains unexplained. Resolving this paradox should teach us something fundamental about brain function: What neural substrate sets this low speed limit on the pace of our existence? Why does the brain need billions of neurons to deal with 10 bits/s? Why can we only think about one thing at a time? We consider plausible explanations for the conundrum and propose new research directions to address the paradox between fast neurons and slow behavior.

They’re measuring a noisy phenomenon, yes, but that’s only half the problem.  The other half of the problem is that society demands answers.  New psychology results are a matter of considerable public interest and you can become rich and famous from them.  In the gap between the difficulty of supply and the massive demand grows a culture of fakery.  The same is true of nutrition— everyone wants to know what the healthy thing to eat is, and the fact that our current methods are incapable of discerning this is no obstacle to people who claim to know.

For a counterexample, look at the field of planetary science.  Scanty evidence dribbles in from occasional spacecraft missions and telescopic observations, but the field is intellectually sound because public attention doesn’t rest on the outcome.

Doesn’t matter, because HPMOR is engaging enough on a chapter-by-chapter basis.  I read lots of books when I was a kid when I didn’t understand the overarching plot.  As long as I had a reasonable expectation that cool stuff would happen in the next chapter, I’d keep reading.  I read “Stand On Zanzibar” repeatedly as a child, and didn’t understand the plot until I reread it as an adult last year.  Same with the detective novel “A Deadly Shade of Gold”.  I read it for the fistfights, snappy dialogue, and insights into adult life.  The plot was lost on me.

In general the human body is only capable of healing injuries that are the kind of thing that, if they were smaller, would still leave the victim alive, in the Stone Age.  If an injury is of a type that would be immediately fatal in the Stone Age, there’s no evolutionary pressure to make it survivable.  For example, we can regrow peripheral nerves, because losing a peripheral nerve means a numb patch and a weak limb, but you could live with this for a few months even if you’re a caveman.  On the other hand, we can’t regrow spinal cord, because a transected spinal cord  is fatal within a day or two even given the finest Stone Age nursing care (it didn’t become survivable until about 1946.). On the third hand, we can heal brain from strokes, even though brain is more complex than spinal cord, because a small stroke is perfectly survivable as long as you have someone to feed you until you get better.  We can survive huge surgical incisions, even though those would be fatal in the Stone Age, because small penetrating wounds were survivable, and the healing mechanisms can just do the same thing all along the incision.  This is why we sew wounds up: to convince the healing mechanisms that it’s only a small cut.

Unfortunately this argument suggests regrowing limbs is impossible.  An amputation is bad but survivable, and after it heals, you can still get around.  But many years of spending a lot of bodily energy on regrowing a limb that is pretty useless for most of that time doesn’t seem worthwhile.

Some particular problems I see:

In humans, there’s no mechanism for a growing limb to connect correctly to an adult injury site.  For example, there’s already a bunch of scar tissue there, which has to be cleared away progressively as the limb grows.  Evolution has not seen fit to provide us with this complex biochemistry, unlike the case of salamanders.

Children have a high level of circulating growth hormone, which tells the arm cells how fast to grow.  If you tried to provide this to an adult, their other bones would also grow, causing deformity (acromegaly).

It’s odd that we can’t grow new teeth when the old ones fall out.  More than once, I mean.  Drilling for cavities makes sense because the enamel (outer tooth layer) is essentially dead, and doesn’t regrow.  But we should be able to grow a whole new tooth from the root when we get a cavity.

To hold the surface out, you need to have a magnetic field tangent to the surface.  But you can’t have a continuous magnetic field tangent to every point on the surface of a sphere.  That’s a theorem of topology, called the Hairy Ball Theorem.  So there has to be some area of the ball that’s unsupported.  I guess if the area is small enough, you just let it dimple inwards in tension.  The balloon would be covered in dimples, like a golf ball.

Thanks for clearing that up.  It sounds like we’re thinking along very similar lines, but that I came to a decision to stop earlier.  From a position inside one of major AI labs, you’ll be positioned to more correctly perceive when the risks start outweighing the benefits.  I was perceiving events more remotely from over here in Boston, and from inside a company that uses AI as a one of a number of tools, not as the main product.

I’ve been aware of the danger of superintelligence since the turn of the century, and I did my “just now orienting to the question” back in the early 2000s.  I decided that it was way too early to stop working on AI back then, and I should just  “monitor for new considerations or evidence or events.”  Then in 2022, Sydney/Bing came along, and it was of near-human intelligence, and aggressively misaligned, despite the best efforts of its creators.  I decided that was close enough to dangerous AI that it was time to stop working on such things.  In retrospect I could have kept working safely in AI for another couple of years, i.e. until today.  But I decided to pursue the “death with dignity” strategy: if it all goes wrong, at least you can’t blame me.  Fortunately my employers were agreeable to have me pivot away from AI; there’s plenty of other work to be done.