Had it turned out that the brain was big because blind-idiot-god left gains on the table, I'd have considered it evidence of more gains lying on other tables and updated towards faster takeoff.
I agree the blackbody formula doesn't seem that relevant, but it's also not clear what relevance Jacob is claiming it has. He does discuss that the brain is actively cooled. So let's look at the conclusion of the section:
Conclusion: The brain is perhaps 1 to 2 OOM larger than the physical limits for a computer of equivalent power, but is constrained to its somewhat larger than minimal size due in part to thermodynamic cooling considerations.
If the temperature-gradient-scaling works and scaling down is free, this is definitely wrong. But you explicitly flag your low confidence in that scaling, and I'm pretty sure it wouldn't work.* In which case, if the brain were smaller,... (read more)
The capabilities of ancestral humans increased smoothly as their brains increased in scale and/or algorithmic efficiency. Until culture allowed for the brain’s within-lifetime learning to accumulate information across generations, this steady improvement in brain capabilities didn’t matter much. Once culture allowed such accumulation, the brain’s vastly superior within-lifetime learning capacity allowed cultural accumulation of information to vastly exceed the rate at which evolution had been accumulating information. This caused the human sharp left turn.
This is basically true if you're talking about the agricultural or industrial revolutions, but I don't think anybody claims evolution improved human brains that fast. But homo sapiens have only been around 300,000 years, which is still quite short... (read more)
Upvoted mainly for the 'width of mindspace' section. The general shard theory worldview makes a lot more sense to me after reading that.
Consider a standalone post on that topic if there isn't one already.
I feel that there's something true and very important here, and (as the post acknowledges) it is described very imperfectly.
One analogy came to mind for me that seems so obvious that I wonder if you omitted it deliberately: a snare trap. These very literally work by removing any slack the victim manages to create.
There's definitely something here.
I think it's a mistake to conflate rank with size. The point of the whole spherical-terrarium thing is that something like 'the presidency' is still just a human-sized nook. What makes it special is the nature of its connections to other nooks.
Size is something else. Big things like 'the global economy' do exist, but you can't really inhabit them—at best, you can inhabit a human-sized nook with unusually high leverage over them.
That said, there's a sense in which you can inhabit something like 'competitive Tae Kwon Do' or 'effective altruism' despite not directly experiencing most of the specific people/places/things involved. I guess it's a mix of meeting random-ish samples... (read more)
Maybe it's an apple of discord thing? You claim to devote resources to a good cause, and all the other causes take it as an insult?
If you really want to create widespread awareness of the broad definition, the thing to do would be to use the term in all the ways you currently wouldn't.
E.g. "The murderer realized his phone's GPS history posed a significant infohazard, as it could be used to connect him to the crime."
If Bostrom's paper is our Schelling point, 'infohazard' encompasses much more than just the collectively-destructive smallpox-y sense.
Here's the definition from the paper.
Information hazard: A risk that arises from the dissemination or the potential dissemination of (true) information that may cause harm or enable some agent to cause harm.
'Harm' here does not mean 'net harm'. There's a whole section on 'Adversarial Risks', cases where information can harm one party by benefitting another party:
In competitive situations, one person’s information can cause harm to another even if no intention to cause harm is present. Example: The rival job applicant knew more and got the job.
ETA: localdeity's comment below points out that it's a pretty bad idea to have a term that colloquially means 'information we should all want suppressed' but technically also means 'information I want suppressed'. This isn't just pointless pedantry.
My gut instinct on metacosmology is that if were a simpler computation than our universe that produced intelligent life, we'd probably be there instead of here. I'm not sure that's valid anthropics, but it still surprises me to see apparent assumption (in posts like this of the opposite conclusion—that there are many universes simpler than ours capable of producing intelligence. (EDIT: that post doesn't actually make that assumption, and I don't have another example ready. Still turned out to be a fruitful question, though)
(Yes, we know cellular automata can implement intelligence, but AFAIK we don't know that they can do it more simply than by implementing a Turing machine stimulating our universe).
Is there an argument I've missed?
The argument here seems to be "humans have not yet discovered true first-principles justifications of the practical models, therefore a superintelligence won't be able to either".
I agree that not being able to experiment makes things much harder, such that an AI only slightly smarter than humans won't one-shot engineer things humans can't iteratively engineer. And I agree that we can't be certain it is possible to one-shot engineer nanobots with remotely feasible compute resources. But I don't see how we can be sure what isn't possible for a superintelligence.