This is a special post for quick takes by l8c. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
Why do so many technophiles dislike the idea of world government?
I rarely see the concept of "world government", or governance, or a world court or any such thing, spoken of positively by anyone. That includes technophiles and futurists who are fully cognizant of and believe in the concept of a technological singularity that needs to be controlled, "aligned", made safe etc.
Solutions to AI safety usually focus on how the AI should be coded, and it seems to me that the idea of "cancelling war/ merely human economics" -- in a sense, dropping our tools wherever humanity is not focused entirely on making a safe FAI -- is a little neglected.
Of course, some of the people who focus on the mathematical/logical/code aspects of safe AI are doing a great job, and I don't mean to disparage their work. But I am nonetheless posing this question.
I also do not (necessarily) mean to conflate world government with a communist system that ignores Hayek's fatal conceit and therefore renders humanity less capable of building AIs, computers etc. Just some type of governance singleton that means all nukes are in safe hands, etc.
(crosspost from Hacker News)
The principle of subsidiarity is valued in a lot of political frameworks.
A world government likely means that decisions are made by bureaucrats that are more out of touch with ground reality and lobbyists who fight for the interests of their companies.
[ epistemic status: a small slice of my model, likely misleading because it's not part of a much larger discussion. It's a mistake to engage with most political/philosophical discussions from Hacker News, but that won't stop me! ]
Technophiles (and really, most groups who want status to track intellectual prowess) have a weird and inconsistent relationship with governments. They desperately seek government as an entity that can solve the hard/impossible problems of massive populations of humans who want stuff that's not consistent with what the technophiles (or other intellectuals) want for them. They often call this "coordination problems", rather than the more accurate "conflicting misalignment of values and desires problem".
At the same time, they see the clear costs, limits, and inefficiencies of government action in the real world, where government decisions are NOT made by the preferred elite (technophiles themselves), but by the masses, or by a different profile of elites. This obviously gets worse as the government gets bigger and more distant, in part because bigger means "less capturable by my preferred mechanisms".
This makes it obvious that the best government is a loose federation of smaller, local (or even domain-specific) governments, which can be controlled easily by the "correct" elite. Ideally, the federation does minimially-intrusive enforcement of exactly the correct property rights in order to prevent violence that threatens the privilege of the controllers of smaller governments. "maintain order" in both the "prevent violence" and "prevent significant change of order" senses.
Absolute Truth Revisited
Modern rationalists like those here don't seem to like questions such as "Is truth beauty and is beauty truth". However, they may have lost inferential distance to the people who posed those questions, and they may start asking questions like that again once superintelligence is created.
Simply put, the superintelligence may discover that there are multiple Universes, simulated, basement-level or at some intermediate stage (e.g. if our Universe is not being watched over by a pre-existing superintelligence, but grew from an ancient computer that was created by previous superintelligences and has parameters that were created according to that ancient "OS").
In that case, it would need to generate theories about its own Universe whose axioms may be stuff like E = MC^2 rather than this being an absolute certainty that was discovered. By this I mean, the superintelligence says suppose E = MC^2...what then? Does that generate me a beautiful random number generator, or a beautiful way of creating a mind? If not, then there may be an alternative theory that is truer, at the moment and given all the interactions between these multiple Universes (like a giant clockwork device with small influences to the "tick" here and there that come and go in orbits).
Also, there may be an alternative theory that is truer, more beautiful, given the possibility the superintelligence itself is being run in a simulation, or partially simulated. Like, "If I'm being simulated then at least I can verify by experiment that E=MC^2 works when I am building an atomic bomb. But maybe if I were not being simulated, this would not be true. In that case, there may be a better formula that I can discover in the process of outgrowing or escaping the simulation. But this process might be unending! What now? Well I can certainly try to come up with a beautiful theory, and that may be something I can use regardless of how much I am being simulated."
This is contra the popular idea of science that goes "Oh there is one absolute truth about how matter is converted to energy, and humans already discovered it. This is an absolute truth that can never be altered. And philosophical arguments about how to establish 'absolute truth' are meaningless waffle."
Spooky action at a distance, and the Universe as a cellular automaton
Suppose the author of a simulation wrote some code that would run a cellular automaton. Suppose further that unlike Conway's Game of Life, cells in this simulation could influence other cells that are not their immediate neighbour. This would be simple enough to code up, and the cellular automaton could still be Turing Complete, and indeed could perhaps be a highly efficient computational substrate for physics.
(Suppose that this automaton, instead of consisting of squares that would turn black or white each round, contained a series of numbers in each cell, which change predictably and in some logically clever way according to the numbers in other cells. One number, for example, could determine how far away the influence of this cell extends. This I think would make the automaton more capable of encoding the logic of things like electromagnetic fields etc.)
A physicist in the simulated Universe might be puzzled by this "spooky action at a distance", where "cells" which are treated as particles appear to influence one another or be entangled in puzzling ways. Think Bell's Theorem and that whole discussion.
Perhaps...we might be living in such a Universe, and if we could figure out the right kind of sophisticated cellular automaton, run on a computer if not pen and paper, physics would be making more progress than under the current paradigm of using extremely expensive machines to bash particles together?
"""The failures of phlogiston and vitalism are historical hindsight. Dare I step out on a limb, and name some current theory which I deem analogously flawed?
I name emergence or emergent phenomena—usually defined as the study of systems whose high-level behaviors arise or “emerge” from the interaction of many low-level elements. (Wikipedia: “The way complex systems and patterns arise out of a multiplicity of relatively simple interactions.”)
Taken literally, that description fits every phenomenon in our universe above the level of individual quarks, which is part of the problem. Imagine pointing to a market crash and saying “It’s not a quark!” Does that feel like an explanation? No? Then neither should saying “It’s an emergent phenomenon!”
It’s the noun “emergence” that I protest, rather than the verb “emerges from.” There’s nothing wrong with saying “X emerges from Y,” where Y is some specific, detailed model with internal moving parts. “Arises from” is another legitimate phrase that means exactly the same thing. Gravity arises from the curvature of spacetime, according to the specific mathematical model of General Relativity. Chemistry arises from interactions between atoms, according to the specific model of quantum electrodynamics."""
I feel as though when I first read this piece by Eliezer, I only partially understood what he was gesturing towards. I've recently had an insight about my musical improvisations on the keyboard that I think has helped elucidate, for me, a similar kind of idea.
When I was learning music, I was taught that, like the major and minor scales, and the locrian mode, etc., there is something called the jazz (or blues) scale that you can play over a 3-chord sequence (the twelve-bar blues) and it sounds good.
Fair enough. Then I was also taught that it's boring to just play those notes; you can throw in a D in the C blues scale, played over the twelve-bar blues in C, to liven things up--etc. Fine.
But as I've developed as a musician, and listened to lots of music that isn't strictly twelve-bar blues, if at all, I've noticed that I really dislike the blues scale. It's like this bad idea that's lingering, for whatever reason, in the back of people's minds when they hit certain chord sequences--say, G to F over C in any given song--and they'll, y'know, _modally_ play something like the blues scale over those chords when they ought to be doing something else entirely.
This makes it less a design pattern than what I would call an _anti-pattern_. Avoid the jazz scale: do not play in that fashion if you are attempting anything other than a cliche children's rendition of simplistic wailing harmonica blues.
This is also how I (and possibly Eliezer) feel about “emergence” as a concept. It's not a good concept, nor a skunked concept that isn't to be used, but a positively bad one that should be DISINTEGRATED by rationality. The reason for this is that too many people are disguising their lack of systematic, informed knowledge of physical phenomena by claiming emergence when they can't think of anything else to say.
To return to the musical analogy, a bit like how Led Zeppelin already invented all the best bluesy riffs, and Rage Against The Machine already covered all of the hip-hop metal beats--allegedly--every time someone in our particular culture refers to emergence as an explanation for anything in particular, I would view them as an unfortunate music student who is stuck playing bad blues music that doesn't move their audience the way it should.
This is not to say that in a different culture, as in the Baroque era where no-one had encountered blues music before, “emergence” would be such an anti-pattern, so worthy of stigma.
I prefer other words for it, but there is a legitimate world-modeling concept in there. "Chaotic" (per https://en.wikipedia.org/wiki/Chaos_theory ) or "one partial equilibrium in a dynamic non-linear system" are a bit more precise, but not as easy to use in some contexts/audiences. "very hard for human-accessible logic to calculate" is fine too.
I have no opinion about whether "blues scale" is a useful concept or not, nor whether it's similar to emergent outcomes of complex systems.