I've always been surprised that there's not more discussion of Dune in rationalist circles, especially considering that:

1. It's a book all about people improving their minds to the point where they become superhuman.

2. It's set in a world where AI Goal Alignment issues are not only widely understood, but are integrated into the foundation of every society.

3. It's ecological science fiction — dedicated to "the dry-land ecologists, wherever they may be" — but what that secretly means is that it's a series of novels about existential risk, and considers the problem on a timescale of tens of thousands of years.

For those of you who are not familiar, Dune is set about 20,000 years in the future. About 10,000 years before the events of the first book, Strong Artificial Intelligence was developed. As one might expect, humanity nearly went extinct. But we pulled together and waged a 100-year war against the machines, a struggle known as the Butlerian Jihad (This is why a butler is “one who destroys intelligent machines”). We succeeded, but only barely, and the memory of the struggle was embedded deep within the human psyche. Every religion and every culture set up prohibitions against "thinking machines". This was so successful that the next ten millennia saw absolutely no advances in computing, and despite the huge potential benefits of defection, coordination was strong enough to prevent any resurgence of computing technology.

Surprisingly, the prohibition against "thinking machines" appears to extend not only to what we would consider to be Strong AI, but also to computers of all sorts. There is evidence that devices for recording journals (via voice recording?) and doing basic arithmetic were outlawed as well. The suggestion is that there is not a single mechanical calculator or electronic memory-storage device in the entire Imperium. There are advanced technologies, but nothing remotely like computers — the Orange Catholic Bible is printed on "filament paper", not stored on a Kindle.

While I appreciate the existential threat posed by Strong AI, I've always been confused about the proscription against more basic forms of automation. The TI-81 is pretty helpful and not at all threatening. Storing records on paper or filament paper has serious downsides. Why does this society hamstring themselves in this way?

The characters have a good deal to say about the Butlerian Jihad, but to me, their answers were always somewhat confusing:

Once men turned their thinking over to machines in the hope that this would set them free. But that only permitted other men with machines to enslave them. (Reverend Mother Gaius Helen Mohiam)

And:

What do such machines really do? They increase the number of things we can do without thinking. Things we do without thinking — there's the real danger. (Leto Atreides II)

This doesn't suggest that literal extinction threat is the only reason for the Jihad. In fact, according to these major characters, it wasn't even the primary reason.

This is not to say that extinction risk isn't on their mind. Here's another idea discussed in the books, and condemned for its obvious x-risk issues:

The Ixians contemplated making a weapon—a type of hunter-seeker, self-propelled death with a machine mind. It was to be designed as a self-improving thing which would seek out life and reduce that life to its inorganic matter. (Leto Atreides II)

Or, more explicitly:

Without me there would have been by now no people anywhere, none whatsoever. And the path to that extinction was more hideous than your wildest imaginings. (Leto Atreides II)

But clearly extinction risk isn't the only thing driving the proscription against thinking machines. If it were, then we'd still have our pocket calculators and still be able to index our libraries using electronic databases. But this society has outlawed even these relatively simple machines. Why?

Goodhart's law states that when a measure becomes a target, it ceases to be a good measure.

What this means is that the act of defining a standard almost always destroys the goal you were trying to define. If you make a test for aptitude, teachers and parents will teach to the test. The parents with the most resources — richest/most intelligent/most well-connected — will find ways to get their children ahead at the expense of everyone else. If you require a specific degree to get a particular job, degree-granting institutions will compete to make the degree easier and easier to acquire, to the point where it no longer indicates quality. If supply is at all limited, then the job-seekers who are richest/most intelligent/most well-connected will be the ones who can get the degree. If you set a particular critical threshold for a statistical measure (*cough*), researchers will sacrifice whatever positive qualities about the rest of their research they can in pursuit of reaching that critical threshold.

Governments, if they endure, always tend increasingly toward aristocratic forms. No government in history has been known to evade this pattern. And as the aristocracy develops, government tends more and more to act exclusively in the interests of the ruling class — whether that class be hereditary royalty, oligarchs of financial empires, or entrenched bureaucracy. (Bene Gesserit Training Manual)

One of the most important things we know from AI Alignment work is that defining a rule or standard that can't be misinterpreted is very tricky. An intelligent agent will work very hard to maximize its own utility function, and will find clever ways around any rules you throw in its way.

One of the ways we have been short-sighted is in thinking that this applies only to strong or general artificial intelligences. Humans are strong general intelligences; if you put rules or standards in their way, they will work very hard to maximize their own utility functions and will find clever ways around the rules. Goodhart's law is the AI Alignment problem applied to other people.

("The real AI Alignment problem ... is other people?")

It's been proposed that this issue is the serpent gnawing at the root of our culture. The long and somewhat confusing version of the argument is here. I would strongly recomend that you read first (or instead) this summary by Nabil ad Dajjal. As Scott says, "if only there were something in between Nabil’s length and Concierge’s", but reading the two I think we can get a pretty good picture.

Here are the first points, from Nabil:

There is a four-step process which has infected and hollowed out the entirety of modern society. It affects everything from school and work to friendships and dating.

In step one, a bureaucrat or a computer needs to make a decision between two or more candidates. It needs a legible signal. Signaling (see Robin Hanson) means making a display of a desired characteristic which is expensive or otherwise difficult to fake without that characteristic; legibility (see James Scott) means that the display is measurable and doesn’t require local knowledge or context to interpret.

I will resist quoting it in full. Seriously, go read it, it's pretty short.

When I finished reading this explanation, I had a religious epiphany. This is what the Butlerian Jihad was about. While AI may literally present an extinction risk because of its potential desire to use the atoms in our bodies for its own purposes, lesser forms of AI — including something as simple as a device that can compare two numbers! — are dangerous because of their need for legible signals.

In fact, the simpler the agent is, the more dangerous it is, because simple systems need their signals to be extremely legible. Agents that make decisions based on legible signals are extra susceptible to Goodhart's law, and accelerate us on our way to the signaling catastophe/race to the bottom/end of all that is right and good/etc.

As Nabil ad Dajjal points out, this is true for bureaucrats as well as for machines. It doesn't require what we normally think of as a "computer". Anything that uses a legible signal to make a decision with no or little flexibility will contribute to this problem.

The target of the Jihad was a machine-attitude as much as the machines. (Leto Atreides II)

As a strong example, consider Scott Aaronson's review of Inadequate Equilibria, where he says:

In my own experience struggling against bureaucracies that made life hellish for no reason, I’d say that about 2/3 of the time my quest for answers really did terminate at an identifiable “empty skull”: i.e., a single individual who could unilaterally solve the problem at no cost to anyone, but chose not to.  It simply wasn’t the case, I don’t think, that I would’ve been equally obstinate in the bureaucrat’s place, or that any of my friends or colleagues would’ve been.  I simply had to accept that I was now face-to-face with an alien sub-intelligence—i.e., with a mind that fetishized rules made up by not-very-thoughtful humans over demonstrable realities of the external world.

Together, this suggests a surprising conclusion: Rationalists should be against automation. I suspect that, for many of us, this is an uncomfortable suggestion. Many rationalists are programmers or engineers. Those of us who are not are probably still hackers of one subject or another, and have as a result internalized the hacker ethic.

If you're a hacker, you strongly believe that no problem should ever have to be solved twice, and that boredom and drudgery are evil. These are strong points, perhaps the strongest points, in favor of automation. The world is full of fascinating problems waiting to be solved, and we shouldn't waste the talents of the most gifted among us solving the same problems over and over. If you automate it once, and do it right, you can free up talents to work on the next problem. Repeat this until you've hit all of humanity's problems, boom, utopia achieved.

The problem is that the talents of the most gifted are being double-wasted in our current system. First, intelligent people spend huge amounts of time and effort attempting to automate a system. Given that we aren't even close to being able to solve the AI Alignment problem, the attempt to properly automate the system always fails, and the designers instead automate the system so that it uses one or more legible signals to make its judgment. Now that this system is in place, it is immediately captured by Goodhart's law, and people begin inventing ways to get around it.

Second, the intelligent and gifted people — those people who are most qualified to make the judgment they are trying to automate — are spending their time trying to automate a system that they are (presumably) qualified to make judgments for! Couldn't we just cut out the middleman, and when making decisions about the most important issues that face our society, give intelligent and gifted people these jobs directly?

So we're 1) wasting intellectual capital, by 2) using it to make the problem it's trying to solve subject to Goodhart's law and therefore infinitely worse.

Give me the judgment of balanced minds in preference to laws every time. Codes and manuals create patterned behavior. All patterned behavior tends to go unquestioned, gathering destructive momentum. (Darwi Odrade)

Attempts to solve this problem with machine learning techniques are possibly worse. This is essentially just automating the task of finding a legible signal, with predictable results. It's hard to say if a neural network will tend to find a worse legible signal than the programmer would find on their own, but it's not a bet I would take. Further, it lets programmers automate more decisions, lets them do it faster, and prevents them from understanding the legible signal(s) they select. That doesn't inspire comfort.

Please also consider the not-literally-automation version, as described by Scott Alexander in his daycare worker example:

Daycare companies really want to avoid hiring formerly-imprisoned criminals to take care of the kids. If they can ask whether a certain employee is criminal, this solves their problem. If not, they’re left to guess. And if they’ve got two otherwise equally qualified employees, and one is black and the other’s white, and they know that 28% of black men have been in prison compared to 4% of white men, they’ll shrug and choose the white guy.

Things like race, gender, and class are all extremely legible signals. They're hard to fake, and they're easy to read. So if society seems more racist/sexist/classist/politically bigoted than it was, consider the idea that it may be the result of runaway automation. Or machine-attitude, as God Emperor Leto II would say.

I mentioned before that, unlike problems with Strong AI, this weak-intelligence-Goodhart-problem (SchwachintelligenzGoodhartproblem?) isn't an existential risk. The bureaucrats and the standardized tests aren't going to kill us in order to maximize the amount of hydrogen in the universe. Right?

But if we consider crunches to be a form of x-risk, then this may be an x-risk after all. This problem has already infected "everything from school and work to friendships and dating" making us "older, poorer, and more exhausted". Not satisfied with this, we're currently doing our best to make it more 'efficient' in the areas it already holds sway, and working hard to extend it to new areas. If taken to its logical conclusion, we may successfully automate nearly everything, and destroy our ability to make any progress at all.

I'll take this opportunity to reiterate what I mean by automation. I suspect that when I say "automate nearly everything", many of you imagine some sort of ascended economy, with robotic workers and corporate AI central planners. But part of the issue here is that Goodhart's law is very flexible, and kicks in with the introduction of most rules, even when the rules are very simple.

Literal machines make this easier — a program that says "only forward job applicants if they indicated they have a Master's Degree and 2+ years of experience" is simple, but potentially very dangerous. But on the other hand, a rule about what sorts of applicants will be considered can be identical when faithfully applied by a single-minded bureaucrat. The point is that the decision has been automated to a legible signal. Machines just make this easier, faster, and more difficult to ignore. All but the most extreme bureaucrats will occasionally break protocol. Automation by machine will never do so.

So we want two closely related things:

1. We want to avoid the possible x-risk from automation.

2. We want to reverse the whole "hollowed out the entirety of modern society" thing and make life feel meaningful again.

The good news is that there are some relatively easy solutions.

First, stop automating things or suggesting that things should be automated. Reverse automation wherever possible. (As suggested by Noitamotua.)

There may be some areas where automation is safe and beneficial. But before automating something, please spend some time thinking about whether or not the legible signal is too simple, whether the automation will be susceptible to Goodhart's law. Only in cases where the legible signal is effectively identical to the value you actually want, or where the cost of an error is low ("you must be this tall to ride" is a legible signal) will this be acceptable.

Second, there are personal and political systems which are designed to deal with this problem. Goodhart's law is powerless against a reasonable person. While you or I might take someone's education into consideration when deciding whether or not to offer them a job, we would weigh it in a complex, hard-to-define way against the other evidence available.

Let's continue the hiring decision metaphor. More important than ignoring credentials is the ability to participate in the intellectual arms race, exactly the problem fixed rules cannot follow (Goodhart's law!). If I am in charge of hiring programmers, I might want to give them a simple coding test as part of their interview. I might ask, "If I have two strings, how do I check if they are anagrams of each other?" If I use the same coding test every time (or I automate it, setting up a website version of the test to screen candidates before in-person interviews), then anyone who knows my pattern can figure out or look up the answer ahead of time, and the question no longer screens for programming ability — it screens for whatever "can figure out or look up the answer ahead of time" is.

But when you are a real human being, not a bureaucrat or automaton, you can vary the test, ask questions that haven't been asked before, and engage in the arms race. If you are very smart, you may invent a test which no one has ever thought of before.

Education is no substitute for intelligence. That elusive quality is defined only in part by puzzle-solving ability. It is in the creation of new puzzles reflecting what your senses report that you round out the definitions. (Mentat Text One)

So by this view, the correct thing to do is to replace automation with the most intelligent people available, and have them personally engaged with their duty — rather than having them acting as an administrator, as often happens under our current system.

Some people ask, what genre is Dune? It's set in the far future; there are spaceships, lasers, and nuclear weapons. But most of the series focuses on liege-vassal relationships, scheming, and religious orders with magic powers. This sounds a lot more like fantasy, right?

Clearly, Dune is Political Science Fiction. Science fiction proposes spectacular advances in science and, as a result, technology. But political thought is also a technology:

...while we don’t tend to think of it this way, philosophy is a technology—philosophers develop new modes of thinking, new ways of organizing the state, new ethical principles, and so on. Wartime encourages rulers to invest in Research and Development. So in the Warring States period, a lot of philosophers found work in local courts, as a sort of mental R&D department.

So what Dune has done is thought about wild new political technologies, in the same way that most science fiction thinks about wild new physical technologies (or chemical, or biological, etc.).

The Confucian Heuristic (which you should read, entirely on its own merits) describes a political system built on personal relationships. According to this perspective, Confucius hated unjust inequality. Unlike the western solution, which is to destroy all forms of inequality, Confucius rejected that as impossible. Instead, he proposed that we recognize and promote gifted individuals, and make them extremely aware of their duties to the rest of us. (Seriously, just read it!)

Good government never depends upon laws, but upon the personal qualities of those who govern. The machinery of government is always subordinate to the will of those who administer that machinery. (The Spacing Guild Manual)

In a way, Dune is Confucian as well, or perhaps Neo-Confucian, as Stephenson might say. It presents a society that has been stable for 10,000 years, based largely on feudal principles, and which has arranged itself in such a way that it has kept a major, lurking x-risk at bay.

It’s my contention that feudalism is a natural condition of human beings…not that it is the only condition or not that it is the right condition…that it is just a way we have of falling into organisations. I like to use the example of the Berlin Museum Beavers.

…

Before World War II there were a number of families of beaver in the Berlin Museum. They were European beaver. They had been there, raised in captivity for something on the order of seventy beaver generations, in cages. World War II came along and a bomb freed some of them into the countryside. What did they do? They went out and they started building dams. (Frank Herbert)

One way of thinking about Goodhart's law is that it says that any automated system can and will be gamed as quickly and ruthlessly as possible. Using human authorities rather than rules is the only safeguard, since the human can participate in the intellectual arms race with the people trying to get around the regulation; they can interpret the rules in their spirit rather than in their letter. No one will get far rules-lawyering the king.

The people who will be most effective at Goodhart-gaming a system will be those with starting advantages. This includes the rich, but also those with more intelligence, better social connections, etc., etc. So one problem with automation is that it always favors the aristocracy. Whoever has advantages will, on average, see them magnified by being the best at gaming automated systems.

The Confucian solution to inequality is to tie aristocrats into meaningful personal relationships with their inferiors. The problem with automation is that it unfairly benefits aristocrats and destroys the very idea of a meaningful personal relationship.

What you of the CHOAM directorate seem unable to understand is that you seldom find real loyalties in commerce. When did you last hear of a clerk giving his life for the company? (A letter to CHOAM, Attributed to The Preacher)

I've argued that we need to use human judgment in place of legible signals, and that we should recruit the most gifted people to do so. But giving all the decision-making power to an intellectual elite comes with its own problems. If we're going to recruit elites to replace our automated decision-making, we should make use of a political technology specifically designed to deal with this situation.

I'm not saying that we need to introduce elective fealty, per se. My short-term suggestion, however, would be that you don't let those in positions of power over you pretend that they are your equal. Choosing to attach yourself to someone powerful in exchange for protection is entirely left to your discretion.

Of course, what I really think we should do is bring back sumptuary laws.

Sumptuary laws keep individuals of a certain class from purchasing or using certain goods, including clothing. People tend to think of sumptuary laws as keeping low-class people from pretending to be high-class people, even if they're rich enough to fake it. The story goes that this was a big problem during the late middle ages, because merchants were often richer than barons and counts, but you couldn't let them get away with pretending to be noble.

The Confucian view is that sumptuary laws can keep high-class people from pretending to be low-class people, and attempting to dodge the responsibilities that come with it. Think of the billionaire chicken farmer wearing overalls and a straw hat. Is he just ol' Joe from up the road? Or was he born with a fortune he doesn't deserve?

Confucians would say that a major problem with our current system is that elites are able to pretend that they aren't elite. They see themselves as, while personally gifted, equal in opportunity to the rest of us, and as a result on an equal playing field. They think that they don't owe us anything, and try to convince us to feel the same way.

I like to think of this as the "Donald-Trump-should-be-forced-to-wear-gold-and-jewels-wherever-he-goes" rule. Or if you're of a slightly different political bent, "If-Zuckerberg-wears-another-plain-grey-T-Shirt-I-will-throw-a-fit-who-does-he-think-he's-fooling" rule.

This viewpoint also strikes a surprising truce between mistake and conflict theorists. Mistake theorists are making the mistake of thinking there is no conflict occurring, of letting "elites" pretend that they're part of the "people". Conflict theorists are making the mistake of thinking that tearing down inequality is desirable or even possible.

If you found any of this interesting, I would suggest that you read Dune and its immediate sequels (up to Chapterhouse, but not the junk Herbert's son wrote). If nothing else, consider that despite being published in 1965, it predicted AI threat and x-risk more generally as a major concern for the future of humanity. I promise there are other topics of interest there.

If you found any of this convincing, I strongly recommend that you fight against automation and legible signals whenever possible. Only fully realized human beings have the ability to pragmatically interpret a rule or guideline in the way it was intended. If we ever crack Strong AI, that may change — but safe to say, at that point we will have a new set of problems!

And in regards to the machines:

War to the death should be instantly proclaimed against them. Every machine of every sort should be destroyed by the well-wisher of his species. Let there be no exceptions made, no quarter shown; let us at once go back to the primeval condition of the race. (Samuel Butler)