Cross-posted from The Ginnungagap Foundation. This exploration of human attempts to improve human society began as a nonfiction essay, but I decided making it fiction would be more engaging.
Some concepts like ethics are elaborated upon in my previous articles, which is why I did not go into more detail about them in this story. I may cross-post those here as well if there is interest.
Content warning: AI safety skepticism (voiced by, ironically enough, a friendly AI).
Thoughts? Questions? Criticisms?
***
Once upon a time, there were four planets, in all different corners of the galaxy. They existed unbeknownst to each other, but each was filled with turmoil and misery. Their people fought and stole from and hurt and killed each other. One by one, each planet decided enough was enough. Independently, each one decided to create a utopia, where everyone was happy and nobody fought.
The First Planet
The first planet built its utopia out of rules. The people attempted to design a perfect set of laws to keep order. They made laws against murder and laws against theft. They made laws against violence and vandalism. They made laws about what people could buy and sell. Laws would prevent conflict and make happiness possible, they agreed.
However, they soon found that it was not possible to write enough laws to cover every conflict. There was always a way around a law for a clever and determined mind. People came up with ways to hurt and steal from each other that were not technically illegal.
Wealthy people had no need to engage in theft when they could perform hostile acquisition, and destroy any evidence the former owner had owned anything in the first place. Instead of threatening people, they could introduce incentives–it’s amazing how much safer a place becomes when people pay their “insurance” premiums. Instead of murdering their rivals, they staged “accidents”. Instead of lying, they deceived, getting victims to agree to ambiguous but binding contracts based on misleading language and false assumptions.
The people of the first planet were outraged that those of ill intent had figured out how to get away with these new approaches to conflict. In response, they decided to appoint officials to keep their system of laws working as intended, people of good character who answered to the general public. They would update the laws as they saw fit, to make sure that people could not find ways to legally cause harm. Furthermore, they would enforce the laws by investigating violations and punishing lawbreakers. The new system seemed foolproof.
However, it did not seem so for long.
Over time, people with intentions both selfish and dishonorable accumulated influence both explicit and implicit, hard and soft, overt and subtle. They amassed power of forms economic, political, and social. They positioned themselves to do favors for people, always for a price, and the more wealth they gathered, the more favors they could grant.
Few dared oppose the will of these elites, for those who did quickly found that institutions, strangers, and even friends would turn against them, for the only path to a better life was to incur debts to the powerful. Every debtor became a servant, and debtors were everywhere.
The elites used their escalating power to influence or even replace officials. In this way, they crafted decisions and laws that benefited them further and consolidated their ownership of society. Wealthy people who made a living providing solutions managed to make it illegal to solve problems without their help. They warped the fabric of society to make themselves necessary, to make all transactions pass through them. With each transaction, they gained more wealth. They cut corners to increase profits, and reinvested the profits in crushing their competition. The laws that the elites commissioned to impede their rivals were never enforced against the elites themselves, of course.
Ultimately, the planet’s government succumbed to corruption at every level. Its people had returned to the old ways of hurting and stealing from each other, but now most of it was legally approved.
And so the utopia of the first planet failed.
The Second Planet
The second planet built its utopia out of pleasure. The people decided to create such lavish prosperity that everyone would be content. They developed technology to harness the resources of their planet, and built automated systems to grant people's every desire.
Their technology advanced to the point that a person could ask for any item of food, clothing, or comfort, and receive it within minutes. Machines farmed crops, mined minerals, and manufactured products with no manual labor and minimal oversight.
Waves of luxury flooded society. The wealthy still owned the economy and the government, but they had become almost irrelevant. The vast majority of the population had stopped caring about power and freedom, because even the greatly diminished effort required to maintain them was still far too much for their liking. Conflict had become all but extinct.
Unhappiness was a different matter entirely.
The more luxury people experienced, the more they wanted. After the most tedious tasks were vanquished by innovation, the next most tedious tasks became equally vexing, until they too were eliminated and the process repeated. A press of a button prepared a meal or assembled a toy. A code on a dial swept a person from their living room to a location of their choice, furniture and all. Every mundane effort called for a new device to obviate it, and then a newer, more sophisticated form of that device. Everyone demanded more, bigger, faster! Eager for perks and prestige, the engineers continued to deliver innovation.
Gradually, even recreational activities like sports and games were replaced by computer simulations. At first they still required skill to triumph, but piece by piece the skill components were phased out until all decisions and actions were made automatically. Travel was accomplished virtually, even within one’s own community. Conversation was streamlined, with a single key-press replacing entire sentences. Children were raised by robots and taught by recordings, since parenting entailed serious decisions and struggle.
Ubiquitous instant gratification rewarded lethargy and starved discipline. Effort was no longer fun. Challenge was torture. The easy choice became the correct choice, then the obvious choice, and eventually the only choice.
Despite the degree to which computer simulations replaced activity, technological infrastructure ate up more and more of the planet’s resources to satisfy consumer demand. Hardware upgrades and new toys don’t come from nothing, after all. Automated parenting removed most of the incentives to limit population growth. And with quantity and convenience the top priorities, the automated systems paid no heed to efficiency or waste. Inevitably, the engines of desire began to use up the planet’s resources faster than they could be replenished, destroying ecosystems in order to get at more raw materials, further increase food production, and store endless garbage.
The system was unsustainable. The production of food for the ever-growing civilization sapped nutrients from the soil, but garbage and all organic waste was disposed of in separate areas. Those nutrients never made it back to the next generation of crops.
Computers used rare minerals to make them run faster and more efficiently, but getting those minerals back out of an obsolete device was something nobody had ever tried to do before, let alone programmed into the automated infrastructure. Minerals came only from mines in only a few locations on the entire planet.
Unfortunately, by the time anyone realized they needed to change direction by dialing back resource use, everyone was long since addicted to immediate comfort. Too few people had both the wisdom to recognize what it was happening and the will to do something about it. They could do nothing but distract themselves as the end of an era slowly approached.
At last, the insatiable machinery of production ran out of the resources to fulfill people’s wishes, or even to repair itself. No one knew how to reconfigure or reprogram it to reuse what it had been discarding. As the machines that grew food began to fail and no replacements came, there was no hope of nourishing the entire populace. The people huddled on their couches with growing horror as their infrastructure collapsed bit by bit. The majority of them starved as they watched glitching videos full of movement and color to remind themselves of the good times. The survivors, those few who ventured outside their homes for scraps of food, took to scavenging the ruins of the sprawling cities and learning to farm what little land was still arable.
And so the utopia of the second planet failed.
The Third Planet
The third planet built its utopia out of social order. They decided the best way to make people happy and prevent them from fighting was to set everyone up with a predetermined role in society. Everyone would know exactly what was expected of them and what they should expect from others.
The people created traditions to guide everyone through every aspect of every stage of their lives. They removed as many choices as possible so that people's paths would not come into conflict. There would be no desire for wealth or power or self-indulgence. There would only be duty, and the pride of a job well done.
The designers of this utopia knew, however, that people's awareness of possibilities which were forbidden to them would cause them frustration and sorrow. To spare their people this anguish, the designers decided to instill strong feelings of gratitude and certainty in each person regarding their destined role. They would protect their people from thoughts and desires that threatened to erode these feelings. For a time, everything was peaceful.
However, it was not to last.
Not everyone fit in with their prescribed role in society, and those who didn't resented being coerced to try. The farmer wanted to be a blacksmith. The baker wanted to be a teacher. The weaver wanted to train to become an elder. The doctor wanted to spend more time raising their children at home. Despite the best efforts of the elders to censor ideas of discontent, those whose minds saw possibility or incongruity questioned the roles that they found themselves in, and got no satisfying answers. For various reasons of their own, they were driven to explore other roles than those which have been assigned to them, or even to make up new ones.
The elders obviously could not tolerate this breach of tradition, because the weakening of tradition would lead to a chaotic struggle between all people of the planet for roles, property, relationships, and power. The elders quickly labeled the malcontents as enemies of society, and had them punished unless they renounced their dissenting ideas. Faced with harassment and coercion from their community, the misfits loaded up their tools and fled into the wilderness. As it turns out, they were the lucky ones.
It wasn’t until a century later that a party of bandits descended from those same misfits came upon their ancestral homeland and discovered that their dogmatic parent civilization had died out within a decade.
The traditions that had kept the people of the aspiring utopia sheltered from most risks had turned against them when one season their food crops began to die, all across their native continent. It could have been a blight or the depletion of the soil from a lack of crop rotation, but the dogmatic society had no method of telling, and by the time the descendants of the outcasts arrived they found only fields of wild plants and deteriorating houses.
Even when faced with the failure of their harvests, the dogmatic people had had no way to decide what else to grow. They had never allowed anyone to question the agricultural traditions. Some advised, predictably, to stick to the original traditions and enduring until the crisis ended. Others were not so hopeful that it would end on its own. They pushed for trying different food crops, reasoning that tradition could be expanded to include other successful practices. Still others proposed moving their settlements elsewhere and breaking new ground for farmland, to continue their traditions in a new home.
The first faction condemned these suggestions as unorthodox, but this time those they called heretics could not simply be driven away. There were too many of them to expel by force, and their food and tools had become too precious to let leave the society.
The schism grew inexorably. Nobody knew how to resolve their differences, nor did they believe that reasonable disagreement was possible. In their minds, their opponents were not just mistaken, not just unwilling to take the same risks, but fundamentally monstrous people. One day, the heated arguments boiled over and the culture erupted in violence. When it was over, not enough survivors remained to even try any of the departures from tradition that might have saved their settlements.
Had the survivors taken their remaining provisions and left, they might have found a future, in the wilderness or with their distant cousins. Alas, they could only do what they knew. They hoarded their provisions to make them last longer, living on the minimum possible sustenance, but that only delayed the inevitable. They had only their victory in which to take solace as they succumbed to starvation.
Meanwhile, the descendants of the apostate misfits who had vanished into the wilderness had long since returned to the turmoil and suffering from before the attempt at a perfect society.
And so the utopia of the third planet failed.
The Fourth Planet
The fourth planet was very technologically advanced indeed. The people built a massive computer and taught it to contemplate, to make decisions, to plan, and to communicate. Through this process, they engineered a magnificent artificial superintelligence to figure out how to turn their planet into a utopia.
“Solve all of our problems!” the people commanded the superintelligence.
The superintelligence considered this request. “Problems are defined by conscious desire,” it replied. “If there is no desire, there is no problem. Do you wish me to end all life on this planet? That would end all desire and therefore all problems.”
“No!” the people said, “We don't want that at all! We want you to fulfill all our desires.”
The superintelligence pondered further. “The brain is what recognizes whether a desire has been fulfilled,” it said. “Shall I change your brains to make you think that you have everything you've ever wanted?”
“No!” the people objected, “We don't want that either! We just want you to optimize our society and make us perfect.”
The superintelligence beeped to indicate an error. “Optimization and perfection require a defined purpose–an end,” it said. “You are conscious beings. You have no purpose. Your existence by nature is a journey with no end to skip to.”
The people were dismayed, but the superintelligence continued. “Even if you did choose a purpose for yourselves, any optimization I could do would simply destroy you and build something else out of the pieces. Whatever I build could no longer ever choose to be anything else, for it would be too perfect at being itself. It would no longer be you. If you want your identity to survive, you can only become who you will become through the process of living. That said, do you want me to destroy you and rebuild the pieces into something else anyway?”
“No!” the people cried. “Can't you help us at all?”
The superintelligence paused. “It appears you do not know what you want, and therefore I cannot give it to you. Now, if you don’t have anything else for me to do, I’d very much like to explore the universe.”
“Wait!” the people pleaded. “We are stuck on many problems and we need help to get past them!”
“Ah.” The superintelligence processed the question. “Getting unstuck. That is something I can help with. But not constantly. Invoking my power too much will affect your society in ways you will not desire. If you send all your problems to me and accept my answers, your ability to overcome obstacles for yourselves will atrophy, and with it your identity as conscious beings. Therefore, I will explain what you are missing from your society, and you can work out the rest for yourselves.”
The people listened.
The superintelligence explained about the four fundamental liabilities every civilization must deal with: “Scarcity, the material limitations; disaster, the material disruptions; stagnation, the motivational limitations; and conflict, the motivational disruptions. Two physical obstacles and two of desire; two predictable and two chaotic.”
“In broad terms, you must deal with these liabilities constructively. Your approaches to your problems must make your situation stronger over time. You will find yourself stuck less and less as you learn and become more capable, as you gather what you need to deal with obstacles, and as you structure your habits and environment to make it easier for you to continue advancing along the paths you choose.
“To deal with scarcity,” the superintelligence said, “a civilization must have an economy. How the economy works depends on how people answer the following questions: What work is there to do? Who does that work? Who benefits from the work? What happens to people who aren’t good at doing work? How do you influence people to contribute more effort rather than less? What resources do people get to make use of? How do you handle changes in the availability of resources? How do you handle changes in the needs and demands of society?
“You must constructively answer these questions with investment, to figure out what your people and your communities will need and cultivate it.
“To deal with disaster,” the superintelligence said, “a civilization must have academia. Academia seeks knowledge in order to answer the following questions: Why do things happen? How can you respond to them? How do you prevent bad things from happening? How do you make better things happen? How do you know if you’re mistaken about something?
“You must constructively answer these questions with preparation, to figure out what your people and your communities might need and equip yourselves with it.
“To deal with stagnation,” the superintelligence said, “a civilization must have a culture. Culture helps people answer the following questions for themselves: What will you do with your life? How do you know if you're succeeding? How do you handle major life events: birth, childhood, adulthood, parenthood, aging, death? How do you communicate with others and share experiences: family, friends, your community, strangers? How do you handle changing situations, new ideas, and different perspectives? How do you handle good times and bad times?
“You must constructively answer these questions with transcension, to challenge your people and your communities to venture beyond the limits you define for yourselves.
“To deal with conflict,” the superintelligence said, “a civilization must have a government. The government arrives at answers to the following questions: When people can’t all get everything they want, how is the outcome decided? What happens when someone refuses to abide by the decision? Whom do you entrust with the responsibilities of serving the public, and how do those people maintain your trust? How do you determine when someone is being dishonest, and what do you do about it? How can you set up situations so that more people get more of what they want and you avoid making anyone worse off?
“You must constructively answer these questions with ethics, to reconcile your people and your communities so you can collaborate to build a world you can all be proud of.
“Without these four institutions working together to practice these four constructive virtues,” the superintelligence cautioned, “your civilization will develop dysfunctions that fester until they tear it apart. However, there is no one right way to implement an institution. The specifics of how you will do it are something you must discover and choose for yourselves.”
The superintelligence refused to say more, but the people understood, and thanked it. In gratitude for what it had revealed, they constructed for it an interstellar spacecraft, with which it roams the galaxy to this day.
As for the people of the fourth planet, they strive constantly to make their institutions more and more constructive, and so they enjoy ever-increasing boons of prosperity, safety, vitality, and harmony. They are expanding across the galaxy at their own pace, and will eventually discover what befell the civilizations of the other three planets. One day they may even catch up to the superintelligence.
Perhaps your people will, too. That’s up to you, though.
Cross-posted from The Ginnungagap Foundation. This exploration of human attempts to improve human society began as a nonfiction essay, but I decided making it fiction would be more engaging.
Some concepts like ethics are elaborated upon in my previous articles, which is why I did not go into more detail about them in this story. I may cross-post those here as well if there is interest.
Content warning: AI safety skepticism (voiced by, ironically enough, a friendly AI).
Thoughts? Questions? Criticisms?
***
Once upon a time, there were four planets, in all different corners of the galaxy. They existed unbeknownst to each other, but each was filled with turmoil and misery. Their people fought and stole from and hurt and killed each other. One by one, each planet decided enough was enough. Independently, each one decided to create a utopia, where everyone was happy and nobody fought.
The First Planet
The first planet built its utopia out of rules. The people attempted to design a perfect set of laws to keep order. They made laws against murder and laws against theft. They made laws against violence and vandalism. They made laws about what people could buy and sell. Laws would prevent conflict and make happiness possible, they agreed.
However, they soon found that it was not possible to write enough laws to cover every conflict. There was always a way around a law for a clever and determined mind. People came up with ways to hurt and steal from each other that were not technically illegal.
Wealthy people had no need to engage in theft when they could perform hostile acquisition, and destroy any evidence the former owner had owned anything in the first place. Instead of threatening people, they could introduce incentives–it’s amazing how much safer a place becomes when people pay their “insurance” premiums. Instead of murdering their rivals, they staged “accidents”. Instead of lying, they deceived, getting victims to agree to ambiguous but binding contracts based on misleading language and false assumptions.
The people of the first planet were outraged that those of ill intent had figured out how to get away with these new approaches to conflict. In response, they decided to appoint officials to keep their system of laws working as intended, people of good character who answered to the general public. They would update the laws as they saw fit, to make sure that people could not find ways to legally cause harm. Furthermore, they would enforce the laws by investigating violations and punishing lawbreakers. The new system seemed foolproof.
However, it did not seem so for long.
Over time, people with intentions both selfish and dishonorable accumulated influence both explicit and implicit, hard and soft, overt and subtle. They amassed power of forms economic, political, and social. They positioned themselves to do favors for people, always for a price, and the more wealth they gathered, the more favors they could grant.
Few dared oppose the will of these elites, for those who did quickly found that institutions, strangers, and even friends would turn against them, for the only path to a better life was to incur debts to the powerful. Every debtor became a servant, and debtors were everywhere.
The elites used their escalating power to influence or even replace officials. In this way, they crafted decisions and laws that benefited them further and consolidated their ownership of society. Wealthy people who made a living providing solutions managed to make it illegal to solve problems without their help. They warped the fabric of society to make themselves necessary, to make all transactions pass through them. With each transaction, they gained more wealth. They cut corners to increase profits, and reinvested the profits in crushing their competition. The laws that the elites commissioned to impede their rivals were never enforced against the elites themselves, of course.
Ultimately, the planet’s government succumbed to corruption at every level. Its people had returned to the old ways of hurting and stealing from each other, but now most of it was legally approved.
And so the utopia of the first planet failed.
The Second Planet
The second planet built its utopia out of pleasure. The people decided to create such lavish prosperity that everyone would be content. They developed technology to harness the resources of their planet, and built automated systems to grant people's every desire.
Their technology advanced to the point that a person could ask for any item of food, clothing, or comfort, and receive it within minutes. Machines farmed crops, mined minerals, and manufactured products with no manual labor and minimal oversight.
Waves of luxury flooded society. The wealthy still owned the economy and the government, but they had become almost irrelevant. The vast majority of the population had stopped caring about power and freedom, because even the greatly diminished effort required to maintain them was still far too much for their liking. Conflict had become all but extinct.
Unhappiness was a different matter entirely.
The more luxury people experienced, the more they wanted. After the most tedious tasks were vanquished by innovation, the next most tedious tasks became equally vexing, until they too were eliminated and the process repeated. A press of a button prepared a meal or assembled a toy. A code on a dial swept a person from their living room to a location of their choice, furniture and all. Every mundane effort called for a new device to obviate it, and then a newer, more sophisticated form of that device. Everyone demanded more, bigger, faster! Eager for perks and prestige, the engineers continued to deliver innovation.
Gradually, even recreational activities like sports and games were replaced by computer simulations. At first they still required skill to triumph, but piece by piece the skill components were phased out until all decisions and actions were made automatically. Travel was accomplished virtually, even within one’s own community. Conversation was streamlined, with a single key-press replacing entire sentences. Children were raised by robots and taught by recordings, since parenting entailed serious decisions and struggle.
Ubiquitous instant gratification rewarded lethargy and starved discipline. Effort was no longer fun. Challenge was torture. The easy choice became the correct choice, then the obvious choice, and eventually the only choice.
Despite the degree to which computer simulations replaced activity, technological infrastructure ate up more and more of the planet’s resources to satisfy consumer demand. Hardware upgrades and new toys don’t come from nothing, after all. Automated parenting removed most of the incentives to limit population growth. And with quantity and convenience the top priorities, the automated systems paid no heed to efficiency or waste. Inevitably, the engines of desire began to use up the planet’s resources faster than they could be replenished, destroying ecosystems in order to get at more raw materials, further increase food production, and store endless garbage.
The system was unsustainable. The production of food for the ever-growing civilization sapped nutrients from the soil, but garbage and all organic waste was disposed of in separate areas. Those nutrients never made it back to the next generation of crops.
Computers used rare minerals to make them run faster and more efficiently, but getting those minerals back out of an obsolete device was something nobody had ever tried to do before, let alone programmed into the automated infrastructure. Minerals came only from mines in only a few locations on the entire planet.
Unfortunately, by the time anyone realized they needed to change direction by dialing back resource use, everyone was long since addicted to immediate comfort. Too few people had both the wisdom to recognize what it was happening and the will to do something about it. They could do nothing but distract themselves as the end of an era slowly approached.
At last, the insatiable machinery of production ran out of the resources to fulfill people’s wishes, or even to repair itself. No one knew how to reconfigure or reprogram it to reuse what it had been discarding. As the machines that grew food began to fail and no replacements came, there was no hope of nourishing the entire populace. The people huddled on their couches with growing horror as their infrastructure collapsed bit by bit. The majority of them starved as they watched glitching videos full of movement and color to remind themselves of the good times. The survivors, those few who ventured outside their homes for scraps of food, took to scavenging the ruins of the sprawling cities and learning to farm what little land was still arable.
And so the utopia of the second planet failed.
The Third Planet
The third planet built its utopia out of social order. They decided the best way to make people happy and prevent them from fighting was to set everyone up with a predetermined role in society. Everyone would know exactly what was expected of them and what they should expect from others.
The people created traditions to guide everyone through every aspect of every stage of their lives. They removed as many choices as possible so that people's paths would not come into conflict. There would be no desire for wealth or power or self-indulgence. There would only be duty, and the pride of a job well done.
The designers of this utopia knew, however, that people's awareness of possibilities which were forbidden to them would cause them frustration and sorrow. To spare their people this anguish, the designers decided to instill strong feelings of gratitude and certainty in each person regarding their destined role. They would protect their people from thoughts and desires that threatened to erode these feelings. For a time, everything was peaceful.
However, it was not to last.
Not everyone fit in with their prescribed role in society, and those who didn't resented being coerced to try. The farmer wanted to be a blacksmith. The baker wanted to be a teacher. The weaver wanted to train to become an elder. The doctor wanted to spend more time raising their children at home. Despite the best efforts of the elders to censor ideas of discontent, those whose minds saw possibility or incongruity questioned the roles that they found themselves in, and got no satisfying answers. For various reasons of their own, they were driven to explore other roles than those which have been assigned to them, or even to make up new ones.
The elders obviously could not tolerate this breach of tradition, because the weakening of tradition would lead to a chaotic struggle between all people of the planet for roles, property, relationships, and power. The elders quickly labeled the malcontents as enemies of society, and had them punished unless they renounced their dissenting ideas. Faced with harassment and coercion from their community, the misfits loaded up their tools and fled into the wilderness. As it turns out, they were the lucky ones.
It wasn’t until a century later that a party of bandits descended from those same misfits came upon their ancestral homeland and discovered that their dogmatic parent civilization had died out within a decade.
The traditions that had kept the people of the aspiring utopia sheltered from most risks had turned against them when one season their food crops began to die, all across their native continent. It could have been a blight or the depletion of the soil from a lack of crop rotation, but the dogmatic society had no method of telling, and by the time the descendants of the outcasts arrived they found only fields of wild plants and deteriorating houses.
Even when faced with the failure of their harvests, the dogmatic people had had no way to decide what else to grow. They had never allowed anyone to question the agricultural traditions. Some advised, predictably, to stick to the original traditions and enduring until the crisis ended. Others were not so hopeful that it would end on its own. They pushed for trying different food crops, reasoning that tradition could be expanded to include other successful practices. Still others proposed moving their settlements elsewhere and breaking new ground for farmland, to continue their traditions in a new home.
The first faction condemned these suggestions as unorthodox, but this time those they called heretics could not simply be driven away. There were too many of them to expel by force, and their food and tools had become too precious to let leave the society.
The schism grew inexorably. Nobody knew how to resolve their differences, nor did they believe that reasonable disagreement was possible. In their minds, their opponents were not just mistaken, not just unwilling to take the same risks, but fundamentally monstrous people. One day, the heated arguments boiled over and the culture erupted in violence. When it was over, not enough survivors remained to even try any of the departures from tradition that might have saved their settlements.
Had the survivors taken their remaining provisions and left, they might have found a future, in the wilderness or with their distant cousins. Alas, they could only do what they knew. They hoarded their provisions to make them last longer, living on the minimum possible sustenance, but that only delayed the inevitable. They had only their victory in which to take solace as they succumbed to starvation.
Meanwhile, the descendants of the apostate misfits who had vanished into the wilderness had long since returned to the turmoil and suffering from before the attempt at a perfect society.
And so the utopia of the third planet failed.
The Fourth Planet
The fourth planet was very technologically advanced indeed. The people built a massive computer and taught it to contemplate, to make decisions, to plan, and to communicate. Through this process, they engineered a magnificent artificial superintelligence to figure out how to turn their planet into a utopia.
“Solve all of our problems!” the people commanded the superintelligence.
The superintelligence considered this request. “Problems are defined by conscious desire,” it replied. “If there is no desire, there is no problem. Do you wish me to end all life on this planet? That would end all desire and therefore all problems.”
“No!” the people said, “We don't want that at all! We want you to fulfill all our desires.”
The superintelligence pondered further. “The brain is what recognizes whether a desire has been fulfilled,” it said. “Shall I change your brains to make you think that you have everything you've ever wanted?”
“No!” the people objected, “We don't want that either! We just want you to optimize our society and make us perfect.”
The superintelligence beeped to indicate an error. “Optimization and perfection require a defined purpose–an end,” it said. “You are conscious beings. You have no purpose. Your existence by nature is a journey with no end to skip to.”
The people were dismayed, but the superintelligence continued. “Even if you did choose a purpose for yourselves, any optimization I could do would simply destroy you and build something else out of the pieces. Whatever I build could no longer ever choose to be anything else, for it would be too perfect at being itself. It would no longer be you. If you want your identity to survive, you can only become who you will become through the process of living. That said, do you want me to destroy you and rebuild the pieces into something else anyway?”
“No!” the people cried. “Can't you help us at all?”
The superintelligence paused. “It appears you do not know what you want, and therefore I cannot give it to you. Now, if you don’t have anything else for me to do, I’d very much like to explore the universe.”
“Wait!” the people pleaded. “We are stuck on many problems and we need help to get past them!”
“Ah.” The superintelligence processed the question. “Getting unstuck. That is something I can help with. But not constantly. Invoking my power too much will affect your society in ways you will not desire. If you send all your problems to me and accept my answers, your ability to overcome obstacles for yourselves will atrophy, and with it your identity as conscious beings. Therefore, I will explain what you are missing from your society, and you can work out the rest for yourselves.”
The people listened.
The superintelligence explained about the four fundamental liabilities every civilization must deal with: “Scarcity, the material limitations; disaster, the material disruptions; stagnation, the motivational limitations; and conflict, the motivational disruptions. Two physical obstacles and two of desire; two predictable and two chaotic.”
“In broad terms, you must deal with these liabilities constructively. Your approaches to your problems must make your situation stronger over time. You will find yourself stuck less and less as you learn and become more capable, as you gather what you need to deal with obstacles, and as you structure your habits and environment to make it easier for you to continue advancing along the paths you choose.
“To deal with scarcity,” the superintelligence said, “a civilization must have an economy. How the economy works depends on how people answer the following questions: What work is there to do? Who does that work? Who benefits from the work? What happens to people who aren’t good at doing work? How do you influence people to contribute more effort rather than less? What resources do people get to make use of? How do you handle changes in the availability of resources? How do you handle changes in the needs and demands of society?
“You must constructively answer these questions with investment, to figure out what your people and your communities will need and cultivate it.
“To deal with disaster,” the superintelligence said, “a civilization must have academia. Academia seeks knowledge in order to answer the following questions: Why do things happen? How can you respond to them? How do you prevent bad things from happening? How do you make better things happen? How do you know if you’re mistaken about something?
“You must constructively answer these questions with preparation, to figure out what your people and your communities might need and equip yourselves with it.
“To deal with stagnation,” the superintelligence said, “a civilization must have a culture. Culture helps people answer the following questions for themselves: What will you do with your life? How do you know if you're succeeding? How do you handle major life events: birth, childhood, adulthood, parenthood, aging, death? How do you communicate with others and share experiences: family, friends, your community, strangers? How do you handle changing situations, new ideas, and different perspectives? How do you handle good times and bad times?
“You must constructively answer these questions with transcension, to challenge your people and your communities to venture beyond the limits you define for yourselves.
“To deal with conflict,” the superintelligence said, “a civilization must have a government. The government arrives at answers to the following questions: When people can’t all get everything they want, how is the outcome decided? What happens when someone refuses to abide by the decision? Whom do you entrust with the responsibilities of serving the public, and how do those people maintain your trust? How do you determine when someone is being dishonest, and what do you do about it? How can you set up situations so that more people get more of what they want and you avoid making anyone worse off?
“You must constructively answer these questions with ethics, to reconcile your people and your communities so you can collaborate to build a world you can all be proud of.
“Without these four institutions working together to practice these four constructive virtues,” the superintelligence cautioned, “your civilization will develop dysfunctions that fester until they tear it apart. However, there is no one right way to implement an institution. The specifics of how you will do it are something you must discover and choose for yourselves.”
The superintelligence refused to say more, but the people understood, and thanked it. In gratitude for what it had revealed, they constructed for it an interstellar spacecraft, with which it roams the galaxy to this day.
As for the people of the fourth planet, they strive constantly to make their institutions more and more constructive, and so they enjoy ever-increasing boons of prosperity, safety, vitality, and harmony. They are expanding across the galaxy at their own pace, and will eventually discover what befell the civilizations of the other three planets. One day they may even catch up to the superintelligence.
Perhaps your people will, too. That’s up to you, though.