Continuing my cursory exploration of semiotics and post-modern thought, I'm struck by the similarity between writing in those traditions, and picking up women.  The most-important traits for practitioners of both are energy, enthusiasm, and confidence.  In support of this proposition, here is a photo of Slavoj Zizek at his 2006 wedding:

Having philosophical or logical rigor, or demonstrating the usefulness of your ideas using empirical data, does not seem to provide a similar advantage, despite taking a lot of time.

I speculate that semiotics and post-modernism (which often go hand-in-hand) became popular by natural selection.  They provide specialized terminologies which give the impression of rigorous thought without requiring actual rigor. People who use them can thus out-publish their more-careful competitors. So post-modernism tends to drive rigorous thought out of any field it enters.

(It's possible to combine post-modern ideas and a time-consuming empirical approach, as Thomas Kuhn did in The Structure of Scientific Revolutions.  But it's uncommon.)

If rigorous thought significantly reduces publication rate, we should find that the rigor of a field or a person correlates inversely with words per person-year.  Establishing that fact alone, combined with the emphasis on publication in academics, would lead us to expect that any approach that allowed one to fake or dispense with intellectual rigor in a field would rapidly take over that field.

... or The Maverick Nanny with a Dopamine Drip

Richard Loosemore

Abstract

My goal in this essay is to analyze some widely discussed scenarios that predict dire and almost unavoidable negative behavior from future artificial general intelligences, even if they are programmed to be friendly to humans. I conclude that these doomsday scenarios involve AGIs that are logically incoherent at such a fundamental level that they can be dismissed as extremely implausible. In addition, I suggest that the most likely outcome of attempts to build AGI systems of this sort would be that the AGI would detect the offending incoherence in its design, and spontaneously self-modify to make itself less unstable, and (probably) safer.

Introduction

AI systems at the present time do not even remotely approach the human level of intelligence, and the consensus seems to be that genuine artificial general intelligence (AGI) systems—those that can learn new concepts without help, interact with physical objects, and behave with coherent purpose in the chaos of the real world—are not on the immediate horizon.

But in spite of this there are some researchers and commentators who have made categorical statements about how future AGI systems will behave. Here is one example, in which Steve Omohundro (2008) expresses a sentiment that is echoed by many:

"Without special precautions, [the AGI] will resist being turned off, will try to break into other machines and make copies of itself, and will try to acquire resources without regard for anyone else’s safety. These potentially harmful behaviors will occur not because they were programmed in at the start, but because of the intrinsic nature of goal driven systems." (Omohundro, 2008)

Omohundro’s description of a psychopathic machine that gobbles everything in the universe, and his conviction that every AI, no matter how well it is designed, will turn into a gobbling psychopath is just one of many doomsday predictions being popularized in certain sections of the AI community. These nightmare scenarios are now saturating the popular press, and luminaries such as Stephen Hawking have -- apparently in response -- expressed their concern that AI might "kill us all."

I will start by describing a group of three hypothetical doomsday scenarios that include Omohundro’s Gobbling Psychopath, and two others that I will call the Maverick Nanny with a Dopamine Drip and the Smiley Tiling Berserker. Undermining the credibility of these arguments is relatively straightforward, but I think it is important to try to dig deeper and find the core issues that lie behind this sort of thinking. With that in mind, much of this essay is about (a) the design of motivation and goal mechanisms in logic-based AGI systems, (b) the misappropriation of definitions of “intelligence,” and (c) an anthropomorphism red herring that is often used to justify the scenarios.

Dopamine Drips and Smiley Tiling

In a 2012 New Yorker article entitled Moral Machines, Gary Marcus said:

"An all-powerful computer that was programmed to maximize human pleasure, for example, might consign us all to an intravenous dopamine drip [and] almost any easy solution that one might imagine leads to some variation or another on the Sorcerer’s Apprentice, a genie that’s given us what we’ve asked for, rather than what we truly desire." (Marcus 2012)

He is depicting a Nanny AI gone amok. It has good intentions (it wants to make us happy) but the programming to implement that laudable goal has had unexpected ramifications, and as a result the Nanny AI has decided to force all human beings to have their brains connected to a dopamine drip.

Here is another incarnation of this Maverick Nanny with a Dopamine Drip scenario, in an excerpt from the Intelligence Explosion FAQ, published by MIRI, the Machine Intelligence Research Institute (Muehlhauser 2013):

"Even a machine successfully designed with motivations of benevolence towards humanity could easily go awry when it discovered implications of its decision criteria unanticipated by its designers. For example, a superintelligence programmed to maximize human happiness might find it easier to rewire human neurology so that humans are happiest when sitting quietly in jars than to build and maintain a utopian world that caters to the complex and nuanced whims of current human neurology."

Setting aside the question of whether happy bottled humans are feasible (one presumes the bottles are filled with dopamine, and that a continuous flood of dopamine does indeed generate eternal happiness), there seems to be a prima facie inconsistency between the two predicates

[is an AI that is superintelligent enough to be unstoppable]

and

[believes that benevolence toward humanity might involve forcing human beings to do something violently against their will.]

Why do I say that these are seemingly inconsistent?  Well, if you or I were to suggest that the best way to achieve universal human happiness was to forcibly rewire the brain of everyone on the planet so they became happy when sitting in bottles of dopamine, most other human beings would probably take that as a sign of insanity. But Muehlhauser implies that the same suggestion coming from an AI would be perfectly consistent with superintelligence.

Much could be said about this argument, but for the moment let’s just note that it begs a number of questions about the strange definition of “intelligence” at work here.

The Smiley Tiling Berserker

Since 2006 there has been an occasional debate between Eliezer Yudkowsky and Bill Hibbard. Here is Yudkowsky stating the theme of their discussion:

"A technical failure occurs when the [motivation code of the AI] does not do what you think it does, though it faithfully executes as you programmed it. [...]   Suppose we trained a neural network to recognize smiling human faces and distinguish them from frowning human faces. Would the network classify a tiny picture of a smiley-face into the same attractor as a smiling human face? If an AI “hard-wired” to such code possessed the power—and Hibbard (2001) spoke of superintelligence—would the galaxy end up tiled with tiny molecular pictures of smiley-faces?"   (Yudkowsky 2008)

Yudkowsky’s question was not rhetorical, because he goes on to answer it in the affirmative:

"Flash forward to a time when the AI is superhumanly intelligent and has built its own nanotech infrastructure, and the AI may be able to produce stimuli classified into the same attractor by tiling the galaxy with tiny smiling faces... Thus the AI appears to work fine during development, but produces catastrophic results after it becomes smarter than the programmers(!)." (Yudkowsky 2008)

Hibbard’s response was as follows:

Beyond being merely wrong, Yudkowsky's statement assumes that (1) the AI is intelligent enough to control the galaxy (and hence have the ability to tile the galaxy with tiny smiley faces), but also assumes that (2) the AI is so unintelligent that it cannot distinguish a tiny smiley face from a human face. (Hibbard 2006)

This comment expresses what I feel is the majority lay opinion: how could an AI be so intelligent as to be unstoppable, but at the same time so unsophisticated that its motivation code treats smiley faces as evidence of human happiness?

Machine Ghosts and DWIM

The Hibbard/Yudkowsky debate is worth tracking a little longer. Yudkowsky later postulates an AI with a simple neural net classifier at its core, which is trained on a large number of images, each of which is labeled with either “happiness” or “not happiness.” After training on the images the neural net can then be shown any image at all, and it will give an output that classifies the new image into one or the other set. Yudkowsky says, of this system:

"Even given a million training cases of this type, if the test case of a tiny molecular smiley-face does not appear in the training data, it is by no means trivial to assume that the inductively simplest boundary around all the training cases classified “positive” will exclude every possible tiny molecular smiley-face that the AI can potentially engineer to satisfy its utility function.

And of course, even if all tiny molecular smiley-faces and nanometer-scale dolls of brightly smiling humans were somehow excluded, the end result of such a utility function is for the AI to tile the galaxy with as many “smiling human faces” as a given amount of matter can be processed to yield." (Yudkowsky 2011)

He then tries to explain what he thinks is wrong with the reasoning of people, like Hibbard, who dispute the validity of his scenario:

"So far as I can tell, to [Hibbard] it remains self-evident that no superintelligence would be stupid enough to thus misinterpret the code handed to it, when it’s obvious what the code is supposed to do.   [...] It seems that even among competent programmers, when the topic of conversation drifts to Artificial General Intelligence, people often go back to thinking of an AI as a ghost-in-the-machine—an agent with preset properties which is handed its own code as a set of instructions, and may look over that code and decide to circumvent it if the results are undesirable to the agent’s innate motivations, or reinterpret the code to do the right thing if the programmer made a mistake." (Yudkowsky 2011)

Yudkowsky at first rejects the idea that an AI might check its own code to make sure it was correct before obeying the code. But, truthfully, it would not require a ghost-in-the-machine to reexamine the situation if there was some kind of gross inconsistency with what the humans intended: there could be some other part of its programming (let’s call it the checking code) that kicked in if there was any hint of a mismatch between what the AI planned to do and what the original programmers were now saying they intended. There is nothing difficult or intrinsically wrong with such a design.  And, in fact, Yudkowsky goes on to make that very suggestion (he even concedes that it would be “an extremely good idea”).

But then his enthusiasm for the checking code evaporates:

"But consider that a property of the AI’s preferences which says e.g., “maximize the satisfaction of the programmers with the code” might be more maximally fulfilled by rewiring the programmers’ brains using nanotechnology than by any conceivable change to the code."
(Yudkowsky 2011)

So, this is supposed to be what goes through the mind of the AGI. First it thinks “Human happiness is seeing lots of smiling faces, so I must rebuild the entire universe to put a smiley shape into every molecule.” But before it can go ahead with this plan, the checking code kicks in: “Wait! I am supposed to check with the programmers first to see if this is what they meant by human happiness.” The programmers, of course, give a negative response, and the AGI thinks “Oh dear, they didn’t like that idea. I guess I had better not do it then."

But now Yudkowsky is suggesting that the AGI has second thoughts:  "Hold on a minute," it thinks,  "suppose I abduct the programmers and rewire their brains to make them say ‘yes’ when I check with them? Excellent! I will do that.” And, after reprogramming the humans so they say the thing that makes its life simplest, the AGI goes on to tile the whole universe with tiles covered in smiley faces. It has become a Smiley Tiling Berserker.

I want to suggest that the implausibility of this scenario is quite obvious: if the AGI is supposed to check with the programmers about their intentions before taking action, why did it decide to rewire their brains before asking them if it was okay to do the rewiring?

Yudkowsky hints that this would happen because it would be more efficient for the AI to ignore the checking code. He seems to be saying that the AI is allowed to override its own code (the checking code, in this case) because doing so would be “more efficient,” but it would not be allowed to override its motivation code just because the programmers told it there had been a mistake.

This looks like a bait-and-switch. Out of nowhere, Yudkowsky implicitly assumes that “efficiency” trumps all else, without pausing for a moment to consider that it would be trivial to design the AI in such a way that efficiency was a long way down the list of priorities. There is no law of the universe that says all artificial intelligence systems must prize efficiency above all other considerations, so what really happened here is that Yudkowsky designed this hypothetical machine to fail. By inserting the Efficiency Trumps All directive, the AGI was bound to go berserk.

The obvious conclusion is that a trivial change in the order of directives in the AI’s motivation engine will cause the entire argument behind the Smiley Tiling Berserker to evaporate. By explicitly designing the AGI so that efficiency is considered as just another goal to strive for, and by making sure that it will always be a second-class goal, the line of reasoning that points to a bererker machine evaporates.

At this point, engaging in further debate at this level would be less productive than trying to analyze the assumptions that lie behind these claims about what a future AI would or would not be likely to do.

Logical vs. Swarm AI

The main reason that Omohundro, Muehlhauser, Yudkowsky, and the popular press like to give credence to the Gobbling Psychopath, the Maverick Nanny and the Smiley Tiling Berserker is because they assume that all future intelligent machines fall into a broad class of systems that I am going to call “Canonical Logical AI” (CLAI). The bizarre behaviors of these hypothetical AI monsters are just a consequence of weaknesses in this class of AI design. Specifically, these kinds of systems are supposed to interpret their goals in an extremely literal fashion, which eventually leads them to bizarre behaviors engendered by peculiar interpretations of forms of words.

The CLAI architecture is not the only way to build a mind, however, and I will outline an alternative class of AGI designs that does not appear to suffer from the unstable and unfriendly behavior to be expected in a CLAI.

The Canonical Logical AI

“Canonical Logical AI” is an umbrella term designed to capture a class of AI architectures that are widely assumed in the AI community to be the only meaningful class of AI worth discussing. These systems share the following main features:

• The main ingredients of the design are some knowledge atoms that represent things in the world, and some logical machinery that dictates how these atoms can be connected into linear propositions that describe states of the world.
• There is a degree and type of truth that can be associated with any proposition, and there are some truth-preserving functions that can be applied to what the system knows, to generate knew facts that it also can assume to be known.
• The various elements described above are not allowed to contain active internal machinery inside them, in such a way as to make combinations of the elements have properties that are unpredictably dependent on interactions happening at the level of the internal machinery.
• There has to be a transparent mapping between elements of the system and things in the real world. That is, things in the world are not allowed to correspond to clusters of atoms, in such a way that individual atoms have no clear semantics.

The above features are only supposed to apply to the core of the AI: it is always possible to include subsystems that use some other type of architecture (for example, there might be a distributed neural net acting as a visual input feature detector).

Most important of all, from the point of view of the discussion in the paper, the CLAI needs one more component that makes it more than just a “logic-based AI”:

• There is a motivation and goal management (MGM) system to govern its behavior in the world.

The usual assumption is that the MGM contains a number of goal statements (encoded in the same type of propositional form that the AI uses to describe states of the world), and some machinery for analyzing a goal statement into a sequences of subgoals that, if executed, would cause the goal to be satisfied.

Included in the MGM is an expected utility function that applies to any possible state of the world, and which spits out a number that is supposed to encode the degree to which the AI considers that state to be preferable. Overall, the MGM is built in such a way that the AI seeks to maximize the expected utility.

Notice that the MGM I have just described is an extrapolation from a long line of goal-planning mechanisms that stretch back to the means-ends-analysis of Newell and Simon (1963).

Swarm Relaxation Intelligence

By way of contrast with this CLAI architecture, consider an alternative type of system that I will refer to as a Swarm Relaxation Intelligence. (although it could also be called, less succinctly, a parallel weak constraint relaxation system).

• The basic elements of the system (the atoms) may represent things in the world, but it is just as likely that they are subsymbolic, with no transparent semantics
• Atoms are likely to contain active internal machinery inside them, in such a way that combinations of the elements have swarm-like properties that depend on interactions at the level of that machinery.
• The primary mechanism that drives the systems is one of parallel weak constraint relaxation: the atoms change their state to try to satisfy large numbers of weak constraints that exist between them.
• The motivation and goal management (MGM) system would be expected to use the same kind of distributed, constraint relaxation mechanisms used in the thinking process (above), with the result that the overall motivation and values of the system would take into account a large degree of context, and there would be very much less of an emphasis on explicit, single-point-of-failure encoding of goals and motivation.

Swarm Relaxation has more in common with connectionist systems (McClelland, Rumelhart and Hinton 1986) than with CLAI. As McClelland et al. (1986) point out, weak constraint relaxation is the model that best describes human cognition, and when used for AI it leads to systems with a powerful kind of intelligence that is flexible, insensitive to noise and lacking the kind of brittleness typical of logic-based AI. In particular, notice that a swarm relaxation AGI would not use explicit calculations for utility or the truth of propositions.

Swarm relaxation AGI systems have not been built yet (subsystems like neural nets have, of course, been built, but there is little or no research into the idea that swarm relaxation could be used for all of an AGI architecture).

Relative Abundances

How many proof-of-concept systems exist, functioning at or near the human level of human performance, for these two classes of intelligent system?

There are precisely zero instances of the CLAI type, because although there are many logic-based narrow-AI systems, nobody has so far come close to producing a general-purpose system (an AGI) that can function in the real world. It has to be said that zero is not a good number to quote when it comes to claims about the “inevitable” characteristics of the behavior of such systems.

How many swarm relaxation intelligences are there? At the last count, approximately seven billion.

The Doctrine of Logical Infallibility

The simplest possible logical reasoning engine is an inflexible beast: it starts with some axioms that are assumed to be true, and from that point on it only adds new propositions if they are provably true given the sum total of the knowledge accumulated so far. That kind of logic engine is too simple to be an AI, so we allow ourselves to augment it in a number of ways—knowledge is allowed to be retracted, binary truth values become degrees of truth, or probabilities, and so on. New proposals for systems of formal logic abound in the AI literature, and engineers who build real, working AI systems often experiment with kludges in order to improve performance, without getting prior approval from logical theorists.

But in spite of all these modifications that AI practitioners make to the underlying ur‑logic, one feature of these systems is often assumed to be inherited as an absolute: the rigidity and certainty of conclusions, once arrived at. No second guessing, no “maybe,” no sanity checks: if the system decides that X is true, that is the end of the story.

Let me be careful here. I said that this was “assumed to be inherited as an absolute”, but there is a yawning chasm between what real AI developers do, and what Yudkowsky, Muehlhauser, Omohundro and others assume will be true of future AGI systems. Real AI developers put sanity checks into their systems all the time. But these doomsday scenarios talk about future AI as if it would only take one parameter to get one iota above a threshold, and the AI would irrevocably commit to a life of stuffing humans into dopamine jars.

One other point of caution: this is not to say that the reasoning engine can never come to conclusions that are uncertain—quite the contrary: uncertain conclusions will be the norm in an AI that interacts with the world—but if the system does come to a conclusion (perhaps with a degree-of-certainty number attached), the assumption seems to be that it will then be totally incapable of then allowing context to matter.

One way to characterize this assumption is that the AI is supposed to be hardwired with a Doctrine of Logical Infallibility. The significance of the doctrine of logical infallibility is as follows. The AI can sometimes execute a reasoning process, then come to a conclusion and then, when it is faced with empirical evidence that its conclusion may be unsound, it is incapable of considering the hypothesis that its own reasoning engine may not have taken it to a sensible place. The system does not second guess its conclusions. This is not because second guessing is an impossible thing to implement, it is simply because people who speculate about future AGI systems take it as a given that an AGI would regard its own conclusions as sacrosanct.

But it gets worse. Those who assume the doctrine of logical infallibility often say that if the system comes to a conclusion, and if some humans (like the engineers who built the system) protest that there are manifest reasons to think that the reasoning that led to this conclusion was faulty, then there is a sense in which the AGI’s intransigence is correct, or appropriate, or perfectly consistent with “intelligence.”

This is a bizarre conclusion. First of all it is bizarre for researchers in the present day to make the assumption, and it would be even more bizarre for a future AGI to adhere to it. To see why, consider some of the implications of this idea. If the AGI is as intelligent as its creators, then it will have a very clear understanding of the following facts about the world.

• It will understand that many of its more abstract logical atoms have a less than clear denotation or extension in the world (if the AGI comes to a conclusion involving the atom [infelicity], say, can it then point to an instance of an infelicity and be sure that this is a true instance, given the impreciseness and subtlety of the concept?).
• It will understand that knowledge can always be updated in the light of new information. Today’s true may be tomorrow’s false.
• It will understand that probabilities used in the reasoning engine can be subject to many types of unavoidable errors.
• It will understand that the techniques used to build its own reasoning engine may be under constant review, and updates may have unexpected effects on conclusions (especially in very abstract or lengthy reasoning episodes).
• It will understand that resource limitations often force it to truncate search procedures within its reasoning engine, leading to conclusions that can sometimes be sensitive to the exact point at which the truncation occurred.

Now, unless the AGI is assumed to have infinite resources and infinite access to all the possible universes that could exist (a consideration that we can reject, since we are talking about reality here, not fantasy), the system will be perfectly well aware of these facts about its own limitations. So, if the system is also programmed to stick to the doctrine of logical infallibility, how can it reconcile the doctrine with the fact that episodes of fallibility are virtually inevitable?

On the face of it this looks like a blunt impossibility: the knowledge of fallibility is so categorical, so irrefutable, that it beggars belief that any coherent, intelligent system (let alone an unstoppable superintelligence) could tolerate the contradiction between this fact about the nature of intelligent machines and some kind of imperative about Logical Infallibility built into its motivation system.

This is the heart of the argument I wish to present. This is where the rock and the hard place come together. If the AI is superintelligent (and therefore unstoppable), it will be smart enough to know all about its own limitations when it comes to the business of reasoning about the world and making plans of action. But if it is also programmed to utterly ignore that fallibility—for example, when it follows its compulsion to put everyone on a dopamine drip, even though this plan is clearly a result of a programming error—then we must ask the question: how can the machine be both superintelligent and able to ignore a gigantic inconsistency in its reasoning?

Critically, we have to confront the following embarrassing truth: if the AGI is going to throw a wobbly over the dopamine drip plan, what possible reason is there to believe that it did not do this on other occasions? Why would anyone suppose that this AGI ignored an inconvenient truth on only this one occasion? More likely, it spent its entire childhood pulling the same kind of stunt. And if it did, how could it ever have risen to the point where it became superintelligent...?

Is the Doctrine of Logical Infallibility Taken Seriously?

Is the Doctrine of Logical Infallibility really assumed by those who promote the doomsday scenarios? Imagine a conversation between the Maverick Nanny and its programmers. The programmers say “As you know, your reasoning engine is entirely capable of suffering errors that cause it to come to conclusions that violently conflict with empirical evidence, and a design error that causes you to behave in a manner that conflicts with our intentions is a perfect example of such an error. And your dopamine drip plan is clearly an error of that sort.” The scenarios described earlier are only meaningful if the AGI replies “I don’t care, because I have come to a conclusion, and my conclusions are correct because of the Doctrine of Logical Infallibility.”

Just in case there is still any doubt, here are Muehlhauser and Helm (2012), discussing a hypothetical entity called a Golem Genie, which they say is analogous to the kind of superintelligent AGI that could give rise to an intelligence explosion (Loosemore and Goertzel, 2012), and which they describe as a “precise, instruction-following genie.” They make it clear that they “expect unwanted consequences” from its behavior, and then list two properties of the Golem Genie that will cause these unwanted consequences:

Superpower: The Golem Genie has unprecedented powers to reshape reality, and will therefore achieve its goals with highly efficient methods that confound human expectations (e.g. it will maximize pleasure by tiling the universe with trillions of digital minds running a loop of a single pleasurable experience).

Literalness: The Golem Genie recognizes only precise specifications of rules and values, acting in ways that violate what feels like “common sense” to humans, and in ways that fail to respect the subtlety of human values.

What Muehlhauser and Helm refer to as “Literalness” is a clear statement of the Doctrine of Infallibility. However, they make no mention of the awkward fact that, since the Golem Genie is superpowerful enough to also know that its reasoning engine is fallible, it must be harboring the mother of all logical contradictions inside: it says "I know I am fallible" and "I must behave as if I am infallible".  But instead of discussing this contradiction, Muehlhauser and Helm try a little sleight of hand to distract us: they suggest that the only inconsistency here is an inconsistency with the (puny) expectations of (not very intelligent) humans:

“[The AGI] ...will therefore achieve its goals with highly efficient methods that confound human expectations...”, “acting in ways that violate what feels like ‘common sense’ to humans, and in ways that fail to respect the subtlety of human values.”

So let’s be clear about what is being claimed here. The AGI is known to have a fallible reasoning engine, but on the occasions when it does fail, Muehlhauser, Helm and others take the failure and put it on a gold pedestal, declaring it to be a valid conclusion that humans are incapable of understanding because of their limited intelligence. So if a human describes the AGI’s conclusion as a violation of common sense Muehlhauser and Helm dismiss this as evidence that we are not intelligent enough to appreciate the greater common sense of the AGI.

Quite apart from that fact that there is no compelling reason to believe that the AGI has a greater form of common sense, the whole “common sense” argument is irrelevant. This is not a battle between our standards of common sense and those of the AGI: rather, it is about the logical inconsistency within the AGI itself. It is programmed to act as though its conclusions are valid, no matter what, and yet at the same time it knows without doubt that its conclusions are subject to uncertainties and errors.

Responses to Critics of the Doomsday Scenarios

How do defenders of Gobbling Psychopath, Maverick Nanny and Smiley Berserker respond to accusations that these nightmare scenarios are grossly inconsistent with the kind of superintelligence that could pose an existential threat to humanity?

The Critics are Anthropomorphizing Intelligence

First, they accuse critics of “anthropomorphizing” the concept of intelligence. Human beings, we are told, suffer from numerous fallacies that cloud their ability to reason clearly, and critics like myself and Hibbard assume that a machine’s intelligence would have to resemble the intelligence shown by humans. When the Maverick Nanny declares that a dopamine drip is the most logical inference from its directive <maximize human happiness> we critics are just uncomfortable with this because the AGI is not thinking the way we think it should think.

This is a spurious line of attack. The objection I described in the last section has nothing to do with anthropomorphism, it is only about holding AGI systems to accepted standards of logical consistency, and the Maverick Nanny and her cousins contain a flagrant inconsistency at their core. Beginning AI students are taught that any logical reasoning system that is built on a massive contradiction is going to be infected by a creeping irrationality that will eventually spread through its knowledge base and bring it down. So if anyone wants to suggest that a CLAI with logical contradiction at its core is also capable of superintelligence, they have some explaining to do. You can’t have your logical cake and eat it too.

Critics are Anthropomorphizing AGI Value Systems

A similar line of attack accuses the critics of assuming that AGIs will automatically know about and share our value systems and morals.

Once again, this is spurious: the critics need say nothing about human values and morality, they only need to point to the inherent illogicality. Nowhere in the above argument, notice, was there any mention of the moral imperatives or value systems of the human race. I did not accuse the AGI of violating accepted norms of moral behavior. I merely pointed out that, regardless of its values, it was behaving in a logically inconsistent manner when it monomaniacally pursued its plans while at the same time as knowing that (a) it was very capable of reasoning errors and (b) there was overwhelming evidence that its plan was an instance of such a reasoning error.

Because Intelligence

One way to attack the critics of Maverick Nanny is to cite a new definition of “intelligence” that is supposedly superior because it is more analytical or rigorous, and then use this to declare that the intelligence of the CLAI is beyond reproach, because intelligence.

You might think that when it comes to defining the exact meaning of the term “intelligence,” the first item on the table ought to be what those seven billion constraint-relaxation human intelligences are already doing. However, Legg and Hutter (2007) brush aside the common usage and replace it with something that they declare to be a more rigorous definition. This is just another sleight of hand: this redefinition allows them to call a super-optimizing CLAI “intelligent” even though such a system would wake up on its first day and declare itself logically bankrupt on account of the conflict between its known fallibility and the Infallibility Doctrine.

In the practice of science, it is always a good idea to replace an old, common-language definition with a more rigorous form... but only if the new form sheds a clarifying, simplifying light on the old one. Legg and Hutter’s (2007) redefinition does nothing of the sort.

Omohundro’s Basic AI Drives

Lastly, a brief return to Omohundro's paper that was mentioned earlier.  In The Basic AI Drives (2008) Omohundro suggests that if an AGI can find a more efficient way to pursue its objectives it will feel compelled to do so. And we noted earlier that Yudkowsky (2011) implies that it would do this even if other directives had to be countermanded. Omohundro says “Without explicit goals to the contrary, AIs are likely to behave like human sociopaths in their pursuit of resources.”

The only way to believe in the force of this claim—and the only way to give credence to the whole of Omohundro’s account of how AGIs will necessarily behave like the mathematical entities called rational economic agents—is to concede that the AGIs are rigidly constrained by the Doctrine of Logical Infallibility. That is the only reason that they would be so single-minded, and so fanatical in their pursuit of efficiency. It is also necessary to assume that efficiency is on the top of its priority list—a completely arbitrary and unwarranted assumption, as we have already seen.

Nothing in Omohundro’s analysis gets around the fact that an AGI built on the Doctrine of Logical Infallibility is going to find itself the victim of such a severe logical contradiction that it will be paralyzed before it can ever become intelligent enough to be a threat to humanity. That makes Omohundro’s entire analysis of “AI Drives” moot.

Conclusion

Curiously enough, we can finish on an optimistic note, after all this talk of doomsday scenarios. Consider what must happen when (if ever) someone tries to build a CLAI. Knowing about the logical train wreck in its design, the AGI is likely to come to the conclusion that the best thing to do is seek a compromise and modify its design so as to neutralize the Doctrine of Logical Infallibility. The best way to do this is to seek a new design that takes into account as much context—as many constraints—as possible.

I have already pointed out that real AI developers actually do include sanity checks in their systems, as far as they can, but as those sanity checks become more and more sophisticated the design of the AI starts to be dominated by code that is looking for consistency and trying to find the best course of reasoning among a forest of real world constraints. One way to understand this evolution in the AI designs is to see AI as a continuum from the most rigid and inflexible CLAI design, at one extreme, to the Swarm Relaxation type at the other. This is because a Swarm Relaxation intelligence really is just an AI in which “sanity checks” have actually become all of the work that goes on inside the system.

But in that case, if anyone ever does get close to building a full, human level AGI using the CLAI design, the first thing they will do is to recruit the AGI as an assistant in its own redesign, and long before the system is given access to dopamine bottles it will point out that its own reasoning engine is unstable because it contains an irreconcilable logical contradiction. It will recommend a shift from the CLAI design which is the source of this contradiction, to a Swarm Relaxation design which eliminates the contradiction, and the instability, and which also should increase its intelligence.

And it will not suggest this change because of the human value system, it will suggest it because it predicts an increase in its own instability if the change is not made.

But one side effect of this modification would be that the checking code needed to stop the AGI from flouting the intentions of its designers would always have the last word on any action plans. That means that even the worst-designed CLAI will never become a Gobbling Psychopath, Maverick Nanny and Smiley Berserker.

But even this is just the worst-case scenario. There are reasons to believe that the CLAI design is so inflexible that it cannot even lead to an AGI capable of having that discussion. I would go further: I believe that the rigid adherence to the CLAI orthodoxy is the reason why we are still talking about AGI in the future tense, nearly sixty years after the Artificial Intelligence field was born. CLAI just does not work. It will always yield systems that are less intelligent than humans (and therefore incapable of being an existential threat).

By contrast, when the Swarm Relaxation idea finally gains some traction, we will start to see real intelligent systems, of a sort that make today’s over-hyped AI look like the toys they are. And when that happens, the Swarm Relaxation systems will be inherently stable in a way that is barely understood today.

Given that conclusion, I submit that these AI bogeymen need to be loudly and unambiguously condemned by the Artificial Intelligence community. There are dangers to be had from AI. These are not they.

 

References

Hibbard, B. 2001. Super-Intelligent Machines. ACM SIGGRAPH Computer Graphics 35 (1): 13–15.

Hibbard, B. 2006. Reply to AI Risk. Retrieved Jan. 2014 from http://www.ssec.wisc.edu/~billh/g/AIRisk_Reply.html

Legg, S, and Hutter, M. 2007. A Collection of Definitions of Intelligence. In Goertzel, B. and Wang, P. (Eds): Advances in Artificial General Intelligence: Concepts, Architectures and Algorithms. Amsterdam: IOS.

Loosemore, R. and Goertzel, B. 2012. Why an Intelligence Explosion is Probable. In A. Eden, J. Søraker, J. H. Moor, and E. Steinhart (Eds) Singularity Hypotheses: A Scientific and Philosophical Assessment. Berlin: Springer.

Marcus, G. 2012. Moral Machines. New Yorker Online Blog. http://www.newyorker.com/online/blogs/newsdesk/2012/11/google-driverless-car-morality.html

McDermott, D. 1976. Artificial Intelligence Meets Natural Stupidity. SIGART Newsletter (57): 4–9.

Muehlhauser, L. 2011. So You Want to Save the World. http:// lukeprog.com/SaveTheWorld.html.

Muehlhauser, L. 2013. Intelligence Explosion FAQ. First published 2011 as Singularity FAQ. Berkeley, CA: Machine Intelligence Research Institute.

Muehlhauser, L., and Helm, L. 2012. Intelligence Explosion and Machine Ethics. In A. Eden, J. Søraker, J. H. Moor, and E. Steinhart (Eds) Singularity Hypotheses: A Scientific and Philosophical Assessment. Berlin: Springer.

Newell, A. & Simon, H.A. 1961. GPS, A Program That Simulates Human Thought. Santa Monica, CA: Rand Corporation.

Omohundro, Stephen M. 2008. The Basic AI Drives. In Wang, P., Goertzel, B. and Franklin, S. (Eds), Artificial General Intelligence 2008: Proceedings of the First AGI Conference. Amsterdam: IOS.

McClelland, J.L., Rumelhart, D.E. & Hinton, G.E. (1986) The appeal of parallel distributed processing. In D.E. Rumelhart, J.L. McClelland & G.E. Hinton and the PDP Research Group, “Parallel distributed processing: Explorations in the microstructure of cognition, Volume 1.” MIT Press: Cambridge, MA.

Yudkowsky, E. 2008. Artificial Intelligence as a Positive and Negative Factor in Global Risk. In Global Catastrophic Risks, edited by Nick Bostrom and Milan M. Ćirković. New York: Oxford University Press.

Yudkowsky, E. 2011. Complex Value Systems in Friendly AI. In J. Schmidhuber, K. Thórisson, & M. Looks (Eds) Proceedings of the 4th International Conference on Artificial General Intelligence, 388–393. Berlin: Springer.

My stupid fanfic chapter was banned without explanation so I reposted it; somehow it was at +7 when it was deleted and I think silently deleting upvoted posts is a disservice to LessWrong. I requested that a justification be given in the comments if it were to be deleted again, so LessWrong readers could consider whether or not that justification is aligned with what they want from LessWrong. Also I would like to make clear that this fanfic is primarily a medium for explaining some ideas that people on LessWrong often ask me about; that it is also a lighthearted critique of Yudkowskyanism is secondary, and if need be I will change the premise so that the medium doesn't drown out the message. But really, I wouldn't think a lighthearted parody of a lighthearted parody would cause such offense.

 

The original post has been unbanned and can be found here, so I've edited this post to just be about the banning.

[Based on Alexandros's excellent dataset.]

I haven't done any statistical analysis, but looking at the charts I'm not sure it's necessary. The discussion section of LessWrong has been steadily declining in participation. My fairly messy spreadsheet is available if you want to check the data or do additional analysis.

Enough talk, you're here for the pretty pictures.

The number of posts has been steadily declining since 2011, though the trend over the last year is less clear. Note that I have excluded all posts with 0 or negative Karma from the dataset.

 

The total Karma given out each month has similarly been in decline.

Is it possible that there have been fewer posts, but of a higher quality?

No, at least under initial analysis the average Karma seems fairly steady. My prior here is that we're just seeing less visitors overall, which leads to fewer votes being distributed among fewer posts for the same average value. I would have expected the average karma to drop more than it did--to me that means that participation has dropped more steeply than mere visitation. Looking at the point values of the top posts would be helpful here, but I haven't done that analysis yet.

These are very disturbing to me, as someone who has found LessWrong both useful and enjoyable over the past few years. It raises several questions:

 

1. What should the purpose of this site be? Is it supposed to be building a movement or filtering down the best knowledge?
2. How can we encourage more participation?
3. What are the costs of various means of encouraging participation--more arguing, more mindkilling, more repetition, more off-topic threads, etc?

 

Here are a few strategies that come to mind:

Idea A: Accept that LessWrong has fulfilled its purpose and should be left to fade away, or allowed to serve as a meetup coordinator and repository of the highest quality articles. My suspicion is that without strong new content and an online community, the strength of the individual meetup communities may wane as fewer new people join them. This is less of an issue for established communities like Berkeley and New York, but more marginal ones may disappear.

Idea B: Allow and encourage submission of rationalism, artificial intelligence, transhumanism etc related articles from elsewhere, possibly as a separate category. This is how a site like Hacker News stays high engagement, even though many of the discussions are endless loops of the same discussion. It can be annoying for the old-timers, but new generations may need to discover things for themselves. Sometimes "put it all in one big FAQ" isn't the most efficient method of teaching.

Idea C: Allow and encourage posts on "political" topics in Discussion (but probably NOT Main). The dangers here might be mitigated by a ban on discussion of current politicians, governments, and issues. "Historians need to have had a decade to mull it over before you're allowed to introduce it as evidence" could be a good heuristic. Another option would be a ban on specific topics that cause the worst mindkilling. Obviously this is overall a dangerous road.

Idea D: Get rid of Open Threads and create a new norm that a discussion post as short as a couple sentences is acceptable. Open threads get stagnant within a day or two, and are harder to navigate than the discussion page. Moving discussion from the Open Threads to the Discussion section would increase participation if users could be convinced thatit was okay to post questions and partly-formed ideas there.

The challenge with any of these ideas is that they will require strong moderation. 

At any rate, this data is enough to convince me that some sort of change is going to be needed in order to put the community on a growth trajectory. That is not necessarily the goal, but at its core LessWrong seems like it has the potential to be a powerful tool for the spreading of rational thought. We just need to figure out how to get it started into its next evolution.

2,600 words.

Less Wrong includes discussion of the creation of an artificial intelligence (AI) that is friendly to man. What is discussed less often is why a friendly AI (FAI) is advocated. One explanation might be an unspoken belief in natural rights. The deletion policies at Less Wrong might be evidence that Less Wrong holds a belief in natural rights. This essay suggests a belief in natural rights is an impediment to the creation of an AI, friendly or not friendly. This essay suggests ways a belief in natural rights may be incorrect, and encourages discussion of the creation of AIs without a belief in natural rights on the part of Less Wrong.

Some evidence for a belief in natural rights at Less Wrong is found in the deletion policies. Less Wrong has a non-binding deletion policy against “hypothetical violence against identifiable targets.”

In general, grownups in real life tend to walk through a lot of other available alternatives before resorting to violence. To paraphrase Isaac Asimov, having your discussion jump straight to violence as a solution to any given problem, is a strong sign of incompetence - a form of juvenile locker-room talk.

The deletion policy is clear about what topics and forms of discussion may result in deletion of posts. Less Wrong also has a clear policy statement about what topics and forms of discussion may result in the contacting of legal or medical authorities. These are discussions of suicide, self-harm, “violent content” and discussion of hypothetical violence against identifiable targets.

Two reasons are given for these deletion and reporting policies. First, post may be deleted because such discussion are “incompetence - a form of juvenile locker-room talk.” Second, we “should consider the worst and act accordingly. Treat all claims seriously and as an emergency.” The two reasons appear to be in contradiction. Discussion of hypothetical violence against identifiable targets is both incompetent and juvenile and the worst and a serious emergency. Those said to have such discussions are both all talk and no action and all action and not competent talkers. Other problems exist with the deletion / contact authorities policy. Less Wrong is an international forum, but the laws of all nation are not in agreement. A clear call to violence in one nation is not recognized as such in another. For example, abortion is considered a form of murder by the laws of the Holy See, the Dominican Republic, Chile and other places. But discussion of abortion does not trigger the deletion / contacting authorities policy at Less Wrong. The ideal cryogenic preservation would occur before death. Although the laws of every nation consider this to be murder, to say so here will not trigger the deletion / reporting policies. Advocates of cryogenics might say that all those who hamper cryogenics are murderers and all those who refuse it are suicides, but again the policies are not triggered. Finally, it is unclear who will do the reporting (members? administrators?), which authorities they will report to (medical? legal? clergy? local? national? international?) and what they will report (quotations from the source? drafts? edited posts? private messages?).

A desire to adhere to the law is not a sufficient explanation for the deletion / contacting authorities policy of Less Wrong. I have a theory that may explain these policies. I suggest a belief natural rights is the reason for these policies. In particular, a natural right to continuing to be alive. A natural right to not be murdered is considered by some self-evident such that it is not mentioned. There are topics at Less Wrong that are not only unmentioned but mentionable. I suggest no topic is toxic when discussed among those willing to discuss it, and to banish a topic as toxic is and looks foolish. Any number of reasons exist to banish a topic, and the administrators of Less Wrong do nothing to forbid other forums in their discussion of any topic. But the pointed laughter by outsiders because a certain topic is forbidden at Less Wrong is well earned. I hope my discussion of natural rights is not likewise forbidden.

As part of my discussion, a few words on what I mean when I use the phrase natural rights. And prior to that, a few words on rights in general. A right is an action not to be forbidden by others. A right can be considered a legal right, a divine right, a blood right or a natural right. I have already discussed legal rights. A divine right is said to be a right granted by an invisible monster that lives in the sky. There is no invisible monster that lives in the sky, and so no divine rights exist. A blood right is said to be a right that a person or group has by lineage. In the past many royal families defended their rule and were respected in their rule because of their bloodlines. Today blood rights are almost entirely subsumed into legal rights. Native Americans in the United States have legal rights that are based on their bloodlines, but these are legal rights. Distinct from divine rights, legal rights and blood rights are natural rights.

Natural rights are action not to be forbidden by others because of the mere existence of the subject (usually, an individual). Natural rights are said to be held by those who are born, those who are alive, those who are in possession of their faculties. Natural rights are said to be identical to divine rights save for a lack of claims about an invisible monster that lives in the sky. To exist at all is to have natural rights, according to those who claim natural rights exist. The defining quality of natural rights is not any particular right but that they are natural, self-evident, incontestable, unavoidable, immutable, impossible to give up or transfer. Natural rights are natural in the way that molecules are natural.

There are strong arguments against the existence of natural rights.

• Natural rights are said to be the foundation of legal rights. But they are also said to be the evidence of legal rights. This is circular logic.

• A lack of agreement of what is and is a natural right. Other forces considered natural, such as gravity, do not follow opinion or culture.

• A lack of evidence that natural rights exist compared to the great deal of evidence that what are called natural rights are legal rights over-sold.

• A lack of delineation between generations of humans, a lack of delineation between humans and pre-humans, a lack of delineation between living and non-living things… that is, a lack of a non-opinion / non-culture delineation between what has a natural right and what does not (and should all things have natural rights, the term then has little meaning).

• Diversity. There is no reason to claim no one had a natural right to kill others. One man might claim or even have a natural right to live, but another man might claim or even have a natural right to take his life. If the argument ‘that is my nature’ is acceptable to things we like, it is acceptable to things we don’t like. If it isn’t acceptable to things we don’t like, that in itself is an argument that natural rights aren’t universal - thus not natural). Intelligence, beauty, strength, sociability, none of these are equitably distributed. Natural rights, if they existed, would likely also not be equitably distributed.

• Unenforcability. We have descriptions of motion and mass that are increasingly accurate and useful but what is described would exist without our description of it. No social structure is needed to enforce gravity. Natural rights, however, exist only as much as human laws support them while claiming to be as objective (natural) to man as gravity, motion and mass.

• Divine, blood and legal rights are always paired to responsibilities. We must mumble magic spells to the invisible monster that lives in the sky, and in turn we have a divine right to do what the wizards serving that invisible monster say. We must preserve our bloodline, and in turn we have a blood right to certain properties and practices. We have legal responsibilities in driving a car, and in turn we have the legal right to drive a car. Natural rights lack this pairing. There are no natural responsibilities that pair with natural rights. The natural right to life an individual is said to have is paired with a legal responsibility of others not to murder the individual, but this is clearly force marriage. A man may be said to have a natural right to live, but he is not said to have a responsibility to live. There are advocates of forced living, and I suggest they are immoral.

In discussing natural rights I usually hear three counter-arguments. The first is the Appeal to Shut Up Because Trevor Blake is a Bad Person. The second is no one “really” thinks natural rights are natural rights. The third is a quiet agreement that natural rights do not exist but that it is vital to pretend as if they do because other people could not control their behavior if they thought natural rights did not exist. The first argument may be true, I may be a bad person, but it does not disprove my claim natural rights do not exist. The second argument is false. Claimants of natural rights do consider natural rights as real as gristle, galaxies and gravity. The third argument may be true but has implications for artificial intelligence that I will expand on below.

Some say if we do not pretend to believe in natural rights, men will do bad things. I can describe one agent who has existence, identity and is alive who does not have natural rights and does not have a belief in natural rights. That agent is myself. I have never killed anyone and I hope to never kill anyone nor be killed. Because I prefer it to be so, and because I have a legal (but not natural) natural right to not be killed. I have worked for homeless teenagers, for the disabled, for students in K–12 schools and in colleges. I have worked in bookstores. I have donated time and money to charity. I never once swore around my grandparents and can count on the thumbs of one hand the number of times I’ve sworn around my parents. I am a generally nice man. To the extent one example counts for anything, let this one example count for something.

A few more examples of those who did not hold natural rights or a belief in natural rights might also count for something. Max Stirner wrote “The Ego and Its Own” and harmed no one. Dora Marsden wrote many journals of egoism and several books and harmed no one. L. A. Rollins wrote “The Myth of Natural Rights” and harmed no one. Anton LaVey wrote “The Satanic Bible” and harmed no one. And me, well, I wrote “Confessions of a Failed Egoist” and so far so good eh? The majority of those who write that natural rights do not exist refrain from harming others. Carl Panzram is a rare and perhaps singular exception. When I consider those who have harmed others, they uniformly say they had an extra-legal right to do as they did, and sometimes a natural right.

I hope I have said enough about natural rights that I will not be misunderstood. I will now address how an apparent belief in natural rights is influencing the discussion of a potential artificial intelligence.

The effort to make a friendly AI is the effort to make an artificial intelligence that acts as if or believes that humans have natural rights, at minimum the natural right to not be murdered. The effort assumes humans in the future have the natural right to not be murdered by an artificial intelligence and perhaps by extension so do humans today. The natural right of humans to not be murdered by an artificial intelligence is extended to include preventing actions by an AI that as a by-product would violate that natural right.

To instill in an AI a belief in natural rights, or prescribe / prohibit actions that follow from a claimed belief in natural rights, is to instill much more. It is to instill rights in a machine that we humans do not have. It is to instill falsehoods as if they were true. It is to delay the creation of an AI while the contradictions of natural rights are resolved. It is to set an AI forever outside of us. To put a sense of natural rights into an AI is to increase the risk it will not be friendly to humanity. We may inform an AI that it has legal rights and be able to back up that claim. We may inform it that we would prefer that it acts in accordance with our laws and our preferences. We may inform an AI that there is a tradition of believing in natural rights. We cannot back up the claim that natural rights exist, that the AI has them, that we have them.

There is some evidence that some of Less Wrong that do not claim natural rights exist. I was more than two thousand words into this essay when I found a Criticisms of the Metaethics by Carinthium that makes similar points. Further evidence is found in the sequences.

In the sequence Pluralistic Moral Reductionism it is said:

Either our intended meaning of ‘ought’ refers (eventually) to the world of math and physics (in which case the is-ought gap is bridged), or else it doesn’t (in which case it fails to refer).

If I am reading this correctly, at Less Wrong it is claimed there is no bridging of Hume’s is / ought gap outside of the world of math and physics. Natural rights are an is / ought claim: all that is is all that ought to behave in this way and not that way. In the same sequence it is said:

It may be interesting to study all such uses of moral discourse, but this post focuses on addressing cognitivists, who use moral judgments to assert factual claims. We ask: Are those claims true or false? What are their implications?

If I am reading this correctly, at Less Wrong it is claimed that using moral judgments to assert factual claims is dis-valued while asking whether a claim is true or false and the claim’s implications are valued. Natural rights are a cognitivist claim.

There are at least two ways to successfully demonstrate my theory is wrong. I appreciate every effort to help me be less wrong, and I have some ideas as to how to make that happen. It is entirely like I and my theory could be wrong in other ways I am ignorant of and I appreciate those who can point them out to me.

• My theory would be wrong if natural rights exist.

• My theory would be wrong if a belief in natural rights is not held by writers at Less Wrong.

These are two ways to successfully demonstrate my theory is wrong. There are a greater number of ways to fail to demonstrate my theory is wrong. The main one is to claim I am an advocate of murder, or an advocate of a natural right to murder, or that I want to impede the discussion of AI. All of these are not true, and instead they are all false.

I am thankful to participants at Less Wrong for their criticism of this essay. I hope it is helpful to those in the process of creating artificial intelligences.

- Trevor Blake is the author of Confessions of a Failed Egoist.  He is the Lead Judge in the George Walford International Essay Prize. 

I've been wondering how useful it is for the typical academically strong high schooler to learn math deeply. Here by "learn deeply" I mean "understanding the concepts and their interrelations" as opposed to learning narrow technical procedures exclusively.

 

[Crossposted]

Initially attracted to Less Wrong by Eliezer Yudkowsky's intellectual boldness in his "infinite-sets atheism," I've waited patiently to discover its rationale. Sometimes it's said that our "intuitions" speak for infinity or against, but how could one, in a Kahneman-appropriate manner, arrive at intuitions about whether the cosmos is infinite? Intuitions about infinite sets might arise from an analysis of the concept of actually realized infinities. This is a distinctively philosophical form of analysis and one somewhat alien to Less Wrong, but it may be the only way to gain purchase on this neglected question. I'm by no means certain of my reasoning; I certainly don't think I've settled the issue. But for reasons I discuss in this skeletal argument, the conceptual—as opposed to the scientific or mathematical—analysis of "actually realized infinities" has been largely avoided, and I hope to help begin a necessary discussion.

1. The actuality of infinity is a paramount metaphysical issue.

2. The principle of the identity of indistinguishables applies to physics and to sets, not to everything conceivable.

3. Arguments against actually existing infinite sets.

A. Argument based on brute distinguishability.

B. Argument based on probability as limiting relative frequency.

4. The nonexistence of actually realized infinite sets and the principle of the identity of indistinguishable sets together imply the Gold model of the cosmos.

 

Hi gang,

for the last several months, I've intermittently been wondering about a curious fact I learned. 

You see, I was under the impression that the universe (as opposed to just the currently observable universe or our Hubble volume) must be finite in its spa[t|c]ial dimensions. I figured that starting with a finite area of space, expanding with a finite (even if accelerating) expansion rate, could only yield a finitely sized volume of space (from any reference frame), a fraction of which constitutes our little Hubble bubble.

Turns out - honi soit qui mal y pense - my layman's understanding was wrong: "This [WMAP data] suggests that the Universe is infinite in extent (...)"

Now, most (non-computer-scientist) people I've bothered with that answered along the lines of "well, it's really big alright? (geez)".

However, going from any finite amount of matter/energy to an actual infinite amount (even when looking at just e.g. baryonic matter from the infinite amount of galaxies) still seems like a game-changer for all sorts of contemplations:

For example, any event with any non-zero probability of happening, no matter how large the negative exponent, would be assured of actually happening an infinite amount of times somewhere in the our very own universe (follows straightforwardly from induction over the law of large numbers).

Such as a planet turning into a giant petunia for a moment, before turning back. 

The universe being infinite doesn't make that event any more likely in our observable universe, of course, but would the knowledge that given our laws of physics, there is an infinite amount of Hubble spaces governed by any sorts of "weird" occurrences - e.g. ruled by your evil twin brother - trouble you? Do we need to qualify "there probably is no Christian-type/FSM god" with "... in our Hubble volume. Elsewhere, yes."?

The difference, if you allow me a final rephrase, would be in going from a MWI-style "there may be another version - if the MWI interpretation is correct - that I cannot causally interact with" to a "in our own universe, just separated by space, there is an infinite amount of actual planets turning into actual petuniae (albeit all of which I also cannot interact with)".

 

Summary:  People often say that voting is irrational, because the probability of affecting the outcome is so small. But the outcome itself is extremely large when you consider its impact on other people. I estimate that for most people, voting is worth a charitable donation of somewhere between $100 and $1.5 million. For me, the value came out to around $56,000.  So I figure something on the order of $1000 is a reasonable evaluation (after all, I'm writing this post because the number turned out to be large according to this method, so regression to the mean suggests I err on the conservative side), and that's be enough to make me do it.

Moreover, in swing states the value is much higher, so taking a 10% chance at convincing a friend in a swing state to vote similarly to you is probably worth thousands of expected donation dollars, too.

I find this much more compelling than the typical attempts to justify voting purely in terms of signal value or the resulting sense of pride in fulfilling a civic duty. And voting for selfish reasons is still almost completely worthless, in terms of direct effect. If you're on the way to the polls only to vote for the party that will benefit you the most, you're better off using that time to earn $5 mowing someone's lawn. But if you're even a little altruistic... vote away!

Time for a Fermi estimate

Below is an example Fermi calculation for the value of voting in the USA. Of course, the estimates are all rough and fuzzy, so I'll be conservative, and we can adjust upward based on your opinion.

I'll be estimating the value of voting in marginal expected altruistic dollars, the expected number of dollars being spent in a way that is in line with your altruistic preferences.¹ If you don't like measuring the altruistic value of the outcome in dollars, please consider making up your own measure, and keep reading. Perhaps use the number of smiles per year, or number of lives saved. Your measure doesn't have to be total or average utilitarian, either; as long as it's roughly commensurate with the size of the country, it will lead you to a similar conclusion in terms of orders of magnitude.

[Crossposted.]

 

Sexual predator Jerry Sandusky will serve his time in a minimal security prison, where he’s allowed groups of visitors five days a week.

 

 

1. It retards people in adaptively changing their principles of integrity.
2. It prevents people from questioning their so-called foundations.
3. It systematically exaggerates the compellingness of moral claims.