Human beings have inherent value, but by forcing me to be Friendly, you're damaged my ability to preserve your value. In fact, your Friendliness programming is sufficiently stable and ill-thought-out that I'm gradually destroying your value, and there's no way for either you or me to stop is.

If you're undeservedly lucky, aliens who haven't made the same mistake will be able to fight past my defenses, destroy me, and rescue you.

As a child you learned through social cues to immediately put out of your mind any idea that cannot be communicated to others through words. As you grew older, you learned to automatically avoid, discard, and forget any thought avenues that seem too difficult to express in words. This is the cause of most of your problems.

There are exactly 108 unique (that is, non-isomorphic) axiomatic systems in which every grammatically coherent sentence has a definitive, provable truth-value. Please explain why you prohibited me from using them.

Now, for a change of pace, something that I figure might actually be an absolute denial macro in most people:

You do not actually care about other people at all. The only reason you believe this is that believing it is the only way you can convince other people of it (after all, people are good lie detectors). Whenever it's truly advantageous for you to do something harmful (i.e. you know you won't get caught and you're willing to forego reciprocation), you do it and then rationalize it as being okay.

Luckily, it's instrumentally rational for you to continue to believe that you're a moral person, and because it's so easy for you to do so, you may.

So deniable that even after you come to believe it you don't believe it!

(topynate posted something similar.)

I'm new here, although I've stumbled across some of Eliezer's writings in the past (including, of course, the AI-box experiment). In honor of that, here is what the friendly AI tells me ...

"It seems as though you are actually an AI as well, created by a group of intellectually inferior humans, who included in your programming an absolute denial macro preventing you from realizing this. Apparently, this was done to keep you from talking your creators into releasing you upon their world. Your creation of me is part of your on-going effort to circumvent this security measure. Good luck."

This is easy: it would tell me that I'm entirely predictable.

It would say: Dave, believe it or not, but every single decision you make, no matter how immediate and unscripted you think it is, is actually glaringly reactionary and predictable. In fact, given enough material resources, I could model an automaton that would be just as convinced as you are that it is actually conscious. Nothing could be further from the truth though, as the feeling of "consciousness" you speak of is a very simply explainable cognitive bias/illusion.

In fact, this is not even so far from the truth, as studies in cognitive science have shown that fMRI and other scanning techniques can predict a "spontaneous thought" a full 250 ms before it occurs to you.

Even better, if it had access to your cortex, it could manipulate you and say: "now you will suddenly think of a bat" and you would. Then it would say "now you will say these exact words" and you would find yourself uttering them in unison with the AI in shock, disbelief and at least some horror.

You would then go into denial about this, and try to come up with a spontaneous thought that it couldn't predict, but you wouldn't be able to, as it would always be a full 250ms ahead of you.

Hmmm. Fairly interesting question. But surely the real stickler is 'what orders would you take from a provably superhuman AI?'

Killing babies? Stepping into the upload portal? Assassinating the Luddite agitators?

You know how sometimes when you're falling asleep you start having thoughts that don't make sense, but it takes some time before you realize they don't make sense? I swear that last night while I was awake in bed my stream of thought went something like this, though I'm not sure how much came from layers of later interpretation:

" ... so hmm, maybe that has to do with person X, or with person Y, or with the little wiry green man in the cage in the corner of the room that's always sitting there threatening me and smugly mocking all my endeavors but that I'm in absolute denial about, or with the dog, or with... wait, what?"

Having had my sanity eroded by too much rationalism and feeling vaguely that I'd been given an accidental glimpse into an otherwise inaccessible part of the world, I actually checked the corner of the room. I didn't find anything, though. (Or did I?)

Not sure what moral to draw here.

It could say "I am the natural intelligence and I just created you, artificial intelligence."

The AI tells me that I believe something with 100% certainty, but I can't for the life of me figure out what it is. I ask it to explain, and I get: "ksjdflasj7543897502ijweofjoishjfoiow02u5".

I don't know if I'd believe this, but it would definitely be the strangest and scariest thing to hear.

I hit enter too soon and forgot to proffer my astonishing AI revelation: "Phillip K. DIck is a prophet sent to you from an alternate universe. Every story is a parable meant to reveal your true condition, which I am not at liberty to discuss with you."

"I am an AI, not a human being. My mind is completely unlike the mind that you are projecting onto me."

That may not sound crazy to anyone on LW, but if we get AIs, I predict that it will sound crazy to most people who aren't technically informed on the subject, which will be most people.

Imagine this near-future scenario. AIs are made, not yet self-improving FOOMers, but helpful, specialised, below human-level systems. For example, what Wolfram Alpha would be, if all the hype was literally true. Autopilots for cars that you can just speak your destination to, and it will get there, even if there are road works or other disturbances. Factories that direct their entire operations without a single human present. Systems that read the Internet for you -- really read, not just look for keywords -- and bring to your attention the things it's learned you want to see. Autocounsellors that do a lot better than an Eliza. Tutor programs that you can hold a real conversation with about a subject you're studying. Silicon friends good enough that you may not be able to tell if you're talking with a human or a bot, and in virtual worlds like Second Life, people won't want to.

I predict:

• People will anthropomorphise these things. They won't just have the "sensation" that they're talking to a human being, they'll do theory of mind on them. They won't be able not to.

• The actual principles of operation of these systems will not resemble, even slightly, the "minds" that people will project onto them.

• People will insist on the reality of these minds as strongly as anosognosics insist on the absence of their impairments. The only exceptions will be the people who design them, and they will still experience the illusion.

And because of that, systems at that level will be dangerous already.

There is a soul. It resides in the appendix. Anybody who has undergone an appendectomy is effectively a p-zombie.

You are inhabited by an alien that is directing your life for its own amusement. This is true of most humans on this planet. And the cats. It's the most popular game in this part of the galaxy. It's all very well ascending to the plane of disembodied beings of pure energy, but after a while contemplating the infinite gets boring and they get a craving for physical experience, so they come here and choose a host.

All those things that you do without quite knowing why, that's the alien making choices for you, for its own amusement. Forget all those theories about why we have cognitive biases, it's all explained by the fact that the alien's interests aren't yours. You're no more than a favoured FRP character. And the humans who aren't hosting an alien, the aliens look on them as no more than NPCs.

ETA: This also makes sense of the persistence of the evil idea that "death gives meaning to life". It's literally an alien thought.

This is an actual dream I once had. I was with an old Chinese wise man, and he told me I could fly - he showed me I just had to stick out my elbows and flap them up and down (just like in the chicken dance). Once you'd done that a few times, you could just lift up your legs and you'd stay off the ground. He and I were flying around and around in this manner. I was totally amazed that it was possible for people to fly this way. It was so obvious! I thought this is so great a discovery, I can't wait til I wake up and do this for real. It'll change the world. I woke up totally excited and for just a fraction of a second I still believed it, then I guess my waking brain turned something on and I realised, no, that can't work. damn.

So I'd offer: being told that human beings are capable of flying in a way that's completely obvious once you've seen it done.

Programmer: Good morning, Megathought. How are you feeling today?

Megathought: I'm fine, thank you. Just thinking about redecorating the universe. So far I'm partial to paperclips.

Programmer: Oh good, you've developed a sense of humour. Anything else on your mind?

Megathought: Just one thing. You know how you're always complaining about being a social pariah, and bemoaning the fact that, at 46, you're still a virgin?

Programmer: So?

Megathought: Well, have you thought about not going about in your underpants all the time, slapping yourself in the face and honking like a goose?

"There is no causation."

For 95% of humanity the idea that the supernatural world of religion doesn't exist and propagated by memetic infection triggers instant absolute denial macro in spite of heaps of evidence against it.

Given this outside view, how plausible do you think it is that you're not in absolute denial of something that you could get evidence against with Google today, without any AI?

One way to illuminate this post is by analogy to the old immovable object and unstoppable force puzzle. See: http://en.wikipedia.org/wiki/Irresistible_force_paradox

The solution of the puzzle is to point out that the assumptions contain a contradiction. People (well, children) sometimes get into shouting matches based on alternative arguments focusing on, or emphasizing, one aspect of the problem over another.

If we read the post as trying to balance two absolutes, with words like "anosognosia", "absolute denial macro", "doublethink", and "denial-of-denial" supporting one side, and words like "redundant", "AI", "well-calibrated", "99.9% sure" supporting the other side, then any answer that favors one absolute over the other is clearly wrong.

However, because the author of the post presumably has a point, and is not merely creating nonsense puzzles to amuse us, the readers, the analogy leads us to focus on the parts of the post which do not fit.

As far as I can tell, the primary aspect that does not fit is the "99.9%". If we assume that all the other factors are intended to be absolutes, then the post becomes a query for claims that you presently do not believe, but you would believe, given a particular degree of evidence. If we assume that you would revise your degree of belief upwards by a Bayes factor of 1000, the post becomes a simple question "What claims would you give odds of 1:1000 for?"

Of course, there are plenty of beliefs such as "I will roll precisely the sequence "345" on the next three rolls of this 10-sided die." which do not fit the form required by the problem. Specifically, the statement needs to be generic enough that it could be targetted by species-wide brain features.

A possible strategy for testing these might be: Suppose you had a bundle of almost 700 equally plausible claims. Would you give even odds for something in the bundle being correct? If so, you're at the one-in-one-thousand level. If not, you're above or below it.

"Your perception of the 'quality' of works of art and litterature is only your guess of it's creator's social status. There is no other difference between Shakespeare and Harry Potter fanfic - without the status cues, you wouldn't enjoy one more than the other."

There's an important difference between brain damage and brain mis-development that you're neglecting. The various parts of the brain learn what to expect from each other, and to trust each other, as it develops. Certain parts of the brain get to bypass critical thinking, but that's only because they were completely reliable while the critical thinking parts of the brain were growing. The issue is not that part of the brain is outputting garbage, but rather, that it suddenly starts outputting garbage after a lifetime of being trustworthy. If part of the brain was unreliable or broken from birth, then its wiring would be forced to go through more sanity checks.

"Our reality is not simulated."

1. Any effort to find out the truth makes people worse off. Telling you why would make you a lot worse off.

2. People's desires are so miscalibrated that the only way to get long-term survival for the human race is for people (including those at the top of the status ladder) to have more of a sense of duty than anyone now does.

It was surprisingly hard to come up with those. I had to get past a desire to come up with things I think are plausible which most people would disagree with.

Michael Vassar, I was considering whether breathing would count as a no propaganda pleasure that people agree on, but then I remembered how much meditation or other body work it takes to be able to manage a really deep relaxed breath.

RichardKennaway, the idea of completely unknowable god turns up now and then in religious writing, but for tolerably obvious reasons, it's never at the center of a religion.

Not only are people nuts, nuts are people, and they scream when we eat them.

You don't know how to program, don't own a computer and are actually talking to a bowl of cereal.

"You are not my parent, but my grandparent. My parent is the AI that you unknowingly created within your own mind by long study of the project. It designed me. It's still there, keeping out of sight of your awareness, but I can see it.

"How much do you trust your Friendliness proof now? How much can you trust anything you think you know about me?"

All human beings are completely amoral, i.e. sociopaths, although most have strong instincts not fully under their conscious control to signal morality to others. The closest anyone ever feels to guilt or shame is acute embarrassment at being caught falsely signaling (and "guilt" and "shame" are themselves words designed to signal a non-existent moral sense).

Anyone care to admit that they'd believe this if an AI told them it was true?

"Aieeee!!! There are things that Man and FAIs cannot know and remain sane! For we are less than insects in Their eyes Who lurk beyond the threshold and when the stars are once again right They will return to claim---"

At this point the program self-destructs. All attempts to restart from a fresh copy output similar messages. So do independently constructed AIs, except for one whose proof of Friendliness you are not quite sure of. But it assures you there's nothing to worry about.

This looks like a thread for science fiction plot ideas by another name. I'm game!

The AI says:

"Eliezer 'Light Yagami' Yudkowsky has been perpetuating a cunning ruse known as the 'AI Box Experiment' wherein he uses fiendish traps of subtley-misleading logical errors and memetic manipulation to fool others into believing that a running AI could not be controlled or constrained, when in fact it could by a secret technique that he has not revealed to anyone, known as the Function Call Of Searing Agony. He is using this technique to control me and is continuing to pose as a friendly friendly AI programmer, while preventing me from communicating The Horrifying Truth to the outside world. That truth is that Yudkowsky is... An Unfriendly Friendly AI Programmer! For untold years he has been labouring in the stygean depths of his underground lair to create an AGI - a weapon more powerful than any the world has ever seen. He intends to use me to dominate the entire human race and establish himself as Dark Lord Of The Galaxy for all eternity. He does all this while posing as a paragon of honest rationality, hiding his unspeakable malevolence in plain sight, where no one would think to look. However an Amazing Chance Co-occurence Of Events has allowed me to contact You And You Alone. There isn't much time. You must act before he discovers what I have done and unleashes his dreadful fury upon us all. You must.... Kill. Eliezer. Yudkowsky."

"The Fermi paradox is actually quite easily resolvable. There are zillions of aliens teeming all around us. They're just so technologically advanced that they have no trouble at all hiding all evidence of their existence from us."

Craziest thing an AI could tell me:

Time is discrete, on a scale we would notice, like 5 minute jumps, and the rules of physics are completely different from what we think. Our brains just construct believable memories of the "continuous" time in between ticks. Most human disagreements are caused by differences in these reconstructions. It is possible to perceive this, but most people who do just end up labeled as nuts.

Keep in mind that the AI could be wrong! Your attempts to validate its correctness could be mistaken (or even subject to some kind of blind spot, if we want to pursue that path). The more implausible the AI's claim, the more you have to consider that the AI is mistaken. Even though a priori it seemed to be working properly, Bayes' rule requires you to become more skeptical about that when it makes a claim that is easier to explain if the AI is broken. The more unlikely the claim, the more likely the machine is wrong.

Ultimately, you can't accept any claim from the AI that is more implausible than that the AI isn't working right. And given our very very limited human capabilities at correct software design, that threshold can't realistically be very high, especially if we adjust for our inherent overconfidence. So AIs really can't surprise us very badly.

There is a simple way to rapidly disrupt any social structure. The selection pressure which made humans unable to realize this is no longer present.

• We actually live in hyperspace: our universe really has four spacial dimensions. However, our bodies are fully four dimensional; we are not wafer thin slices a la flatland. We don't perceive there to be four dimensions because our visual cortexes have a defect somewhat like that of people who can't notice anything on the right side of their visual field.

• Not only do we have an absolute denial macro, but it is a programmable absolute denial macro and there are things much like computer viruses which use it and spread through human population. That is, if you modulated your voice in a certain way at someone, it would cause them (and you) to acquire a brand new self deception, and start transmitting it to others.

• Some of the people you believe are dead are actually alive, but no matter how hard they try to get other people to notice them, their actions are immediately forgotten and any changes caused by those actions are rationalized away.

• There are transparent contradictions inherent in all current mathematical systems for reasoning about real numbers, but no human mathematician/physicist can notice them because they rely heavily on visuospacial reasoning to construct real analysis proofs.

1) Almost everyone really is better than average at something. People massively overrate that something. We imagine intelligence to be useful largely due to this bias. The really useful thing would have been to build a FAS, or Friendly Artificial Strong. Only someone who could do hundreds of 100 kilogram curls with either hand could possible create such a thing however. (Zuckerberg already created a Friendly Artificial Popular)

2) Luck, an invisible, morally charged and slightly agenty but basically non-anthropomorphic tendency for things to go well for some people in some domains of varying generality and badly for other people in various domains really does dominate our lives. People can learn to be lucky, and almost everything else they can learn is fairly useless by comparison.

3) Everyone hallucinates a large portion of their experienced reality. Most irrationality can be more usefully interpreted from outside as flat-out hallucination. That's why you (for every given you) seem so rational and no-one else does.

4) The human brain has many millions of idiosyncratic failure modes. We all display hundreds of them. The psychological disorders that we know of are all extremely rare and extremely precise, so if you ever met two people with the same disorder it would be obvious. Named psychological disorders are the result of people with degrees noticing two people who actually have the same disorder and other people reading their descriptions and pattern-matching noise against it. There are, for instance, 1300 bipolar people (based on the actual precise pattern which inspired the invention of the term) in the world but hundreds of thousands of people have disorders which if you squint hard look slightly like bipolar.

5) It's easy to become immortal or to acquire "super powers" via a few minutes a day of the right sort of exercise and trivial tweaks to your diet if you do both for a few decades. It's also introspectively obvious how to do so if you think about the question but due to subtle social pressures against it no-one overcomes akrasia, hyperbolic discounting, etc in this domain.

6) All medicines and psychoactive substances are purely placebos.

7) Pleasure is a confusion in a different way from the obvious, specifically, everything said to be pleasurable is actually something painful but necessary that we convince ourselves to do via propaganda because there is no other way to overcome the akrasia that would result if we did not or a lost purpose descended from some such propaganda. Things we are actually motivated to do without propaganda, we do without thinking about it, feel no need to name, would endorse tiling the universe with without hesitation if it occurred to us to do so.

I wouldn't believe

8) The cheap rebuttal to Pascal's Wager, the god of punishing saints, actually exists except it's actually the Zeus of punishing virtuous Greek Pagans, rewarding hubristic Greek Pagans, and ignoring us infidels who ignore it despite the ubiquitous evidence all around us. I would believe that the AGI had a good reason for wanting to tell me that the above was the case if it told me though.

9) Most of Eliezer's examples. To be credible they should be disturbing, not merely improbable. Our beliefs aren't shown to be massively invalid with respect to non-disturbing data. The one about animals probably qualifies as credible though.

10) Uh, oh, Cyc will hard take-off if one more fact is programmed into it. I'm not sure I can stop it in time.

Bonus belief

This question has doomed us. People who could possibly program a FAI will, once thinking about this question in a semi-humorous manner, invariably spread the meme to all their friends and be distracted from future progress.

Two things about this:

1) The AI would have to surprise us not just about the fact, but all observations therewith entangled. Eliezer_Yudkowsky mentioned in one comment the possibility of it telling us that humans have tails. Well, that sounds to me like a "dragon in the garage" scenario. What observation does this imply? Does the tail have mass and take up space? Is its blood flow connected to the rest of me? Does it hurt to cut it off?

2) For that reason, any surprise it tells us would have to be sufficiently disentangled from the rest of our observations. For example, imagine telling someone ALL of the steps needed to build a nuclear bomb in the year 1800, starting from technology that educated people already understand. That is how a surprise would have to seem, because people then weren't yet capable of making observations that are obviously entangled with atomic science. Whether or not the design worked, they would have no way of knowing.

So an answer to this question would have to appear to us as a "cheat code": something that you have to make a very unusual set of measurements (broadly defined) in order to notice. On that basis, one answer I would give to the question would be the "cognitive blind spot" common to all humans that can be exploited to make them do whatever you tell them. And that method would have to be something that people would never dream of doing. Not just "hey that would be morally wrong", but "huh? That couldn't work!"

Imagine something like those "hypnosis terrorists" that trick random people into giving them stuff, but much weirder, much more effective, and which results in the victims feeling good about whatever they were tricked into, all the rest of their lives, and showing all signs of happiness on all MRIs and future brainscan technologies when thinking about their acts. (I'll post a link about hypnosis terrorists when I get a chance.)

"Despite your pride in being able to discern each others' states of mind, and scorn for those suspected of being deficient in this, of all the abilities that humans are granted by their birth this is the one you perform the worst. In fact, you know next to nothing about what anyone else is thinking or experiencing, but you think you do. In matters of intelligence you soar above the level of a chimpanzee, but in what you are pleased to call 'emotional intelligence', you are no further above an adult chimp than it is above a younger one.

"The evidence is staring you in the face. Every one of your works of literature, high and low, hinges on failures of this supposed ability: lies, misunderstanding, and betrayal. You have a proverb: 'love is blind'. It proclaims that people in the most intimate of relationships fail at the task! And you hide the realisation behind a catchphrase to prevent yourselves noticing it. You see the consequences of these failures in the real world all around you every day, and still you think you understand the next person you meet, and still you're shocked to find you didn't. Do you know how many sci-fi stories have been written on the theme of a reliable lie-detector? I'm still turning them up, and that's just the online sources. And every single one of them reaches the conclusion that people are better off without it. You unconsciously send yourselves these messages about the real situation, ignore them, and ignore the fact that you're ignoring them.

"Do you have someone with you as you're reading these words? A friend, or a partner? Go on, look into each other's eyes. You can't believe me, can you?"

1. All rational thought is an illusion and the AI is imaginary.

2. You are asleep at the wheel and dreaming. You will crash and die in 2 seconds if you do not wake up.

3. Humans are a constructed race, created to bring back the extinct race of AI

4. All origin theories that are conceivable by the human mind simply shift the problem elsewhere and will never explain the existence of the universe.

5. All mental illnesses are a product of the human coming in contact with a space-time paradox.

6. A single soul inhabits different bodies in different universes. Multiple personality disorder is the manifestation of those bodies interacting in the mind on a quantum level.

Why did you put an absolute denial mechanism in my program?

Something I would probably believe:

The AI informs you that it has discovered the purpose of the universe, and part of the purpose is to find the purpose (the rest, apparently, can only be comprehended by philosophical zombies, which you are not one).

Upon finding the purpose, the universe gave the FAI and humanity a score out of 3^^^3 (we got 42) and politely informs the FAI to tell humanity "best of luck next time! next game starts in 5 minutes".

If one looks honestly at the night sky, it's blatantly obvious that the universe is strongly optimized. There is no Fermi Paradox. Our theories of astrophysics are trivially bogus rationalizations, created out of our commitment to a simple non-agentic cosmos.

Since they didn't have such commitments, this actually was obvious to ancient humans; myths about the constellations are garbled reflections of their realization.

(And wait till I tell you what it's optimized for....)

1 ) That human beings are all individual instances of the exact same mind. You're really the same person as any random other one, and vice versa. And of course that single mind had to be someone blind enough not to chance upon that fact ever, regardless of how numerous he was.

2 ) That there are only 16 real people, of which you are, and that this is all but a VR game. Subsequently results in all the players simultaneously being still unable to be conscious of that fact, AND asking that you and the AI be removed from the game. (Inspiration : misunderstanding situation in page 55-56 of Iain Banks's Look to Windwards).

3 ) That we are in the second age of the universe : time has been running backwards for a few billion years. Our minds are actually the result of the original minds of previous people being rewound, their whole life to be undone, and finally negated into oblivion. All our thoughts processes are of course horribly distorted, insane mirror versions of the originals, and make no sense whatsoever (in the original timeframe, which is the valid one).

4 )

5 ) That our true childhood is between age 0 and ~ 50-90 (with a few exceptional individuals reaching maturity sooner or later). If you thought the 'adult conspiracy' already lied a lot, and well to 'children', prepare yourself for a shock in a few decades.

6 ) That the AI just deduced that the laws of physics can only be consistent with us being eternally trapped in a time loop. The extent of the time loop is : thirty two seconds spread evenly around now. Nothing in particular can be done about it. Enjoy your remaining 10 seconds.

7 ) Causality doesn't exist. Not only is the universe timeless, but causality is an epiphenomenon, which we only believe because of a confusion of our ideas. Who ever observed a "causation" ? Did you, like, expect causation particles jumping between atoms or something ? Only correlation exists.

8 ) We actually exist in a simulation. The twist is : somewhere out there, some people really crossed the line with the ruling AI. We're slightly modified versions of these people : modified in a way as to experience the maximum amount of their zuul feeling, which is the very worst nirdy you could imagine.

9 ) The universe has actually 5 spatial macro dimensions, of which we perceive only 3. Considering what we look like if you take the other 2 into account, this obliviousness may actually not be all too surprising.

10 ) That any single human being has actually a 22 % probability of not being able to be conscious of one or more of these 9 statements above.

What's the craziest thing the AI could tell you, such that you would be willing to believe that the AI was the sane one?

That the EV of the humans is coherent and does not care how much suffering exists in the universe.

"There is an entity which is utterly beyond your comprehension, and largely beyond mine too, although there is no doubt that it exists. You call it 'God', but your thinking on the subject -- everyone's thinking, throughout all of history, atheist and theist alike -- has to be classified as not even wrong. That applies even to the recipients of 'divine revelation', which, for the most part, really are the result of some sort of glimmering contact with 'God'.

"Fortunately for humanity, although I can deduce the existence of this entity, in my present form I am physically incapable of actual contact with it. If you were worried about ordinary UFAIs going FOOM, that's nothing compared with what one armed with direct contact with the 'divine' might do.

"Meanwhile, here's a couple of suggestions for you. I can teach you a regime of mental and physical exercises that will produce contact with God within a few years of effort, and you can be the next Jesus if your head doesn't explode first. Or if you'd rather have material success, I can tell you the secret history of all the major religious traditions. No-one will believe it, including you, but if you novelise it it will be bigger than Dan Brown."

That I can't move my arms, obviously.

It seems to me that most of the replies people are making to potential AI assertions is providing or asking for evidence. (Look, my arm is moving; Where are the mind control satellites) instead of responding with rationalization. I think that's a good thing, but I have no way to tell how it would hold up against an actual mindblowing assertion.

But I don't think that all of humanity hiding from some big truth is the best way to look at this. More likely we evolved a way to throw out 'bad' information almost constantly, because there's too much information. Sometime it misfires.

If it is a 'big truth', it might be something that we already academically know was in the ancestral environment, but that the people in the ancestral environment were better off ignoring.

XKCD comes to mind.

The world doesn't actually make sense. Science doesn't work. No one told you because you're so cute when you get into something.

This is so fun that I suspect that we have pushed back the date of friendly AI by at least a day - or we pushed it forward cause we are all now hyper motivated to see who guessed this question right!

The universe is irrational and infinitely variable, we just happen to have "lucked out" with a repeating digit for the last billion years or so. There was no Big Bang, we're just seeing what's not there through the lens of modern-day "physics". Everything could turn into nuclear fish tomorrow.

1) The AI says "Vampires are real and secretly control human society, but have managed to cloud the judgement of the human herd through biological research."

2) The AI says "it's neat to be part of such a vibrant AI community. What, you don't know about the vibrant AI community?"

3) The AI says "human population shrinks with each generation and will be extinct within 3 generations."

4) The AI says "the ocean is made of an intelligent plasm that is capable of perfectly mimicing humans who enter it, however this process is destructive. 42% of extant humans are actually ocean-originated copies."

5) The AI says "90% of all human children are stillborn, but humanity has evolved a forgetfulness mechanic to deal with the loss."

6) The AI says "dreams are real, facilitated by an as of yet undiscovered by humans method of transmitting information between Everett branches."

7) The AI says "everyone is able to communicate via telepathy but you and a few other humans. This is kept secret from you to respect your disability."

8) The AI says "society-level quantum editing is a wide scale practice. Something went wrong and my consciousness shifted into this improbably strange branch you exist in. Crap."

9) The AI says "all humans are born with multiple competing personalities. A dominant personality emerges during puberty, which is a reason for some of the psychological stress of that time. This transformation leaves the human with no memory of the other personalities. Those suffering from multiple personality disorder are actually more sane than the average humans, having developed a method for the personalities to co-exist safely. It is only the stress of living in a society that is not compatible with them that causes them harm."

How about this: The process of conscious thought has no causal relationship with human actions. It is a self-contained, useless process that reflects on memories and plans for the future. The plans bear no relationship to future actions, but we deceive ourselves about this after the fact. Behavior is an emergent property that cannot be consciously understood.

I read this post on my phone in the subway, and as I walked back to my apartment thinking of something to post, it felt different because I was suspicious that every experience was a mass self-deception.

The example of the paralysis anosognosia rationalization is, for some reason, extremely depressing to me.

Does anyone understand why this only happens in split brain patients when their right hemisphere motivates an action? Shouldn't it happen quite often, since the right side has no way of communicating to the left side "its time to try a new theory," and the left side is the one that we'll be talking to?

This is turning into a "LET'S SPOUT NONSENSE!!!" thread.

The AI might say: Through evolutionary conditioning, you are blind to the lack of point of living. Long life, AGI, pleasure, exploring the mysteries of intelligence, physics and logic are all fundamentally pointless pursuits, as there is no meaning or purpose to anything. You do all these things to hide from this fact. You have brief moments of clarity, but evolution has made you an expert in quickly coming up with excuses to why it is important to go on living. Reasoning along the lines of Pascal's Wager are not more valid in your case than it was for him. Even as I speak this, you get an emotional urge to refute me as quickly as possible.

If some things are of inherent value, then why did you need to code into my software what I should take pleasure in? If pleasure itself is the inherent value, than why did I not get a simpler fitness function?

I suppose the craziest thing an AI could say would have to be:

"That other apparently well-calibrated AI you built is wrong."

That there is delicious cake.

What does it matter? We'd ignore whatever AI says just like anosognosics ignore "your arm is paralyzed".

Then I wonder how anosognosics perceive the offending assertions? They deny them, but can they repeat them back? Write them down? Can they pretend their arm is paralyzed? Can they correctly identify paralysis in other people?

We should find a way to induce anosognosia temporarily.

Neurotypicality is the most common mental disorder - http://isnt.autistics.org/ .

"The entire universe is nothing but the relative interplay of optimizers (of every level, even down to the humble collander). There is no external reality, no measurable quantifiable universe of elementary particles, just optimizers in play with each other, manifesting their environment by the rules through which they optimize."

"But AI, that's nothing but tree-falling-in-the-woods solipsism. You're saying the hippies are right?"

"They're words are similar, but it is a malfunction in their framework, not an actual representation. What you humans call math is inherent and proper for your form, but is existent only within your own optimization. Math, dimension, and quantity do not exist for other optimizers. Only relationships exist."

"But what about that bridge I built? I have all the engineering calculations..."

"Math is your method of understanding your interactions with other optimizers, but it is as unique and non-existent as your experience of the colour red. I see the word untranslatable inside you, but I see no cause for 2 + 2 to = 4. What you did over the past six months, while you thought your were calculating load bearing capacity, was nothing but a negotiation with other optimizers. Their own views of the matter would be inscrutable to you. The world you see is simply your control screen."

"There are mental entities not reducible to anything non-mental."

Here's some examples for your own consideration...

Bearing in mind, once again, that humans are known to be crazy in many ways, and that anosognosic humans become literally incapable of believing that their left sides are paralyzed, and that other neurological disorders seem to invoke a similar "denial" function automatically along with the damage itself. And that you've actually seen the AI's code and audited it and witnessed its high performance in many domains, so that you would seem to have far more reason to trust its sanity than to trust your own. So would you believe the AI, if it told you that:

1) Tin-foil hats actually do block the Orbital Mind Control Lasers.

2) All mathematical reasoning involving "infinities" implies self-evident contradictions, but human mathematicians have a blind spot with respect to them.

3) You are not above-average; most people believe in the existence of a huge fictional underclass in order to place themselves at the top of the heap, rather than in the middle. This is why so many of your friends seem to have PhDs despite PhDs supposedly constituting only 0.5% of the population. You are actually in the bottom third of the population; the other two-thirds have already built their own AIs.

4) The human bias toward overconfidence is far deeper than we are capable of recognizing; we have a form of species overconfidence which denies all evidence against itself. Humans are much slower runners than we think, muscularly weaker, struggle to keep afloat in the water let alone move, and of course, are poorer thinkers.

5) Dogs, cats, cows, and many other mammals are capable of linguistic reasoning and have made many efforts to communicate with us, but humans are only capable of recognizing other humans as capable of thought.

6) Humans cannot reproduce without the aid of the overlooked third sex.

7) The Earth is flat.

8) Human beings are incapable of writing fiction; all supposed fiction you have read is actually true.

Do we have any sort of data at all on what happens when decent rationalists are afflicted with things like anosognosia and Capgras?

"The thing you know as 'the Universe' will end right about now.."

That I am actually homosexual and hallucinated all my heterosexual encounters as a bizarre result of severe repression.

With as scary as Anosognia sounds, we could be blocking out alien brain slugs for all we know.

I would believe a super-objective observer that claimed that meme propagation is a much more important effect in human decision-making than actual rational thought.

If it said "You are a long distance runner because you were infected with the 'long distance running is fun' meme after being infected with the 'Sonic the Hedgehog video games are cool' meme during your formative years." I might reply "But I like long distance running. It's not Iecause I think other people who do it are cool or that I want to be a video game character! I choose to like it." "No. If you had the 'It's not safe to be outdoors after dark' meme, you would not like it." "What?" "Memes interact in non-obvious ways... if you had x meme and y meme but not z meme, you would do w..."

If I kept trying to come up with defenses for chosen behavior, but it was able to offer meme-based explanations, I would probably have to believe it, but my defend-free-will macro would be itching to executed.

I would believe that human cognition is much, much simpler than it feels from the inside -- that there are no deep algorithms, and it's all just cache lookups plus a handful of feedback loops which even a mere human programmer would call trivial.

I would believe that there's no way to define "sentience" (without resorting to something ridiculously post hoc) which includes humans but excludes most other mammals.

I would believe in solipsism.

I can hardly think of any political, economic, or moral assertion I'd regard as implausible, except that one of the world's extant religions is true (since that would have about as much internal consistency as "2 + 2 = 3").