Comment Permalink

That was part of a line of reasoning that was introduced by "I used to think that". Pay more attention to context.

You have to read all 5 sentences in that paragraph, to read about the new position as I just described it. The original postion was much more wrong.

See in context

-1

Lie to me?

by pwno

24th Jun 2009

2 min read

-1

I used to think that, given two equally capable individuals, the person with more true information can always do at least as good as the other person. And hence, one can only gain from having true information. There is one implicit assumption that makes this line of reason not true in all cases. We are not perfectly rational agents; our mind isn’t stored in a vacuum, but in a Homo sapien brain.There are certain false beliefs that benefit you by exploiting your primitive mental warehouse, e.g., self-fulfilling prophecies.

Despite the benefits, adopting false beliefs is an irrational practice. If people never acquire the maps that correspond the best to the territory, they won’t have the most accurate cost-benefit analysis for adopting false beliefs. Maybe, in some cases, false beliefs make you better off. The problem is you'll have a wrong or sub-optimal cost-benefit analysis, unless you first adopt reason.

Also, it doesn’t make sense to say that the rational decision could be to “have a false belief” because in order to make that decision, you would have to compare that outcome against “having a true belief.” But in order for a false belief to work, you must truly believe in it — you cannot deceive yourself into believing the false belief after knowing the truth! It’s like figuring out that taking a placebo leads to the best outcome, yet knowing it’s a placebo no longer makes it the best outcome.

Clearly, it is not in your best interest to choose to believe in a falsity—but what if someone else did the choosing? Can’t someone whose advice you rationally trust be the decider of whether to give you false information or not (e.g. a doctor deciding whether you receive a placebo or not)? They could perform a cost-benefit analysis without diluting the effects of the false belief. We only want to know the truth, but prefer to be unknowingly lied to in some cases.

Which brings me to my question: do we program an AI to only tell us the truth or to lie when the AI believes (with high certainty) the lie will lead us to a net benefit over our expected lifetime?

Added: Keep in mind that knowledge of the truth, even for a truth-seeker, is finite in value. The AI can believe that the benefit of a lie would outweigh a truth-seeker's cost of being lied to. So unless someone values the truth above anything else (which I highly doubt), would a truth-seeker ever choose only to be told the truth from the AI?

Self-Deception

Personal Blog

-1

New Comment

32 comments, sorted by

top scoring

Click to highlight new comments since: Today at 5:56 PM

[-]jimrandomh15y60

This topic has already been done to death, and then some.

[-]pwno15y20

Ah, my bad.

[-]Vladimir_Nesov15y10

This fact should be fixed by some articles on the wiki, with references to the previous discussions (so that you can substantiate you claim with a link, explaining to any newcommer where to look). This point seems to fit in the concepts of Truth and Self-deception; the latter article is currently almost empty.

ETA: Costs of rationality is a closer match.

[-]ThoughtDancer15y10

The wiki is a good starting tool, but it's not yet as fully developed as I would like. I'm still working to develop sufficient background knowledge of the discussions, assumptions, and definitions used in Less Wrong so as to be sufficiently confident in commenting.

So I will forgive the occasions when someone who sincerely wants information and thoughtful reactions stumbles into spaces that have already been well-trodden.

Nevertheless, the wiki itself isn't yet fully developed with interconnections and links to definitions: until such internal tagging is complete, newer people will sometimes fail to find what they are searching for and will instead ask it directly.

I welcome these questions being asked, and if only as a sign that Less Wrong does not encourage self-censorship (which, I gather from conversations elsewhere, may have been a concern on Overcoming Bias).

[-]Vladimir_Nesov15y10

Of course the wiki is still in its infancy. All the more reason to shape it by contributing your own synthesis of the discussed concepts, especially when these concepts leave the impression of having been discussed to death.

[-]A1987dM11y40

It’s like figuring out that taking a placebo leads to the best outcome, yet knowing it’s a placebo no longer makes it the best outcome.

Unless what counts for whether the placebo works is what your System 1 believes, what counts for your rational cost-benefit analysis is what your System 2 believes, and you somehow manage to keep them separate (which is hard but not always impossible).

[-]cousin_it15y10

Yep, done to death. Eliezer's answer: The Third Alternative. I hereby disclose that I downvoted you.

[-]Vladimir_Nesov15y00

Which brings me to my question: do we program an AI to only tell us the truth or to lie when the AI believes (with high certainty) the lie will lead us to a net benefit over our expected lifetime?

This question is too anthropomorphic. Since you clearly mean a strong AI, it's no longer a question-answering machine, it's an engine for a supervised universe. At which point you ought to talk about the organization of the future, for example of fun theory.

[-]Furcas15y00

Well, the AI wouldn't only have to predict how the lie will benefit the person who hears it, but also how the actions that result from holding a false belief might affect other individuals.

The above quibble aside, the answer to your question is pretty trivial. To a person who values the truth, knowledge is a benefit and will therefore be part of the AI's 'net benefit' calculation. Consequently, the kind of person who would want to program an AI to only tell him the truth would never be lied to by an AI that does a net benefit calculation.

The only question that remains is if we would want to program an AI to always tell the truth to people who want to be deceived at least some of the time. In other words, do we want other individuals to believe whatever they like, as long as it's a net benefit to them and doesn't affect the rest of us? My answer would be yes.

Of course, in the real world, popular false beliefs, such as religious ones, often do not lead to a net benefit for those who hold them, and even more often affect the rest of us negatively.

[-]pwno15y00

To a person who values the truth, knowledge is a benefit and will therefore be part of the AI's 'net benefit' calculation.

But knowledge of the truth has a finite value. What if the AI believed that the benefit of a lie would outweigh a truth-seeker's cost of being lied to?

So the question is, would any rational truth-seeker choose to only be told the truth by the AI?

[-]Furcas15y00

A person doesn't have to 'infinitely value' truth to always prefer the truth to a lie. The importance put on truth merely has to be greater than the importance put on anything else.

That said, if the question is, is there a human, or has there ever been a human who values truth more than anything else, the answer is almost certainly no. For example, I care about the truth a lot, but if I were given the choice between learning a single, randomly chosen fact about the universe, and being given a million dollars, I'd pick the cash without too much hesitation.

However, as Eliezer has said many times, human minds only represent a tiny fraction of all possible minds. A mind that puts truth above anything else is certainly possible, even if it doesn't exist yet.

[-]pwno15y00

Now that we know we programmed an AI that may lie to us, our rational expectations will make us skeptical of what the AI says, which is not ideal. Sounds like the AI programmer will have to cover up the fact that the AI does not always speak the truth.

[+]timtyler15y-50

Moderation Log