Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Comment author: James_Miller 02 February 2013 07:38:10PM *  15 points [-]

An accurate emotion = "I'm angry because I should be angry because she is being really, really mean to me."

A useful emotion = "Showing empathy towards someone being mean to me will minimize the cost to me of others' hostility."

Comment author: AdeleneDawner 02 February 2013 07:40:30PM 4 points [-]

Where's that 'should' coming from? (Or are you just explaining the concept rather than endorsing it?)

Comment author: Luke_A_Somers 22 January 2013 10:38:59PM 4 points [-]

I don't see why it'd need to be personal, but being useful would be a good lure.

The question is to where to go from there. Can you Scheherezade it out?

Comment author: AdeleneDawner 24 January 2013 07:14:08AM 3 points [-]

This was basically my answer - I can't play as an AI using this strategy, for obvious reasons, but an AI that used its one sentence to give a novel and easily-testable solution to a longstanding social problem of some sort (or an easily-testable principle that suggests one or more novel solutions) would probably get at least a second sentence from me (though not a typed response; that seems to open up a risky channel). Especially if the AI in question didn't actually have access to a lot of information about human culture or me personally and had to infer that a solution like that would be useful from near-base principles - that's not proof of Friendliness, but an AI using its one guaranteed communication to do something that has a decent chance of improving the world per our definition without any prompting whatsoever sure looks suspiciously like Friendly to me.

In response to comment by WrongBot on Just One Sentence
Comment author: [deleted] 05 January 2013 02:37:36AM *  15 points [-]

"physical laws" and "universe" maybe suppose too much background.

I cross-pollinate your thing with EY's:

"If you test theories by how precisely they predict experimental results, you will learn how to make powerful weapons."

EDIT: My latest version is "If you test theories by how precisely they predict experimental results, you will unlock the secrets of the ancients." Which fixes a few bugs.

In response to comment by [deleted] on Just One Sentence
Comment author: AdeleneDawner 05 January 2013 04:12:50AM 11 points [-]

That parses as 'do not let others conduct experiments'. Probably not what you're aiming for.

Comment author: [deleted] 05 January 2013 02:53:47AM 8 points [-]

"there is a copy of all the knowledge of the ancients buried at the <insert trasure map>"

At first I was like, "lulz I can just insert coordinates", but then I realized they don't have GPS. Serious dark ages...

The only unambiguously specifiable terrestrial spot I can think of is the south pole, but there must be something better.

In response to comment by [deleted] on Just One Sentence
Comment author: AdeleneDawner 05 January 2013 04:10:41AM 10 points [-]

If you have the resources to put something at the south pole, you probably have the resources to scatter a couple dozen stonehenges/pyramids/giant stone heads around; then you don't have to specify unambiguously, plus redundancy is always good.

In response to Failed Utopia #4-2
Comment author: Allan_Crossman 21 January 2009 11:24:26PM 8 points [-]

Is this a "failed utopia" because human relationships are too sacred to break up, or is it a "failed utopia" because the AI knows what it should really have done but hasn't been programmed to do it?

Comment author: AdeleneDawner 30 December 2012 05:23:48AM 0 points [-]

I think it's a failed utopia because it involves the AI modifying the humans' desires wholesale - the fact that it does so by proxy doesn't change that it's doing that.

(This may not be the only reason it's a failed utopia.)

Comment author: Ursus 26 December 2012 07:39:34AM 0 points [-]

Thanks for the response.

However, I think you misunderstood what I was attempting to say. I see I didn't use the term "filtered evidence", and am wondering if my comment showed up somewhere other than the article "what evidence filtered evidence": http://lesswrong.com/lw/jt/what_evidence_filtered_evidence/ Explaining how I got a response so quickly when commenting on a 5 year old article! If so, my mistake as my comment was then completely misleading!

When the information does not come from a filtered source, I agree with you. If I find out that there is evidence that will be in the up (or down) direction of a belief, this will modify my priors based on the degree of entanglement between the source and the matter of the belief. After seeing the evidence then the probability assessment will on average remain the same; if it was weaker/stronger it will be lower/higher (or higher/lower, if evidence was downward) but it will of course not pass over the initial position before I heard of the news, unless of course it turns out to be evidence for the opposite direction.


What my question was about was filtered evidence. Filtered evidence is a special case, in which that entanglement between the source and matter of the belief = 0. Using Eliezer_Yudkowsky's example from "the bottom line": http://lesswrong.com/lw/js/the_bottom_line/ The only entanglement between the claim of which sort of evidence I will be presented with, "box B contains the diamond!", is with whether the owner of box A or box B bid higher (owner of box B, apparently).

Not with the actual content of those boxes (unless there is some relation between willingness to bid and things actually entangled with presence/absence of a diamond).

Therefore, being told of this will not alter my belief about whether or not box B is the one that contains the diamond. This is the lead-in to the questions posed in my previous/first post.

Knowing that the source is filtered, every single piece of evidence he gives you will support the conclusion the diamond is in box B. Yet you simply cannot expect every single piece of evidence to on average increase your belief that it is in box B. While the arguments are actually entangled, their selective presentation means P(E)=1 and P(~E)=0. You can't balance the equations other than by not modifying your beliefs at all with each piece of evidence. Probability of the diamond being in box B = probability of the diamond being in box B given that you are shown evidence, meaning the evidence doesn't matter. The direction of the evidence that passes through the filter (the clever arguer) is entangled with which box-owner bid more money. Not with which box actually contains the diamond. Thus, it sounds like you simply should not modify your beliefs when faced with a clever arguer who filters evidence. No entanglement between evidence's direction and the truth/conservation of expected evidence.

My problem: not taking into account evidence entangled with reality doesn't sit well with me. It sounds as though it should ultimately be taken into account, but I can't immediately think of an effective process by which to do it.


Using drugs instead of boxes if that is an example you prefer: imagine a clever arguer hired by Merck to argue about what a great drug Rofecoxib is. The words "cardiovascular", "stroke", and "heart attack" wont ever come up. With the help of selectively drawing from trials, a CA can paint a truly wonderful picture of the drug that has limited baring on reality.

Before seeing his evidence he tells you "Rofecoxib is wonderful!" This shouldn't modify your belief, as it only tells you he is on Merck's payroll. Now how do you appropriately modify your belief on the drug's quality and merits with the introduction of each piece of evidence this clever arguer presents to you?

Comment author: AdeleneDawner 26 December 2012 04:27:57PM 1 point [-]

Actually, it's my bad - I found your comment via the new-comments list, and didn't look very closely at its context.

As to your actual question: Being told that someone has evidence of something is, if they're trustworthy, not just evidence of the thing, but also evidence of what other evidence exists. For example, in my scenario with gwern's prank, before I've seen gwern's web page, I expect that if I look the mentioned drug up in other places, I'll also see evidence that it's awesome. If I actually go look the drug up and find out that it's no better than placebo in any situation, that's also surprising new information that changes my beliefs - the same change that seeing gwern's "April Fools" message would cause, in fact, so when I do see that message, it doesn't surprise me or change my opinion of the drug.

In your scenario, I trust Merck's spokesperson much less than I trust gwern, so I don't end up with nearly so strong of a belief that third parties will agree that the drug is a good one - looking it up and finding out that it has dangerous side effects wouldn't be surprising, so I should take the chance of that into account to begin with, even if the Merck spokesperson doesn't mention it. This habit of keeping possible information from third parties (or information that could be discovered in other ways besides talking to third parties, but that the person you're speaking to wouldn't tell you even if they'd discovered it) into account when talking to untrustworthy people is the intended lesson of the original post.

Comment author: Ursus 26 December 2012 03:16:14AM *  1 point [-]

I'm trying to incorporate this with conservation of expected evidence: http://lesswrong.com/lw/ii/conservation_of_expected_evidence/

For example: "On average, you must expect to be exactly as confident as when you started out. Equivalently, the mere expectation of encountering evidence—before you've actually seen it—should not shift your prior beliefs." -EliezerYudkowsky AND "Your actual probability starts out at 0.5, rises steadily as the clever arguer talks (starting with his very first point, because that excludes the possibility he has 0 points)" -EliezerYudkowsky

Appear to be contradictions, given that each point= a piece of evidence (shininess of box, presence of blue stamp, etc).

The cherry picking problem appears to be similar to the witch trial problem. In the latter any piece of evidence is interpreted to support the conclusion, while in the former evidence is only presented if it supports the conclusion.

You can't expect your probabilities to on average be increased before seeing/hearing the evidence.

I think only if you do have a large background of knowledge, with a high probability that you are already aware of any given piece of evidence. But if you hear a repeat evidence, it simply shouldn't alter your probabilities, rather than lower it. I'm having a hard time coming up with a way to properly balance the equation.

The only thing I can think of is if you count the entire argument as one piece of evidence, and use a strategy like suggested by g for updating your priors based on the entire sum?

But you don't necessarily listen to the entire argument. Knowing about hypothetical cut off points below which they wont spend the time to present and explain evidence means with enough info you could still construct probabilities. If time is limited, can you update with each single piece of evidence based on strength relative to expected?

What if you are unfamiliar with the properties of boxes and how they are related to likelihood of the presence of a diamond? Any guesstimates seem like they'd be well my abilities at least.

Unless I already know a lot, I have a hard time justifying updating my priors at all based on CA's arguments. If I do know a lot, I still can't think of a way to justifiably not expect the probability to increase, which is a problem. Help, ideas?

PS. Thankfully not everyone is a clever arguer. Ideally, scientists/teachers teaching you about evolution (for example) will not be selective in giving evidence. The evidence will simply be lopsided because of nature being lopsided in how it produces evidence (entangled with truth). I don't think one has to actually listen to a creationist, assuming it is known that the scientists/source material the teacher is drawing from are using good practice.

Also, this is my first post here, so if I am ignorant please let me know and direct me to how I can improve!

Comment author: AdeleneDawner 26 December 2012 04:47:44AM 3 points [-]

Someone claiming that they have evidence for a thing is already evidence for a thing, if you trust them at all, so you can update on that, and then revise that update on how good the evidence turns out to be once you actually get it.

For example, say gwern posts to Discussion that he has a new article on his website about some drug, and he says "tl;dr: It's pretty awesome" but doesn't give any details, and when you follow the link to the site you get an error and can't see the page. gwern's put together a few articles now about drugs, and they're usually well-researched and impressive, so it's pretty safe to assume that if he says a drug is awesome, it is, even if that's the only evidence you have. This is a belief about both the drug (it is particularly effective at what it's supposed to do) and what you'll see when you're able to access the page about it (there will be many citations of research indicating that the drug is particularly effective).

Now, say a couple days later you get the page to load, and what it actually says is "ha ha, April Fools!". This is new information, and as such it changes your beliefs - in particular, your belief that the drug is any good goes down substantially, and any future cases of gwern posting about an 'awesome' drug don't make you believe as strongly that the drug is good - the chance that it's good if there is an actual page about it stays about the same, but now you also have to factor in the chance that it's another prank - or in other words that the evidence you'll be given will be much worse than is being claimed.

It's harder to work out an example of evidence turning out to be much stronger than is claimed, but it works on the same principle - knowing that there's evidence at all means you can update about as much as you would for an average piece of evidence from that source, and then when you learn that the evidence is much better, you update again based on how much better it is.

Comment author: Ezekiel 25 December 2012 11:39:08AM 14 points [-]

Correct me if I'm wrong, but it looks like you're talking about anti-deathism (weak or strong) as if it was a defining value of the LessWrong community. This bothers me.

If you're successful, these rituals will become part of the community identity, and I personally would rather LW tried to be about rationality and just that as much as it can. Everything else that correlates with membership - transhumanism, nerdiness, thinking Eliezer is awesome - I would urge you not to include in the rituals. It's inevitable that they'd turn up, but I wouldn't give them extra weight by including them in codified documents.

As an analogy, one of the things that bugged me about Orthodox Judaism was that it claims to be about keeping the Commandments, but there's a huge pile of stuff that's done just for tradition's sake, that isn't commanded anywhere (no, not even in the Oral Lore or by rabbinical decree).

Comment author: AdeleneDawner 25 December 2012 05:38:49PM 9 points [-]

What would a ritual that's just about rationality and more complex than a group recitation of the Litany of Tarsky look like?

Comment author: gelisam 24 December 2012 03:40:49PM *  2 points [-]

After finding the source and the issue list, I found instructions which indicate that there is, after all, non-zero engineering resources for lesswrong development. Specifically, somebody is sorting the incoming issues into "issues for which contributions are welcome" versus "issues which we want to fix ourselves".

The path to becoming a volunteer contributor is now very clear.

Comment author: AdeleneDawner 25 December 2012 03:35:03AM *  3 points [-]

non-zero engineering resources

effectively zero

Getting someone to sort a list, even on an ongoing basis, is not functionally useful if there's nobody to take action on the sorted list.

Comment author: Eliezer_Yudkowsky 24 December 2012 02:17:18AM 4 points [-]

Yeah seriously. What if violence is the right thing to do?

Then discussing it on the public Internet is the wrong thing to do. I can't compare it to anything but juvenile male locker-room boasting.

Comment author: AdeleneDawner 24 December 2012 02:31:58PM 2 points [-]

Actually, I can think of at least one type of situation where this isn't true, though it seems unwise to explain it in public and in any case it's still not something you'd want associated with LW, or in fact happening at all in most cases.

View more: Next