Growing up, as a kid, I was always told that every sapient life is precious, everything that thinks and knows itself -

Yes, this is a tale about lies-told-to-children.  You'll probably figure it out yourself before too long.  For now, just listen.

Where was I?  Right.  As children, we were always told that every sapient life is precious.  It was told to us by the teachers, and shown to us in children's television - though I saw less children's television than most children in our age cohort - children's TV was censored where I grew up, though, of course, I didn't find that out until much later -

I see you're starting to guess under what sort of circumstances I grew up.  Go ahead, write down the prediction if you want.  Maybe you already see where this entire thing is headed.  But you asked me for a story about the lies I was told as a child, and that's what you're getting.  It's not my fault, if a lot of stories like that are predictable; people who lie to children have other things to optimize for than unpredictability.

So where was I?  Right.  I grew up in a remote village of about three thousand people, the sort that's more hills than houses.  Charming travel-pathways that cut through forests.  Not everyone knows everyone, but you sure know somebody who knows anybody.

Children's television in my region was censored, though of course they didn't tell us that as children.  But the children's television that we saw had aliens and monsters and creatures of fantasy, with four legs or fourteen legs, three faces or no face at all, and all of them were treated by the television show as having lives that meant something.  Sometimes in the children's show there were alien monsters who only thought their own kind of life was valuable, and then maybe you couldn't trade with them as friends.  Maybe they'd already lied to you once and you couldn't trust them enough to bargain with them, maybe you couldn't talk to them at all.  But their lives still had meaning to the story's human protagonists, even some aliens whose lives had no meaning to themselves.  You didn't cause them pain if there was any way to avoid it; you didn't kill them unless their biology was sufficiently similar to human that you were confident in your ability to cryopreserve them afterwards.

The shows never spelled it out, never said, 'And this is because of a universal rule in every case that sapient life has value.'  Our teachers said that explicitly, though.

And they treated every one of us children, too, as if our lives had meaning.

Except the children with the red hair; those dirty reds.

You're nodding along with a knowing look, I see.  Was it what you predicted?  Not exactly, maybe, but rough ballpark?  I suppose I'll find out when we open your prediction afterwards.

The red-haired children hardly needed the red hair, as their targeting-mark; they looked different from the rest of us in other ways too.  When I was old enough to first ask, I was told that they were the children's children of people who'd been exiled from a faraway city for committing terrible crimes there, who'd been given sanctuary by the grace and mercy of our own benevolent kind.  The red-haired children tended bigger than the rest of us, with more adult facial structures, to the point where you could've maybe mistaken them for very small adults in disguise.  The red-haired adults, what few of them we ever saw, were correspondingly huge and muscular.  You could see, in retrospect - if you were actually trying to think at all, which we weren't really - how somebody might have felt threatened by such big muscular people, even while graciously granting them sanctuary.

There weren't many of the red-haired children being educated alongside us; a handful, four or six.  I can't recall how many by counting names, because they kept to themselves and did not try to be friends with the rest of us.

They were slower than the rest of us in class to answer.  On the rare occasions a teacher called on them, they'd often get the question wrong.  We were kids, young kids, so of course we didn't ask ourselves anything like "Is this in fact an intrinsic deficit of intelligence or is it a self-fulfilling prophecy about who gets more effort from the teachers?" or come up with any experiments to test that one way or another.  We just wordlessly thought that red-haired kids were stupider; and that this too was a universal rule just like gravity.

We did not, in fact, treat our red-headed fellow kids all that well.  We were of an age where kids take their cues from adults without carefully rethinking everything they're seeing.  We noticed how the older kids treated red-haired kids, we noticed how teachers treated red-haired kids, we noticed the huge red-headed adults who were silently sweeping the hallways and not doing any intellectual labor.  We noticed how the adult reds got casually shoved aside by other adults or even non-red-headed older kids, and how the red-headed adults just silently took that.

There were names to call them, 'dirty reds', worse things than that, scatological profanities to giggle over amongst ourselves.

Now and then you'd see a Security Officer come by and ask some reds some questions.  One time Security took one of the janitors away, and then after that, nobody ever saw him again.  I think one of the kids did ask, in class, what happened to that guy, and the teacher shut her right down and said that any questions about dirty reds or for that matter Security were things best asked in private if you asked at all.

And meanwhile the television shows, those that we got to watch, went right on teaching the lesson that all sapient life is precious, with no exceptions for fourteen legs or not having a face.

Eventually, of course, it started coming to a point, and then it did come to a point.

It started to come to a point, at the point where a red-haired kid was called on in class and answered a question wrong, and the teacher asked if his parents were too busy stealing other people's books to teach him how to read.  The red-haired kid didn't say anything back, but I flinched, visibly.

It came to the point, two days after that, when I was walking home from class, and I heard a groan from off the pathway home, what sounded like a moan of pain.

I left the pathway and ran around a hill to find one of those dirty reds hiding behind it, with blood all over his left pants-leg.

He asked me not to get an adult.

He said that he was hiding from Security.

He asked me to help him walk, help him get away.

It didn't feel real.  It felt like I was inside one of the children's television shows.

Of course, in children's television shows, they always show the heroes reminding themselves that things are real and that they've got to do what's right, because it's real, so I knew that I needed to remember that this was real because that's what you do when you're inside a television show.

I think I was probably very scared, though I don't remember noticing myself being scared.

I asked him what he'd done to get Security looking for him.

He said that he had, a few days ago, said something about red-haired people deserving better treatment than they currently got, around a non-red-haired person he'd thought, hoped, was a friend.

I gave him a hand so he could stand up, on the leg that wasn't covered with blood, and then he leaned on me and we hopped away through the hills until we got to where a red-haired woman - you saw fewer of those - whispered a thank-you to me and took him away with herself.

I ran back to the pathway and ran home, though I was still late, of course.  My dad asked me where I'd been and I said I'd seen a funny-looking butterfly and run off to chase it.  I remember believing, even then, that he knew I was lying, but dad didn't ask me any more questions, and I didn't tell him anything.

About an hour later, Security knocked on our door and asked everyone if they'd seen a red-haired person who looked like - and of course the picture was of the man I'd helped to get away.

I said no, I hadn't seen him.  But because I was a kid and kids that age aren't taught theory-of-deception, I asked what the man had done and if he was considered dangerous.  And I didn't think, until too late, about whether that was something I was much more likely to ask if -

The Security officer asked me if I maybe wanted to change my mind about having seen the fugitive.

I gave him my best surprised look and said no.

The Security officer noted that Security officers get special training in reading emotions, and I seemed pretty frightened to him.

I said yes, I was, because the Security officer was suggesting that I was lying and that was scary.

The Security officer said he knew perfectly well, at this point, that I was lying.  But I wouldn't end up in trouble if I showed him where the fugitive went and identified anyone else he was with.

I said that he didn't know what he was talking about.

The Security officer gave me a sort of stern look and said that he'd detected another lie, and did I really want to get in trouble for some dirty red.

I told him that I wasn't stupid and I knew he was bluffing, to try to trick me, because he suspected me, even though I hadn't done it.

He took a photo out of his pocket and showed it to me.

It was me helping the red-haired man walk on his one good leg.

Why, said the Security.  He just looked sad, now.  Why had I done it?  Why was a dirty red worth it?

And I remember, by that point, that I'd noticed I was scared, and I think I was trying to get out of it - by proving that I was, in the end, obeying adult authority - when I said that we'd all been told in class that every sapient life is precious, everything that thinks and knows itself, that was the rule we'd been given, and nobody had reasonably argued at any point that there was an exception for people with red hair, and also we'd all been told that hurting people is wrong and you shouldn't let social conformity push you into it.

The rest of it went the way you'd expect.

The Security officer smiled.

My parents rushed in and hugged me and told me I'd been so brave and so good and scored in what would've been the upper 5th percentile twenty years ago for the age where I started to object and not go along with it anymore; and explained about Civilization needing to test some kids now and then, to find out how well we were doing environment-wise and heredity-wise on people's kindness and resistance to conformity-pushed cruelty; and test against an earlier-reported bug where general rules about fair and okay treatment of people would somehow end up not being applied to some subgroup; and our little village was settling an important conditional prediction market from twenty years earlier, that had millions of labor-hours wagered on it; and that children growing up to be good people was a vital figure-of-merit for all of Civilization and lots of big policy decisions turned around it, which was why it had been worth specializing our village to do Science about that, and they hoped I understood all that and wouldn't tell the other children right away.  There wasn't actually any such thing as Security, and if there ever was it would mean that it was time to overthrow the government immediately.

I nodded along in a wise, understanding, and rather numb fashion.  I think the main thing I said, at the end, was that I'd better be getting paid for this, and they all laughed and said of course I was, lots of money, at least as much as my parents were getting, because children are sapient beings too.

So that's my story about the-lies-we-tell-to-children.  And the part that I value now the most, even more than the money I got then and when I was older, even more than knowing that I was good and brave in the only sort of real test that most people in Civilization ever get, is that I approximately always win any Lies-Told-To-Children storytelling night.

New Comment
97 comments, sorted by Click to highlight new comments since:
Some comments are truncated due to high volume. (⌘F to expand all)Change truncation settings

I liked this, but I observe that I don't think it would have gotten nearly as much karma if it had been written by someone without a reputation already.

...which suggests that we're maybe doing something wrong?

Or is this in fact more informative, because we know that it came from Eliezer?

What the system is missing is the "ratio between reads to upvotes". When you just look at upvotes, that's a retrospective quality signal conflated with the prospective calculation people do about whether to try reading. The latter and earlier decision is obviously going to be username-dependent.

Yeah, I've experimented with showing that piece of information, but sadly it made people feel super self-conscious about even clicking on any post, because it would kind of be an implicit downvote. It's also kind of hard to determine what counts as a view (does any click count? Or any time someone reads to the end? Or sometime someone spends more than 10 seconds on the page?). 

Plausible we should experiment more with it. Or maybe do kind of a thing where you can get that piece of information but it's slightly inconvenient (maybe putting it on-hover is good enough).

3Mati_Roy
for unrelated reasons, I would also like a feature where I can mark a post as read (although that's probably not common enough to be used as direct signal, but could still be a proxy maybe)
3CraigMichael
Maybe if they spent the estimated amount of time on the page to read the article (or more), that counts as a view.
5Pattern
There's also something to be said about the impact of strong upvotes.
2RedFishBlueFish
Yeah, maybe we could show ratio of strong upvotes to upvotes

It feels like an important metric is "karma per view" or "karma per read" or "karma per user-minute-looking-at-text" something similar. Currently, we can't gauge that, and so when someone who gives a strong prior that their post will be worth reading posts, that post will get more views which means more upvotes even if they have a similar "karma per read".

EDIT: EY's post loaded the second I posted this, but I promise it was an independent invention

And ironically, your post has substantially less karma than Eliezer's does, despite saying basically the same thing!

3Pattern
Karma also involves strong upvotes (and that's before getting into 'people with more karma have their votes counted more). I'm reading this right now at EY: 37 karma, 19 votes (I don't know what the base line is for his default upvote of himself*.) Alex Hollow: 18 karma 11 votes. If we assume both consist of upvotes, then it seems pretty clear this a result of a) lots of strong upvotes (those votes aren't counting once). b) people whose 'upvote' increases karma by two, with a few strong upvotes thrown in, or c) a mixture of a and b. *His karma is 116,019 though.

I note that when I ran the Conor Moreton experiment, it seemed to me to confirm that karma and attention does accrue to good writing without prior reputation.  Perhaps it takes a couple of rounds, to get past people's ordinary why-should-this-be-promoted-to-my-attention-above-everything-else-competing-for-that-attention heuristics, but.

Nobody had a clue who Conor Moreton was, and after a week, he was being listened to.

It definitely suggests karma is an imperfect measure of quality/usefulness but I don't think that automatically means we're making the wrong trade-offs. 

(these thoughts are more general than this particular post, which I think is a little different just for being fiction).

Some posts require more energy than others to get the value out of them. When you're first looking at a post, it can be hard to estimate what the total return on energy is. In particular "deeply challenging and thought provoking" can look a lot "total bullshit", especially if the writer hasn't put a ton of skill and effort into making their challenging post easier to access. When faced with a new post that is definitely going to take a lot of energy and has an uncertain payoff, considering the reputation of the writer can be a legit useful heuristic.

This has a bunch of negative consequences, but it's really unclear to me if those outweigh the benefits, or if something could be done to improve the pareto frontier.

6ChristianKl
I do find more value in reading it given that it came from Eliezer. I do care about what Eliezer writes. 
5viluon
I'm delighted I read the post in the curated newsletter without noticing who was the author and only then decided to head here to upvote it. I wonder if moving the authorship information to the end of the newsletter influences the reader's willingness to upvote the post – perhaps an opportunity for an A/B test?
-3Dagon
Yeah, there are a few posters who'll get lots of upvotes regardless of the quality or utility of the post.  And more comments and discussion as well, which is a kind of self-fulfilling utility of the post.   I don't think it indicates we're doing something wrong, just the way cults like LW work.  Ok, that's way too harsh, but if you think it's important, that probably means you're taking karma too seriously.  Meaningless internet points are meaningless.
-9rank-biserial

I often wonder what effect the idea of Santa Claus has on children's cognitive development. We have created a vast conspiracy to deceive young children, spanning every possible layer of authority: your parents are in on it, your teachers are in on it, mass media and corporations are in on it, even the government is in on it. All working together to support the idea that a man from the North Pole is going to fly around the world and bring you presents.

As a child gets older, maybe pieces of the story start to not make sense. The child expresses a little doubt, but is quickly reassured. The child may think, what's more likely: that my doubts are misplaced, or that everyone in the world I've been taught to trust is wrong or lying to me?

Then finally the child's ability to suspend reason is pushed to its breaking point. Yes, everyone else is really is wrong. My own comprehension of the world is a more reliable source of truth than what I've been instructed to believe. The child finally confronts their parents. "You're right," the child is told. "You've passed the test. You're on the inside now. Don't tell your little sister."

The whole Santa Claus business is either a good idea or a bad idea. If it's a good idea, it could be taken further and further until it starts being a bad idea to go any farther than that!

8astridain
I have a slightly different perspective on this — I don't know how common this is, but looking back on my feelings on Santa Claus as a young child, they had more to do with belief-in-belief than with an "actual" belief in an "actual" Santa. It was religious faith as I understand it; I wanted, vaguely, to be the sort of kid who believed in Santa Claus; I looked for evidence that Santa Claus was real, for theories of how he could be real even if magic wasn't. So the lesson it taught me when I stopped believing in the whole thing was more of an insight about what it was like inside religious people's heads.
4redbird
Young kids don’t make a clear distinction between fantasy and reality. The process of coming to reject the Santa myth helps them clarify the distinction. It’s interesting to me that young kids function as well as they do without the notions of true/false, real/pretend! What does “belief” even mean in that context? They change their beliefs from minute to minute to suit the situation. Even for most adults, most beliefs are instrumental: We only separate true from false to the extent that it’s useful to do so!
4Duncan Sabien (Deactivated)
The above strikes me as more true than false, but not true thanks to some combination of making its claim too strongly/too universally/via a kind of typical-mind channel. If I had been trying to convey [my own version of this claim], I would have written something like: ... these hedges and caveats might feel like nitpicks, but they feel pretty important to me personally for not immediately losing track of what's true! =P
3jmh
You might like watching The Hog Father -- a twisted (sort of) remake of the Santa story. The punchline (as I recall) was basically how can we expect anyone to believe the big lies (morality, justice, truths?) if the small ones are not learned.

Don't tag "dath ilan", please; tags are shown at top and that's a humongous spoiler.

Hmm... Maybe there should be a way to spoiler-hide tags on posts? Cause not tagging it dath ilan would mean it wouldn't be discoverable through the tag page, which also isn't great.

Make a stub page with the dath ilan tag that links to this one?

2Multicore
Or put a spoilered link to this post in the dath ilan tag's wiki text?
4Adam Zerner
That would be nice, but it strikes me as something that would be somewhat low on the priority list.
9Kinrany
Honestly "fiction" was enough of a spoiler. "As a child, we were always told that every sapient life is precious." made it a certainty.

I didn't look at the tags before reading. I did notice it was fiction pretty quickly but "is this dath ilan" was still a live question for me until the reveal. (Though Eliezer might want to continue writing some non-dath ilan fiction occasionally, if he wants that to continue to be a likely thought process.)

29eB1
I didn't notice the author, nor the tags. My thought when the article asked for a prediction was that they grew up in a Jainist community of some kind, although I realized that Jainists may not have their own television shows, and the probability of someone from LessWrong having grown up in a Jainist community is probably less likely than that they grew up in a cult of some kind. But cults are even less likely to have their own television shows, I surmised, at which point I decided the point about television shows was probably rhetorical. That's when I decided the entire thing might be rhetorical, so it wasn't actually worth making a prediction.
1Dweomite
I got to the part about writing down a prediction, which prompted me to pay closer attention, and it was only at that point I noticed the "fiction" tag.
2gbear605
That's supposed to be a spoiler? I thought it was obvious from like the third paragraph.
7MichaelDickens
I thought it was obviously fiction, but I didn't know that it was set in Dath Ilan, and the fact that it's set in Dath Ilan would give away that the red hair thing is fake.
[-]Jiro320

If there is actually such a thing as Security, the best thing to do is whatever you think will keep you (or other people) alive. Mouthing off to a security officer about your actual reason for saving the redhead is behavior that is suboptimal because it would lead to your death. So testing kids to "see if they'll reject conformity" is actually testing kids to see if they'll pointlessly risk their life.

Furthermore, the belief that every sapient being is precious is something that the child was told. In other words, the child cannot claim "well, nobody reasonably argued for the exception" because nobody argued for the rule either!

And I don't see why the child doesn't say "you lied to me about Security. How do I know you're not lying to me about (insert every random thing about the society that the kid might find slightly questionable)." For that matter, the kid could think that "there's no security" is itself a lie and that the whole thing is a Hundred Flowers Campaign--the actual purpose is not to test for compassion, it's to detect troublemakers by giving them a few weeks to fully reveal themselves. It's your world, which isn't actually a Hundred Flowers Campaign, but how does the kid know, now that he's been lied to at least once?

Quite a good story. But I think at this point I would quite like Eliezer to make some sort of statement about to what degree he endorses Dath Ilan, ethically speaking.  As a fictional setting it's a great machine for fleshing out thought experiments, of course, but it seems downright dystopian in many ways. 

(I mean, the fact that they're cryopreserving everyone and have AGI under control means they're morally "preferable" to Earth, but that's sort of a cheat. For example, you could design an alt. history where the world is ruled by a victorious Third Reich, but where Hitler got super into cryopreservation in his old age, and poured a lot of resources and authority into getting the populations under his control to accept it too. Probably in the long run that world is "preferable" to the world where the Allies win but billions more brains rot — but it's still not much of a utopia in any useful sense.)

Of course, Eliezer previously stuck the non-consensual-sex thing in Three Worlds Collide, as an attempt to simulate the “future societies will likely trivialize things we consider unthinkable and there's no way to tell what” effect. I suspect — hope? — some of the ickier parts ... (read more)

3jimrandomh
A surprising number of people seem to have missed what the point of this was in Three Worlds Collide.  It's not a prediction about future (human) societies. AFAICT it's there to remind people that changing values is actually bad. That when we talk about, for example, AIs getting random values instead of inheriting our human values, that we should not think of this like we think of a foreign country's cultural quirks, we should think of this as terrifying and revolting. This is a misconception that a lot of people actually have, and TWC as a whole is aimed squarely at dispelling it.
9Audere
I strongly disagree that this was the point of this in TWC and would be highly surprised if Eliezer agreed with you. For one thing, the parties involved in nonconsensual sex in TWC seem to be having a perfectly fine time. I also wouldn't be surprised if someone raping an Ancient such that they have a terrible awful no-good time would fall under some other crime and still get the perpetrator arrested.
2astridain
I don't think those are contradictory? It can both be "there would be value drift" and "this might be quite bad, actually". Anyway, whatever the actual actual spirit of that bit in TWC, that doesn't change my question of wanting some clarity on whether the worse bits of Dath Ilan are intended in the same spirit.

Predictions over the course of my reading:

Guessed "dath ilan" from title.

Switched prediction to "Amenta" at mention of red hair.

Switched back to "dath ilan" (with much higher certainty) at the first use of capital-C "Civilization".

What were the lives of the red-headed children like? Were they in the know about the nature of the experiment? I suppose they must have been, but then how could such small kids be such good actors? I'm doubtful about the power of stories versus someone's experience, but I liked the story :) 

[-]gjm917

It's explicitly called out that the red-headed children "tended bigger than the rest of us, with more adult facial structures, to the point where you could've maybe mistaken them for very small adults in disguise". I suspect EY's intention is that they are in fact small adults in disguise.

I missed that!

I had assumed that they were neanderthals. 

I had wondered if they were hobbits.

5lsusr
I didn't figure that out. I assumed (incorrectly) that the red-headed children appeared more like adults because they lived in a harsher environment.

I said, at the end, was that I'd better be getting paid for this, and they all laughed and said of course I was, lots of money, at least as much as my parents were getting, because children are sapient beings too.

This seems like a rather hypocritical thing to say, unless dath ilan had some clever idea for how to implement this compensation that I'm failing to see right now.

If I was a subject in this experiment, there would be no amount of money you could pay me to retroactively agree that this was a fair deal. There's just nothing money can buy that would be worth the years of deception and the hours of mortal terror. 

If it was earth it'd be different, because earth has absolutely dire problems that can be solved by money, and given enough millions, that'd take precedence over my own mental wellbeing. But absent such moral obligations, it's just not worth it for me.

So do parents surreptitiously ask their children what sum of money they'd demand as compensation for participating in a wide variety of hypothetical experiments, some real, some fake, years before they move to a town like this? Seems rather impractical and questionable, considering how young the children would be when they made their choice.

All dath ilani kids grow up with enough adult gaslighting that they don't just grow up to believe anything authority tells them.

This was just the brand on offer in a particular village.

That seems to make it worse, not better?

3benjamincosman
Your feelings about lies depend on the context - for example I assume you'd be willing to play the game "Two Truths and a Lie", and you would not feel harmed by the certainty that someone is lying to you? In fact people enjoy the game, since seeing through the lie is a fun puzzle. Now outside of games like that, the majority of the time someone lies to you on Earth, it's to profit at your expense - they want to take your stuff, your vote, etc. So with the exception of games, you've quite reasonably developed strong negative emotions about being lied to, and those emotions may transfer even to the rarer cases where the lie isn't directly hurting you. But dath ilan is extremely high trust and high coordination; from childhood you will experience that the vast majority of the times someone lies to you, it's clearly-in-retrospect for your own benefit, and the vast majority of the remaining times, it's solidly for the benefit of Civilization and they're willing to eventually tell you the truth and pay for your inconvenience. So while you still try to see through lies whenever you can, it's much more like the Earth game setting: most lies are just harmless puzzles. So you don't grow up with the same internalized feelings that anyone lying to you is hurting you? (Which also reduces the price they'd have to pay you, since they're not trying to compensate you for an Earth-level of negative emotions.)

I am fairly confident that I would incredibly strongly dislike being lied to like this even if it were „for my own benefit“. The source of my disgust for nonconsensual lies does not seem to me to stem from a history of such lies hurting me. Rather, they just feel inherently hurtful. That's on top of the distress I would feel from years of keeping my mouth shut and my head down regarding the fake discrimination while secretly crying about it sometimes.

Also, there‘s still the hours of mortal terror that this scenario entails.

Given your perspective, you may enjoy: Lies Told To Children: Pinocchio, Which I found posted here.

Personally I think I'd be fine with the bargain, but having read that alternative continuation, I think I better understand how you feel.

4localdeity
At least in this scenario, I don't think the lies are for your benefit, they're for Civilization; and the adults involved do think it's appropriate to compensate you financially for being put through it, which implies they do believe the costs it imposes on you exceed the benefits to you. The other scenarios alluded to in Eliezer's above comment... well, I'd be interested to see how they play out.  I could believe that having multiple experiences of "being gaslit and eventually figuring it out" was a good thing on average... but I expect there'd be individual cases where it sucked pretty hard.  (If dath ilan was really good at psychology, maybe they could tell who would respond well vs badly beforehand?)  Details matter: exactly what is being lied about, how important it is, how long it goes, how pervasive it is (e.g. does the child have any completely trustworthy friends who haven't been in on any of the scams?), and so on.  For example, if it was "Here's a once-a-year festival at which there will be the following cool physical event", and the claimed event is physically impossible (say, it violates conservation of momentum) and faked (which the child is hoped to figure out after learning some physics), that seems relatively harmless.

A thing that feels a bit confused in this discussion is... currently the default state is that adults gaslighting kids... just happens all the time? One possible world you could try to engineer is a world where this never happens. Another world is one where it happens deliberately in a controlled fashion that teaches valuable life lessons and leaves children more resilient. The question is whether the former is actually tractable.

9Eliezer Yudkowsky
False! It just means that Civilization is benefiting and considers itself obliged to share a fair portion of those gains.
3localdeity
Oh, I see.  Perhaps something isomorphic to the following: we suppose Civilization gains $1 billion in value from the results, and we imagine the kids were in a position to negotiate a payment for their services rendered, and we figure they could have gotten $N hundred million, so we decide they deserve that much and divide it among the kids.  (Maybe the parents did some actual negotiating, on their own behalves at least.)

Presumably, from behind the Rawlsian veil, you might accept this deal in exchange for avoiding dystopia. I have not read the Dath Ilan glowfic, but I infer they are genetically/memetically engineered to me much more psychologically robust than us. 

Also note, we do much worse to children all the time. 

But yeah, it's sort of icky to live in a town where all the adults are conspiring to lie to you, but the vast majority of children in this world are not being lied to. And I think they made a lot of money, presumably the amount of money this rather-competent society predicted would be their "cheerful price".

I would certainly accept such treatment for 2 million dollars, for example. 

Though, this fellow seems to enjoy having this story to tell - so perhaps his cheerful price was rather low!

7Lucius Bushnaq
On earth, 200 million and I might consider it, though it sure wouldn‘t be my cheerful price. On dath ilan, not for any sum. Even fiat access to all economic output wouldn‘t be worth it. If you try to solve this with prediction, and have any kind of feedback mechanism in place where the project gets docked money in proportion to how much predicted cheerful prices diverged from occasionally measured actual cheerful prices, I expect your market to tell you that this project is prohibitively costly, because you can‘t get the chance of including children like me small enough. In addition, I don‘t know about you, but I would have objections to this situation even if a perfect/extremely good prediction mechanism was in place. Correlating events with my actual preferences is one reason I want people to ask for my consent before doing things to me, and perfect prediction takes care of that. But it is not the only reason. I also value being the person with final say inherently.  So if I were to be denied my right to deny consent, and told in the same sentence that of course I‘m a sapient being too and my preferences matter, it would taste rather bitter.
5Pattern
I'm wondering if I would accept a price even if I got to opt in to an experiment. Might be, to overthrow a government.
1NoriMori1992
  Yes, but most of it doesn't happen because approximately everyone in the world got together and shared all their knowledge and thought really hard about it and talked about it and then still decided it was a good idea.
[-]lc130

There's just nothing money can buy that would be worth the years of deception and the hours of mortal terror. 

Ridiculous. I would do this at least a dozen times for a billion dollars. This is practically the default society in human history. You would get over it.

I think I can model my own preferences better than you can, thank you very much. Regardless of whether I‘d „get over it“ or not, this experience would bother me more than anything extraordinary I can think of that I could plausibly buy in dath ilan‘s economy would please me.

1[comment deleted]

I say this because I think it relates to / could be within your same world:

One of my favorite current story ideas is to write a story where there's a girl who lives in a world where each society is running an experiment about the best way to live life. In school, she finds out that while most societies are aware of the experiment that's being run, her society is one of the few where the experiment is kept secret from the people in it.

As she grows up, she realizes how poor her society is, and how little technology they have. It improves very quickly over the years, but when she was younger they didn't even have washing machines or dishwashers.

So, she starts digging to try to figure out why it is that all their technology was so bad, and why her society started off so poor, and begins advocating for the idea that whatever experiment is being run, it should be shut down, because their quality of life is so much lesser than other societies.

Eventually, the movement she starts begins gaining momentum, and the entire society begins to revolt against being in the experiment. When she's finally able to bring her case to the main governing decision body (the council), she finds out what the ... (read more)

I don't understand one thing about the universe. The test and the fact that this story wind "lies-told-to-children" storytelling night, seems to indicate that most children never hear those dirty-red lies. However, if that causes those children to be raised, to learn, differently than most of the planet, this only mean that the test checks whether those children are actually good, not whether the whole planet education system raises good children. This is especially important as I can imagine and fear that seeing this discrimination for years, then having ... (read more)

3Rasmus Eide
IIUC, this particular village was for testing specific genetic tendencies (Does it also have in it's purpose to test epigenetic ones, or are those controlled for somehow?), there are presumably other villages where they use different setups to test various phases of upbringing and education on the matter, probably using twin studies to separate it form the genetic factors tested in this one. 
3sketerpot
Not just genetic tendencies; it was also meant to look at shared environment, like the TV shows subtly but thoroughly pushing a certain worldview, and the words from teachers saying it explicitly. From the story: they wanted "to find out how well we were doing environment-wise and heredity-wise on people's kindness and resistance to conformity-pushed cruelty".
2Pattern
What affects epigenetics?
2jimrandomh
The details of the experimental setup aren't fully spelled out, but I think you can infer from the mention of children's TV being censored that there's a connected experiment going on where these children are getting different TV shows than the broader society. So this could be measuring which TV programs produce more moral children, with two groups both receiving the same discrimination-against-reds test but watching different TV shows.

I've a regret that the requirement to write down thoughts only came once. Clearly, there was no way I ever could see where it goes at the time it was requested. However, later, as soon as Red was mentioned, I guessed the end, the tests. Still cried reading it. I suspect that if I didn't know who the author was, I would not have guessed, that seems to unusual for most author authors I love

(I was misled initially because I wound up thinking more Star Trek redshirts or possibly behavioral genetics than red teaming; out-of-universe reasoning can be tricky.)

Related story: https://slatestarcodex.com/2014/06/05/dont-be-an-asch-hole/

It is in fact an Amenta reference (Alicorn/glowfic setting in which red-haired people are discriminated against).

There wasn't actually any such thing as Security, and if there ever was it would mean that it was time to overthrow the government immediately.

I held back tears at this part.

I think that the idea of dath ilan being better at solving racism than earth social media is really valuable (in basically every different way that dath ilan stories are valuable, which is a wide variety of extremely different reasons). It should be covered again, at projectlawful at least, but this is a huge deal, writing more of it can achieve a wide variety of goals, and it definitely isn't something we should sleep on or let die here.

I feel like Conservative Judaism does this pretty well.

In Reform Judaism, everyone is in on the joke. Which means there's no obstacle to overcome, nothing to learn. Reform Jews win no points on Lies-Told-To-Children storytelling night.

Orthodox Jews play the game too seriously. Like, you get a bunch of people who win the game, and they tend to be interesting people who honed an important skill, but far too many don't get it during their lifetimes. Which means that they have these awful ideas like "we are God's chosen people" and "the world exists so we can ... (read more)

1Kenny
Is Conservative Judaism a 'distinct thing'? I know a few 'Orthodox Jews' and one of them seemed (or so my memory vaguely reports) to admit to me that they'd 'seen thru the lies', but the others didn't and I'd expect they would have if this was indeed a "formative experience" in the sense you seem to be implying.
3Rand0
Yes, Conservative Judaism is a distinct movement from Reform and Orthodox Judaism, generally regarded as existing in between the two in terms of theology and practice.  I don't quite understand the second question: Most Orthodox Jews never see through the lies, so they don't have that formative experience. But for those who do leave their faith, it tends to be an important part of their identity. (Note that I don't really know what percentage of Conservative and Reform Jews believe in God. I do think that the two movements have converged in beliefs and practices to a considerable degree over the last 50 years. Also, this is a very late response: I don't comment here often.)
2Kenny
Thanks for the info! And no worries about the (very) late response – I like that people fairly often reply at all (beyond same-day or within a few days) on this site; makes the discussions feel more 'timeless' to me. The second "question" wasn't a question, but it was due to not knowing that Conservative Judaism is distinct from Orthodox Judaism. (Sadly, capitalization is only relatively weak evidence of 'proper-nounitude'.)

The child does not even ask after the red headed kids? If the red-headed kids are real, sentient beings, this experiment is going to damage near everyone involved; we know the social problems and trauma caused by the Stanford prison experiment, and that was consensual, informed and contained.

The experiment also fundamentally underestimates what it takes to resist indoctrination. If your government systematically oppresses a group, and wants you to go along to it, they will not teach you universal rights spanning across these groups and so set you up to be ... (read more)

2Muga Sofer
The red-headed kids were adults in disguise.

Curated. 

>! At first, I thought about this piece as a comment on educational systems, but that's not really it. The post is about how a society attempts to audit itself and its education. Our community is not at the scale where this setyp is feasible (nor do we necessarily have the time), but it does get me thinking about what are reasonable tests that demonstrate whether we're morally and epistemologically on track. 

Growing up, as a kid, I was always told that every sapient life is precious, everything that thinks and knows itself -

Between this beginning and the title, I thought this was going someplace completely different.

Excellent short story!

Curious if anyone has any non-plot-related takeaways that might be enlightening.

2Pattern
How would the results compare if they were non-human?

When you realize that they totally have a city somewhere that's running the control trial.

This is the first thing I've read set in dath ilan and I think I'd enjoy reading more.

I see the tag wiki lists several other writings in dath ilan, would anyone here care to suggest a reading order for me?

1Kenny
I think date-posted is a fine 'default' order for reading the other posts. Some of the other posts with this tag are more indirect, i.e. not stories but about stories.

Was this an attempt to flesh out why Dath Ilani would have low posteriors on there being a conspiracy? 

So i read a lot of dah-ilani glowfics the previous weeks, and yet, i didn't guess right. i didn't stop to put numbers on it, so i can in retrospective (and still not sure it's actually good idea to put numbers on all things). and it was 0.9 that the stroy is about kid losing trust in adults because they was told a lie, and 0.3 that after that, it turned out they should trust adults and this distrust is bad (like teens that think all drugs are not dangerous because adults exaggerate the harm of the less-harmful ones). in that situation, i was basically 50-5... (read more)

Really engaging! Was especially gripped at the bluffing section, and laughed out loud in shock at the reveal. Really enjoyed it :)

Seems like the real test would be to do it without the television shows?

I think it would be more accurate to say that the test was meant to check whether the TV shows were effective than whether the children had a maximal inherent tendency towards virtuousness.

2Pattern
I think they're a way to make an experiment easier to do. (It's not actually clear what the point of the experiment is - to figure out how to shape behavior, or what.)
1Measure
Maybe there are multiple test conditions, or maybe this is just the one that the market settled on.
3sketerpot
There's also variation in the amount of children's television watched by people in the test population. The protagonist mentions seeing "less children's television than most children in our age cohort", and it's a safe bet that this was tracked as part of the data set.
[+][comment deleted]10
[+][comment deleted]10