Self-Integrity and the Drowning Child
I think the key to the drowning child parable is the ability of others to judge you. I can't judge you for not donating a huge portion of your income to charity, because then you'll bring up the fact that I don't donate a huge portion of my own income to charity. Sure, there are people who do donate that much, but they are few enough that it is still socially safe to not donate. But I can judge you for not saving the child, because you can't challenge me for not saving them - I was not there. This means that not saving the child poses a risk to your social status, which can greatly tilt the utility balance in favor of saving them.
But how exactly do you do this without hammering down on the part that hammers down on parts? Because the part that hammers down on parts really has a lot to offer, too, especially when it notices that one part is way out of control and hogging the microphone, or when it sees that one part is operating outside of the domain in which its wisdom is applicable.
(Your last paragraph seems to read "and now, dear audience, please see that the REAL problem is such-and-such a part, namely the part that hammers down on parts, and you may now proceed to hammer down on this part at will!")
You can apply the lesson to that conclusion as well, avoid hammering down on the part that hammers down on parts. The point is not to belittle it, but to reform it so that it's less brutishly violent and gullible, so that the parts of mind it gardens and lives among can grow healthy together, even as it judiciously prunes the weeds.
I agree with and appreciate the broad point. I'll pick on one detail because I think it matters.
this whole parable of the drowning child, was set to crush down the selfish part of you, to make it look like you would be invalid and shameful and harmful-to-others if the selfish part of you won [...]
It is a parable calculated to set at odds two pieces of yourself... arranging for one of them to hammer down the other in a way that would leave it feeling small and injured and unable to speak in its own defense.
This seems uncharitable? Singer's thought experiment may have had the above effects, but my impression's been that it was calculated largely to help people recognize our impartially altruistic parts—parts of us that in practice seem to get hammered down, obliterated, and forgotten far more often than our self-focused parts (consider e.g. how many people do approximately nothing for strangers vs. how many people do approximately nothing for themselves).
So part of me worries that "the drowning child thought experiment is a calculated assault on your personal integrity!" is not just mistaken but yet another hammer by which people will kick down their own altruistic parts—the parts of us that protect those who are small and injured and unable to speak in their own defense.
Yeah ok. But this essay was posted on Earth. And on Earth I read it as response to a percieved failure-mode of an Effective Altruism philosophy.
At a pond, my niece was in a child floaty, reached too far and flipped over into the water. I slammed my half-eaten sandwich on my brother's chest, hoping he would grab it and ran into the water and saved her.
She was fine and I got to finish my sandwich.
Regarding the direct example
I feel like it's self-subverting. There's an old canard about https://www.watersafetymagazine.com/drowning-doesnt-look-like-drowning/ Given how staggeringly disproportionate the utility losses are in this scenario I think even a 1% chance of my assumption that 'I have 15 seconds to undress' would lead to death means I should act immediately.
In general when thinking about superfast reflex decisions vs thought out decisions: Obey the reflex unless your ability to estimate the probabilities involved has really low margins of error. My gut says X but my slow, super weak priors-that-have-never-been-adjusted-by-real-world-experience-about-this-first-time-in-my-life-situation say Y... Yeah just go with X. Reflect on the outcome later and maybe come up with a Z that should have been the gut/reflex response.
There's an old video game Starcraft 2 advice from Day9 that's surprisingly applicable in life: Plan your game before the game, in game follow the plan even if it seems like it's failing, after the game review and adjust your plan. Never plan during the game, speed is of the essence and the loss of micro and macro speed will cost you more than a bad plan ...
1Humans don't swim very well wearing lots of clothing. Take off your suit before going into the water.
I actually keep thinking this in the back of my own mind every time I run into this parable, so thank you for stating it out loud. (I expect if a child brought it up, the Watcher credited them for noticing further consequences, then asked to assume the Less Convenient Possible World where this is not the case.)
Trying to summarize for those of us not fond of long-winded parables.
- A single moral agent is a bad model of a human,
- Multiple agents with individual utilities/rules/virtues is a more accurate model.
- It's useful to be aware of this, because Tarski, or else your actions don't match your expectations.
I worry that "parts as people" or even "parts as animals" models are putting people on the wrong path to self-integrity, and I did my best to edit this whole metaparable to try to avoid suggesting that any point.
I worry that "parts as people" or even "parts as animals" models are putting people on the wrong path to self-integrity
I'd very much love to hear more about this. (Including from others, both for and against.)
Oops... I guess I misunderstood what you meant by "two pieces of yourself".
Anyway, I really like the part
you failed to understand and notice a kind of outside assault on your internal integrity, you did not notice how this parable was setting up two pieces of yourself at odds, so that you could not be both at once, and arranging for one of them to hammer down the other in a way that would leave it feeling small and injured and unable to speak in its own defense
because it attends to the feelings and not just to the logic: "hammer down the other in a way that would leave it feeling small and injured".
I could have designed an adversarial lecture that would have driven everybody in this room halfway crazy - except for Keltham
I... would love to see one of those, unless you consider it an infohazard/Shiri's scissor.
> I could have designed an adversarial lecture that would have driven everybody in this room halfway crazy - except for Keltham
I... would love to see one of those, unless you consider it an infohazard/Shiri's scissor.
I think this might just mean using the drowning child argument to convince the students they should be acting selflessly all the time, donating all their money above minimal subsistence, etc.
If the people on the other side of the argument ended up behaving coherently, rather than twisting themselves into knots and burning themselves out as their inner gears ground against themselves in unresolvable circles, it wouldn't be much of an adversarial lecture, would it?
Regarding pieces of oneself, consider the ideas of IFS (internal family systems). "Parts" can be said to attenuate to different concerns and if one can distract from others then an opportunity to maximize utility across dimensions may be missed. One might also suggest that attenuation to only one concern over time can result in a slight movement towards disintegration as a result of increasingly strong feelings about "ignored" concerns. Integration or alignment, with every part joining a cooperative council is often considered a goal and personification can assist some in more peaceably achieving that. I personally found the suggestion to personify felt weird and false.
I unfortunately have very little of substance to add, but a strong upvote was not quite enough.
There is something in here of Iron Hufflepuff, and I'm exceedingly grateful to Eliezer for dignifying and validating it so unequivocally in this meta-parable. I expect to link to this fairly frequently over the next decade.
An interesting difference between the drowning child situation and the "could donate to effective charity to save children's lives" situation is that the person who happens to be walking by that pond has a non-transferable opportunity to save a child's life for $500 (or whatever the cost of the clothes are, plus some time cost, and the inconvenience of getting wet and muddy). In the case of effective charity, even if one declines to donate, other people will still have the same opportunity. In the case of the drowning child, the fact that you are the only one who can act makes jumping in to save the child somehow seem more urgent. If you don't save the child, then you'd be somehow "wasting" a valuable opportunity. For a mostly selfish person who values all lives other than their own at less than $500, the opportunity would be valuable to others but useless for themselves.
If the going rate is $1000 to save a child through effective charity, then a mostly altruistic person would be willing to pay a mostly selfish person $600 to compensate for the costs of their clothes. There would have to be some "honour" involved, since the selfish person couldn't exactly unsave the child after the fact. If they could make the deal work anyway, then the mostly selfish person would have succeeded in selling non-transferable opportunity for $100, and it would be worthwhile for them to save the drowning child.
I would like to distinguish between money burnt and money transferred. The $500 are burnt, assuming you handpicked homegrown cotton and knitted it into clothes. The $1000 are also burnt, assuming that nobody extracts rent on saving children. The $600 are merely transferred, and so the altruist may be willing to pay more even than $1000, if he expects the profits spent in ways he still moderately endorses.
My objection lies in the second part of the drowning child parable. The part where someone geographically distant is considered identical to the child in front of me, and money is considered identical to the actions of saving. It's some sort of physics being the same everywhere intuition being inappropriately applied. Of course distance in time, space, or inference create uncertainty. Of course uncertainty reduces expected value and possibly even brings the sign of the action into question if the expected variance is high enough.
A literal drowning child puts a limit on your commitment. Save this child, and your duty is discharged. When we apply this moral intuition to all the other issues in the world, our individual obligation suddenly becomes all-consuming.
Furthermore, a literal drowning child is an accident. It represents a drastic exception to the normal outcomes of your society. Your saving action is plugging a hole in a basically sound system. Do our moral intuitions stem from a consequentialist goal to save all lives that can be saved? Or do they stem from an obligation to maintain a healthy, caring, and more-or-less self-sufficient society?
To me, the best interpretation of the drowning child parable extended to a global level is that it gives me a sense of moral glee. Holy smokes! The mere act of donating money, or of doing direct work in a powerful cause for good, can save lives just the way that a more conventional heroic action can! How cool!
I'd import Eliezer's concept of a "cheerful price," but in reverse. Instead of being paid in money to cheerfully take an action I'd otherwise rather not do, I am being paid in lives saved to cheerfully give some money I'd otherwise rather not donate. A life saved for a mere $10,000? A bargain at twice the price!
Furthermore, a literal drowning child is an accident. It represents a drastic exception to the normal outcomes of your society.
This is a good point. I never noticed it before.
Equally importantly IMO, it argues for transfer from a context where the effect of your actions is directly perceptionally obvious to one where it is unclear and filters through political structures (e.g., aid organizations and what they choose to do and to communicate; any governments they might be interacting with; any other players on the ground in the distant country) that will be hard to model accurately.
Curated. To generalize, as the stakes continue to seem high ("most important century"-level high), it's easy to feel an immense obligation to act and to give it all up for the sake of the future. This meta-parable reminds us that humans aren't made solely of parts that give everything up, and that it's a matter of self-integrity to not do so.
not selfish and unselfish components in their utility function, but parts of themselves in some less Law-aspiring way than that
Utility functions don't model all agents; we should look at a larger space. I expect it to better model not just a human but also a council of humans or a multiverse of acausal traders. I expect this also to say how an AGI should handle uncertainty about preferences.
There should be a natural way to aggregate a distribution of agents into an agent, obeying the obvious law that an arbitrarily deeply nested distribution comes out the ...
Something with a utility function, if it values an apple 1% more than an orange, if offered a million apple-or-orange choices, will choose a million apples and zero oranges. The division within most people into selfish and unselfish components is not like that, you cannot feed it all with unselfish choices whatever the ratio. Not unless you are a Keeper, maybe, who has made yourself sharper and more coherent; or maybe not even then, who knows?
I fear that this parable encourages a view whereby the utility function "should" factorize over intuiti...
TLDR: if we model a human as a collection of sub-agents rather than single agent, how do we make normative claims about which sub-agents should or shouldn't hammer down others? There's no over-arching set of goals to evaluate against, and each sub-agent always wants to hammer down all the others.
If I'm interpreting things right, I think I agree with the descriptive claims here, but tentatively disagree with the normative ones. I agree that modeling humans as single agents is inaccurate, and a multi-agent model of some sort is better. I also agree that the ...
This post felt like a great counterpoint to the drowning child thought experiment, and as such I found it a useful insight. A reminder that it's okay to take care of yourself is important, especially in these times and in a community of people dedicated to things like EA and the Alignment Problem.
In case anyone else, like me, followed the link near the start of OP to the story from which this is excerpted, and is wondering whether (having had a bunch of updates on 2021-10-24 but none since) it's likely to be dead: I had a look at its pattern of updates and it's very bursty, with gaps on the order of 1-3 weeks between bursts, so the current ~1w of no updates is not strong evidence against there being more future updates to come.
(It seems like the majority of fictional works posted online end up being abandoned, and I generally prefer not to read things of non-negligible length that stop in mid
The Watcher spoke on, then, about how most people have selfish and unselfish parts—not selfish and unselfish components in their utility function, but parts of themselves in some less Law-aspiring way than that.
I guess it's appropriate that children there learn about utility functions before learning about multiplication.
Perhaps the parable could have been circumvented entirely by never teaching the children that such a thing as a “utility function” existed in the first place. I was mildly surprised to learn that the dath ilani used the concept at all, rather than speaking of preferences directly. There are very few conversations about relative preference that are improved by introducing the phrase “utility function.”
Utility functions are very useful for solving decision problems with simple objectives. Human preference is not one of these, but we can often fit a utility function that approximately captures it in a particular situation, which is useful for computing informed suggestions for decisions. The model of one's preference that informs fitting of utility functions to it for use in contexts of particular decision problems could also be called a model of one's utility function, but that terminology would be misleading.
The error is forgetting that on human level, all utility functions you can work with are hopelessly simplified approximations, maps of some notion of actual preference, and even an understanding of all these maps considered altogether is a hopelessly simplified approximation, not preference itself. It's not even useful to postulate that preference is a utility function, as this is not the form that is visible in practice when drawing its maps. Still, having maps for a thing clarifies what it is, better than not having any maps at all, and better yet when maps stop getting confused for the thing itself.
I reject the parable/dilemma for another reason: in the majority of cases, I don't think it's ethical to spend so much money on a suit that you would legitimately hesitate to save a drowning child if it put the suit at risk?
If you're so rich that you can buy tailor-made suits, then sure, go save the child and buy another suit. If you're not... then why are you buying super-expensive tailor-made suits? I see extremely few situations where keeping the ability to play status games slightly better would be worth more than saving a child's life.
(And yes, there'...
This applies to integrity of a false persona just as well, a separate issue from fitting an agentic persona (that gets decision making privileges, but not self-ratification privileges) to a human. Deciding quite clearly who you are doesn't seem possible without a million years of reflection and radical cognitive enhancement. The other option is rewriting who you are, begging the question, a more serious failure of integrity (of a different kind) whose salience distracts from the point of the dath ilani lesson.
I prefer hypocrisy to cruelty.
More gennerally I think this just misses the point of drowning child. The argument is not that you have this set of preferences and therefore you save the child, the argument is that luxury items are not of equal moral worth to the life of a child. This can be made consistent with taking off your suit first if think the delay has a sufficiently small probability of leading to the death of child and you think the death of a child and the expensive suit are comparable.
(Excerpted from "mad investor chaos and the woman of asmodeus", about an unusually selfish dath ilani, "Keltham", who dies in a plane accident and ends up in Cheliax, a country governed by D&D!Hell. Keltham is here remembering an incident from his childhood.)
And the Watcher told the class a parable, about an adult, coming across a child who'd somehow bypassed the various safeguards around a wilderness area, and fallen into a muddy pond, and seemed to be showing signs of drowning (for they'd already been told, then, what drowning looked like). The water, in this parable, didn't look like it would be over their own adult heads. But - in the parable - they'd just bought some incredibly-expensive clothing, costing dozens of their own labor-hours, and less resilient than usual, that would be ruined by the muddy water.
And the Watcher asked the class if they thought it was right to save the child, at the cost of ruining their clothing.
Everyone in there moved their hand to the 'yes' position, of course. Except Keltham, who by this point had already decided quite clearly who he was, and who simply closed his hand into a fist, otherwise saying neither 'yes' nor 'no' to the question, defying it entirely.
The Watcher asked him to explain, and Keltham said that it seemed to him that it was okay for an adult to take an extra fifteen seconds to strip off all their super-expensive clothing and then jump in to save the child.
The Watcher invited the other children to argue with Keltham about that, which they did, though Keltham's first defense, that his utility function was what it was, had not been a friendly one, or inviting of further argument. But they did eventually convince Keltham that, especially if you weren't sure you could call in other help or get attention or successfully drag the child's body towards help, if that child actually did drown - meaning the child's true life was at stake - then it would make sense to jump in right away, not take the extra risk of waiting another quarter-minute to strip off your clothes, and bill the child's parents' insurance for the cost. Or at least, that was where Keltham shifted his position, in the face of that argumentative pressure.
Some kids, at that point, questioned the Watcher about this actually being a pretty good point, and why wouldn't anyone just bill the child's parents' insurance.
To which the Watcher asked them to consider hypothetically the case where insurance refused to pay out in cases like that, because it would be too easy for people to set up 'accidents' letting them bill insurances - not that this precaution had proven to be necessary in real life, of course. But the Watcher asked them to consider the Least Convenient Possible World where insurance companies, and even parents, did need to reason like that; because there'd proven to be too many master criminals setting up 'children at risk of true death from drowning' accidents that they could apparently avert and claim bounties on.
Well, said Keltham, in that case, he was going right back to taking another fifteen seconds to strip off his super-expensive clothes, if the child didn't look like it was literally right about to drown. And if society didn't like that, it was society's job to solve that thing with the master criminals. Though he'd maybe modify that if they were in a possible-true-death situation, because a true life is worth a huge number of labor-hours, and that part did feel like some bit of decision theory would say that everyone would be wealthier if everyone would sacrifice small amounts of wealth to save huge amounts of somebody else's wealth, if that happened unpredictably to people, and if society was also that incompetent at setting up proper reimbursements. Though if it was like that in real life instead of the Least Convenient Possible World, it would mean that Civilization was terrible at coordination and it was time to overthrow Governance and start over.
This time the smarter kids did not succeed in pushing Keltham away from his position, and after a few more minutes the Watcher called a halt to it, and told the assembled children that they had been brought here today to learn an important lesson from Keltham about self-integrity.
Keltham is being coherent, said the Watcher.
Keltham's decision is a valid one, given his own utility function (said the Watcher); you were wrong to try to talk him into thinking that he was making an objective error.
It's easy for you to say you'd save the child (said the Watcher) when you're not really there, when you don't actually have to make the sacrifice of what you spent so many hours laboring to obtain, and would you all please note how none of you even considered about whether or not to spend a quarter-minute stripping off your clothes, or whether to try to bill the child's parents' insurance. Because you were too busy showing off how Moral you were, and how willing to make Sacrifices. Maybe you would decide not to do it, if the fifteen seconds were too costly; and then, any time you spent thinking about it, would also have been costly; and in that sense it might make more sense given your own utility functions (unlike Keltham's) to rush ahead without taking the time to think, let alone the time to strip off your expensive fragile clothes. But labor does have value, along with a child's life; and it is not incoherent or stupid for Keltham to weigh that too, especially given his own utility function - so said the Watcher.
Keltham did have enough dignity, by that point in his life, not to rub it in or say 'told you so' to the other children, as this would have distracted them from the process of updating.
The Watcher spoke on, then, about how most people have selfish and unselfish parts - not selfish and unselfish components in their utility function, but parts of themselves in some less Law-aspiring way than that. Something with a utility function, if it values an apple 1% more than an orange, if offered a million apple-or-orange choices, will choose a million apples and zero oranges. The division within most people into selfish and unselfish components is not like that, you cannot feed it all with unselfish choices whatever the ratio. Not unless you are a Keeper, maybe, who has made yourself sharper and more coherent; or maybe not even then, who knows? For (it was said in another place) it is hazardous to non-Keepers to know too much about exactly how Keepers think.
It is dangerous to believe, said the Watcher, that you get extra virtue points the more that you let your altruistic part hammer down the selfish part. If you were older, said the Watcher, if you were more able to dissect thoughts into their parts and catalogue their effects, you would have noticed at once how this whole parable of the drowning child, was set to crush down the selfish part of you, to make it look like you would be invalid and shameful and harmful-to-others if the selfish part of you won, because, you're meant to think, people don't need expensive clothing - although somebody who's spent a lot on expensive clothing clearly has some use for it or some part of themselves that desires it quite strongly.
It is a parable calculated to set at odds two pieces of yourself (said the Watcher), and your flaw is not that you made the wrong choice between the two pieces, it was that you hammered one of those pieces down. Even though with a bit more thought, you could have at least seen the options for being that piece of yourself too, and not too expensively.
And much more importantly (said the Watcher), you failed to understand and notice a kind of outside assault on your internal integrity, you did not notice how this parable was setting up two pieces of yourself at odds, so that you could not be both at once, and arranging for one of them to hammer down the other in a way that would leave it feeling small and injured and unable to speak in its own defense.
"If I'd actually wanted you to twist yourselves up and burn yourselves out around this," said the Watcher, "I could have designed an adversarial lecture that would have driven everybody in this room halfway crazy - except for Keltham. He's not just immune because he's an agent with a slightly different utility function, he's immune because he instinctively doesn't switch off a kind of self-integrity that everyone else in this class needs to learn to not switch off so easily."