Sunzi said: The art of war is of vital importance to the State. It is a matter of life and death, a road either to safety or to ruin. Hence it is a subject of inquiry which can on no account be neglected.
The art of war, then, is governed by five constant factors, to be taken into account in one's deliberations, when seeking to determine the conditions obtaining in the field. These are: (1) The Moral Law; (2) Heaven; (3) Earth; (4) The Commander; (5) Method and discipline.
The Moral Law causes the people to be in complete accord with their ruler, so that they will follow him regardless of their lives, undismayed by any danger.
The very first factor in the very first chapter of The Art of War is about the importance of synchronous goals between agents and represented. It is instrumental in preserving the state. It is also instrumental in preserving the state (sic).
Sun Tzu replied: "Having once received His Majesty's commission to be the general of his forces, there are certain commands of His Majesty which, acting in that capacity, I am unable to accept."
Accordingly, he had the two leaders beheaded, and straightway installed the pair next in order as leaders in their place.
A metaphor.
ridiculously strong aversion
The iron is hot, some feel fear.
just suggesting an improvement
You aren't though.
You're expressing belief in a possible downside of current practice. We can say, unconditionally and flatly, that it is a downside, if real, without it being right to minimize that downside. To your credit, you also argue that effects on the average influenced person are less valuable than is generally thought, which if true would be a step towards indicating a change in policy would be good.
But beyond that, you don't articulate what would be a superior policy, and you have a lot of intermediary conclusions to establish to make a robust criticism.
You aren't though.
Correct, I was imprecise. I'm listing a downside and listing nonobvious considerations that make it more of a downside than might be assumed.
A short reply to the Book of Eliezer and a comment on the Book of Luke.
No one wants to save the world. You must thoroughly research this. Those who think they truly think they want to truly want to save the world, in reality they're actually just horribly afraid of the consequences of not saving the world. And that is a world of difference.
Eliezer, you know that ridiculously strong aversion to lost purposes and sphexishness that you have?1 Sometimes, very rarely, other people have that too. And most often it is a double-negative aversion. I am sure you know as much as very nearly anyone what it feels like to work from the inside of a triple-negative motivation system by default, for fear of being as evil and imperfect as every other human in history, among other less noble fears. You quickly learn to go meta to escape the apparently impossible double-binds—if going meta isn't itself choosing a side—but by constantly moving vertically you never practice pushing to the left or to the right, or choosing which responsibility to sacrifice in the first place. And even if you could, why would you want to be evil?
And for this rare kind of person, telling them to stop obsessing over prudence or to just try to make marginal contributions, immediately gets pattern-matched to that ages-old adage: "The solution is easy, just shut up and be evil.". Luckily it is this kind of person we can make the most use of, when it comes to the big crunch time—if we're not already in it.
1We do not yet know how to teach this skill, and no one can be a truly aspiring rationalist without it, even if they can still aspire to perfection. That does mean I believe there are like maybe 5 truly aspiring rationalists in this community, a larger set of falsely aspiring rationalists, a further much larger set of of truly aspiring aspiring "rationalists", and a further much much larger set of falsely aspiring aspiring "rationalists". (3, 30, 300, 3000, say.) I don't think anyone thinks about this nearly enough, because no one has any affordance—no affordance to not not-think about it—especially not when they're thinking fuzzy happy thoughts about creating aspiring rationalists or becoming a rationalist.