truly aspiring rationalists
Does that mean people for whom rationalism is a near-terminal goal that cannot become a lost purpose? Do you use "rationalism" somewhat like the way Charlie Sheen might use "winning", as "rationalism" is often used here?
If yes and no, then to what end rationalism?
If yes and yes, then you value that for someone who, in relation to various things, wants them, that they have as a cherished thing achieving their own wants? Is people having such a nearly-terminal goal a correspondingly deep value of yours, or is it more instrumental? Either way, is coming to value that one of the smaller changes I could make to turn my values towards consistency (and how much more change than coming to consciously value it would I have to do if it is not emergent from my existing values)? If so, at what level would I be valuing that, presumably the same as you do, no? It isn't enough to have a passing devotion to wanting that, that which I want, I should get it?
If this is unclear or badly off-target, let it indicate the magnitude of my confusion as to what you meant.
And for this rare kind of person, telling them
This comes to mind.
[09:28] Eliezer: if I had to take a real-world action, like, guessing someone's name with a gun to my head
[09:29] Eliezer: if I had to choose it would suddenly become very relevant that I knew Michael was one of the most statistically common names, but couldn't remember for which years it was the most common, and that I knew Michael was more likely to be a male name than a female name
[09:29] Eliezer: if an alien had a gun to its head, telling it "I don't know" at this point would not be helpful
[09:29] Eliezer: because there's a whole lot I know that it doesn't
[09:30] X: ok
[09:33] X: what about a question for which you really don't have any information?
[09:33] X: like something only an alien would know
[09:34] Eliezer: if I have no evidence I use an appropriate Ignorance Prior, which distributes probability evenly across all possibilities, and assigns only a very small amount to any individual possibility because there are so many
[09:35] Eliezer: if the person I'm talking to already knows to use an ignorance prior, I say "I don't know" because we already have the same probability distribution and I have nothing to add to that
[09:35] Eliezer: the ignorance prior tells me my betting odds
[09:35] Eliezer: it governs my choices
[09:35] X: and what if you don't know how to use an ignorance prior
[09:36] X: have never heard of it etc
[09:36] Eliezer: if I'm dealing with someone who doesn't know about ignorance priors, and who is dealing with the problem by making up this huge elaborate hypothesis with lots of moving parts and many places to go wrong, then the truth is that I automatically know s/he's wrong
[09:36] Eliezer: it may not be possible to explain this to them, short of training them from scratch in rationality
[09:36] Eliezer: but it is true
[09:36] Eliezer: and if the person trusts me for a rationalist, it may be both honest and helpful to tell them, "No, that's wrong"
Eliezer obviously wouldn't be telling them to shut up and be evil; he'd be intending to tell the person he'd infer he was talking to that, and if this "rare person" couldn't learn about Eliezer's actual intent by inferring the message Eliezer had intended to communicate to who Eliezer ought to have thought he was talking to, the person would be rarely dense.
So that part of Eliezer's message is not flawed, so I'm not sure why you thought it needed addressing.
This assumes I'm reading this post correctly, something I'm not confident of.
Does that mean people for whom rationalism is a near-terminal goal that cannot become a lost purpose?
Maybe in some way, but not in the way that you interpret it to mean... I emphasize the importance of noticing lost purposes, which is central to both epistemic and instrumental rationality. Elsewhere in this thread I re-wrote the post without the cool links, if you're interested in figuring out what I originally meant. I apologize for the vagueness.
As for your second critique, I'm not claiming that Eliezer's message is particularly flawed, just suggestin...
A short reply to the Book of Eliezer and a comment on the Book of Luke.
No one wants to save the world. You must thoroughly research this. Those who think they truly think they want to truly want to save the world, in reality they're actually just horribly afraid of the consequences of not saving the world. And that is a world of difference.
Eliezer, you know that ridiculously strong aversion to lost purposes and sphexishness that you have?1 Sometimes, very rarely, other people have that too. And most often it is a double-negative aversion. I am sure you know as much as very nearly anyone what it feels like to work from the inside of a triple-negative motivation system by default, for fear of being as evil and imperfect as every other human in history, among other less noble fears. You quickly learn to go meta to escape the apparently impossible double-binds—if going meta isn't itself choosing a side—but by constantly moving vertically you never practice pushing to the left or to the right, or choosing which responsibility to sacrifice in the first place. And even if you could, why would you want to be evil?
And for this rare kind of person, telling them to stop obsessing over prudence or to just try to make marginal contributions, immediately gets pattern-matched to that ages-old adage: "The solution is easy, just shut up and be evil.". Luckily it is this kind of person we can make the most use of, when it comes to the big crunch time—if we're not already in it.
1We do not yet know how to teach this skill, and no one can be a truly aspiring rationalist without it, even if they can still aspire to perfection. That does mean I believe there are like maybe 5 truly aspiring rationalists in this community, a larger set of falsely aspiring rationalists, a further much larger set of of truly aspiring aspiring "rationalists", and a further much much larger set of falsely aspiring aspiring "rationalists". (3, 30, 300, 3000, say.) I don't think anyone thinks about this nearly enough, because no one has any affordance—no affordance to not not-think about it—especially not when they're thinking fuzzy happy thoughts about creating aspiring rationalists or becoming a rationalist.