Not wanting to become superior because that might make you overconfident...
...is not what I'm talking about.
The self-critiquing-reversibility test is designed specifically to prevent apparent self improvements which are not actual self improvements and from which you cannot retreat. If the test is passed then it should give you more room to play and explore because you actually have a safety net in the form of a "bailout option".
The test is designed to prevent you from, for example, getting addicted to a purported nootropic that turns out to be more like crystal meth than like caffeine. Avoiding "belief in the value of irrational belief" is another place where the heuristic might be applied.
For Bella, thing vampires can't do include turning off their desire for blood, or changing their emotional connection to their mates. These are, in some sense, "permanent utility function tweaks" rather that simple "optimization power upgrades".
If Harry had applied the test in the first handful of chapters of MoR, he would have asked McGonagall if it was possible for him to explore the wizarding world but then back out somehow if he decided it was better to be a muggle instead of a wizard after educating himself about the costs and benefits of both states. The best answer from McGonagall (though I don't think she can actually do this, which may be relevant) is "Here, let me take veritaserum... Now... Yes, easily, because memories can be erased with an obliviation spell and returning to a naive state will be basically the same as never having learned about the wizarding world in the first place, but you'll find that the cost benefit analysis is unambiguously positive because of things like X and Y which appeal to you right now. The biggest downsides are P and Q and similar issues which are obviously negligible in the face of X and Y."
keep in mind she has evidence that this irreversible transition would make her better at improving
Absolutely. Resilience and naive optimization are often in conflict.
The highest expected value strategy in investing is to put all your money in the single investment that itself has the highest expected value (assuming the opportunity is large enough that your whole contribution doesn't push the project very far down its marginal utility curve so the last dollar invested will have lower return on investment than some other investment). Nonetheless an index fund can be a better strategy based on variance estimates and more or less sophisticated risk of ruin calculations combined with the value of "avoiding ruin". Nearly all billionaires are massively "over invested" in their own companies and they frequently stop being billionaires for this very reason. The fortune 500 has substantial turnover decade-over-decade in part because a company has to sacrifice some resilience to get onto that list and in the long run (since corporations are potentially immortal), a lack of resilience catches up to them.
This is what I was trying to get at with the link about causal density. Applying the epistemic reversibility test too diligently can be inferred from first principles to hurt you if you are in a "get big fast" regime where the only survivors are lucky risk takers. Or maybe it can hurt you for some other reason I don't know about yet that will make more sense to me if I apply it some day and then get hurt in a novel way...
And, honestly speaking, for any given heuristic I consciously apply, I expect to gain some benefit, while generally expecting to get hurt sometimes. If I keep doing novel stuff with an eye towards rational self improvement it seems inevitable that I'll get hurt in a way I wasn't expecting - however it seems reasonable to suppose the damage will be limited because I'm on the lookout for it. In working in this area at all I'm either implicitly or explicitly guesstimating that there is an upside to "rationality in general" that beats the downside.
Rationally speaking, it would make sense to make the risks of active rationality cultivation explicit and then subject the the calculation to conscious analysis, and then abandon active rationality cultivation if the expected value is honestly negative. It is precisely the fact that rationality basically demands this kind of bailout analysis at some point that has generally helped me to feel safe(ish) when experimenting with this particular package of memes.
The highest expected value strategy in investing is to put all your money in the single investment that itself has the highest expected value (assuming the opportunity is large enough that your whole contribution doesn't push the project very far down its marginal utility curve so the last dollar invested will have lower return on investment than some other investment).
I wanted to comment on this example: the benefits to index funds are more than in variance. Trading costs make it a superior long-term strategy to managed funds / researching your own sto...
This is Part 2 of the discussion of Alicorn's Twilight fanfic Luminosity.
LATE BREAKING EDIT: Part 3 exists now, so new comment threads should be started there rather than here.
In the vein of the Harry Potter and the Methods of Rationality discussion threads this is the place to discuss anything relating to Alicorn's Twilight fanfic Luminosity. The fanfic is also archived on Alicorn's own website.
Here is Part 1 of the discussion. Previous discussion is hidden so deeply within the first Methods of Rationality thread that it's difficult to find even if you already know it exists.
Similar to how Eliezer's fanfic popularizes material from his sequences Alicorn is using the insights from her Luminosity sequence.
The fic is really really good but there is a twist part way through that makes the fic even more worth reading than it already was, but that makes it hard to talk about because to even ask if someone is twist-aware with any specific hints is difficult. The twist is in the latter half of the story. If you are certainly not post-twist and want to save the surprise, then you should stop reading here and fall back to Part 1 discussion or to the fic itself.
If you think you're pretty sure you are post-twist and are safe to read the rest of this, try reading this rot13'ed hint and see if what you've read matches this high level description of the twist...
Rqjneq unf qvfpbirerq gur frperg gung Vfnoryyn jnf xrrcvat sebz uvz "sbe uvf bja tbbq" bhg bs srne bs Neb ernqvat Rqjneq'f zvaq. Va gur nsgrezngu, fbzrguvat unf punatrq nobhg gurve eryngvbafuvc gung znl unir pnhfrq lbh gb pel sbe n juvyr, naq juvpu znlor urycf gb rzbgvbanyyl qevir ubzr gur pbzovarq zrffntr bs YJ'f negvpyrf nobhg "fbzrguvat gb cebgrpg" naq "ernfba nf n zrzrgvp vzzhar qvfbeqre" naq gur jnl gurl pna fvzhygnarbhfyl nccyl gb crbcyr jub unir abguvat zber va gur jbeyq guna fbzr fvatyr crefba jub gurl ybir.
If the answer to the hint is obvious, then just to be sure that there is not a double illusion of transparency at work, here is the cutoff point spelled out explicitly:
Gur phgbss cbvag sbe cbfgvat urer vf gung lbh unir ernq hc gb puncgre svsgl svir (va gur snasvpgvba irefvba) be puncgre gjragl rvtug ba Nyvpbea'f jrofvgr jurer Rqjneq jnf cebonoyl vapvarengrq, Vfnoryyn fheivirf na nggrzcgrq vapvarengvba, naq fur unf gb ortha gb jbex bhg jung gb qb jvgu gur jerpxntr bs gur erfg bs ure "rgreany" yvsr.