This is a special post for quick takes by Lucent. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
Could rationalists be the worst people to be put in charge of keeping an AI in the box? We can assume:
I've seen AI box transcripts in which the consensus was: the AI stayed in the box because the AI player was too "dumb" to be swayed by the nuanced, rational arguments. Instead of attributing stubbornness to lack of intelligence, is it a deliberate installation by evolution to defend against clever wordplay and manipulation? Is rationalism the ultimate "cult" as its defining feature is a volitional removal of a safeguard in place to prevent domination of values and goals by those with superior wordplay?
Are there loopholes in this mechanism specifically for one's parents? Is that why the values given to you by your parents, whether about religion or automobile brands, are so difficult to displace? If religion's propagation is primarily through a parental trust loophole closed to others, would that be easy to test by seeing if a preference for Ford or Chevy is just as generationally robust?
I haven't heard a nonrationalist say "what you are saying sounds smart but I don't know enough about the topic to evaluate your argument, so I won't let myself get convinced" but have heard that from a rationalist (as in someone coming to the LessWrong community weekend).
It's not rational to let yourself be argued into anything when faced with a powerful and potentially manipulative actor.
Previously, Tourette's occupied the same category in my mind as stuttering, a lifelong organic brain disease or wiring fault not amenable to coaching or psychiatry. However, I'm questioning this after watching The Vow, an NXIVM documentary illustrating, with video, a childhood Tourette's sufferer be "cured" by some sort of negative reinforcement training. Upon looking up the Wikipedia, I'm surprised to find a prognosis of "80% will experience improvement to disappearance of tics beginning in late teens." Further, there's been an extreme uptick in Tourette's experienced by children simply from watching TikTok videos of others' tics, including taking on the tics they're exposed to.
Together, these three data points have shifted my belief that Tourette's, and perhaps stuttering, is not some physical wiring defect, but a named conversion disorder. Perhaps psychiatry already knew this and was being polite, but I did not. I understand the need to group similar behavior disorders into diseases, regardless of somatic or organic origin. However, I also see the danger in naming (or renaming) somatoform disorders like delusional parasitosis as Morgellons, giving legitimacy and cause célèbre to what are otherwise social contagions with no underlying organic etiology.
I can't speak for others, but if one day I woke up with tics, I suspect my recovery path would be quite different depending on whether I was told, "this is a habit you can unlearn or will likely soon outgrow" or, "this is a serious, unique brain disease, and you deserve sympathy and accommodation."