It's a good point, re: some of the gap being that it's hard to concretely visualize the world in which AGI isn't built. And also about the "we" being part of the lack of concreteness.
I suspect there're lots of kinds of ethical heuristics that're supposed to interweave, and that some are supposed to be more like "checksums" (indicators everyone can use in an embodied way to see whether there's a problem, even though they don't say how to address it if there is a problem), and others are supposed to be more concrete.
For some more traditional examples:
- There're heuristics for how to tell whether a person or organization is of bad character (even though these heuristics don't tell how how to respond if a person is of bad character). Eg JK Rowling's character Sirius's claim that you can see the measure of a person by how they treat their house-elves (which has classical Christian antecedents, I'm just mentioning a contemporary phrasing).
- There're heuristics for how countries should be, e.g. "should have freedom of speech and press" or (longer ago) "should have a monarch who inherited legitimately."
It would be too hard to try to equip humans and human groups for changing circumstances via only a "here's what you do in situation X". It's somewhat easier to do it (and traditional ethical heuristics did do it) by a combination of "you can probably do well by [various what-to-do heuristics]" and "you can tell if you're doing well by [various other checksum-type heuristics]. Ethics is help to let us design our way to better plans, not to only always give us those plans.
1Another place where I'll think and act somewhat differently as a result of this conversation:
- It's now higher on my priority list to try to make sure CFAR doesn't act as a "gateway" to all kinds of weird "mental techniques" (or quasi-cults who use "mental techniques"). Both for CFAR's new alumni, and for social contacts of CFAR's new alumni. (This was already on some lists I'd made, but seeing Adele derive it independently bumped it higher for me.)
1Okay, but: it's also find individuals who are willing to speak for heuristic C, in a way I suspect differs from what it was like for leaded gasoline and from what I remember as a kid in the late 80's about the ozone layer.
It's a fair point that I shouldn't expect "consensus", and should've written and conceptualized that part differently, but I think heuristic C is also colliding with competing ethical heuristics in ways the ozone situation didn't.
I listed the cases I could easily list of full-blown manic/psychotic episodes in the extended bay area rationalist community (episodes strong enough that the person in most cases ended up hospitalized, and in all cases ended up having extremely false beliefs about their immediate surroundings for days or longer, eg “that’s the room of death, if I walk in there I’ll die”; "this is my car" (said of the neighbor's car)).
I counted 11 cases. (I expect I’m forgetting some, and that there are others I plain never knew about; count this as a convenience sample, not an exhaustive inventory.)
Of these, 5 are known to me to have involved a psychedelic or pot in the precipitating event.
3 are known to me to have *not* involved that.
In the other 3 cases I’m unsure.
In 1 of the cases where I’m unsure about whether there were drugs involved, the person had taken part in a several-weeks experiment in polyphasic sleep as part of a Leverage internship, which seemed to be part of the precipitating event from my POV.
So I’m counting [between 6 and 8] out of 11 for “precipitated by drugs or an imprudent extended sleep-deprivation experiment” and [between 3 and 5] out of 11 for “not precipitated by doing anything unusually physiologically risky.”
(I’m not here counting other serious mental health events, but there were also many of those in the several-thousand-person community across the last ten years, including several suicides; I’m not trying here to be exhaustive.)
(Things can have multiple causes, and having an obvious precipitating physiological cause doesn’t mean there weren’t other changeable risk factors also at play.)
Your guess above, plus: the person's "main/egoic part", who has have mastered far-mode reasoning and the rationalist/Bayesian toolkit, and who is out to "listen patiently to the dumb near-mode parts that foolishly want to do things other than save the world," can in some people, with social "support" from outside them, help those parts to overpower other bits of the psyche in ways that're more like tricking and less like "tug of wars", without realizing they're doing this.
1My own guesses are that CFAR mostly paid an [amount of attention that made sense] to reducing psychosis/mania risks in the workshop context, after our initial bad experience with the mania/psychosis episode at an early workshop when we did not yet realize this could be a thing.
The things we did:
- tried to screen for instablity;
- tried to warn people who we thought might have some risk factors (but not enough risk factors that we were screening them out) after accepting them to the workshop, and before they'd had a chance to say yes. (We’d standardly say something like: “we don’t ask questions this nosy, and you’re already in regardless, but, just so you know, there’s some evidence that workshops of all sorts, probably including CFAR workshops, may increase risks of mania or psychosis in people with vulnerability to that, so if you have any sort of psychiatric history you may want to consider either not coming, or talking about it with a psychiatrist before coming.”)
- try to train our instructors and “mentors” (curriculum volunteers) to notice warning signs. check in as a staff regularly to see if anyone had noticed any warning signs for any participants. if sensible, talk to the participant to encourage them to sleep more, skip classes, avoid recreational drugs for awhile, do normal grounding activities, etc. (This happened relatively often — maybe once every three workshops — but was usually a relatively minor matter. Eg this would be a person who was having trouble sleeping and who perhaps thought they had a chance at solving [some long-standing personal problem they’d previously given up on] “right now” a way that weirded us out, but who also seemed pretty normal and reasonable still.)
I separately think I put a reasonable amount of effort into organizing basic community support and first aid for those who were socially contiguous with me/CFAR who were having acutely bad mental health times, although my own capacities weren’t enough for a growing community and I mostly gave up on the less near-me parts around 2018.
It mostly did not occur to me to contemplate our cultural impact on the community’s overall psychosis rate (except for trying for awhile to discourage tulpas and other risky practices, and to discourage associating with people who did such things, and then giving up on this around 2018 when it seemed to me there was no real remaining chance of quarantining these practices).
I like the line of inquiry about “what art of rationality might be both good in itself, and increase peoples’ robustness / decrease their vulnerability to mania/psychosis-type failure modes, including much milder versions that may be fairly common in these parts and that are still bad”. I’ll be pursuing it. I take your point that I could in principle have pursued it earlier.
If we are going to be doing a fault analysis in which we give me and CFAR responsibility for some of our downstream memetic effects, I’d like CFAR to also get some credit for any good downstream memetic effects we had. My own guess is that CFAR workshops:
- made it possible for EA and “the rationalist community” to expand a great deal without becoming nearly as “diluted”/“normie” as would’ve happened by default, with that level of immigration-per-year;
- helped many “straw lesswrongers” to become more “agenty” and realize “problems are for solving” instead of sort of staring helplessly at their todo lists and desires, and that this part made the rationalist community stronger and healthier
- helped a fair number of people to become less “straw EA” in the sense of “my only duty is to do the greatest good for the greatest number, while ignoring my feelings”, and to tune in a bit more to some of the basics of healthy life, sometimes.
I acknowledge that these alleged benefits are my personal guesses and may be wrong. But these guesses seem on par to me with my personal guess that patterns of messing with one’s own functioning (as from “CFAR techniques”) can erode psychological wholeness, and I’m afraid it’ll be confusing if I voice only the negative parts of my personal guesses.
I'm most worried about 2, I think there's potentially something toxic about the framing of "rationality habits" in general, which has previously led to a culture of there being all these rationality "tricks" that would solve all your problems ... which in turn leads to people uncritically trying dubious techniques that fuck them up.
Could you say a bit more here, please?
(not a direct response, but:) My belief has been that there are loads of people in the bay area doing dubious things that mess them up (eg tulpas, drugs, weird sex things, weird cult things -- both in the rationalist diaspora, and in the bay area broadly), but this is mostly people aiming to be edgy and do "weird/cool/powerful" things, not people trying CFAR techniques as such.
though IDC feels similar flavoured and is an original.
Awkwardly, while IDC is indeed similar-flavored and original to CFAR, I eventually campaigned (successfully) to get it out of our workshops because I believe, based on multiple anecdotes, that IDC tends to produce less health rather than more, especially if used frequently. AWC believes Focusing should only be used for dialog between a part and the whole (the "Self"), and I now believe she is correct there.
Also we understand basic arithmetic around here, which goes a long way sometimes.