Prediction and planning remain incredibly distinct as structures of cognitive work,
I disagree. (Partially.) For a unitary agent who is working with a small number of possible hypotheses (e.g., 3), and a small number of possible actions, I agree with your quoted sentence.
But let’s say you’re dealing with a space of possible actions that’s much too large to let you consider each exhaustively, e.g. what blog post to write (considered concretely, as a long string of characters).
It’d be nice to have some way to consider recombinable pieces, e.g. “my blog post could include idea X”, “my blog post could open with joke J”, “my blog post could be aimed at a reader similar to Alice”.
Now consider the situation as seen by the line of thinking that is determining: “should my blog post be aimed mostly at readers similar to Alice, or at readers similar to Bob?”. For this line of thinking to do a good estimate of ExpectedUtility(post is aimed at Alice), it needs predictions about whether the post will contain idea X. However, for the line of thinking that is determining whether to include idea X (or the unified agent, at those moments when it is actively considering this), it’’ll of course need good plans (not predictions) about whether to include X, and how exactly to include X.
I don’t fully know what a good structure is for navigating this sort of recombinable plan space, but it might involve a lot of toggling between “this is a planning question, from the inside: shall I include X?” and “this is a prediction question, from the outside: is it likely that I’m going to end up including X, such that I should plan other things around that assumption?”.
My own cognition seems to me to toggle many combinatorial pieces back and forth between planning-from-the-inside and predicting-from-the-outside, like this. I agree with your point that human brains and bodies have all kinds of silly entanglements. But this part seems to me like a plausible way for other intelligences to evolve/grow too, not a purely one-off humans idiosyncrasy like having childbirth through the hips.
Also we understand basic arithmetic around here, which goes a long way sometimes.
It's a good point, re: some of the gap being that it's hard to concretely visualize the world in which AGI isn't built. And also about the "we" being part of the lack of concreteness.
I suspect there're lots of kinds of ethical heuristics that're supposed to interweave, and that some are supposed to be more like "checksums" (indicators everyone can use in an embodied way to see whether there's a problem, even though they don't say how to address it if there is a problem), and others are supposed to be more concrete.
For some more traditional examples:
- There're heuristics for how to tell whether a person or organization is of bad character (even though these heuristics don't tell how how to respond if a person is of bad character). Eg JK Rowling's character Sirius's claim that you can see the measure of a person by how they treat their house-elves (which has classical Christian antecedents, I'm just mentioning a contemporary phrasing).
- There're heuristics for how countries should be, e.g. "should have freedom of speech and press" or (longer ago) "should have a monarch who inherited legitimately."
It would be too hard to try to equip humans and human groups for changing circumstances via only a "here's what you do in situation X". It's somewhat easier to do it (and traditional ethical heuristics did do it) by a combination of "you can probably do well by [various what-to-do heuristics]" and "you can tell if you're doing well by [various other checksum-type heuristics]. Ethics is help to let us design our way to better plans, not to only always give us those plans.
1Another place where I'll think and act somewhat differently as a result of this conversation:
- It's now higher on my priority list to try to make sure CFAR doesn't act as a "gateway" to all kinds of weird "mental techniques" (or quasi-cults who use "mental techniques"). Both for CFAR's new alumni, and for social contacts of CFAR's new alumni. (This was already on some lists I'd made, but seeing Adele derive it independently bumped it higher for me.)
1Okay, but: it's also find individuals who are willing to speak for heuristic C, in a way I suspect differs from what it was like for leaded gasoline and from what I remember as a kid in the late 80's about the ozone layer.
It's a fair point that I shouldn't expect "consensus", and should've written and conceptualized that part differently, but I think heuristic C is also colliding with competing ethical heuristics in ways the ozone situation didn't.
I listed the cases I could easily list of full-blown manic/psychotic episodes in the extended bay area rationalist community (episodes strong enough that the person in most cases ended up hospitalized, and in all cases ended up having extremely false beliefs about their immediate surroundings for days or longer, eg “that’s the room of death, if I walk in there I’ll die”; "this is my car" (said of the neighbor's car)).
I counted 11 cases. (I expect I’m forgetting some, and that there are others I plain never knew about; count this as a convenience sample, not an exhaustive inventory.)
Of these, 5 are known to me to have involved a psychedelic or pot in the precipitating event.
3 are known to me to have *not* involved that.
In the other 3 cases I’m unsure.
In 1 of the cases where I’m unsure about whether there were drugs involved, the person had taken part in a several-weeks experiment in polyphasic sleep as part of a Leverage internship, which seemed to be part of the precipitating event from my POV.
So I’m counting [between 6 and 8] out of 11 for “precipitated by drugs or an imprudent extended sleep-deprivation experiment” and [between 3 and 5] out of 11 for “not precipitated by doing anything unusually physiologically risky.”
(I’m not here counting other serious mental health events, but there were also many of those in the several-thousand-person community across the last ten years, including several suicides; I’m not trying here to be exhaustive.)
(Things can have multiple causes, and having an obvious precipitating physiological cause doesn’t mean there weren’t other changeable risk factors also at play.)
Your guess above, plus: the person's "main/egoic part", who has have mastered far-mode reasoning and the rationalist/Bayesian toolkit, and who is out to "listen patiently to the dumb near-mode parts that foolishly want to do things other than save the world," can in some people, with social "support" from outside them, help those parts to overpower other bits of the psyche in ways that're more like tricking and less like "tug of wars", without realizing they're doing this.
1My own guesses are that CFAR mostly paid an [amount of attention that made sense] to reducing psychosis/mania risks in the workshop context, after our initial bad experience with the mania/psychosis episode at an early workshop when we did not yet realize this could be a thing.
The things we did:
- tried to screen for instablity;
- tried to warn people who we thought might have some risk factors (but not enough risk factors that we were screening them out) after accepting them to the workshop, and before they'd had a chance to say yes. (We’d standardly say something like: “we don’t ask questions this nosy, and you’re already in regardless, but, just so you know, there’s some evidence that workshops of all sorts, probably including CFAR workshops, may increase risks of mania or psychosis in people with vulnerability to that, so if you have any sort of psychiatric history you may want to consider either not coming, or talking about it with a psychiatrist before coming.”)
- try to train our instructors and “mentors” (curriculum volunteers) to notice warning signs. check in as a staff regularly to see if anyone had noticed any warning signs for any participants. if sensible, talk to the participant to encourage them to sleep more, skip classes, avoid recreational drugs for awhile, do normal grounding activities, etc. (This happened relatively often — maybe once every three workshops — but was usually a relatively minor matter. Eg this would be a person who was having trouble sleeping and who perhaps thought they had a chance at solving [some long-standing personal problem they’d previously given up on] “right now” a way that weirded us out, but who also seemed pretty normal and reasonable still.)
I separately think I put a reasonable amount of effort into organizing basic community support and first aid for those who were socially contiguous with me/CFAR who were having acutely bad mental health times, although my own capacities weren’t enough for a growing community and I mostly gave up on the less near-me parts around 2018.
It mostly did not occur to me to contemplate our cultural impact on the community’s overall psychosis rate (except for trying for awhile to discourage tulpas and other risky practices, and to discourage associating with people who did such things, and then giving up on this around 2018 when it seemed to me there was no real remaining chance of quarantining these practices).
I like the line of inquiry about “what art of rationality might be both good in itself, and increase peoples’ robustness / decrease their vulnerability to mania/psychosis-type failure modes, including much milder versions that may be fairly common in these parts and that are still bad”. I’ll be pursuing it. I take your point that I could in principle have pursued it earlier.
If we are going to be doing a fault analysis in which we give me and CFAR responsibility for some of our downstream memetic effects, I’d like CFAR to also get some credit for any good downstream memetic effects we had. My own guess is that CFAR workshops:
- made it possible for EA and “the rationalist community” to expand a great deal without becoming nearly as “diluted”/“normie” as would’ve happened by default, with that level of immigration-per-year;
- helped many “straw lesswrongers” to become more “agenty” and realize “problems are for solving” instead of sort of staring helplessly at their todo lists and desires, and that this part made the rationalist community stronger and healthier
- helped a fair number of people to become less “straw EA” in the sense of “my only duty is to do the greatest good for the greatest number, while ignoring my feelings”, and to tune in a bit more to some of the basics of healthy life, sometimes.
I acknowledge that these alleged benefits are my personal guesses and may be wrong. But these guesses seem on par to me with my personal guess that patterns of messing with one’s own functioning (as from “CFAR techniques”) can erode psychological wholeness, and I’m afraid it’ll be confusing if I voice only the negative parts of my personal guesses.
Yes; this (or something similar) is why I suspect that "'believing in' atoms" may involve the same cognitive structure as "'believing in' this bakery I am helping to create" or "'believing in' honesty" (and a different cognitive structure, at least for ideal minds, from predictions about outside events). The question of whether to "believe in" atoms can be a question of whether to invest in building out and maintaining/tuning an ontology that includes atoms.