upskilling or career transition grants, especially from LTFF, in the last couple of years

Interesting; I'm less aware of these.

How are they falling short?

I'll answer as though I know what's going on in various private processes, but I don't, and therefore could easily be wrong. I assume some of these are sort of done somewhere, but not enough and not together enough.

• Favor insightful critiques and orientations as much as constructive ideas. If you have a large search space and little traction, a half-plane of rejects is as or more valuable than a guessed point that you knew how to even generate.
• Explicitly allow acceptance by trajectory of thinking, assessed by at least a year of low-bandwidth mentorship; deemphasize agenda-ish-ness.
• For initial exploration periods, give longer commitments with less required outputs; something like at least 2 years. Explicitly allow continuation of support by trajectory.
• Give a path forward for financial support for out of paradigm things. (The Vitalik fellowship, for example, probably does not qualify, as the professors, when I glanced at the list, seem unlikely to support this sort of work; but I could be wrong.)
• Generally emphasize judgement of experienced AGI alignment researchers, and deemphasize judgement of grantmakers.
• Explicitly asking for out of paradigm things.
• Do a better job of connecting people. (This one is vague but important.)

(TBC, from my full perspective this is mostly a waste because AGI alignment is too hard; you want to instead put resources toward delaying AGI, trying to talk AGI-makers down, and strongly amplifying human intelligence + wisdom.)

grantmakers have tried pulling that lever a bunch of times

What do you mean by this? I can think of lots of things that seem in some broad class of pulling some lever that kinda looks like this, but most of the ones I'm aware of fall greatly short of being an appropriate attempt to leverage smart young creative motivated would-be AGI alignment insight-havers. So the update should be much smaller (or there's a bunch of stuff I'm not aware of).

(FWIW this was my actual best candidate for a movie that would fit, but I remembered so few details that I didn't want to list it.)

I'm struggling to think of any. Some runners-up:

• Threads (1984) because Beyond the Reach of God.

• Bird Box because Contrapositive Litany of Hodgell.

• Ghostbusters because whatever is real is lawful; it's up to you to Think Like Reality, and then you can bust ghosts.

• The Secret Life of Walter Mitty (1947, NOT 2013) because at some point you have to take the Inside View.

• Pi (1998) because The Solomonoff Prior is Malign.

Cf. Moneyball.

Emotions are hardwired stereotyped syndromes of hardwired blunt-force cognitive actions. E.g. fear makes your heart beat faster and puts an expression on your face and makes you consider negative outcomes more and maybe makes you pay attention to your surroundings. So it doesn't make much sense to value emotions, but emotions are good ways of telling that you value something; e.g. if you feel fear in response to X, probably X causes something you don't want, or if you feel happy when / after doing Y, probably Y causes / involves something you want.

we've checked for various forms of funny business and our tools would notice if it was happening.

I think it's a high bar due to the nearest unblocked strategy problem and alienness.

I agree that when AGI R&D starts to 2x or 5x due to AI automating much of the process, that's when we need the slowdown/pause)

If you start stopping proliferation when you're a year away from some runaway thing, then everyone has the tech that's one year away from the thing. That makes it more impossible that no one will do the remaining research, compared to if the tech everyone has is 5 or 20 years away from the thing.

10 more years till interpretability? That's crazy talk. What do you mean by that and why do you think it? (And if it's a low bar, why do you have such a low bar?)

"Pre-AGI we should be comfortable with proliferation" Huh? Didn't you just get done saying that pre-AGI AI is going to contribute meaningfully to research (such as AGI research)?

I think you might have been responding to

Susan could try to put focal attention on the scissor origins; but one way that would be difficult is that she'd get pushback from her community.

which I did say in a parenthetical, but I was mainly instead saying

Susan's community is a key substrate for the scissor origins, maybe more than Susan's interaction with Robert. Therefore, to put focal attention on the scissor origins, a good first step might be looking at her community--how it plays the role of one half of a scissor statement.

Your reasons for hope make sense.

hope/memory of the previous society that (Susan and Tusan and Vusan) and (Robert and Sobert and Tobert) all shared, which she has some hope of reaccessing here

Anecdata: In my case it would be mostly a hope, not a memory. E.g. I don't remember a time when "I understand what you're saying, but..." was a credible statement... Maybe it never was? E.g. I don't remember a time when I would expect people to be sufficiently committed to computing "what would work for everyone to live together" that they kept doing so in political contexts.

(generic comment that may not apply too much to Mayer's work in detail, but that I think is useful for someone to hear:) I agree with the basic logic here. But someone trying to follow this path should keep in mind that there's philosophically thorniness here.

A bit more specifically, the questions one asks about "how intelligence works" will always be at risk of streetlighting. As an example/analogy, think of someone trying to understand how the mind works by analyzing mental activity into "faculties", as in: "So then the object recognition faculty recognizes the sofa and the doorway, and it extracts their shapes, and sends their shapes to the math faculty, which performs a search for rotations that allow the sofa to pass through the doorway, and when it finds one it sends that to the executive faculty, which then directs the motor-planning faculty to make an execution plan, and that plan is sent to the motor factulty...". This person may or may not be making genuine progress on something; but either way, if they are trying to answer questions like "which faculties are there and how do they interoperate to perform real-world tasks", they're missing a huge swath of key questions. (E.g.: "how does the sofa concept get produced in the first place? how does the desire to not damage the sofa and the door direct the motor planner? where do those desires come from, and how do they express themselves in general, and how do they respond to conflict?")

Some answers to "how intelligence works" are very relevant, and some are not very relevant, to answering fundamental questions of alignment, such as what determines the ultimate effects of a mind.