A lot of rationalists seem to assume that “as you gain more IQ points, learn more facts, and think harder, you'll naturally converge to the optimal theory of ethics.” Like the OP, I don't think this is true in general.
Even if you’re a moral antirealist who only cares about figuring out the most ethical policy by your own lights, I think whatever you “truly” mean by “ethics” is likely substantially different from what you'd get if you actually instantiated your favorite reflection process. Even if that process is "get smarter, learn more, then think hard about what the reflection process should be and do that"!
Here are some gestures at my reasoning:
- It's now common wisdom among rationalists that an arbitrarily-initialized mind, elevated to superintelligence, won't necessarily be ethical.[1] So if you stipulate that the mind is a smart human with a genuine desire to be ethical, elevated to superintelligence using some “reflection process” endorsed by that human, would that fix the problem? It's not at all clear to me that it would!
- Your reflection process is only a proxy for your “true values.” By Goodhart’s law, optimizing really hard for whatever theory of ethics comes out of this reflection process will lead to something different from your true values, perhaps catastrophically so.
- Your reflection process is underspecified and path-dependent. Slight variants of the same reflection process will vehemently disagree with each other.
- For example, if your plan for solving ethics starts with “learn a bunch of facts,” what order do you learn these facts? Do you read world history, Western philosophy, Eastern philosophy, LessWrong? Everything you learn changes you, and frames everything you learn later. Even if you start by giving yourself a perfect memory and one billion IQ points, does that really ensure you’ll give every idea a fair shot regardless of when you encounter it? What does “fair shot” even mean?
- We only have “ethical data” about what we’ve actually experienced (whether individually or as a society). Trying to make conclusions about what you’d truly value in very different situations is like extrapolating a trendline out of distribution: the further away you get from existing data, the more likely that something will go wrong. You should treat your moral intuitions about very unfamiliar scenarios as a signal with very high variance.
- Imagine being a high schooler who thinks “I really value being a doctor!” because your parents say they’d be proud of you. You invest 10+ years of your life into schooling, you start your first month as a doctor, and you finally realize you hate it.
- Now imagine thinking “I really value happiness!” because how could anyone not be happy about more happiness? You send off the von Neumann probes, they tile the entire universe with hedonium, and you finally realize this was not at all what you wanted. (Oh wait, you can’t realize it, because your body has been converted to hedonium.)
- See also Scott Alexander’s analogy to the Bay Area transit system in The Tails Coming Apart as a Metaphor for Life.
If I had to guess, I’d say that most people's ideal reflection process probably involves thinking really hard, but also things like emotional processing, and plenty of other things I haven’t thought of. It's very tough to say.
Despite the name, “self-correction” often originates from other people. It's highly unlikely that one person sitting in a room (or even hundreds of rationalists sitting in Lighthaven) would converge on the ultimate theory of ethics. I think one major reason for society’s mysterious moral progress is the gradual propagation of ideas through a distributed network of humans, all with different blind spots, who can deliberate and point out each others’ errors.[2] Given enough time and space for healthy competition, the ideas that help society thrive in the long term should hopefully rise to the top.
If possible, it seems pretty robustly good to give society more time to deliberate and correct themselves rather than immediately locking in an ethical reflection process to optimize for. To get more data on out-of-distribution ethical scenarios, we may need some form of iterative deployment:[3] inching forward a bit, seeing what happens, and deliberating about what to do next.
But even if we somehow institute a slow, pluralistic reflection process, this is just another reflection process. It may lead to some values that our idealized selves would find really bad to optimize to the limit. One workaround to the dilemma of finding the perfect reflection process is to regularize: just don’t optimize too hard for any set of values! We can start by making changes that pretty much everyone agrees are not unethical: ending poverty, reverting climate change, replacing factory-farmed meat with plant-based alternatives that taste just as good.[4] At least this won’t cause harm relative to the status quo.[5]
- ^
- ^
One of the downsides about offloading large parts of society’s cognition to AIs is that this network gets dominated by a few hugely-prolific, mode-collapsed nodes.
- ^
Just a slower version than OpenAI’s current approach.
- ^
Not sure if the last one passes the “pretty much everyone agrees it’s not unethical” bar. Maybe this rule needs tweaking…
- ^
Even more controversial version of that take: maybe we should precommit to pin our regularization to the values of humanity as of 2026. If almost everyone in the world changes their values to something that 2026 humans actively hate, that likely means that society has been eaten by some crazy totalizing memeplex. On the other hand, if past civilizations had done this, they’d probably lock in values like worshipping God and wives being obedient to their husbands, so this idea clearly needs some work.
1Even if you’re a moral antirealist who only cares about figuring out the most ethical policy by your own lights, I think whatever you “truly” mean by “ethics” is likely substantially different from what you'd get if you actually instantiated your favorite reflection process. Even if that process is "get smarter, learn more, then think hard about what the reflection process should be and do that"!
This sounds to me like you're claiming that (at least for humans?) it's very hard to have one's values properly preserved/extrapolated across self-improvement/instrumental convergence?
(Other than that,) I think I agree with your comment (or at least most of it / the spirit of it), but/except regarding an assumption that I think lies behind (e.g.) this:
Your reflection process is only a proxy for your “true values.” By Goodhart’s law, optimizing really hard for whatever theory of ethics comes out of this reflection process will lead to something different from your true values, perhaps catastrophically so.
The assumption that I think lies behind this is that humans have such a thing as "true values" that can tell you what is good / how to do good in full generality or something. We don't. Humans have values, but the further you deviate from familiar circumstances, the less their behavior looks like already having values, and it looks more like constructing values at runtime, by somehow extending them into the new territory. There are a lot of open questions with indeterminate answers about how to extend your values into the new territory; you can "genuinely choose" to do it one way or another.
In a sense, this is retrospectively obvious if you think of humans as results of a blind selection process that imbued them with shards of desire that don't compose into something too coherent once they leave the ancestral environment.
(Maybe you already think this, but it wasn't clear to me from reading your comment.)
I think there are multiple legitimate ways that someone's values could evolve, but some ways are illegitimate. A reflection process should probably reject slavery and avoid joining cults, but maybe it doesn't matter which exact level of libertarianism it suggests.
People mean something when they talk about "ethics" and "true values," even if there's no objective truth of the matter. I'm talking about whatever it is they mean.
Vladimir Nesov has a suggestion here about how this could be done[1]. I don't think it quite works, but to the extent that it is effective, it can be extended beyond just the influence of superintelligence to other types of new territory (and superintelligence as well, since Nesov's proposal requires a Sysop[2], though presumably with a lot of transhumanist 3+1- or 4-volume locked out by Nesov's design).
One problem is that human value might be inherently multidimensional. I think of it as the want/like/approve distinction. We seem to have separate mechanisms in our brains for 1) enjoying something in the moment, 2) wanting to do it before, and 3) approving of it afterward. It's possible to want something without enjoying it (like a person with OCD wanting to close the door exactly ten times), enjoy something without wanting it (people have said that they've reached very enjoyable meditative states but feel zero motivation to reach them again), enjoy something without approving it (porn), approve something without enjoying it (exercise), and all other combinations. This is the reason why "revealed preference" doesn't work: a person's actions are dictated disproportionally by the "want" dimension, but a good theory of value should incorporate all three. If we optimize one over the others, the tails will come apart.
A nice toy example is video games, where people are attracted to them because of the graphics, then stay because of the gameplay, and then have a warm afterglow and want to discuss afterward because of the story. Which of the three should contribute the most to the "true" quality rating of a videogame - "want", "like", or "approve"? Is this question philosophically meaningful? Will more reflection solve it?
Is this question philosophically meaningful?
Assuming you mean something like "Can philosophy answer this question?" I think "Maybe, but we probably won't know until we do a lot more philosophy." To put it another way, I think it's very plausible (but far from certain) that we can eventually answer questions like this one, given enough competent reflection, and I want to make sure we definitively find out before we make any irreversible choices based on what we think our values are.
Hi Wei,
I agree with your concerns about the sorry state of our current abilities that would need to be remedied before making major irreversible decisions about the future of humanity. In fact I agree with you so much that I don't really see our approaches as alternatives. Instead, I'd see yours as a useful lens on key things that would need to be solved during such a reflection, or a highlighting of how we would have to be changed in the process of a Long Reflection.
Your list (1)–(8) is focused on improving our abilities rather than gaining knowledge, though it is largely on our abilities to gain that knowledge. So represents more of a suggestion of how to go about gaining the knowledge than an outside alternative. (e.g. I'd endorse both the Long Reflection and the Long Self-Correction, and see them as parts of the same thing).
The similarity was made particularly clear by your paragraph:
My main hope for a Long Self-Correction eventually succeeding rests on the fact that humans have seemingly, mysteriously, made progress on these issues over a very long period of time, so if we preserve the environment in which we can seemingly do this, and not give anyone or anything the power to permanently derail such progress, then maybe we can continue to snowball The Correction until we reach a point when we can rightly justify reshaping the universe according to our volition.
This is pretty much exactly how I see the Long Reflection: that the very messy process of moral improvement through philosophy, politics, civil society, religion (and more) has led to dramatic improvements over the course of thousands of years. And I'm saying that we shouldn't cut off that process now, but continue it. (I think there are ways we could do better than the messy laissez faire approach, but attempts to formalise the process could also increase the chance of getting stuck in traps, so I'd be cautious about doing so.)
A difference from your Long Self-Correction is that I also include gaining the scientific knowledge to understand the limits of what can be achieved and the trade-offs encountered in particular routes forwards (especially when the paths involve irreversible steps), though you could also talk about improving our abilities to be able to gain and usefully apply such knowledge.
Thanks, Toby, I highly appreciate your perspective. I'm honestly a bit surprised by our apparent convergence in views here, given that this part of your book sounds a lot more positive/optimistic than my post and you didn't talk about most of the flaws I listed, except something like my #1: "The study of the positive is at a much earlier stage of development." I'm curious if this was a matter of choice of emphasis or presentation, or if your views have substantively changed since writing the book.
Perhaps another potential difference between us is that I see existential security as being secured or securable only after the Long Self-Correction has largely finished (instead of as a precondition for the Long Reflection as you wrote in your book), because even if we secured ourselves against all other risks, humanity's flaws imply that in the early stages there is seemingly no way to ensure that the process doesn't go off-rails, locking us into a basin of attraction that ends up in a wrong destination (according to true normativity or actually competent reflection), or we just choose to end the process early out of collective overconfidence. (Both of these could constitute an existential risk in the sense of "permanently destroy humanity's future potential".)
Finally, I wonder if you can clear up something I've always wondered about the Long Reflection - was it meant to be pre-ASI, post-ASI, or agnostic about this? (See this comment for additional context.)
I'd say my views have evolved, but are quite continuous with my original views. More a sharpening and changes of emphasis than a disagreement with the stated view in The Precipice. One reason I didn't express things more like your (2)–(8) is that I hadn't explicitly thought about many of them. I've been appreciating your writing on this here and your podcast interview. I'm not 100% convinced of your position (e.g. I'm not even quite sure how to make 'humans are bad at philosophy' into a meaningful statement), but I'm very sympathetic to it and see your evolving views on these matters as some of the most interesting writing about humanity's deeper predicament.
One reason I'm optimistic is because of humanity's progress over the last three thousand years (often in very difficult circumstances where the number of people with the ability and freedom to think about these things was often tiny and disconnected). My claim is basically that if you remove the traps where we could get stuck and then give the process long enough, we should get there. This is a reason why I think it could take a long time (I'd think ideally thousands of years, though I've calculated millions of years as an upper bound for how long we have before we start losing appreciable fractions of the future).
You could well be right about the existence of various traps during the Long Reflection. I still think my ordering is roughly right — put out the current fires and develop good fire defenses before settling down to think things through. It is partly because I don't think we should set up a formal structure for the process that I'm not as concerned about the initial setup dooming it. e.g. we didn't have a formal structure for philosophical progress over the last three thousand years. I'd expect traps to be slow moving enough that one can learn about them and avoid them during the process, though this still does mean that the structure of First eliminate all traps Second do Long Reflection is a simplification.
I did envisage it as being pre-ASI and think that Will did too. This is partly because the risk we undergo in getting to ASI doesn't have to be borne. In an ideally managed world we'd instead develop lots of narrower and safer tools to help (think Wikipedia and theorem provers and Mathematica). In my view, in the Long Reflection we delay potentially existential technologies/choices until we can be sure the risk is low or that it can't be avoided or work out how to make a version with low risk. Then we can adopt those that we've assessed as worth adopting. (Some people like Robin Hanson argue against a straw man that the Long Reflection would ban all new technology/choices until proven safe or until the entire reflection is completed, but neither Will nor I have advanced such a proposal. Instead the focus is on radical technologies and choices, such as ASI or a diaspora of humanity throughout the galaxy.)
I really like the idea of branding this better, and I think we can do even better than what's been proposed here. Let's brainstorm in replies to this comment?
I originally meant to propose a new name/idea for intellectual discussion among rationalists/EAs (where things like Orwellian connotations aren't as important as the literal/logical meanings), but given that it has a chance of spreading further I suppose PR considerations should also be taken into account. But contra @bits I do want the name to be somewhat "negative", i.e., suggest that we're starting from a very flawed state, should be very wary of doing anything highly consequential in our current state, and whatever process we undertake has a very real chance of failure.
The Long Ponder?
It’s pretty similar to the Long Reflection but to me has slightly more connotation of puzzlement. Plus it has a nice ring to it.
I note that “correction” also refers to stocks going down. That's not a great connotation to have, especially in the unfortunate futures where it becomes political.
I chatted with an LLM a little. My favorites were:
- The Long Reformation
- The Great Reconstruction
- The Long Becoming
These all to me have a character of fixing ourselves and realigning with a new philosophy. "The Long Correction" sounds a little Orwellian; GLaDOS would "correct your behavior".
I currently like "The Long (self-)Correction" better than these. Part of what I like about it is it feels less pretentious, like, Reflection and Reconstruction sound like fancy things fancy philosopher-kings do, instead of a bumbling-but-persistent/hopeful thing that imperfect people can do.
If you believe (as Wei says) that it will be known as "The Long Correction", then it sounds Orwellian (and similar to "correctional facility") and I currently believe is a non-starter.
It... is going to be a grand mission that humanity goes on together? I don't get the idea of calling grand & important things unimportant names when they are in fact grand & important. Let us not call our era the age of reason, or the enlightenment, for that is too grand; let us call it the age of getting-slightly-better. I think it is good to give grand projects appropriately grand names.
Let us not call our era the age of reason, or the enlightenment, for that is too grand; let us call it the age of getting-slightly-better.
The Long Self-Correction seems appropriately grand to me. :) Age of Reason and the Enlightenment seem to be instances of the very thing I criticize in the OP, "badly calibrated about our philosophical and strategic competence". Have you tried reading some of the philosophy from those times?
I think I understand where you're coming from, but (to me) these sound a lot like a communist dictator's euphemism for a famine!
lol!
I suspect whether it sounds Orwellian depends on what it's actually describing, and whether it's natural to construe it as its opposite.
All of these, and Long Self-Correction, have specific connotations that I expect the public won't like.
Long Self-Correction: we must be corrected, i.e. we are bad. Jails are "correctional facilities." We need giga universal-jail.
Long Reformation: invokes the Reformation. At least nobody has strong feelings about Protestantism vs Catholicism.
Great Reconstruction: we are currently deconstructed, perhaps having just undergone some particular tragedy
Long Becoming: Lovecraftian concern about what exactly I, you, we, are becoming
It's easy to criticize, so more ideas:
- The Great Debate
- The New Long Horizon
- Prolonged Revelation
- Long Dawn of Humanity (of Civilization, of the Future)
Fable comes up with (curated selection):
- The Long Cultivation
- The Long Ascent
- The Long Deliberation
I start to feel over-indexed on the "long" part here. CEV is a nice idea because it's well-specified independent of how long it takes.
"Cultivation" does seem to capture something more positive than "correction."
3
FWIW also satisfies Wei's desideratum of "being somewhat 'negative'", starting from a very flawed state.
1Plugging the concept of viatopia again because it's a nice explanatory partner to a Long Reflection or Long Self-Correction. A viatopia is a world that will, with near-certainty, perform a Long Correction and then implement the desired future. Or, a Long Correction is what you would do if you found yourself in a viatopia when your current sentient inhabitants are not yet ethically developed enough to commit to the project of engineering the future.
(I'd also be very happy to see someone coin another term that means what is gestured at by viatopia.)
It's not a strict definition, because a society undergoing the Long Reflection might be more insecure than what is necessary to qualify as a viatopia.
I'd like to see people having the public conversation: do we need to achieve existential security before we slow everything down and do a Long Reflection? And more generally, at what level of technology would the world be comfortable stopping?
It's possible that AI Pause advocates will need to argue for a concrete technological vision, something like: "With the level of AI we have today in the year 2031, just applying the current models will continue to produce breakthroughs in medicine, manufacturing, etc. enough to cure cancer and bring the world economic baseline up to an American middle-class outcome. We are committed to continuing to fund this progress while still restricting frontier AI development."
I think that having that conversation at least lets people imagine a concrete world state from which a Long Reflection can happen. And then there's an easier way to point at what is being talked about: it's the answer to the question "and then what?"
Yeah, viatopia is another idea in the same cluster, but it's explicitly framed as post-ASI:
Yet almost no one has articulated a positive vision for what comes after superintelligence. Few people are even asking, “What if we succeed?” Even fewer have tried to answer.
Same post also says it's meant to be a generalization of Long Reflection:
Viatopia is a more general concept: the long reflection is one proposal for what viatopia would look like, but it need not be the only one.
However my memory says that the Long Reflection was meant to be pre-ASI, and looking back at where it was originally proposed (Toby Ord's The Precipice), that still seems like the most plausible interpretation, although it wasn't fully explicit about this. (Note that the Long Self-Correction is also meant to be pre-ASI, since I think humans are too flawed/unsafe to try to build ASI.) Quoting from Toby's book (bolding added by me):
How can humanity have the greatest chance of achieving its potential? I think that at the highest level we should adopt a strategy proceeding in three phases:2
- Reaching Existential Security
- The Long Reflection
- Achieving Our Potential
On this view, the first great task for humanity is to reach a place of safety—a place where existential risk is low and stays low. I call this existential security.
It has two strands. Most obviously, we need to preserve humanity’s potential, extracting ourselves from immediate danger so we don’t fail before we’ve got our house in order. This includes direct work on the most pressing existential risks and risk factors, as well as near-term changes to our norms and institutions.
But we also need to protect humanity’s potential—to establish lasting safeguards that will defend humanity from dangers over the longterm future, so that it becomes almost impossible to fail.3 Where preserving our potential is akin to fighting the latest fire, protecting our potential is making changes to ensure that fire will never again pose a serious threat.4 This will involve major changes to our norms and institutions (giving humanity the prudence and patience we need), as well as ways of increasing our general resilience to catastrophe. This needn’t require foreseeing all future risks right now. It is enough if we can set humanity firmly on a course where we will be taking the new risks seriously: managing them successfully right from their onset or sidestepping them entirely.
Note that existential security doesn’t require the risk to be brought down to zero. That would be an impossible target, and attempts to achieve it may well be counter-productive. What humanity needs to do is bring this century’s risk down to a very low level, then keep gradually reducing it from there as the centuries go on. In this way, even though there may always remain some risk in each century, the total risk over our entire future can be kept small.5 We could view this as a form of existential sustainability. Futures in which accumulated existential risk is allowed to climb toward 100 percent are unsustainable. So we need to set a strict risk budget over our entire future, parceling out this non-renewable resource with great care over the generations to come.
Ultimately, existential security is about reducing total existential risk by as many percentage points as possible. Preserving our potential is helping lower the portion of the total risk that we face in the next few decades, while protecting our potential is helping lower the portion that comes over the longer run. We can work on these strands in parallel, devoting some of our efforts to reducing imminent risks and some to building the capacities, institutions, wisdom and will to ensure that future risks are minimal.6
A key insight motivating existential security is that there appear to be no major obstacles to humanity lasting an extremely long time, if only that were a key global priority. As we saw in Chapter 3, we have ample time to protect ourselves against natural risks: even if it took us millennia to resolve the threats from asteroids, supervolcanism and supernovae, we would incur less than one percentage point of total risk.
The greater risk (and tighter deadline) stems from the anthropogenic threats. But being of humanity’s own making, they are also within our control. Were we sufficiently patient, prudent and coordinated, we could simply stop imposing such risks upon ourselves. We would factor in the hidden costs of carbon emissions (or nuclear weapons) and realize they are not a good deal. We would adopt a more mature attitude to the most radical new technologies—devoting at least as much of humanity’s brilliance to forethought and governance as to technological development.
In the past, the survival of humanity didn’t require much conscious effort: our past was brief enough to evade the natural threats and our power too limited to produce anthropogenic threats. But now our longterm survival requires a deliberate choice to survive. As more and more people come to realize this, we can make this choice. There will be great challenges in getting people to look far enough ahead and to see beyond the parochial conflicts of the day. But the logic is clear and the moral arguments powerful. It can be done.
If we achieve existential security, we will have room to breathe. With humanity’s longterm potential secured, we will be past the Precipice, free to contemplate the range of futures that lie open before us. And we will be able to take our time to reflect upon what we truly desire; upon which of these visions for humanity would be the best realization of our potential. We shall call this the Long Reflection.7
@Toby_Ord @wdmacaskill in case they want to weigh in on this.
I just learned about what a viatopia was from your linked article, but I certainly agree that there should be more discussion on this. In general, I think that focusing too much on terminal values can lead to internal disagreement on matters which don't bear much significance in the present, especially when most of us already agree on certain things that are desirable in the medium-term (such as the ones you've listed e.g. curing cancer).
And more generally, at what level of technology would the world be comfortable stopping?
I'm not entirely sure what year I think would be optimal, but since frontier LLMs see upgrades every few months, I feel as though we only scratch the surface of what we can do with our current models before the next generation is already out. As such, I think that even our current LLMs + improved knowledge on how to make use of them (gathered over years) could already offer significant help toward achieving those medium-term goals.
In any case, it would be nice if viatopia had more attention, as it seems like a promising way to do "one thing at a time" and potentially reduce the confusion (and risk of error) of trying to develop full-length plans from the outset.
(I know that those conversations have happened here and in plenty of living rooms. I'd be very interested in links to anything one degree more public, if anyone is aware of them!)
Curated. I found this a nice, evocative concept that conveys a worldview + problem + solution, that expanded the range of how I think about my theory-of-victory for an existential win.
1An acknowledged scope limitation is not the same as a flaw, and I'm not sure which of these you mean by "problem." The explicit purpose of an AI Pause is to create time for some intentionally unspecified longer-term solution, on the grounds that there are many competing theories as to what such a solution looks like, whereas needing time to implement--and decide between them--is a common factor.
Regarding the reflection challenge, what about approaching it from the other direction? That is, what it would take to redesign the environment such that human propensities are favorable, rather than something needing correction?
One way of categorizing knowledge building is as:
1. Evolutionary = lots of parts, iterated in parallel, keeping what works in context.
2. Engineered = stacking modular abstractions. Tested against and developed for a context, but more fundamentally held to a standard of internal consistency.
Human flaws can be mostly understood as primarily thinking according to evolved processes, which run into systemic problems when out of distribution, then using engineered thinking processes to correct for this distributional shift...but the latter is stretched way beyond its capacity because it was only designed for mild and temporary out-of-distribution moments. We could deal with this system failing by strengthening up our engineered thinking methods, improving mental flexibility to the point where it can handle everything we can expect to have thrown at it...or we can look for ways to lighten the cognitive load. One could call the latter approach "social refactoring."
A high level example of what social refactoring might look like: computing the distributional shift on the societal equilibrium of any given innovation as an externalized cost, which then gets folded in to the more generalized externalized cost tax that (in this hypothetical world) fixed all the more legible global threats. Such an incentive realignment is upstream of the refactor itself, which is the resulting adaptation, where specifics are harder to predict (but maybe a worthwhile project nonetheless).
An acknowledged scope limitation is not the same as a flaw, and I'm not sure which of these you mean by "problem." The explicit purpose of an AI Pause is to create time for some intentionally unspecified longer-term solution, on the grounds that there are many competing theories as to what such a solution looks like, whereas needing time to implement--and decide between them--is a common factor.
Good point/question. My intended meaning is closer to the former, and basically the "problem" is referring to the fact that I lacked a good handle for the concept that I often want to invoke, with "AI Pause" and "Long Reflection" being the closest ones. I think what you say here makes sense and I don't intend for my new handle to replace "AI Pause" where "AI Pause" is more appropriate.
The rest of your comment seems like an interesting idea for solving part of the challenge, worth keeping in mind, and fleshing out and discussing in the future, when we have this "generalized externalized cost tax" that we can build your idea into.
Maybe something like a "long re-friending"? "Re-friending" (or just "friending") seems to me to capture the "coherent", "grow up further together" part, and to be less authoritarian-sounding than "correction." Goes well with claims AI made by labs are minority elites trying to run away with the world's decisions and stakes, better if we figure out how to be friends across factions first.
I feel like the reasons why MIRI circa 2014 moved away from talking about "Friendly AI" (because it was too anthropomorphic, etc.) and instead imported "alignment" to talk about the ~same issue also apply here. The "(re)friend(ing)" frame may make sense insofar as it's meant to be used internally, but probably not a great choice for something that is intended to be more widely broadcast.
"Long alignment" seems not terrible, but there's the whole "alignment to what", as well as "alignment of what". You can align yourself/[your values] to a Landian monstrosity, which you most likely don't want.
How about "long healing"? It is spiritually/conotationally similar to "correction", but I think it satisfies your desideratum of not having authoritarian connotations. It denotes things fitting well together ("alignment"), but also that not all ways of "fitting well together" are "created equal", and that there are better/[more "natural"] and worse/[less "natural"] ways for things to fit well together.
Reading your footnote 2, I got reminded of Rutger Bregman's The School for Moral Ambition. Rutger Bregman's "Moral Ambition" project (the book and the School for Moral Ambition) is one real attempt at this, though notably not from inside EA. He explicitly reframes doing good as the ambitious/high-status choice rather than a guilt-driven sacrifice. He is targeting elite-university students headed for McKinsey/consulting/finance, running selective competitive fellowships modeled on prestige-recruiting machinery, and defining moral ambition as "the will to be among the best, but with different measures of success."
What's suggestive is that this had to happen outside EA, partly as an explicit critique of it (Bregman frames EA as too guilt-coded, too analysis-paralyzed, not aspirational enough).
That said, Bregman doesn't engage with the actual meta-question you're raising — whether openly naming status as a lever backfires. He just uses the lever; there's no visible theorizing about whether the explicitness itself is corrosive.
Thanks for the pointer, which I didn't know about, but yeah I mostly mean discussing/analyzing the relationship between status seeking and altruism, strategizing how to best take advantage of it, including how to avoid problems caused by misalignment between the two (like in this example), and not just going ahead and pulling the lever as you put it.
My main issue is that some of the flaws are far from being "hard but soluble by a Novel Insight And Long Self-Correction": insoluble as stated, too easy or outright erroneous.
- Mankind being good at long-horizon strategy would require mankind to have a far greater skill at noticing the emergence of x-risk from novel places, like AI or the LHC. The problem is that x-risks are so rare that a very detailed description of all real risks with which IIRC 100 billion humans who ever existed have come up is well fittable in ~300 pages in Russian.
- The case against early MIRI's attempt to race towards Friendly AI would have to dismiss The Counterfactual Quiet AGI Timeline[1] and real-world worse actors like xAI.
- I find it unlikely that careful strategising is disvalued in making high-level decisions where it actually matters. What could be erroneously valued is strategising to keep one's position, like failing to inform a higher-level official about a problem.
- I wonder what is even doable about zero-sum values and how they would affect a world where a distribution[2] of power is locked in, as is likely to happen[3] in an ASI-ruled world.
- Being easy to manipulate could be similar to lacking the ability or the data necessary to come up with a better theory. For example, Newton's mechanics was an excellent approximation of reality and was replaced with the paradigm involving spacetime after all variants of the Michelson–Morley experiment failed to notice Earth's motion related to the aether.
My closest candidate solution to these problems is not some Philosophical Insight From The Future, but broadly educating people.
- ^
My main objection is that the existence of safety-pilled AI labs might have had a higher bus factor than ablating Yudkowsky.
- ^
Including a uniform-like one, as happens in the Epilogue of AI 2040.
- ^
However, resources of Earth or the Solar System can also be quickly reallocated between humans.
I like the idea of having a concept for this, though "Long Self-Correction" is probably not the most catchy term.
Why isn't there a version of EA that explicitly talks about how to leverage people's status motivations to do more good for the world? It's very possible that explicit talk about status is actually counterproductive at least in the short run, e.g. it heightens status motivations and makes people less altruistic, but then do we just march into the future while blindfolding ourselves to this aspect of human nature?
Wait, what? As far as I understood it, that was explicitly a main consideration in the 10% pledge Schelling point, though it's possible that I may have just came up with that argument myself and misattributed it.
This post got me thinking, though greater intelligence does not necessarily entail greater morality, I wonder whether moral coherence may be required for continued, open-ended intellectual development. Significant moral incoherence might eventually force an intelligence to correct itself, suppress further reflection, stagnate, or ultimately collapse if it develops too many contradictions. If so, maybe this would act as a limit on the orthogonality thesis (i.e., intelligence and values may vary independently across bounded levels of capability, while some value systems remain incompatible with indefinite cognitive development). From that perspective, the goal may be to help the combined AI–humanity system avoid becoming trapped in these developmental dead ends by discovering more coherent moral models (i.e., ones capable of supporting continued intellectual growth).
summary of the flaws that I have in mind:
- not having a workable moral framework (consequentialism, deontology, virtue ethics all having serious problems
The FTX debacle shows we gave a workable framework. If SBF had followed the rules, it wouldn't have happened.
As far as practical, good enough ethics goes, deontology, maybe with consequentialist justification, wins. Every organised society uses it.
A lot of rationalists seem to assume that “as you gain more IQ points, learn more facts, and think harder, you’ll naturally converge to the optimal theory of ethics.” Like the OP, I don’t think this is true in general.
A lot of rationalists also assume that you can since ethics by a solipsistic process that doesn't take other people into account
One problem is that human value might be inherently multidimensional
It clearly varies a lot, and therefore might as well be called human values.
What do values have to do with ethics, anyway? There is probably some relationship, and probably not simple two way identity.
Rationalists always equate ethics with values, rather than obligations, or virtues. There appears to be no specific reason for this. There are obvious things examples of values that aren't ethically relevant, eg. I can value tutti frutti over vanilla, but it is not ethically relevant. The Three Word Theory, "Morality is values" doesn't hone in on the topic, and probably leaves out important things like obligation.
in practice, human morality is a kind of status game that actively disvalues careful strategy and philosophy in most places
I would build on this: human morality is deeply corrupted by politics, and our political culture is arguably worse than it's been in decades.
long reflection = i want that. please. please let me have that. you have given a name to a thing that i have deeply, deeply wanted for longer than i can remember.
long self-correction = hello, human resources?
All of these seem relevant but as a starting point I would put something like most humans having a lot of technical debt (https://sashachapin.substack.com/p/review-meditation-from-cold-start), not feeling safe and okay in there here and now. Maybe not being enlightened. And then we could get to work on the difficult problems you bring up. Of course, there is the trap here of going off the rails using spirituality or me already being off the rails.
I propose the Long Self-Correction[1] as an alternative name/idea/concept to AI Pause and Long Reflection.
Problem with AI Pause: Pause until when, and for what purpose? Presumably to make AI (that we'll build later) safer, but the deeper problem is that humans aren't safe, and can't safely serve as builders, overseers, or alignment targets for powerful AIs.
Problem with Long Reflection: It seems to imply that the main problem with humans is that we just haven't had enough time to think, that reflection is the main thing we need to do more of, and then we can get on with building powerful AIs or other technologies. Or that if we build aligned AIs that sincerely help us think a lot more, or do the thinking for us, then things will turn out fine.
So I think we need a catchy handle for a related but distinct idea, that humans aren't ready to build AIs or other extremely powerful technologies, because we're currently too flawed, in a variety of ways, and it will take a long process (which may or may not end up succeeding) to fix those flaws.
A summary of the flaws that I have in mind:
(This list focuses on key bottlenecks that seem hard to fix even with AI assistance or intelligence enhancement, and isn't meant to be a complete list of human flaws / safety problems. It ignores e.g. that the median human is ignorant of many important issues, and that we're currently quite bad at complex large-scale coordination such as passing/implementing close-to-optimal government policies.)
My main hope for a Long Self-Correction eventually succeeding rests on the fact that humans have seemingly, mysteriously, made progress on these issues over a very long period of time, so if we preserve the environment in which we can seemingly do this, and not give anyone or anything the power to permanently derail such progress, then maybe we can continue to snowball The Correction until we reach a point when we can rightly justify reshaping the universe according to our volition.
It will probably be shortened to "The Long Correction" at some point if it catches on, similar to how "outer space" is now often just "space".
Why isn't there a version of EA that explicitly talks about how to leverage people's status motivations to do more good for the world? It's very possible that explicit talk about status is actually counterproductive at least in the short run, e.g. it heightens status motivations and makes people less altruistic, but then do we just march into the future while blindfolding ourselves to this aspect of human nature?