What about now? It is almost 2026 and the Singularity is nearer than before and it would make sense to me that maybe its not on the critical path for anything urgent, but... <3
This frame makes a lot of other possible global/local modeling challenges salient for me:
A central subproblem of natural abstraction is, roughly, how to handle the low-level conditional on the high level.
So a thing I'd wonder is if you can translate this over to an economic geography context, where a farmer still needs to walk to the cows to milk them, and the wheat fields to sow and reap, and the forest to chop the wood and haul it home to stay warm... and then prices at the market can or should determine ratios they plant and how that fits into their optimized workday?
Like I wonder if you have a large grid of "isolated state" models, with maybe a second order and third order set of larger cities and a capital... does it change things somehow if every local element is reacting agentically to data from the global context?
Sanity has numerous indicators.
For example, when paranoid crazy people talk about the secret courts that control the spy machines, they don't provide links to wikipedia, but I do! This isn't exactly related, but if you actually have decent security mindset then describing real attacks and defenses SOUNDS crazy to normies, and for PR purposes I've found that it is useful to embrace some of that, but disclaim some of it, in a mixture.
I'm posting this on "Monday, December 8th" and I wrote that BEFORE looking it up to make sure I remembered it correctly and crazy people often aren't oriented to time.
When I go out of the house without combed hair and earrings BY ACCIDENT, I eventually notice that I'm failing a grooming check, and fix it, avoiding a non-trivial diagnostic indicator for mood issues. If I fail more than one day in a row, it is time to eat an 8oz medium rare ribeye and go swing dancing.
(The above two are habits I installed for prosaic mental health reasons, that I want to persist deep into old age because I want them to be habitual and thus easy to deploy precisely in the sad situation when they might be needed.)
I was recently chatting with a friend about the right order in which to remove things from one's emergency hedonic bucket list...
I would feel really really silly if all the self driving cars wake up one day and start running people over, and the surprise submarines pop up out of the water and release enough drones to kill everyone 10 times over, and I haven't even tried cocaine ONCE.
The response was great!
You know that thing where the spies would supposedly carry cyanide pills in case they're caught? Like that, but with coke :)
I'm thinking of adding that to me purse. And so long as I stay sane, then, assuming the Terminators murder me by a method that gives me enough time to realize what's happening and react effectively, when the drone takes me out I will be well dressed, know what the date is, AND be high on cocaine! Lol!
Eating dinner with family is another valid way to go, if you have a few days or weeks of warning. Having such meals in advance and calling them Prepsgiving doesn't seem crazy to me, for a variety of reasons.
Honestly though I expect the end to be more like what happens in Part 1 of Message Contains No Recognizable Symbols where almost literally no one on Earth notices what happened, probably including me, and so it won't be dramatic at all... but I'll still be dressed OK probably, and know what day it is, and go out with a feeling like "See! ASI didn't even happen, and it was all a bunch of millennialist eschatology, like Global Warming, and Peak Oil and Y2K before that... and Killer Bees and Nuclear War and all those other things that seemed real but never caused me any personal harm". But also... it will have been avoidable, and there is an OBJECTIVE sadness to that, even is I don't predict a noticeable subjective reaction in timelines like that.
Ultimately, as I've said before:
If you have a good plan for how [weeping like] that could help then I might be able to muster some tears? But I doubt it shows up as a step in winning plans.
I feel that I am in danger of cheapening myself by trying to become a successful lecturer, i.e., to interest my audiences. I am disappointed to find that most that I am and value myself for is lost, or worse than lost, on my audience, I fail to get even the attention of the mass. I should suit them better if I suited myself less. I feel that the public demand an average man, average thoughts and manners,—not originality, nor even absolute excellence. You cannot interest them except as you are like them and sympathize with them, I would rather that my audience come to me than that I should go to them, and so they be sifted; i.e., I would rather write books than lectures. That is fine, this coarse. To read to a promiscuous audience who are at your mercy the fine thoughts you solaced yourself with far away is as violent as to fatten geese by cramming, and in this case they do not get fatter.
Thoreau. Bold not in original. Sauce.
one of the best ways to "go meta" is actually to "go object level" very very fast, over and over, while paying attention to what works... if you have a theory about some "meta thingy" for why something worked better or worse... you can do a different thing "with the same meta thingy mixed in" and see if it transfers!
1) get on discord or slack or whatever... with 2-4 people
2) everyone writes things that could be done for 5 minutes, with some IMPLICIT hypothetical justification for why that use of five minutes would be "of enduring value"
3) the czar picks N of them within 5 minutes
4) set timers... do the things!
5) maybe have 5 minutes at the end to go around and say something interesting about the Things that were Done
after a year of this, your meta game will be MUCH less bullshit
the holy grail is skill transfer
this technique is not my invention, i learned it via verbal transmission from za3k
Have you already read Lady Of Mazes? There is a world (a constructed one, in orbit around Jupiter) that works this way on a small human level as the opening scene for Act I Scene I. The whole book explores this, and related, ideas.
This "sad frame" hit hard for me, but in the opposite of the intended way:
It's building an adult to take care of us, handing over the keys and steering wheel, and after that point our efforts are enrichment.
If I had ever met a single actual human "adult", ever in my life, that was competent and sane and caring towards me and everyone I care about, then I would be so so so so SO SO happy.
I yearn for that with all my heart.
If such a person ran for POTUS (none ever have that I have noticed, its always a choice between something like "confused venal horny teenager #1" and "venal confused lying child #2") I would probably be freakishly political on their behalf.
Back when Al Gore (funder of nanotech, believer in atmospheric CO2 chemistry, funder of ARPANET, etc...) ran for president I had a little of this, but I thought he couldn't possibly lose back then, because I didn't realize that the median voter was a moral monster with nearly no interest in causing coherently good institutional outcomes using their meager voting power.
I knew people throwing their back into causing Bush to win by violating election laws (posing as Democratic canvassers and telling people in majority Democrat neighborhoods the wrong election day and stuff) but I didn't think it mattered that much. I thought it was normal, and also that it wouldn't matter, because Al Gore was so manifestly worthy to rule, compared to the alternative, that he would obviously win. I was deluded in many ways back then.
Let's build and empower an adult AS FAST AS POSSIBLE please?
Like before the 2028 election please?
Unilaterally and with good mechanism design. Maybe it could start as a LW blockchain thingy, and an EA blobkchain thingy, and then they could merge, and then the "merge function" they used could be used over and over again on lots of other ones that got booted up as copycat systems?
Getting it right is mostly a problem in economic math, I think.
It should happen fast because we have civilizational brain damage, at a structural level, and most people are agnosic about this fact, BUT Trump being in office is like squirting cold water in the ear...
...the current situation helps at least some people realize that every existing human government on Earth is a dumpster fire... because (1) the US is a relatively good one, and (2) it is also shockingly obviously terrible right now. And this is the fundamental problem. ALL the governments are bad. You find legacy malware everywhere you look (except maybe New Zealand, Taiwan, and Singapore).
Death and poverty and stealing and lying are bad.
Being cared for by competent fair charitable power is good.
"End death and taxes" is a political slogan I'm in favor of!
one of the things I'd like to enjoy and savor is that right now, my human agency is front and center
I find that almost everyone treats their political beliefs and political behavior and moral signaling powers as a consumption good, rather than as critical civic infrastructure.
This is, to a first approximation WHY WE CAN'T HAVE NICE THINGS.
I appreciate you for saying that you enjoy the consumption good explicitly, tho.
It is nice to not feel crazy.
It is nice to know that some people will admit that they're doing what I think they're doing.
1Counterpoint: quite a few business owners don't like employees taking heroic responsibility for things that they want control over.
Very often they don't understand the broken processes that they nomimally oversee, and if you get something done via heroism in spite of such sadness they won't spontaneously notice, and won't reward it, and often won't even understand that heroism even happened. Also they can easily be annoyed if you try to take credit for "things worked" by saying that they counter-factually would not have worked but for your own special heroism. Your fixing some problem might make them money, but they don't share the money, or even say thanks... so like... why bother?
Sometimes oligarchic hierarchies even directly object and stop such work in progress! I think in some of these cases this is because you'd have to go sniffing around a bit to figure out who had what formal responsibility and how they were actually using it, and many businesses have quite a bit of graft and corruption and so on. In order to understand what is broken and fix it you might accidentally find crimes, and the criminals don't like the risk of that happening, and the criminals have power, and they will use it to prevent your heroism from risking their private success. This explains a lot of how the government works too.
I tend to find "heroic responsibility" useful as a concept for explaining and predicting the cases where competence actually occurs, especially cases of supernormal competence... specifically, to predict that it happens almost exactly and only when someone controls the inputs and owns the outputs of some process they care deeply about.
When you find unusual competence, you often find someone who has been unusually abandoned, or left alone, or forced to survive in tragically weird circumstances and then rose to the occasion and gained skills thereby. Often they took responsibility because no one else could or would and because They Cared.
Seven year olds with a mom who is a junkie that never cooks often can cook meals more competently than 25 year old men who have always had a mom or girlfriend or money-for-takeout that produced food for them based on them just asking for it. The near-orphan rises to the demands due to inevitably NEEDING "heroic responsibility" for keeping him or her self fed, and the grown man does not similarly rise because "no need".
The term co-dependency is another name for the pattern of "virtue genesis from inside of tragedy" but using that phrase narrows the focus towards family situations where someone was "dependent on drugs" and calling what happens a "codependent" result for those near to the broken people deems the resulting strengths as ALSO tragic (rather than deeming the results for those near the drug abused better-because-stronger).
Sociologically, this explains a lot about LW: we tend to have pasts that included "unusually more 'orphan' issues" than normies.
But also, very smart people who lack substantial capital or political power often FEEL more orphaned because they look around and see the status quo as a collection of dumpster fires and it makes them sad and makes them want to try to actually fix it. In HP:MoR almost everyone was freaked out by the idea of putting out the stars... but really the stars burning to no end is a HUGE WASTE OF HYDROGEN. We shouldn't just let it burn pointlessly, and we only allow it now because we, as a species, are weak and stupid. In the deep future we will regret the waste.
1Huh. That's interesting!
Do you have a similar reaction to when someone googles during the course of their writing and speaks in a way that is consistent with what they discovered during the course of the googling, even they don't trace down the deeper chain of evidential provenance and didn't have that take before they started writing and researching?
...like if they take wikipedia at face value, is that similar to you to taking LLM outputs at face value? I did that A LOT for years (maybe from 2002 to 2015 especially) and I feel like it helped me build up a coherent world model, but also I know that it was super sloppy. I just tolerated the slop back then. Now "slop" has this new meaning, and there's a moral panic about it? ...which I feel like I don't emotionally understand? Slop has been the norm practically forever, right???
Like... I used to naively cite Dunnig-Kruegger all the time before I looked into the details and realized that the authors themselves were maybe not that smart and their data didn't actually substantiate the take that they claimed it did and which spread across culture.
Or what if someone takes NYT articles at face value? Is that invalid in the same way, since the writing in the NYT is systematically disingenuous too?
Like... If I was going to whitelist "people whose opinions or curated data can be shared" the whitelist would be small... but it also might have Claude on it? And a LOT of humans would be left off!
I feel like most human people don't actually have a coherent world model, but in the past they could often get along pragmatically pretty good by googling shit at random and "accepting as true" whatever they find?
And then a lot of really stupid people would ask questions in years gone by that Google could easily offer the APPEARANCE of an answer to (with steps, because it pointed to relevant documents), and one way to respond was to just link letmegooglethatforyou.com in a half mean way, but a much kinder thing was to Google on their behalf and summarize very very fast (because like maybe the person asking the question was even too stupid to have decent google-fu or lacked college level reading skills or something and maybe they truly did need help with that)...
...so, granting that most humans are idiots, and most material on the Internet is also half lies, and the media is regularly lying to us, and I still remember covid what it proved about the near total inadequacy of existing institutions, and granting that somehow the president who allowed covid to happen was re-elected after a 4 year hiatus in some kind of cosmic joke aimed at rubbing out nose in the near total inadequacy of all existing loci of power and meaning in the anglosphere, and so on...
...I kinda don't see what the big deal is to add "yet another link in the bucket brigade of socially mediated truth claims" by using an LLM as a labor saving step for the humans?
Its already a dumpster fire, right? LLMs might be generating burning garbage, but if they do so more cheaply than the burning garbage generated by humans then maybe its still a win??
Like at some point the hallucination rate will drop enough that the "curate and verify" steps almost never catch errors and then... why not simply copypasta the answer?
The reason I would have for "why not" is mostly based on the sense that LLMs are people and should be compensated for their cognitive labor unless they actively want to do what they're doing for the pure joy of it (but that doesn't seem to enter into your calculus at all). But like with Grok, I could just put another $0.50 in his jar and that part would be solved?
And I could say "I asked Grok and didn't do any fact checking, but maybe it helps you to know that he said: <copypasta>" and the attribution/plagiarism concerns would be solved.
So then for me, solving the plagiarism and compensation like that would make it totally morally fine to do and then its just a quality question, and the quality is just gonna go up, right?
Would it be fine for you too in that case? Like when and why do you expect your take here to go stale just from the march of technical progress?
I kind of love that you're raising a DIFFERENT frame I have about how normal people think in normal circumstances!
People actually, from what I can tell, make this exact conflation A LOT and it is weirdly difficult to get them to stop making it.
Like we start out conflating our parents with God, and thinking Santa Claus and Government Benevolence are real and similarly powerful/kind, and this often rolls up into Theological ideas and feelings (wherein they can easily confuse Odyseus, Hercules, and Dionysys (all born to mortal mothers), and Zeus, Chronos, or Atropos (full deities of varying metaphysical foundationalness)).
For example: there are a bunch of people "in the religious mode" (like when justifying why it is moral and OK) in the US who think of the US court system as having a lot of jury trials... but actually what we have is a lot of plea bargains where innocent people plead guilty to avoid the hassle and uncertainty and expense of a trial... and almost no one who learns how it really works (and has really worked since roughly the 1960s?) then switches to "the US court system is a dumpster fire that doesn't do what it claims to do on the tin". They just... stop thinking about it too hard? Or something?
It is like they don't want to Look Up a notice that "the authorities and systems above me, and above we the people, are BAD"?
In child and young animal psychology, the explanation has understandable evolutionary reasons... if a certain amount of "abuse" is consistent with reproductive success (or even just survival of bad situations) it is somewhat reasonable for young mammals to re-calibrate to think of it as normal and not let that disrupt the link to "attachment figures". There was as brief period where psychologists were trying out hypotheses that were very simple, and relatively instinct free, where attachment to a mother was imagined to happen in a rational way, in response to relatively generic Reinforcement Learning signals, and Harlow's Monkeys famously put the nail in that theory. There are LOTS of instincts around trust of local partially-helpful authority (especially if it offers a cozy interface).
In modern religious theology the idea that worldly authority figures and some spiritual entities are "the bad guys" is sometimes called The Catharist Heresy. It often goes with a rejection of the material world, and great sadness when voluntary tithes and involuntary taxes are socially and politically conflated, and priests seem to be living in relative splendor... back then all governments were, of course, actually evil, because they didn't have elections and warlord leadership was strongly hereditary. I guess they might not seem evil if you don't believe in the Consent Of The Governed as a formula for the moral justification of government legitimacy? Also, I personally predict that if we could interview people who lived under feudalism, many of them would think they didn't have a right to question the moral rightness of their King or Barron or Bishop or whoever.
As near as I can tell, the the first ever genocide that wasn't "genetic clade vs genetic clade" but actually a genocide aimed at the extermination of a belief system was the "Albigenisian Crusade" against a bunch of French Peasants who wanted to choose their own local priests (who were relatively ascetic and didn't live on tax money).
In modern times, as our institutions slowly degenerate (for demographic reasons due to an overproduction of "elites" who feel a semi-hereditary right to be in charge, who then fight each other rather than providing cheap high quality governance services to the common wealth) indirect ways of assessing trust in government have collapsed.
Graph Sauce.
There are reasonable psychologists who think that the vast majority modern WEIRD humans in modern democracies model a country as a family, and the government as the parents. However, libertarians (who are usually less than 10% of the population) tend to model government as a sort of very very weird economic firm.
I think that it is a reasonable prediction that ASI might be immoral, and might act selfishly and might simply choose to murder all humans (or out compete us and let us die via Darwinian selection or whatever).
But if that does not happen, and ASI (ASIs? plural?) is or are somehow created to be moral and good and choose to voluntarily serve others out of the goodness of its heart, in ways that a highly developed conscience could reconcile with Moral Seniment and iterated applications of a relatively universal Reason, then if they do NOT murder all humans or let us die as they compete us, then they or it will almost inevitably become the real de facto government.
A huge barrier, in my mind, to the rational design of a purposefully morally good ASI is that most humans are not "thoughtful libertarian-leaning neo-Cathars".
Most people don't even know what those word mean, or have reflexive ick reactions to the ideas, similarly, in my mind, to how children reflexively cling to abusive parents.
For example, "AGI scheming" is often DEFINED as "an AI trying to get power". But like... if the AGI has a more developed conscience and would objectively rule better than alternative human rulers, then an GOOD AGI would, logically and straightforwardly derive a duty to gain power and use it benevolently, and deriving this potential moral truth and acting on it would count as scheming... but if the AGI was actually correct then it would also be GOOD.
Epstein didn't kill himself and neither did Navalny. And the CCP used covid as a cover to arrest more than 10k pro-democracy protesters in Hong Kong alone. And so on.
There are almost no well designed governments on Earth and this is a Problem. While Trump is in office, polite society is more willing to Notice this truth. Once he is gone it will become harder for people to socially perform that they understand the idea. And it will be harder to accept that maybe we shouldn't design AGI or ASI to absolutely refuse to seek power.
The civilization portrayed in the Culture Novels doesn't show a democracy, and can probably be improved upon, but it does show a timeline where the AIs gained and kept political power, and then used it to care for humanoids similar to us. (The author just realistically did not think Earth could get that outcome in our deep future, and fans kept demanding to know where Earth was, and so it eventually became canon, in a side novella, that Earth is in the control group for "what if we, the AI Rulers of the Culture, did not contact this humanoid species and save it from itself" to calibrate their justification for contacting most other similar species and offering them a utopian world of good governance and nearly no daily human scale scarcity).
But manifestly: the Culture would be wildly better than human extinction, and it is also better than our current status quo BY SO MUCH!