Sounds interesting and like something I might miss if true. I would be interested in examples.

I was engaging with this, because I thought maybe you are advocating for some kind of doubles think, and I might spell you out of that, but this doesn't seem to be the case. I am not interested to get deep into that religion argument (too many different people and different religions). Yes, there are some topics like ethics where most people don't benefit from reasoning about them explicitly, and even smart people tend to get very confused by them. I remember I was confused for days by my first introduction to ethics.

Reframing your thoughts such that you don't step on your own toes is great. I am not quite sure what you are trying to argue for with the religion point. Do you actually endorse this on a practical basis? Have you convinced yourself that you need to yourself belief in untrue things or that it is good if other people are believing untrue things in just the right way they cancel each other out? Believing in God seems a particularly bad example. Having high alieve you can solve a problem that lots of people haven't solved before you might be fine for minutes or a few days. But I can't see how you would get psychological benefits from believing in a god or fate and whatever and that not messing up your epistemics.

Thank you for this post! My impression is that this post makes real progress at identifying some of the upstream cruxes between people's models of how AGI is going to go.

As examples, if you look at @So8res’s AGI ruin scenarios are likely (and disjunctive), I claim that a bunch of his AGI ruin scenarios rely on his belief that alignment is hard. I think that belief is correct! But still, it makes his argument less disjunctive than it might seem. Likewise, I now recognize that my own What does it take to defend the world against out-of-control AGIs? sneaks in a background assumption that alignment is hard (or alignment tax is high) in various places.

This seems correct to me.

But I figured out that I can occupy that viewpoint better if I say to myself: “Claude seems nice, by and large, leaving aside some weirdness like jailbreaks. Now imagine that Claude keeps getting smarter, and that the weirdness gets solved, and bam, that’s AGI. Imagine that we can easily make a super-Claude that cares about your long-term best interest above all else, by simply putting ‘act in my long-term best interest’ in the system prompt or whatever.” Now, I don’t believe that, for all the reasons above, but when I put on those glasses I feel like a whole bunch of the LLM-focused AGI discourse—e.g. writing by Paul Christiano, OpenPhil people, Redwood people, etc.—starts making more sense to me.

That seems to represent well the viewpoint that I now have even less faith in, but that didn't seem strictly ruled out to me 4 years ago when GPT-3 had been around for a while, and it seemed likely we were headed for more scaling:

39 Nothing we can do with a safe-by-default AI like GPT-3 would be powerful enough to save the world (to ‘commit a pivotal act’), although it might be fun. In order to use an AI to save the world it needs to be powerful enough that you need to trust its alignment, which doesn’t solve your problem.

• What exactly makes people sure that something like GPT would be safe/unsafe?
• If what is needed is some form of insight/break through: Some smarter version of GPT-3 seems really useful? The idea that GPT-3 produces better poetry than me while GPT-5 could help to come up with better alignment ideas, doesn't strongly conflict with my current view of the world?

Emacs has Eliza still built in by default of course :)

1. This is much less true of my participation in the study where I was more conscientious, but I feel like historically a lot of my AI speed-up gains were eaten by the fact that while a prompt was running, I'd look at something else (FB, X, etc) and continue to do so for much longer than it took the prompt to run.

The temptation to multitask strikes me as a very likely cause of the loss of productivity. It is why I virtually never use reasoning models except for deep research.

And on a more micro-level, living knowing that I and everyone else have one year left to live, and that it's my fault, sounds utterly agonizing.

Earlier you say:

or frankly even if anyone who continues to exist after I die has fun or not or dies or not, because I will be dead, and at that point, from my prospective, the universe may as well not exist anymore.

How are these compatible? You don't care if all other humans die after you die unless you are responsible?

Why wait for mice? If I understand correctly, a lot of the symptoms common to aging also show up in Saccharomyces cerevisiae. The goal of the synthetic yeast project 2.0 was among other things creating chromosomes without retrotransposons, and they recently finished synthesising all 16 chromosomes, so all that is left should be assembling them? If those synthetic cells do or don't show typical signs of aging, that should be a strong sign either way, right?

Deep Research found this PDF. Search for "I was wrong" in the PDF.

The last year, my median conversation was about as entertaining as yours. The top 10% conversations are fun-in-their-own-right at that moment already because my brain anticipates some form of long-term value (with the exception of cracking jokes). I don't know if all those conversations would count as "casual". As intellectually stimulating as the Task Master TV-show is funny. Conversation is more heavy tailed than movies though. Long term value includes: learning or teaching (learning some new technical thing that's usually not written down anywhere (Podcasts tend to be better for that), getting a pointer about something to learn about, teaching something technical in the anticipation that the other person is actually going to do anything with that knowledge, incorporating the generating function behind someone's virtues/wisdom), thinking out loud with someone else in the expectation that this might lead to an interesting idea, gossip, life stories (sometimes preventing you from harm from people/situations that can't be trusted. Sometimes just illuminating parts of life you'd know less about). My most fun conversation had me grinning for 30 minutes after still, and my heartbeat after that time was also still 10–20 beats higher than usual.

My median conversations at parties over my entire life are probably less entertaining than your median ones. My bar for an interesting conversation also rose when I stumbled over the wider rationalist sphere. I remember two conversations from before that era where the main important information was essentially just "there are other smart people out there, and you can have interesting conversations with them where you can ask the questions you have etc.". One was at a networking event for startup founders, and the other was a Computer Science PhD student showing me his work and the university campus (same conversation that got my heart-beat up).