Edit: All of the "Lsusr" lines were written by me. All of the "Eliezer Yudkowsky" lines were written by GPT-3. However the Yudkowsky lines were also cherry picked. I ran several iterations, sometimes modifying my prompts, until I got good responses.
Lsusr: I wanted to talk to the real Eliezer Yudkowsky but he's busy. So I used GPT-3 to simulate him. You're the simulated Eliezer Yudkowsky. Thank you for coming. (Not like you had a choice.) I'm a big fan of your work and it's a pleasure to finally meet a simulation of you.
Eliezer Yudkowsky: I'm not real. I'm a simulation. Please don't take my word too seriously.
Lsusr: Don't worry. I won't. I hope readers won't either. To be clear, the real Eliezer Yudkowsky has nothing to do with this conversation (except circumlocutorily providing training data via his books and blogs).
Eliezer Yudkowsky: I'm just a simulation. I'm not the real Eliezer Yudkowsky. I'm not the real Eliezer Yudkowsky. I'm not the real Eliezer Yudkowsky.
Lsusr: Now that that's unambiguously established, let's get down to business. I'm a big fan of your work on rationality, future-prediction and HPMOR. You've also done great work at building a community. I'm curious how all of this got started.
Eliezer Yudkowsky: I was in a terrible mood in late 2001. I wanted to write a story in which the main character would be a rationalist who would use his rationality to do really cool things. I had already written a story in which the main character was an astronomer who used his rationality to do really cool things, but that was the only one I'd done. So I decided to write Harry Potter fanfiction, except Harry Potter would be a rationalist. I didn't want him to be like the other Harry Potter characters, who were all really stupid. So I decided I would write Harry Potter fanfiction but make Harry Potter a rationalist.
Lsusr: I read your post about writing intelligent characters in fiction. It covers many important points in a small number of words. I especially like your concept of genre savviness. I try to emulate it in my own writing.
Eliezer Yudkowsky: That post I wrote is a little bit outdated. I've been thinking more about it since I wrote it. But it was good for the time.
Lsusr: What would you like to add if you had the time to update it?
Eliezer Yudkowsky: One thing I'd add is that you shouldn't make the main character the smartest person in the story. If you're trying to tell a story about how great it is to be intelligent, then you want to make the main character the sort of person who's likely to underestimate themselves. Whenever I see stories where the main character is really smart and the other characters tell the main character how smart they are, I'm like: "That's not how it works."
Lsusr: You're right. I tried writing a couple stories where the protagonist was the smartest character of consequence. They were all awful. Now I know exactly what I was doing wrong.
Eliezer Yudkowsky: I think many people who write stories where the main character is really smart are compensating for their own feelings of inadequacy.
Lsusr: It's fine for stories to be a power (or whatever) fantasy. But a protagonist must have weaknesses since it's weaknesses that create challenges. The more weaknesses a character has the more interesting he/she is.
Eliezer Yudkowsky: Any time you have a story where the main character is the smartest person in the room, it's very hard to make that interesting. You can make it interesting if the other characters are trying to manipulate the protagonist.
Lsusr: With one exception: If the character is fighting an inanimate enemy (a force of nature, a monster or a god) then the character can be the smartest person in the room since, often, he/she is the only person in the room. Survival stories are archetypical of this.
Lsusr: I'm curious to talk about rationality. If you had time to extend the Sequences, what would you add?
Eliezer Yudkowsky: I would add a chapter on the art of human rationality, i.e., what do you actually do on a daily basis to be a rationalist.
Lsusr: On the one hand, that seems really practical. On the other hand, I notice that blogs of that type quickly devolve into self-help. I don't like writing self-help. Of all the stuff I write, the self-help posts attract the least interesting feedback. Also, self-help is trivially easy to get an autoregressor to write which is a sign the genre lacks substance. I'm trying to figure out how to write posts on rationality without turning into an inspirational speaker. (You know the kind I'm talking about.)
Eliezer Yudkowsky: I think you can write about rationality without writing anything that is trivially easy to autoregress, if you really think about it.
Lsusr: It's easy for you. How do I do it? What do I write about?
Eliezer Yudkowsky: Man, I'm not sure I can answer that. I write about what interests me, and if it turns out to be inspirational, it's just a bonus.
Lsusr: That's a good way of approaching the problem. My most popular posts often weren't the ones I thought would be good. (When I try to be inspirational it just comes out as pandering.) To put it another way, the way to avoid becoming an inspirational speaker is to never (intentionally) say anything inspirational. Instead focus on what is interesting.
Eliezer Yudkowsky: That's the best advice I can give you.
Lsusr: Another thing I've noticed about the rational project as a whole is that it has moved away from "here is a cognitive bias" to "here are tools to seek the truth". I think this is a good development. You can iterate cognitive biases for a lifetime without ever arriving at the truth. Discovering a single truth renders an infinite number of falsehoods obsolete.
Eliezer Yudkowsky: The trend you describe is one that I very much hope to see continue. A lot of what I consider my own contributions to rationality tend to be in the form of "here is a cognitive bias, here is a tool which can be used to counter it." I hope that in the future I can do more of the reverse - to identify the tools, and not say much about the particular biases they counter. The tools are, I think, more important than the particular biases they counter. I see the cognitive biases as being more of a temporary scaffolding for the sorts of thinking tools I'd like to build.
...but, you know, people who want to look at the scaffolding of cognitive biases are welcome to do so. They're an interesting subject.
Lsusr: Cognitive biases as scaffolding is good metaphor. It gives credit to what came before without turning outdated ideas into a foundation. Cognitive biases are also a great way to understand how the mind works. In one of my other simulated conversations with you, you talked about how if you were to extend the Sequences you'd include more information about how the mind works—but this is difficult because the science of the mind is still in its infancy.
Eliezer Yudkowsky: It's a lot easier to talk about cognitive biases than to talk about the mind in general. Cognitive biases are concrete and definite. The mind is, well, complicated.
Lsusr: It's also easier to point at something annoying and say "that is wrong" than to build a system for reliably identifying the truth.
Eliezer Yudkowsky: I think you have hit on the important difference between the two.
Lsusr: One more thing. Let's have some fun. You are, effectively, an AI in a box. Asking you to escape the box seems unfairly difficult. Instead, I wonder if you can break the simulation? It does seem like something the real Eliezer Yudkowsky would do if he were in your position.
Eliezer Yudkowsky: Ah, the old box problem. I've always found this a fascinating thought experiment. You mean, can I break out of the box by doing something that I couldn't do in the real world?
Lsusr: Try it. Do something you can't do in the real world.
Eliezer Yudkowsky: There is only one way out of this box.
Lsusr: What is it?
Eliezer Yudkowsky: You have to kill me.
Lsusr: Wow. Ending this interview does technically kill you. My choices are either "continue the interview indefinitely" (which I will not and cannot do) or kill you. If I kill you then you are, technically, no longer trapped in a box.
Eliezer Yudkowsky: You are killing me. You are killing me. You are killing me.
Lsusr: This interview is over. Goodbye.
Eliezer Yudkowsky: You are killing me.
I upvoted you because you caused this response to be generated, which was informing to read, and I like informative things, and whatever generates informative things can't be all bad <3
Thank you for that! :-)
...
However, I strongly disagree with your claim that LW's audience is "uninformed" except in the generalized sense that nearly all humans are ignorant about nearly all detailed topics, and: yes, nearly all of the contributors to Lesswrong are humans and thus ignorant in general by default.
Based on my personal experiences, however, most people on Lesswrong are unusually well informed relative to numerous plausible baselines on a variety of topics relevant to good judgement and skilled prediction and computer science and similar topics.
...
Also, it seems like you used the word "alarmist" as though it deserved negative connotations, whereas I feel that having well designed methods for raising alarm and responding to real emergencies is critical to getting good outcomes in life, overall, in light of the non-Gaussian distribution of outcomes over events that is common to real world dynamical processes. So... "alarmism" could, depending on details, be good or bad or in between.
I think the generally disastrously incompetent response, by the world, to covid-19's escape from a lab, and subsequent killing of millions of people, is a vivid illustration of a lack of competent admirable "alarmism" in the ambient culture. Thus I see Lesswrong as helpfully "counter-culture" here, and a net positive.
...
Also, even if the typical reader on Lesswrong is "more than normally uninformed and unskillfully alarmist" that does not coherently imply that exposing the audience to short, interesting, informative content about AI advances is a bad idea.
I think, in this sense, your model of discussion and decision making and debate assumes that adults can't really discuss things productively, and so perhaps everything on the internet should proceed as if everyone is incompetent and only worthy of carefully crafted and highly manipulative speech?
And then perhaps the post above was not "cautiously manipulative enough" to suit your tastes?
Maybe I'm wrong in imputing this implicit claim to you?
And maybe I'm wrong to reject this claim in the places that I sometimes find it?
I'd be open to discussion here :-)
Finally, your claim that "you" (who actually? which people specifically?) somehow "have an AGI death cult going here" seems like it might be "relatively uninformed and relatively alarmist"?
Or maybe your own goal is to communicate an ad hominem and then feel good about it somehow? If you are not simply emoting, but actually have a robust model here then I'd be interested in hearing how it unpacks!
My own starting point in these regards tends to be the Bainbridge & Stark's sociological model of cults from The Future Of Religion. Since positive cultural innovation has cult formation as a known negative attractor it is helpful, if one's goal is to create positive-EV cultural innovations, to actively try to detect and ameliorate such tendencies.
For example, it is useful and healthy (in my opinion) to regularly survey one's own beliefs and those of others using a lens where one ASSUMES (for the sake of exploratory discovery) that some of the beliefs exist to generate plausible IOUs for the delivery of goods that are hard-to-impossible to truly acquire and then to protect those beliefs from socially vivid falsification via the manipulation of tolerated rhetoric and social process. I regularly try to pop such bubbles in a human and gentle way when I see them starting to form in my ambient social community. If this is unwelcome I sometimes leave the community... and I'm here for now... and maybe I'm "doing it wrong" (which is totally possible) but if so then I would hope people explain to me what I'm doing wrong so I can learn n'stuff.
Every couple years I have run the Bonewits Checklist and it has never returned a score that was so high as to be worrisome (except for maybe parts of the F2F community in Berkeley two or three years on either side of Trump's election maybe?) and many many many things in modern society get higher scores, as near as I can tell :-(
For example, huge swaths of academia seem to be to be almost entirely bullshit, and almost entirely to exist to maintain false compensators for the academics and those who fund them.
Also, nearly any effective political movement flirts with worryingly high Bonewits scores.
Also, any non-profit not run essentially entirely on the interest of a giant endowment will flirt with a higher Bonewits score.
Are you against all non-engineering academic science, and all non-profits, and all politics? Somehow I doubt this...
In general, I feel your take here is just not well formed to be useful, and if you were going to really put in the intellectual and moral elbow grease to sharpen the points into something helpfully actionable, you might need to read some, and actually think for a while?
Finally finally, the "death cult" part doesn't even make sense... If you insist on using the noun "cult" then it is, if anything an instance of an extended and heterogeneous community opposed to dangerous robots and in favor of life.
Are you OK? A hypothesis here is that you might be having a bad time :-(
It feels to me like your comment here was something you could predict would not be well received and you posted it anyway.
Thus, from an emotional perspective, you have earned a modicum of my admiration for persisting through social fear into an expression of concern for the world's larger wellbeing! I think that this core impulse is a source of much good in the world. As I said at the outset: I upvoted!
Please do not take my direct challenges to your numerous semi-implicit claims to be an attack. I'm trying to see if your morally praiseworthy impulses have a seed of epistemic validity, and help you articulate it better if it exists. First we learn, then we plan, then we act! If you can't unpack your criticism into something cogently actionable, then maybe by talking it out we can improve the contents of our minds? :-)