I came up with this after watching a science fiction film, which shall remain nameless due to spoilers, where the protagonist is briefly in a similar situation to the scenario at the end. I'm not sure how original it is, but I certainly don't recall seeing anything like it before.

Imagine, for simplicity, a purely selfish agent. Call it Alice. Alice is an expected utility maximizer, and she gains utility from eating cakes. Omega appears and offers her a deal - they will flip a fair coin, and give Alice three cakes if it comes up heads. If it comes up tails, they will take one cake away her stockpile. Alice runs the numbers, determines that the expected utility is positive, and accepts the deal. Just another day in the life of a perfectly truthful superintelligence offering inexplicable choices.

The next day, Omega returns. This time, they offer a slightly different deal - instead of flipping a coin, they will perfectly simulate Alice once. This copy will live out her life just as she would have done in reality - except that she will be given three cakes. The original Alice, however, receives nothing. She reasons that this is equivalent to the last deal, and accepts.

(If you disagree, consider the time between Omega starting the simulation and providing the cake. What subjective odds should she give for receiving cake?)

Imagine a second agent, Bob, who gets utility from Alice getting utility. One day, Omega show up and offers to flip a fair coin. If it comes up heads, they will give Alice - who knows nothing of this - three cakes. If it comes up tails, they will take one cake from her stockpile. He reasons as Alice did an accepts.

Guess what? The next day, Omega returns, offering to simulate Alice and give her you-know-what (hint: it's cakes.) Bob reasons just as Alice did in the second paragraph there and accepts the bargain.

Humans value each other's utility. Most notably, we value our lives, and we value each other not being tortured. If we simulate someone a billion times, and switch off one simulation, this is equivalent to risking their life at odds of 1:1,000,000,000. If we simulate someone and torture one of the simulations, this is equivalent to risking a one-in-a-billion chance of them being tortured. Such risks are often acceptable, if enough utility is gained by success. We often risk our own lives at worse odds.

If we simulate an entire society a trillion times, or 3^^^^^^3 times, or some similarly vast number, and then simulate something horrific - an individual's private harem or torture chamber or hunting ground - then the people in this simulation *are not real*. Their needs and desires are worth, not nothing, but far less then the merest whims of those who are Really Real. They are, in effect, zombies - not quite p-zombies, since they are conscious, but e-zombies - reasoning, intelligent beings that can talk and scream and beg for mercy but *do not matter*.

My mind rebels at the notion that such a thing might exist, even in theory, and yet ... if it were a similarly tiny *chance*, for similar reward, I would shut up and multiply and take it. This could be simply scope insensitivity, or some instinctual dislike of tribe members declaring themselves superior.

Well, there it is! The weirdest of Weirdtopias, I should think. Have I missed some obvious flaw? Have I made some sort of technical error? This is a draft, so criticisms will likely be encorporated into the final product (if indeed someone doesn't disprove it entirely.)

Here's the biggest thing that I've been working on for the last several months:

Responses to Catastrophic AGI Risk: A Survey
Kaj Sotala, Roman Yampolskiy, and Luke Muehlhauser

Abstract: Many researchers have argued that humanity will create artificial general intelligence (AGI) within the next 20-100 years. It has been suggested that this may become a catastrophic risk, threatening to do major damage on a global scale. After briefly summarizing the arguments for why AGI may become a catastrophic risk, we survey various proposed responses to AGI risk. We consider societal proposals, proposals for constraining the AGIs’ behavior from the outside, and for creating AGIs in such a way that they are inherently safe.

This doesn't aim to be a very strongly argumentative paper, though it does comment on the various proposals from an SI-ish point of view. Rather, it attempts to provide a survey of all the major AGI-risk related proposals that have been made so far, and to provide some thoughts on their respective strengths and weaknesses. Before writing this paper, we hadn't encountered anyone who'd have been familiar with all of these proposals - not to mention that even we ourselves weren't familiar with all of them! Hopefully, this should become a useful starting point for anyone who's at all interested in AGI risk or Friendly AI.

The draft will be public and open for comments for one week (until Nov 23rd), after which we'll incorporate the final edits and send it off for review. We're currently aiming to have it published in the sequel volume to Singularity Hypotheses.

EDIT: I've now hidden the draft from public view (so as to avoid annoying future publishers who may not like early drafts floating around before the work has been accepted for publication) while I'm incorporating all the feedback that we got. Thanks to everyone who commented!

In late spring this year, Luke tasked me with writing a summary and analysis of the Hanson-Yudkowsky FOOM debate, with the intention of having it eventually published in somewhere. Due to other priorities, this project was put on hold for the time being. Because it doesn't look like it will be finished in the near future, and because Curiouskid asked to see it, we thought that we might as well share the thing.

I have reorganized the debate, presenting it by topic rather than in chronological order: I start by providing some brief conceptual background that's useful for understanding Eliezer's optimization power argument, after which I present his argument. Robin's various objections follow, after which there is a summary of Robin's view of how the Singularity will be like, together with Eliezer's objections to that view. Hopefully, this should make the debate easier to follow. This summary also incorporates material from the 90-minute live debate on the topic that they had in 2011. The full table of contents:

1. Introduction
2. Overview
3. The optimization power argument
   1. Conceptual background
   2. The argument: Yudkowsky
   3. Recursive self-improvement
   4. Hard takeoff
   5. Questioning optimization power: the question of abstractions
   6. Questioning optimization power: the historical record
   7. Questioning optimization power: the UberTool question
4. Hanson's Singularity scenario
   1. Architecture vs. content, sharing of information
   2. Modularity of knowledge
   3. Local or global singularity?
5. Wrap-up
6. Conclusions
7. References

Here's the link to the current draft, any feedback is welcomed. Feel free to comment if you know of useful references, if you think I've misinterpreted something that was said, or if you think there's any other problem. I'd also be curious to hear to what extent people think that this outline is easier to follow than the original debate, or whether it's just as confusing.

Planned top-level post -- any feedback very much welcome.

Obviously a followup to: On the Care and Feeding of Young Rationalists

My very first top-level post on LW was a solicitation for advice/feedback/discussion on the topic of rationalist parenting. I'd like to revisit the topic now.

Goals

First of all, let's talk about goals. I can think of four.

1. Produce thriving, intelligent, rational, happy, good-hearted children who become thriving, intelligent, rational, happy, good-hearted adults.
2. Have your children enjoy their childhoods
3. Enjoy raising your children.
4. Closely tied to 2 and 3 -- actually have a good relationship with your children. Like them and have them like you.

How to Run a Successful Less Wrong Meetup is a guide that I've been working on, based on lukeprog's instructions, for the last week and a half. As it says in the beginning:

This document is written for anyone who wants to organize a Less Wrong meetup. We expect that this document will help you regardless of whether you want to start a new group or improve an existing one. We have tried to write each section so that it applies in either case.

Here's the table of contents:

• Why organize a meetup?
• How to build your team of heroes
  • The organizer
  • The welcomer
  • The learning coach
  • The content provider
  • The visionary
  • The networker
• How to announce and organize your meetups
  • Choosing a venue
  • Making the announcement
  • The first meetup
• Long-term meetup group maintenance
  • Retain members by being a social group
  • Conflicts within the group
    • Learn to recognize status conflicts
  • Group norms and epistemic hygiene
• Meetup content
  • Discussions and Presentations
    • Presentations
    • Topical Discussions
    • Meta Discussion
  • Games and Exercises
    • Aumann’s Thunderdome
    • Biased Co-operation
    • Behavioral Analysis
    • Bluffing Games
    • Bust-a-Distortion
    • Calibration Game
    • Cause and Belief
    • Five-Minute Debiasing
    • Hypothetical Apostasies
    • Paranoid Debating
    • Precommit to Updates
    • Rationalization Game
    • Rejection Therapy
    • Repetition Game
    • Status Exercises
    • Zendo
  • General Bacchanalia
  • Example activities at real meetup groups
• Projects

This is a draft version, so feedback would be most welcome, particularly on things like:

• Is this useful?
• Is there something that should be covered isn't covered at all yet?
• Do you have new games & exercises to suggest?
• Do you have any other content to suggest to any other section?
• Do you disagree with some of the advice given?
• Do you disagree on way something has been worded?
• Etc.

The link above will take you to a Google Docs copy of the document, with the ability to add comments to the draft. Feel free to comment on the guide either as traditional LW comments or by attaching comments to the document itself: both are fine.

EDIT: Here's the most recent version, though without the commenting ability.

EDIT2: The most recent version as of April 11th, with commenting enabled.

EDIT3: First non-draft version; see also this thread.

In J. Michael Straczynski's science fiction TV show Babylon 5, there's a character named Lennier. He's pretty Spock-like: he's a long-lived alien who avoids displaying emotion and feels superior to humans in intellect and wisdom. He's sworn to always speak the truth. In one episode, he and another character, the corrupt and rakish Ambassador Mollari, are chatting. Mollari is bored. But then Lennier mentions that he's spent decades studying probability. Mollari perks up, and offers to introduce him to this game the humans call poker.

If I roll 15 fair 6-sided dice, take the ones that rolled 4 or higher, roll them again, and sum up all the die rolls... what is the probability that I drop at least one die on the floor?

There are two different ways of using probability. When we think of probability, we normally think of neat statistics problems where you start with numbers, do some math, and end with a number. After all, if we don't have any numbers to start with, we can't use a proven formula from a textbook; and if we don't use a proven formula from a textbook, our answer can't be right, can it? But there's another way of using probability that's more general: a probability is just an estimate, produced by the best means available, even if that's a guess produced by mere intuition. To distinguish these two types, let's call the former kind *formal probabilities*, and the latter kind *informal probabilities*.

An informal probability summarizes your state of knowledge, no matter how much or how little knowledge that is. You can make an informal probability in a second, based on your present level of confidence, or spend time making it more precise by looking for details, anchors, reference classes. It is perfectly valid to assign probabilities to things you don't have numbers for, to things you're completely ignorant about, to things that are too complex for you to model, and to things that are poorly defined or underspecified. Giving a probability estimate does not require *any* minimum amount of thought, evidence, or calculation. Giving an informal probability  is not a claim that any relevant mathematical calculation has been done, nor that any calculation is even possible.

I present here the case for assigning informal probabilities, as often as is practical. If any statement crosses your mind that seems especially important, you should put a number on it. Routinely putting probabilities on things has significant benefits, even if they aren't very accurate, even if you don't use them in calculations, and even if you don't share them. The process of assigning probabilities to things tends to prompt useful observations and clarify thinking; it eases the transition into formal calculation when you discover you need it, and provides a sanity check on formal probabilities; having used probabilities makes it easier to diagnose mistakes later; and using probabilities lets you quantify, not just confidence, but also the strength and usefulness of pieces of evidence, and the expected value of avenues of investigation. Finally, practice at generating probabilities makes you better at it.