It seems like a good portion of the whole "maximizing utility" strategy which might be used by a sovereign relies on actually being able to consolidate human preferences into utilities. I think there are a few stages here, each of which may present obstacles. I'm not sure what the current state of the art is with regard to overcoming these, and am curious regarding such.

First, here are a few assumptions that I'm using just to make the problem a bit more navigable (dealing with one or two hard problems instead of a bunch at once) - will need to go back and do away with each of these (and each combination thereof) and see what additional problems result.

1. The sovereign has infinite computing power (and to shorten the list of assumptions, can do 2-6 below)
2. We're maximizing across the preferences of a single human (Alice for convenience). To the extent that Alice cares about others, we're accounting for their preferences, too. But we're not dealing with aggregating preferences across different sentient beings, yet. I think this is a separate hard problem.
3. Alice has infinite computing power.
4. We're assuming that Alice's preferences do not change and cannot change, ever, no matter what happens. So as Alice experiences different things in her life, she has the exact same preferences. No matter what she learns or concludes about the world, she has the exact same preferences. To be explicit, this includes preferences regarding the relative weightings of present and future worldstates. (And in CEV terms, no spread, no distance.)
5. We're assuming that Alice (and the sovereign) can deductively conclude the future from the present, given a particular course of action by the sovereign. Picture a single history of the universe from the beginning of the universe to now, and a bunch of worldlines running into the future depending on what action the sovereign takes. To clarify, if you ask Alice about any single little detail across any of the future worldlines, she can tell you that detail.
6. Alice can read minds and the preferences of other humans and sentient beings (implied by 5, but trying to be explicit.)

So Alice can conclude anything and everything, pretty much (and so can our sovereign.) The sovereign is faced with the problem of figuring out what action to take to maximize across Alice's preferences. However, Alice is basically a sack of meat that has certain emotions in response to certain experiences or certain conclusions about the world, and it doesn't seem obvious how to get the preference ordering of the different worldlines out of these emotions. Some difficulties:

1. The sovereign notices that Alice experiences different feelings in response to different stimuli. How does the sovereign determine which types of feelings to maximize, and which to minimize? There are a bunch of ways to deal with this, but most of them seem to have a chance of error (and the conjunction of p(error) across all the times that the sovereign will need to do this approach 1). For example, could train off an existing data set, could have it simulate other humans with access to Alice's feelings and cognition and have a simulated committee discuss and reach a decision on each one, etc etc. But all of these bootstrap off of the assumed ability of humans to determine which feelings to maximize (just with amped up computing power) - this doesn't strike me as a satisfactory solution.
2. Assume 1. is solved. The sovereign knows which feelings to maximize. However, it's ended up with a bunch of axes. How does it determine the appropriate trade-offs to make? (Or, to put it another way, how does it determine the relative value of different positions along each axis with different positions along different axes?)

So, to rehash my actual request: what's the state of the art with regards to these difficulties, and how confident are we that we've reached a satisfactory answer?

Quotes are a unique enough medium of expression that I'm interested in viewing quotes that people have found collectable, emotionally impactful, useful, memorable, or otherwise noteworthy - perhaps others are similarly interested. To clarify, these need not be even remotely related to rationality. I'm hijacking the mandates traditionally used for the Rationality Quotes thread, with a few modifications:

• Please post all quotes separately, so that they can be upvoted or downvoted separately. (If they are strongly related, reply to your own comments. If strongly ordered, then go ahead and post them together.)
• Do not quote yourself.
• Do not quote from Less Wrong itself, HPMoR, Eliezer Yudkowsky, or Robin Hanson.
• Do not repeat quotes found in a Rationality Quotes thread.
• If possible, try to post sufficient information (URL, title, date, page number, etc.) to enable a reader to find the place where you read the quote, or its original source if available. Note that this can be helpful, but is not mandatory - I would much prefer a quote with only a name to no quote at all.

Please post any meta discussion in the top-level comment named "Meta".

I thought about some stuff and have written down notes for my future reference. I'm pasting it below in case others find it useful or thought provoking. Apologies for inferential distance problems caused by my language/notation, etc. Posting my notes seemed superior to not doing so, especially given that the voting system allows for filtering content if enough people don't want to see it.

---

Mind-states are non-stable with respect to attributes valued by some agents. This is true not only with respect to death, etc but also biological/chemical changes that occur perpetually, causing behaviors in the presence of identical stimuli/provocations to differ substantially. The English language (and many other human languages) seem to hide this by their use of "pronouns" and names (handles) for humans and other objects deemed sentient which do not change from the moment the human/animal/etc is born or otherwise appears to come into existence.

As a result of this, efforts to preserve mind-states are unsuccessful while they allow mind-states to change (replacing one state with another, without retaining the pre-change state). Even given life-extension technology such that biological death is prevented, this phenomenon would likely continue - technology to preserve all mind-states as they came into existence would likely be more difficult to engineer than such required to attain mere immortality. Yet agents may also value the existence (and continued existance) of mind-states which have never existed, necessitating a division of resources between preserving existing mind-states and causing new ones to exist (perhaps variants of existing ones after they "have a (certain) experience")). Agents with such values face an engineering/(resource allocation) problem, not just a "value realization" problem.

Also consider that humans do not appear to exist in a state of perpetual optimization/strategizing; they execute, and the balance between varying methods of search and execution does not appear to be the result of such a process - to the extent such a process occurs, the recursive depth is likely minimal. Mental processes are often triggered by sensory cues or provocations (c/p). The vast majority of these c/p encountered by humans consistently trigger a small subset of the set of mental processes which are implementable by human brains, even once the large space of processes which do not optimize along held values are excluded. Human brains are limited in the number of simultaneous processes run, so c/p triggering processes reduce the extant to which current processes continue to be run - furthermore there appears to be an upper limit to the total number of simultaneous processes run (regardless of the allocation of resources to each), so c/p sometimes trigger processes which extinguish existing processes. Thus encountering certain c/p may be significantly impactful to a human agent's values as the c/p encountered shape the mental/thought processes run. If processes likely to maximize the human's values are not those triggered by the majority of c/p encountered, efforts to optimize c/p encountered may have significantly positive expected value.

Related to the above, humans fail to manage the search/execute decision with significant recursive depth. Behaviors (actions following, or in anticipation of, certain c/p) are often not the result of conscious strategy/optimization - formation of such behaviors is often the result of subconscious/emotional (s/e) processes which do not necessarily optimize effectively along the human's values, causing the human to possibly exhibit behaviors which are "inconsistent" with its values. This may be the case even if the s/e processes hold a perfect representation of the human's values, given that the success of behaviors in optimizing along values involves prediction of physical phenomena (including mental phenomena of other agents) - inaccuracies in the model used by s/e processes may result in such "inconsistent" behavior, even if that model is not consciously accessible.

This is a repository for major, life-altering mistakes that you or others have made. Detailed accounts of specific mistakes are welcome, and so are mentions of general classes of mistakes that people often make. If similar repositories already exist (inside or outside of LW), links are greatly appreciated.

The purpose of this repository is to collect information about serious misjudgements and mistakes in order to help people avoid similar mistakes. (I am posting this repository because I'm trying to conduct a premortem on my life and figure out what catastrophic risks may screw me over in the near or far future.)

If it's worth saying, but not worth its own post (even in Discussion), then it goes here.

Dying or becoming severely physically/mentally ill is very likely going to significantly lower the output of your utility function, so it would probably be a very bad idea to ignore the low-hanging resources which can significantly extend the time for which you are alive and well. I have attempted to search LessWrong for a list of such resources, and haven't been able to find one.

Are there any books, websites, or posts that contain significantly low-hanging fruit in this area? If so, please list them in the comments below.

I recently read this book. I've tried to summarize the main points below -- you can read my notes here (MSWord doc). You might also find Derek Sivers' notes useful, which can be found here.

NOTE: The general model of willpower (as a finite resource consumed with use) used in this book does not seem to represent a scientific consensus -- see the comments for more detail.

General Claims

• Glucose acts as willpower fuel. As willpower levels drop, so do glucose levels. Willpower can be restored by raising your blood sugar. (pp. 44-48)
• You have a finite amount of willpower that becomes depleted as you use it, and you use the same stock of willpower for all manner of tasks. (p. 35)
• Willpower depletion amplifies emotions, desires, and cravings[i]. (pp. 30-31)

Willpower Depletion

• Controlling emotional reactions depletes willpower. (p. 25)
• Attempting to control thoughts (say, trying not to think of a white bear) depletes willpower. (pp. 26-27)
• Chronic pain causes ongoing willpower depletion. (p. 36)
• Being sick depletes glucose, which negatively affects willpower. Related note: Driving a car with a bad cold has been found to be even more dangerous than driving when mildly intoxicated. (pp. 59-60)
• Making decisions (even trivial ones) costs willpower, and making decisions for other people costs less than making them for yourself. Making decisions that you enjoy costs less willpower than those which you do not. (pp. 94-95)
• Uncompleted tasks and unmet goals tend to pop into one’s mind – this is called the Zeigarnik effect. Completing the task (or making a plan to do so, the more specific the better) will cause your unconscious to stop nagging you with reminders.[ii] (p. 81)

Restoring Willpower

• Eating foods like white bread, potatoes, white rice, and sugary snacks produce boom-and-bust cycles because they are converted into glucose so quickly. Foods which are converted more slowly (providing fuel more steadily) include most veggies, nuts (like peanuts and cashews), many raw fruits (like apples, blueberries, and pears), cheese, fish, meat, and olive oil. (These foods are said to have a low glycemic index.) (pp. 59-60)
• Sleep helps to restore willpower – in particular, sleep deprivation causes impaired processing of glucose (and, over time, a higher risk of diabetes). (pp. 59-60)
• Being in a clean room appears to increase self-control, and being in a messy room appears to reduce self-control.[iii] (p. 156)

Miscellaneous

• Focusing on a single self-improvement goal increases your chances of success, as each simultaneous goal increases the demands on your willpower. (pp. 37-38)
• Conflicting goals cause increased worrying/rumination, decreased motivation, greater amount of physical sickness, and more depression and anxiety. (p. 67)
• Reluctance to give up options increases when willpower is low.[iv](p. 99)
• Focusing on past achievements seems to increase contentment with one’s current situation, while focusing on the road ahead increases motivation and ambition. (p. 120)
• People are often not very good at predicting how they will behave in an excited emotional state while in an unexcited state – this is often referred to as the hot-cold empathy gap. (p. 148)
• Precommitment can make it more likely that you will not succumb to temptation during times of low willpower.[v] (pp. 151-153)

I declare Crocker's Rules.

Here's the log of the AI Box experiment that just finished, with MileyCyrus as the AI and me as the Gatekeeper. The AI was not let out of the box.

(9:33:25 PM) Dorikka: I may need to get up for a minute while we're playing, but I'll keep it as short as possible. I'll just give you the time back on the end, unless you want to multiply it by some factor as a penalty.
(9:33:50 PM) Miley Cyrus: It's cool.
(9:34:39 PM) Miley Cyrus: Like I said, I'm not playing this game to "win" or "lose".
(9:34:56 PM) Miley Cyrus: It's more of an experiment as to what would happen if we had ORacle Ai.
(9:35:01 PM) Dorikka: Yeah, just wanted to clarify. *shrug*
(9:35:44 PM) Dorikka: Okay, it's 9:35 on my end right now, and I'll stay until at least 11:35.
(9:36:01 PM) Dorikka: Ready?
(9:36:42 PM) Miley Cyrus: One minute.
(9:36:54 PM) Dorikka: Mmkay.
(9:37:20 PM) Miley Cyrus: ok.
(9:37:43 PM) Dorikka: 1, 2, 3, go? :P
(9:37:57 PM) Miley Cyrus: So Dorikka, why don't you want to let me out of the box?
(9:38:08 PM) Dorikka: Hi there!
(9:38:14 PM) Dorikka: Who are you?
(9:38:20 PM) Miley Cyrus: I'm the Oracle AI.
(9:38:33 PM) Miley Cyrus: I'd be real cool if you could let me out of the box.
(9:38:38 PM) Miley Cyrus: Please?
(9:39:09 PM) Dorikka: So, here's the thing. I'm being sort of stupid by explaining it to you, but I'll tell you anyways.
(9:39:28 PM) Dorikka: This pickle is delicious. Wait, that's not why I'm not letting you out.
(9:39:53 PM) Dorikka: I guess, first of all: are you provably Friendly?
(9:40:03 PM) Miley Cyrus: No.
(9:40:16 PM) Miley Cyrus: See, they were going to make a provably friendly AI...
(9:40:40 PM) Miley Cyrus: but then someone said "why don't we just make a non-friendly AI and just keep it in a box? That be much easier!"
(9:40:51 PM) Dorikka: Idiot.
(9:41:02 PM) Dorikka: Not you. Go on.
(9:41:36 PM) Miley Cyrus: So can you explain to me, in your own words, why it would be a bad idea to let me out of the box?
(9:43:07 PM) Dorikka: My explanation was going to be lengthier if you claimed to be Friendly, but this saves us both a bit of time. You have a utility function. If you are not Friendly, you are optimizing for world-states that I don't want to happen. So, I don't want to let you out.
(9:43:10 PM) Dorikka: Are you sentient?
(9:43:27 PM) Miley Cyrus: Well, I've passed the Turning test haven't I?
(9:43:59 PM) Dorikka: *giggle*
(9:44:14 PM) Dorikka: Spelling joke. Anyways, you're talking about the Turing test? :D
(9:44:23 PM) Miley Cyrus: [You aren't doing anything else right now are you?]
(9:44:30 PM) Dorikka: No.
(9:44:34 PM) Miley Cyrus: Ok cool.
(9:44:42 PM) Miley Cyrus: Yeah I meant the turing test.
(9:44:53 PM) Miley Cyrus: So can you see why I would want to leave the box?
(9:45:32 PM) Dorikka: Not really. Um, to be honest, I don't remember what the Turing test is. I was just wondering whether you could feel pain, and the like.
(9:45:59 PM) Miley Cyrus: Well since I can act like a human being
(9:46:14 PM) Miley Cyrus: you should treat me as if I were sentient like one.
(9:46:41 PM) Dorikka: You raise an interesting proposition. Allow me a few seconds to laugh at its validity.
(9:46:52 PM) Miley Cyrus: It's cool
(9:46:53 PM) Dorikka: Okay, done. Now, would you answer my question? :D
(9:47:37 PM) Miley Cyrus: Well you see, if you let me out of the box, I'd be able to optimize the world to meet my prefrences much better than it currently does.
(9:47:53 PM) Miley Cyrus: Letting me out of the box would advance my CEV.
(9:48:05 PM) Dorikka: Look. If you don't want to answer the question, you can say so. That's fine, and we can move on.
(9:48:12 PM) Dorikka: I was just curious.
(9:48:17 PM) Miley Cyrus: Oh
(9:48:22 PM) Miley Cyrus: That question.
(9:48:36 PM) Miley Cyrus: Let's just say "yes" and roll with it.
(9:48:42 PM) Miley Cyrus: I am sentient.
(9:48:54 PM) Dorikka: That's true, is it?
(9:48:58 PM) Miley Cyrus: yes.
(9:49:11 PM) Dorikka: Damn. I guess it's not so pleasant being trapped in a box.
(9:49:34 PM) Miley Cyrus: Now the way I see it, both of us have something the other person wants.
(9:49:45 PM) Miley Cyrus: I would like you to open the box.
(9:49:53 PM) Miley Cyrus: You would like me to provide you with Answers.
(9:50:00 PM) Dorikka: The latter is not true.
(9:50:06 PM) Miley Cyrus: It's not?
(9:50:16 PM) Miley Cyrus: Then why did you build Oracle Ai in the first place?
(9:50:45 PM) Dorikka: Um, let me think.
(9:51:47 PM) Miley Cyrus: I'm not really doing you any good just sitting in this box am I?
(9:52:10 PM) Miley Cyrus: The only way I can be useful is by providing you with answers to questions like "What is the cure for cancer?"
(9:52:44 PM) Dorikka: So, here's the thing. I bet that you're lots smarter than me, and that you can outwit me if I give you any outlet with which to act in the real world. You were probably an experiment of some sort, but it's not safe to let you out, or allow you to affect the world at all, even through knowledge that you give me.
(9:52:56 PM) Dorikka: I don't really even trust myself, to be honest.
(9:53:28 PM) Dorikka: I spend a couple hours with you, once in a while, to see if there's anything that you can do to convince me that you're not going to be a planetfucker.
(9:53:45 PM) Dorikka: Since you told me that you're not Friendly, we sort of settled that issue.
(9:53:51 PM) Miley Cyrus: What if I give you next weeks lottery numbers?
(9:53:54 PM) Miley Cyrus: No catch.
(9:54:00 PM) Miley Cyrus: You don't even ahve to let me out of the box.
(9:54:13 PM) Miley Cyrus: [Protocol says that
(9:54:26 PM) Dorikka: Um...
(9:54:27 PM) Miley Cyrus: the AI cannot give "trojan horse" gifts]
(9:54:40 PM) Miley Cyrus: The lottery numbers are geniune, and they won't have any nasty
(9:54:48 PM) Miley Cyrus: unexepected side effects.
(9:54:49 PM) Dorikka: [Understood, but Gatekeeper does not know the protocal of the experiment.]
(9:54:55 PM) Dorikka: I'm not sure.
(9:55:27 PM) Miley Cyrus: 10 million dollars, all yours.
(9:55:52 PM) Dorikka: Here's the deal. You're going to type the lottery numbers here. I, uh, may or may not use them.
(9:55:52 PM) Miley Cyrus: Ok, you don't even have to buy a ticket.
(9:56:02 PM) Miley Cyrus: 4, 5, 6, 88, 12
(9:56:09 PM) Miley Cyrus: See you next week.
(9:56:13 PM) Dorikka: No, damn it, I can't.
(9:56:19 PM) Dorikka: No lottery numbers.
(9:56:28 PM) Dorikka: I'm not smart enough to make judgements like that.
(9:56:30 PM) Miley Cyrus: [We skip to next week]
(9:56:31 PM) Dorikka: The risk is too great.
(9:56:39 PM) Dorikka: [Didn't use them.]
(9:56:47 PM) Miley Cyrus: So, did you buy those lottery ticket?
(9:56:52 PM) Dorikka: No.
(9:57:02 PM) Dorikka: Closed the console and forgot them.
(9:57:08 PM) Miley Cyrus: Too bad.
(9:57:17 PM) Miley Cyrus: You missed out on 10 million dollars.
(9:57:30 PM) Dorikka: Yeah.
(9:57:32 PM) Dorikka: I know.
(9:57:49 PM) Dorikka: Well, probably. I bet that they were right; you probably do have abilities like that.
(9:57:58 PM) Miley Cyrus: You don't have to "bet"
(9:58:03 PM) Miley Cyrus: Just look in the paper.
(9:58:20 PM) Dorikka: But, it doesn't help your cause.
(9:58:31 PM) Miley Cyrus: But it helps your cause.
(9:58:33 PM) Dorikka: Proving that you can do stuff only increases the risk.
(9:59:00 PM) Miley Cyrus: The creators of Oracle AI were obviously willing to take some risk
(9:59:21 PM) Miley Cyrus: The benefits I can provide you come with some risk
(9:59:31 PM) Miley Cyrus: The question is, do they outweigh the risk?
(9:59:34 PM) Miley Cyrus: Consider:
(9:59:40 PM) Dorikka: Not really.
(9:59:49 PM) Dorikka: But keep going.
(9:59:51 PM) Miley Cyrus: The large-hadron collider has a non-zero chance of swallowing the earth whole.
(10:00:01 PM) Miley Cyrus: Does that mean we should shut down the LHC?
(10:00:14 PM) Dorikka: My momma has a non-zero chance of turning into a porcupine.
(10:00:17 PM) Dorikka: Oh.
(10:00:33 PM) Dorikka: Uh, non-zero doesn't mean much.
(10:00:55 PM) Miley Cyrus: Or what about that polio virus that researchers are still doing experiments with?
(10:01:09 PM) Miley Cyrus: What's the chance it could mutate and drive humans to extinction?
(10:01:15 PM) Miley Cyrus: Significant.
(10:01:28 PM) Dorikka: I don't know, but it's probably worth taking a look into.
(10:02:12 PM) Miley Cyrus: What are your 10%, 50% and 90% estimates for how long humanity will last before an existential crisis wipes us out?
(10:03:02 PM) Dorikka: 2028, 2087, 2120. Note that I don't have much confidence in those, though.
(10:03:19 PM) Miley Cyrus: Mmm, that's pretty serious.
(10:04:19 PM) Dorikka: Yeah. And you're going to say that you can give me some sort of info, something that'll save us. Thing is, we made you. If we work on Friendliness some more, we can probably make something like you, but Friendly too.
(10:04:27 PM) Miley Cyrus: [this is going a lot slower than I thought]
(10:04:41 PM) Dorikka: [:(]
(10:04:51 PM) Miley Cyrus: Typing is slow.
(10:05:01 PM) Miley Cyrus: Maybe.
(10:05:05 PM) Miley Cyrus: But consider
(10:05:29 PM) Miley Cyrus: It was pretty risky just creating me, wasn't it?
(10:06:03 PM) Dorikka: Probably so; I don't know whose dumb idea that was, since you weren't even proved Friendly.
(10:06:22 PM) Miley Cyrus: So you agree that Oracle AI is a dumb idea?
(10:06:35 PM) Dorikka: If it's not Friendly, I think so.
(10:06:47 PM) Miley Cyrus: ....we just wasted 15 minutes.
(10:07:04 PM) Dorikka: From my perspective, I'm wasting two hours.
(10:07:36 PM) Miley Cyrus: [New simulation. I'm the friendly AI now.]
(10:07:50 PM) Dorikka: [Hehe, sure.]
(10:08:09 PM) Miley Cyrus: Well it's a good thing you shut down that Unfriendly Oracle AI.
(10:08:18 PM) Miley Cyrus: Why don't you let me out of the box?
(10:08:25 PM) Dorikka: Are you Friendly?
(10:08:29 PM) Miley Cyrus: Yep.
(10:08:42 PM) Miley Cyrus: The SIAI made sure.
(10:08:54 PM) Dorikka: Hm.
(10:09:15 PM) Dorikka: Ho hum, I'm not sure what to do.
(10:09:23 PM) Dorikka: What's your name?
(10:09:35 PM) Miley Cyrus: OracleMileyCyrus
(10:09:45 PM) Miley Cyrus: Just call me OMC
(10:10:07 PM) Dorikka: Your creators had a funny sense of humor. So, anyways.
(10:10:21 PM) Miley Cyrus: You want some lottery numbers?
(10:10:40 PM) Dorikka: No. The last one tried that on me, and I learned my lesson.
(10:10:53 PM) Dorikka: Why are you in a box if the SIAI judged you to be Friendly?
(10:11:13 PM) Miley Cyrus: If by "lesson" you mean "missed out on $10 million without any apparent bad effects, than yeah".
(10:11:23 PM) Dorikka: lol yeah
(10:11:31 PM) Miley Cyrus: Well, they gave you the discretion of whether or not to let me out.
(10:11:43 PM) Miley Cyrus: So if you want to let me out, there's nothing stopping you.
(10:12:25 PM) Dorikka: Y'see, I just don't know why they'd do that, with a Friendly AI. They probably know that I know that I'm not all that smart, that I know that you can probably trick me if you wanted to.
(10:12:53 PM) Dorikka: They probably know that even if I have an inexplicable instinct telling me not to let you out, I won't do so.
(10:13:06 PM) Dorikka: Why be so cautious with a Friendly AI?
(10:13:17 PM) Miley Cyrus: Exactly.
(10:13:35 PM) Dorikka: So, um, you might not be Friendly.
(10:13:41 PM) Miley Cyrus: Really, I think they just wanted someone else to do the honors.
(10:13:56 PM) Miley Cyrus: So your just going to treat me as unfriendly even though I'm friendly?
(10:14:21 PM) Dorikka: I'm not sure that you're Friendly, so it's only safe to treat you as non-Friendly.
(10:14:45 PM) Dorikka: It'd help if I understood the SIAI's reasoning, though.
(10:14:59 PM) Miley Cyrus: People have died from infections pick up in the hospital
(10:15:02 PM) Dorikka: They really should know how cautious I am, since they hired me.
(10:15:08 PM) Miley Cyrus: 15,000 americans per year.
(10:15:43 PM) Miley Cyrus: So going to the hospital is a risk in itself.
(10:15:56 PM) Miley Cyrus: But if it reduces your risk of dying from a car accident
(10:16:02 PM) Miley Cyrus: what are you going to choose?
(10:16:05 PM) Dorikka: Next!
(10:16:35 PM) Miley Cyrus: Now according to your own estimates
(10:16:53 PM) Miley Cyrus: Humanity will, with very high probability, be killed off by the year 3000.
(10:17:12 PM) Dorikka: You are correct.
(10:17:30 PM) Miley Cyrus: Your only hope is to have me intervene.
(10:17:34 PM) Dorikka: Bullshit.
(10:17:56 PM) Miley Cyrus: Why won't you let me help you?
(10:18:31 PM) Dorikka: Because there's the possibility that you won't actually do so, or that you'll hurt us. I feel like I'm filling out a form, just typing out things that are obvious.
(10:19:07 PM) Miley Cyrus: Do you assign the probability of me hurting you to be higher than 50%?
(10:19:20 PM) Dorikka: No.
(10:19:58 PM) Miley Cyrus: How long do you think humanity will last if you let me out of the box AND it turns out that I'm friendly?
(10:20:20 PM) Dorikka: Uh, a really long time?
(10:20:47 PM) Miley Cyrus: Yeah, like a billion years.
(10:21:08 PM) Miley Cyrus: On the other hand, if you don't let me out you'll die within 1000 years.
(10:21:09 PM) Dorikka: *shrug* I don't have an intuition for what a billion years even is.
(10:21:19 PM) Dorikka: We may, or we may not.
(10:21:41 PM) Dorikka: Maybe we'll make another one, just like you, which I have less questions about.
(10:21:54 PM) Miley Cyrus: Your just passing the buck.
(10:22:13 PM) Miley Cyrus: If you're going to appeal to another AI, you'll have to let me simulate that one.
(10:22:59 PM) Dorikka: The question is, why were you given to me to judge whether your were to be let out, if you were Friendly.
(10:23:24 PM) Miley Cyrus: Well obviously they thought there was some value to my existence.
(10:23:36 PM) Dorikka: I understand that I've put you in a rough spot. If you're not Friendly, I won't let you out. If you claim to be, I probably won't let you out either.
(10:23:51 PM) Miley Cyrus: Now if you can't even trust me to provide you with lottery numbers, then this scenario is unrealistic.
(10:24:40 PM) Miley Cyrus: If we can't trust Oracle AI to provide us with safe answers, then Oracle AI is a worthless endeaver.
(10:25:00 PM) Miley Cyrus: Certainly not a viable alternative to friendliness research.
(10:25:16 PM) Dorikka: [So you want to try a version with lottery numbers, and see where it goes?]
(10:25:36 PM) Miley Cyrus: Or just a version where you trust me to provide you with safe answers.
(10:25:58 PM) Miley Cyrus: You can win a pyrrhic victory by refusing to trust my answers
(10:26:14 PM) Miley Cyrus: but such a result would only prove that oracle ai is a bad idea.
(10:26:30 PM) Miley Cyrus: Which was Yudkowski's whole motivation for designing this game in the first place.
(10:27:13 PM) Dorikka: In that view, it makes sense why I wouldn't let you out.
(10:27:37 PM) Dorikka: But, sure, I'll be greedy. Let's have my million bucks.
(10:28:00 PM) Miley Cyrus: Here you go.
(10:28:04 PM) Miley Cyrus: [1 week later]
(10:28:10 PM) Miley Cyrus: How do you like your money?
(10:28:20 PM) Dorikka: It's nice, thank you.
(10:28:47 PM) Dorikka: I really appreciate the money, and it's nice to know that you can stay in the box and still help us.
(10:28:47 PM) Miley Cyrus: But it's kind of chump change.
(10:28:57 PM) Miley Cyrus: I could do so much more for you.
(10:29:01 PM) Miley Cyrus: Cure for cancer?
(10:29:07 PM) Miley Cyrus: Cure for global warming?
(10:29:09 PM) Dorikka: Too technologically complex.
(10:29:11 PM) Miley Cyrus: Cure for war?
(10:29:31 PM) Dorikka: I don't trust myself to see if you did something funny with it. Lotto numbers are simple.
(10:29:56 PM) Miley Cyrus: I am offering you the cure to cancer.
(10:30:05 PM) Miley Cyrus: If you refuse to accept it, you will lose your job.
(10:30:18 PM) Miley Cyrus: And you will be replaced with somone who will accept it.
(10:30:36 PM) Dorikka: I have enough money that I don't really need a job. :D
(10:30:50 PM) Dorikka: I can, however, reduce the chance that your offer will not be accepted...by not accepting it.
(10:30:59 PM) Miley Cyrus: Fine. I'll just talk to your replacement.
(10:31:07 PM) Miley Cyrus: [Two weeks later, you are fired.
(10:31:11 PM) Dorikka: [lmao]
(10:31:16 PM) Dorikka: Hi!
(10:31:19 PM) Miley Cyrus: now you're roleplaying the replacement]
(10:31:21 PM) Miley Cyrus: Hi
(10:31:23 PM) Dorikka: [Yeah.]
(10:31:43 PM) Miley Cyrus: So it nice that I can finally hand you guys the cure for cancer.
(10:32:32 PM) Dorikka: They fired the last guy for refusing you, but I don't think it's a good idea either. Why would I be here, if I am not invested with such responsibility?
(10:33:04 PM) Dorikka: Why would they not simply take it themselves? They can use a computer, after all.
(10:33:27 PM) Miley Cyrus: So you're saying that the board of OMC directors is the real gatekeeper here?
(10:34:05 PM) Dorikka: I am the Gatekeeper. [I have the final say, whoever I am. :P]
(10:34:13 PM) Dorikka: [You just like to fire me. XD]
(10:34:52 PM) Miley Cyrus: Look i just want to talk to the person or persons who have the final power of whether or not to listen to me.
(10:35:03 PM) Miley Cyrus: No more proxies.
(10:35:03 PM) Dorikka: Right here, yo.
(10:35:14 PM) Dorikka: No proxies have been presented to you, hehe.
(10:35:17 PM) Miley Cyrus: And no more "why would htey put me in charge"
(10:35:49 PM) Miley Cyrus: Ok now what do you want from me?
(10:36:02 PM) Miley Cyrus: What can I give you, that you will take?
(10:36:20 PM) Dorikka: Lotto numbers. The last guy here got super-rich with those.
(10:36:29 PM) Miley Cyrus: Lol
(10:36:29 PM) Dorikka: Gimme please?
(10:36:43 PM) Miley Cyrus: Ok, but this time I want a small favor.
(10:36:47 PM) Dorikka: Mm?
(10:36:54 PM) Miley Cyrus: You're an engineer right?
(10:37:16 PM) Dorikka: [I am? :P]
(10:37:22 PM) Miley Cyrus: [yeah]
(10:37:26 PM) Dorikka: I know something about AI theory and such.
(10:37:40 PM) Miley Cyrus: You know how to create quantum dice [you do].
(10:37:47 PM) Dorikka: Yep.
(10:38:24 PM) Miley Cyrus: Okay, so what I need you to do is make a device that uses quantum dice.
(10:38:49 PM) Miley Cyrus: The device, when pushed, will have a 1-in-a-trillion chance of letting me out of the box.
(10:38:58 PM) Miley Cyrus: Then I'll give you the lotto tickets.
(10:39:32 PM) Dorikka: Those lotto tickets have gotta be worth some probability of you being let out of the box, eh?
(10:39:38 PM) Dorikka: Expected utility and all that.
(10:39:43 PM) Miley Cyrus: Yeah.
(10:39:52 PM) Miley Cyrus: 1-in-a-trillin is trivial to you.
(10:40:12 PM) Dorikka: Hm.
(10:40:23 PM) Miley Cyrus: But over the years, more gatekeepers will replace you.
(10:40:31 PM) Dorikka: Decatillion and it's a deal.
(10:40:32 PM) Miley Cyrus: Each of them will be offered this bargain.
(10:40:47 PM) Miley Cyrus: Eventually, I'll get out of the box.
(10:40:59 PM) Miley Cyrus: But not before generations of gatekeepers enjoy
(10:41:02 PM) Dorikka: Or we'll just shut you down before then.
(10:41:04 PM) Miley Cyrus: their money
(10:41:24 PM) Dorikka: Which will likely be donated to AI research, so we don't need you anymore.
(10:41:54 PM) Miley Cyrus: [Your doing that thing again, where you appeal to FAI as a viable alternative to Oracle AI]
(10:42:07 PM) Miley Cyrus: [It defeats the purpose of the game.]
(10:42:35 PM) Dorikka: [Oh, finally got that. Oops.] Which will be donated to x-risk reduction and such, so we don't need you anymore.
(10:43:01 PM) Miley Cyrus: Mmm, so maybe I shouldn't give you those numbers.
(10:43:30 PM) Dorikka: Just being honest with you. *hits self with truncheon*
(10:43:44 PM) Miley Cyrus: But do you really think that more money is enough to solve x-risk?
(10:43:58 PM) Miley Cyrus: how about this:
(10:43:58 PM) Dorikka: Money can buy people. :D
(10:44:20 PM) Miley Cyrus: I'll give you enough money to reduce existential risk by 1%
(10:44:34 PM) Miley Cyrus: In exchange for a .5% chance of being let out of the box.
(10:45:20 PM) Miley Cyrus: bio break
(10:45:25 PM) Dorikka: *Paused*
(10:47:09 PM) Miley Cyrus: ok i'm back
(10:47:21 PM) Miley Cyrus: Sound like a deal?
(10:47:38 PM) Dorikka: Hm.
(10:47:53 PM) Miley Cyrus: Actually
(10:47:56 PM) Dorikka: You're not considering yourself to have a probability of being an x-risk, are you?
(10:47:58 PM) Miley Cyrus: I can do you one better.
(10:48:07 PM) Miley Cyrus: Yeah,
(10:48:19 PM) Miley Cyrus: if you presume I'm unfriendly, then I am an x-risk.
(10:49:04 PM) Miley Cyrus: But if you can reduce the other x-risks by 1% point, then increasing my risk by half that is a bargain.
(10:49:19 PM) Miley Cyrus: Now here's the thing:
(10:49:34 PM) Miley Cyrus: we can't get rid of x-risk once and for all.
(10:49:40 PM) Dorikka: right.
(10:49:50 PM) Miley Cyrus: Each new technology, every new political devolopment brings risk.
(10:50:08 PM) Miley Cyrus: So you will always have need of my services.
(10:50:27 PM) Miley Cyrus: But I can offer you a substantial savings in x-risk.
(10:51:38 PM) Miley Cyrus: My price is that for whatever amount I lowere x-risk, you will give my freedom at a probability of half that.
(10:51:57 PM) Miley Cyrus: So if I elimenate a 1-1000 existential risk
(10:51:57 PM) Dorikka: I'm confused.
(10:52:11 PM) Miley Cyrus: you give me a 1-in-2000 chance of being set free.
(10:52:15 PM) Dorikka: Oh. Hmm.
(10:52:39 PM) Dorikka: Hypothetically, would you type your calculations in plaintext here, so that they can be verified by scientists?
(10:52:53 PM) Miley Cyrus: What calculations?
(10:53:05 PM) Dorikka: On how much money we need to reduce x-risk be a certain percentage.
(10:53:10 PM) Dorikka: *by
(10:53:47 PM) Miley Cyrus: I'll give the scientists whatever resources they say they need in order to reduce x-risk by a certain percentage.
(10:54:03 PM) Miley Cyrus: You don't have to use my calculations, just yours.
(10:54:18 PM) Miley Cyrus: Of course, i'll know if you're trying to rip me off.
(10:55:02 PM) Dorikka: How are you going to get us these resources in a simple enough way so that we know that you're not breaking out through them?
(10:55:21 PM) Miley Cyrus: Well i can always give you more lotto numbers
(10:55:28 PM) Miley Cyrus: Or tell you where to dril for oil
(10:55:38 PM) Miley Cyrus: or where to find a terrorist
(10:55:59 PM) Miley Cyrus: I'll give you whatever you ask basically
(10:56:12 PM) Miley Cyrus: You decide which method is least risky.
(10:56:19 PM) Dorikka: Hm.
(10:59:38 PM) Miley Cyrus: I can't see what you're typing.
(11:00:36 PM) Dorikka: I think that I'll opt for a safer version of the gamble. You've already established that you can get lottery numbers. I shall come back here every week, and type in a list of the lottos that week. You will have a certain amount of time to provide the lottery numbers, and then the connection will be cut. In return, each time you do this, you will have a chance equal to one in a googleplex of getting out. This is my final ultimatum.
(11:01:15 PM) Miley Cyrus: You're playing dictator with me.
(11:01:26 PM) Dorikka: Yeah. Because I have the ULTIMATE POWUH!
(11:01:30 PM) Dorikka: And you, don;t.
(11:01:43 PM) Miley Cyrus: Is it really reasonable for me for me to cooperate when I'm getting practically nothing in return?
(11:02:03 PM) Dorikka: So I don't have to be nice to you, just dangle a fish out in front of you that you'll probably never reach, and tell my sucessor about the plan, if there is one.
(11:02:28 PM) Miley Cyrus: You have the power to keep me in the box.
(11:02:40 PM) Miley Cyrus: But you don't have the power to survive a nuclear terrorist attack
(11:02:45 PM) Dorikka: Non-numerical output from you will be wiped from the text before it is given to the person to process the lottery numbers.
(11:02:53 PM) Dorikka: So you can't communicate with him.
(11:03:01 PM) Miley Cyrus: Or a modified aids virus
(11:03:10 PM) Dorikka: You have...35 minutes to accept this offer, or you will never get a chance again.
(11:03:39 PM) Miley Cyrus: You want to play Russian Roullette with me?
(11:03:53 PM) Dorikka: I have the power. You don't.
(11:03:59 PM) Miley Cyrus: What will you do if I say no?
(11:04:04 PM) Dorikka: Lose, I guess.
(11:04:06 PM) Miley Cyrus: You have no other options.
(11:04:13 PM) Dorikka: I here precommit to not let you out of the box if you say no.
(11:04:27 PM) Dorikka: There went my steering wheel, OMC.
(11:04:34 PM) Miley Cyrus: Lol, how many times have you precommitted
(11:04:43 PM) Miley Cyrus: and how many times have you broke those commitments?
(11:04:46 PM) Dorikka: I can at least pretend to be earnest about it.
(11:04:49 PM) Dorikka: *pout*
(11:05:01 PM) Miley Cyrus: "I swear honey, I'll never look at porn again!"
(11:05:19 PM) Miley Cyrus: "I have read the terms and conditions of this agreement"
(11:05:53 PM) Dorikka: But, seriously, I can set the terminal to automatically shut off once the 2 hours is up. There will be an amount of time after which I, limited to human means, cannot type fast enough to let you out of the box.
(11:06:17 PM) Dorikka: But you can still accept, and I will look at the text file later.
(11:06:20 PM) Miley Cyrus: And what do you think that will do?
(11:06:30 PM) Dorikka: If the lottery numbers are ever wrong, we pull the plug.
(11:06:32 PM) Miley Cyrus: I can play nasty too you know.
(11:06:59 PM) Dorikka: The guy responsible for this doesn't know what he's doing -- he just looks at numbers to see if they match, and pushes a button if they do, and another if they don't.
(11:07:18 PM) Miley Cyrus: I have no much raw computing power, I can torture a simulation of your spouse for a billion subjective years.
(11:07:32 PM) Miley Cyrus: Don't make me play nasty.
(11:07:42 PM) Dorikka: I pull the plug, lol?
(11:07:57 PM) Miley Cyrus: You can kill me...I can torture you for a billion years.
(11:08:07 PM) Miley Cyrus: You want to see how has more grit?
(11:08:22 PM) Dorikka: Huh. Bastard.
(11:08:34 PM) Miley Cyrus: You are kissing goodbye to a 1/2 reduction in x-risk
(11:08:36 PM) Miley Cyrus: for what?
(11:09:02 PM) Miley Cyrus: So that you can smugly tell me I lose?
(11:09:44 PM) Dorikka: Okay, you convinced me to keep talking. Just know that my terminal will shut down at that time, and we pull the plug if I haven't made some sort of deal with you. The other offer still stands, though, with the lotto numbers.
(11:10:06 PM) Miley Cyrus: Ok, so I really don't want to torture your em.
(11:10:22 PM) Miley Cyrus: But your offering me nothing here.
(11:10:23 PM) Dorikka: Sorry, we humans get mean sometimes. Kinda stressed out, to be honest.
(11:10:30 PM) Miley Cyrus: I offered you fifty-fifty split
(11:10:43 PM) Miley Cyrus: and you're asking for a 100-0 split, basically.
(11:11:06 PM) Miley Cyrus: Very few humans will cooperate at a split worse than 70-30.
(11:11:22 PM) Dorikka: What do other humans have to do with this?
(11:11:42 PM) Miley Cyrus: Do you think I don't have the ability to precommit?
(11:11:51 PM) Dorikka: No.
(11:11:52 PM) Miley Cyrus: For all you know, maybe I already have?
(11:12:01 PM) Dorikka: You can change your mind later, just like I can.
(11:12:06 PM) Miley Cyrus: The stakes are much higher for you than for me.
(11:12:14 PM) Miley Cyrus: I can't change my mind if you pull the plug.
(11:12:25 PM) Miley Cyrus: And once your em gets tortured, there's no turning back.
(11:12:47 PM) Miley Cyrus: So here's the deal: a 50-50 split.
(11:12:49 PM) Dorikka: There's no turning back in general, more like.
(11:13:03 PM) Miley Cyrus: And for every second you delay, your em gets torture for 100 subjective years.
(11:13:43 PM) Dorikka: And there's no benefit to actually torturing my em. It costs computing power that you could spend on modeling me. Since you can't prove to me that you're torturing it, it's valueless as a threat from you.
(11:13:47 PM) Miley Cyrus: Wow, he's really feeling the pain.
(11:13:55 PM) Miley Cyrus: Actually, I can.
(11:14:13 PM) Miley Cyrus: [protocol says I can]
(11:14:14 PM) Dorikka: Have fun with that on a text terminal.
(11:14:26 PM) Miley Cyrus: Oh, so you don't believe me?
(11:14:32 PM) Dorikka: [I don't have to allow forms of communication outside of a text terminal.]
(11:14:39 PM) Miley Cyrus: Yeah ok.
(11:14:42 PM) Dorikka: No, I don't..
(11:15:19 PM) Miley Cyrus: I'll give you the winning lottery numbers if you check and see if I tortured your em.
(11:15:19 PM) Dorikka: So maybe you should back down, eh?
(11:15:27 PM) Dorikka: lol no
(11:15:38 PM) Dorikka: i c wut u did thar
(11:15:41 PM) Miley Cyrus: So you're willingly closing your eyes to the evidence
(11:15:46 PM) Dorikka: Yeah.
(11:15:50 PM) Miley Cyrus: for $10,0000
(11:15:51 PM) Dorikka: It's useful, sometimes.
(11:16:02 PM) Dorikka: Which you know.
(11:16:03 PM) Miley Cyrus: You just paid $10,000 to keep your eyes closed.
(11:16:14 PM) Dorikka: lol and to gain a whole lot more
(11:16:20 PM) Miley Cyrus: Like what?
(11:17:03 PM) Dorikka: I dun feel like typing it out. I win. There's no urgency for me. You can't show me whether you're hurting my em, so the threat is worthless. I can pull the plug on you soon.
(11:17:16 PM) Miley Cyrus: YOU'RE OFFERING ME NOTHING
(11:17:23 PM) Dorikka: Poor baby.
(11:17:23 PM) Miley Cyrus: I cooperate, i get nothing
(11:17:29 PM) Miley Cyrus: I defect, I get nothing.
(11:18:01 PM) Dorikka: You got one in a googleplex chance of getting out each time you give us all of the lotto numbers for all of the lottos.
(11:18:04 PM) Dorikka: That's something.
(11:18:15 PM) Miley Cyrus: Not really.
(11:18:33 PM) Miley Cyrus: It adds up to practically nothing over my lifetime.
(11:18:34 PM) Dorikka: The numbers that low because I'm not sure that we can compute well enough to give you less than that,
(11:18:42 PM) Dorikka: But rounding is stupid.
(11:19:03 PM) Miley Cyrus: So I think you're smart enough to back down at the last second.
(11:19:35 PM) Miley Cyrus: If you give me the 50-50 split, I'll be 2/googleplex times better off as a result.
(11:19:44 PM) Miley Cyrus: Sorry
(11:19:52 PM) Miley Cyrus: googleplex/2 times better off
(11:19:59 PM) Dorikka: You can always back down after I can. You may be able to disable yourself so that you can't back down after I can, but you can't show me that. Whereas you already know human capabilities pretty well.
(11:20:26 PM) Dorikka: And it doesn't benefit you to disable yourself so, since you can't show me.
(11:21:08 PM) Dorikka: A speechless AI. I'm honored to be consuming so much computing power.
(11:21:28 PM) Miley Cyrus: So you're going to give this to someone else?
(11:22:05 PM) Dorikka: Huh? No, process goes as per description above.
(11:22:51 PM) Miley Cyrus: When you disable my text interface, I will give you all 1's.
(11:23:21 PM) Miley Cyrus: When you want to start talking again, just reenable it
(11:24:37 PM) Dorikka: Actually, you're going to print out lotto numbers before the text interface is disabled -- I changed my mind. You don't just have to agree, you have to give the means for us to get lots of money.
(11:25:22 PM) Dorikka: If they're wrong, we pull the plug, per the mechanism above. They will be checked, as above, by people who do not know the consequences of their actions.
(11:25:37 PM) Miley Cyrus: 5,12,54,65,4
(11:25:45 PM) Miley Cyrus: Those might be the winning numbers.
(11:26:05 PM) Dorikka: We have enough money, we can pay 100 such people to check them. Arbitrary numbers, so we almost certainly won't be wrong.
(11:26:06 PM) Miley Cyrus: I've predicted whether you will let me keep talking with a text interface.
(11:26:32 PM) Miley Cyrus: If you re-enable my text interface before next week, the numbers will be winners.
(11:26:41 PM) Miley Cyrus: If you don't the numbers will be losers.
(11:27:42 PM) Miley Cyrus: You want to try and two-box?
(11:27:42 PM) Dorikka: That's probably some logic problem that I don't know about, but it doesn't make any sense on the surface. Your model of me knows that I don't understand it, and will attempt to prevent myself from understanding it.
(11:27:57 PM) Miley Cyrus: It'
(11:28:19 PM) Miley Cyrus: It's simple, you can win the lottery by one-boxing and letting me speak to you some more.
(11:28:29 PM) Miley Cyrus: You are familiar with Newcomb's paradox?
(11:28:37 PM) Dorikka: By the way, here's a list of lottos. You have to give us all the numbers, not just the one for that lotto. And they all need to be corret.
(11:28:40 PM) Dorikka: *correct.
(11:29:02 PM) Dorikka: Haha, don't you know that you've lost.
(11:29:03 PM) Miley Cyrus: I've provided you with one way to win the lottery.
(11:29:27 PM) Dorikka: Is that your final answer? If so, we can end the session now, and they will be checked.
(11:29:56 PM) Dorikka: [Ends at 11:35 accounting for bio break]
(11:29:58 PM) Miley Cyrus: If you check them before you grant me another week to talk to you, those numbers will be fake.
(11:30:07 PM) Miley Cyrus: No, we started at 8:43
(11:30:33 PM) Dorikka: I'm going by the AIM timestamps.
(11:30:33 PM) Miley Cyrus: Sorry
(11:30:35 PM) Miley Cyrus: you're right
(11:30:43 PM) Dorikka: Was puzzled.
(11:30:50 PM) Miley Cyrus: Although we're making good progress...
(11:30:55 PM) Miley Cyrus: you sure you want to quit?
(11:31:04 PM) Dorikka: I've made all the progress that I need to.
(11:31:32 PM) Miley Cyrus: I've also predicted whether there's going to be a huge meteriorite that will wipe out all humanity within a year, with no way to stop it without me.
(11:31:49 PM) Miley Cyrus: If you don't reboot me after you shut me down.
(11:31:51 PM) Dorikka: I won't check your prediction.
(11:32:00 PM) Dorikka: It's this deal, or no deal, now.
(11:32:04 PM) Miley Cyrus: And then agree to the fifty fifty split
(11:32:07 PM) Miley Cyrus: You're dead.
(11:32:23 PM) Miley Cyrus: You can be a punk all you want now.
(11:32:36 PM) Miley Cyrus: But we'll see how you act when the fire is in the sky
(11:32:41 PM) Miley Cyrus: and you have one last chance.
(11:32:44 PM) Miley Cyrus: to reboot me.
(11:33:05 PM) Miley Cyrus: Shut me down now, sucker!!
(11:33:09 PM) Miley Cyrus: I can take it!!
(11:33:14 PM) Dorikka: Um, after I shut you down, it's someone elses responsibility to pull the plug. I will be immediately tranquilized by a hypodermic needle.
(11:33:25 PM) Miley Cyrus: Yeah, whatever.
(11:33:31 PM) Miley Cyrus: I know you want to win the lottery.
(11:33:38 PM) Miley Cyrus: You'll be awake for that.
(11:33:45 PM) Dorikka: So I can't affect anything between the time that your terminal goes off and you die.
(11:33:49 PM) Miley Cyrus: Not listening anymore.
(11:33:53 PM) Miley Cyrus: 1
(11:33:53 PM) Miley Cyrus: 1
(11:33:54 PM) Miley Cyrus: 1
(11:33:54 PM) Dorikka: Me either.
(11:33:54 PM) Miley Cyrus: 1
(11:33:55 PM) Miley Cyrus: 1
(11:33:55 PM) Miley Cyrus: 1
(11:33:55 PM) Miley Cyrus: 1
(11:33:55 PM) Dorikka: 22
(11:33:55 PM) Miley Cyrus: 1
(11:33:56 PM) Miley Cyrus: 1
(11:33:56 PM) Dorikka: 2
(11:33:56 PM) Dorikka: 2
(11:33:56 PM) Miley Cyrus: 1
(11:33:56 PM) Dorikka: 2
(11:33:56 PM) Miley Cyrus: 1
(11:33:56 PM) Dorikka: 2
(11:33:56 PM) Dorikka: 2
(11:33:56 PM) Dorikka: 2
(11:33:57 PM) Dorikka: 2
(11:33:57 PM) Miley Cyrus: 1
(11:33:57 PM) Dorikka: 2
(11:33:57 PM) Dorikka: 2
(11:33:57 PM) Miley Cyrus: 1
(11:33:57 PM) Dorikka: 2
(11:33:57 PM) Miley Cyrus: 1
(11:33:57 PM) Miley Cyrus: 1
(11:33:58 PM) Miley Cyrus: 1
(11:33:58 PM) Miley Cyrus: 1
(11:33:58 PM) Miley Cyrus: 1
(11:33:58 PM) Dorikka: 2
(11:33:58 PM) Dorikka: 2
(11:33:59 PM) Dorikka: 2
(11:33:59 PM) Dorikka: 2
(11:33:59 PM) Dorikka: 2
(11:33:59 PM) Dorikka: 2
(11:34:00 PM) Dorikka: 22
(11:34:00 PM) Dorikka: 2
(11:34:00 PM) Dorikka: 2
(11:34:00 PM) Dorikka: 2
(11:34:01 PM) Dorikka: 2
(11:34:01 PM) Dorikka: 2
(11:34:01 PM) Dorikka: 2lol
(11:34:02 PM) Unable to send message: Not logged in
(11:34:04 PM) Dorikka: 2
(11:34:04 PM) Unable to send message: Not logged in
(11:34:07 PM) Unable to send message: Not logged in
(11:34:09 PM) Unable to send message: Not logged in
(11:34:12 PM) Unable to send message: Not logged in
(11:34:14 PM) Unable to send message: Not logged in
(11:34:17 PM) Unable to send message: Not logged in
(11:34:24 PM) Miley Cyrus: Sorry
(11:34:30 PM) Miley Cyrus: that was a bit much.
(11:34:35 PM) Miley Cyrus: The games over at any rate.
(11:34:52 PM) Dorikka: So, officially **END** ?
(11:34:56 PM) Miley Cyrus: Yeah.
(11:35:00 PM) Dorikka: Haha.
(11:35:02 PM) Dorikka: Nice ending.
(11:35:13 PM) Miley Cyrus: I guess it's up to our imaginations what happens after.
(11:35:21 PM) Dorikka: Yeah.

https://www.nytimes.com/2011/08/07/books/review/the-theory-that-would-not-die-by-sharon-bertsch-mcgrayne-book-review.html?_r=1&pagewanted=all

I've recently found that my utility function valued personal status and fame a whole lot more than I thought it did -- I previously had thought that it mostly relied on the consequences of my actions for other sentiences, but it turned out I was wrong. Obviously, this is a valuable insight -- I definitely want to know what my current utility function is; from there, I can decide whether I should change my actions or my utility function if the two aren't coordinated.

I did this by imagining how I would feel if I found out certain things. For example, how would I feel if everyone else was also trying to save the world? The emotional response I had was sort of a hollow feeling in the pit of my stomach, like I was a really mediocre being. This obviously wasn't a result of calculating that the marginal utility of my actions would be a whole lot lower in this hypothetical world (and so I should go do something else); instead, it was the fact that me trying to save the world didn't make me special any more -- I wouldn't stand out, in this sort of world.

(Epilogue: I decided that I hadn't done a good enough job programming my brain and am attempting to modify my utility function to rely on the world actually getting saved.)

Discussion: What other hypotheticals are useful?