As part of a broader project of promoting rationality, Raelifin and I had some luck in getting media coverage of rationality-informed approaches to probabilistic thinking (1, 2), mental health (1, 2), and reaching life goals through finding purpose and meaning (1, 2). The media includes mainstream media such as the main newspaper in Cleveland, OH; reason-oriented media such as Unbelievers Radio; student-oriented media such as the main newspaper for Ohio State University; and self improvement-oriented media such as the Purpose Revolution.

This is part of our strategy to reach out both to mainstream and to niche groups interested in a specific spin on rationality-informed approaches to winning at life. I wanted to share these here, and see if any of you had suggestions for optimizations of our performance, connections with other media channels both mainstream and nice, and any other thoughts on improving outreach. Thanks!

A putative new idea for AI control; index here.

In a previous post, I talked about an AI operating only on a virtual world (ideas like this used to be popular, until it was realised the AI might still want to take control of the real world to affect the virtual world; however, with methods like indifference, we can guard against this much better).

I mentioned that the more of the AI's algorithm that existed in the virtual world, the better it was. But why not go the whole way? Some people at MIRI and other places are working on agents modelling themselves within the real world. Why not have the AI model itself as an agent inside the virtual world? We can quine to do this, for example.

Then all the restrictions on the AI - memory capacity, speed, available options - can be specified precisely, within the algorithm itself. It will only have the resources of the virtual world to achieve its goals, and this will be specified within it. We could define a "break" in the virtual world (ie any outside interference that the AI could cause, were it to hack us to affect its virtual world) as something that would penalise the AI's achievements, or simply as something impossible according to its model or beliefs. It would really be a case of "given these clear restrictions, find the best approach you can to achieve these goals in this specific world".

It would be idea if the AI's motives were not given in terms of achieving anything in the virtual world, but in terms of making the decisions that, subject to the given restrictions, were most likely to achieve something if the virtual world were run in its entirety. That way the AI wouldn't care if the virtual world were shut down or anything similar. It should only seek to self modify in way that makes sense within the world, and understand itself existing completely within these limitations.

Of course, this would ideally require flawless implementation of the code; we don't want bugs developing in the virtual world that point to real world effects (unless we're really confident we have properly coded the "care only about the what would happen in the virtual world, not what actually does happen).

Any thoughts on this idea?

School taught me to write banal garbage because people would thumbs-up it anyway. That approach has been interfering with me trying to actually express my plans in writing because my mind keeps simulating some imaginary prof who will look it over and go "ehh, good enough".

Looking good enough isn't actually good enough! I'm trying to build an actual model of the world and a plan that will actually work.

Granted, school isn't necessarily all like this. In mathematics, you need to actually solve the problem. In engineering, you need to actually build something that works. But even in engineering reports, you can get away with a surprising amount of shoddy reasoning. A real example:

Since NodeJS uses the V8 JavaScript engine, it has native support for the common JSON (JavaScript Object Notation) format for data transfer, which means that interoperability between SystemQ and other CompanyX systems can still be fairly straightforward (Jelvis, 2011).

This excerpt is technically totally true, but it's also garbage, especially as a reason to use NodeJS. Sure, JSON is native to JS, but every major web programming language supports JSON. The pressure to provide citable justifications for decisions which were made for reasons more like "I enjoy JavaScript and am skilled with it," produces some deliberately confirmation-biased writing. This is just one pattern—there are many others.

I feel like I need to add a disclaimer here or something: I'm a ringed engineer, and I care a lot about the ethics of design, and I don't think any of my shoddy thinking has put any lives (or well-being, etc) at risk. I also don't believe that any of my shoddy thinking in design reports has violated academic integrity guidelines at my university (e.g. I haven't made up facts or sources).

But a lot of it was still shoddy. Most students are familiar with the process of stating a position, googling for a citation, then citing some expert who happened to agree. And it was shoddy because nothing in the school system was incentivizing me to make it otherwise, and I reasoned it would have cost more to only write stuff that I actually deeply and confidently believed, or to accurately and specifically present my best model of the subject at hand. I was trying to spend as little time and attention as possible working on school things, to free up more time and attention for working on my business, the productivity app Complice.

What I didn't realize was the cost of practising shoddy thinking.

Having finished the last of my school obligations, I've launched myself into some high-level roadmapping for Complice: what's the state of things right now, and where am I headed? And I've discovered a whole bunch of bad thinking habits. It's obnoxious.

I'm glad to be out.

(Aside: I wrote this entire post in April, when I was finished my last assignments & tests. I waited awhile to publish it so that I've now safely graduated. Wasn't super worried, but didn't want to take chances.)

Better Wrong Than Vague

So today.

I was already aware of a certain aversion I had to planning. So I decided to make things a bit easier with this roadmapping document, and base it on one my friend Oliver Habryka had written about his main project. He had created a 27-page outline in google docs, shared it with a bunch of people, and got some really great feedback and other comments. Oliver's introduction includes the following paragraph, which I decided to quote verbatim in mine:

This document was written while continuously repeating the mantra “better wrong than vague” in my head. When I was uncertain of something, I tried to express my uncertainty as precisely as possible, and when I found myself unable to do that, I preferred making bold predictions to vague statements. If you find yourself disagreeing with part of this document, then that means I at least succeeded in being concrete enough to be disagreed with.

In an academic context, at least up to the undergrad level, students are usually incentivized to follow "better vague than wrong". Because if you say something the slightest bit wrong, it'll produce a little "-1" in red ink.

Because if you and the person grading you disagree, a vague claim might be more likely to be interpreted favorably. There's a limit, of course: you usually can't just say "some studies have shown that some people sometimes found X to help". But still.

Practising being "good enough"

Nate Soares has written about the approach of whole-assed half-assing:

Your preferences are not "move rightward on the quality line." Your preferences are to hit the quality target with minimum effort.

If you're trying to pass the class, then pass it with minimum effort. Anything else is wasted motion.

If you're trying to ace the class, then ace it with minimum effort. Anything else is wasted motion.

My last two yearly review blog posts have followed structure of talking about my year on the object level (what I did), the process level (how I did it) and the meta level (my more abstract approach to things). I think it's helpful to apply the same model here.

There are lots of things that humans often wished their neurology naturally optimized for. One thing that it does optimize for though is minimum energy expenditure. This is a good thing! Brains are costly, and they'd have to function less well if they always ran at full power. But this has side effects. Here, the relevant side effect is that, if you practice a certain process for awhile, and it achieves the desired object-level results, you might lose awareness of the bigger picture approach that you're trying to employ.

So in my case, I was practising passing my classes with minimum effort, and not wasting motion, following the meta-level approach of whole-assed half-assing. But while the meta-level approach of "hitting the quality target with minimum effort" is a good one in all domains (some of which will have much, much higher quality targets) the process of doing the bare minimum to create something that doesn't have any obvious glaring flaws, is not a process that you want to be employing in your business. Or in trying to understand anything deeply.

Which I am now learning to do. And, in the process, unlearning the shoddy thinking I've been practising for the last 5 years.

Related LW post: Guessing the Teacher's Password

(This article crossposted from my blog)

From the costs of rationality wiki:

Becoming more epistemically rational can only guarantee one thing: what you believe will include more of the truth . Knowing that truth might help you achieve your goals , or cause you to become a pariah. Be sure that you really want to know the truth before you commit to finding it; otherwise, you may flinch from it.

The reason that truth seeking is often seen as being integral to rationality is that in order to make optimal decisions you must first be able to make accurate predictions. Delusions, or false beliefs, are self-imposed barriers to accurate prediction. They are surprise inducers. It is because of this that the rational path is often to break delusions, but you should remember that doing so is a slow and hard process that is rife with potential problems.

Below I have listed three scenarios in which a person could benefit from considering the costs of truth seeking. The first scenario is when seeking a more accurate measurement is computationally expensive and not really required. The second scenario is when you know that the truth will be emotionally distressing to another person and that this person is not in an optimal state to handle this truth. The third scenario is when you are trying to change the beliefs of others. It is often beneficial if you can understand the costs involved for them to change their beliefs as well as their perspective. This allows you to become better able to actually change their beliefs rather than to just win an argument.

Scenario 1: computationally expensive truth

We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil. Yet we should not pass up our opportunities in that critical 3%. – Donald Knuth

If optimization requires significant effort and only results in minimal gains in utility, then it is not worth it. If you only need to be 90% sure that something is true and you are currently 98% sure that it is, then it is not worth spending some extra effort to get to 99% certainty. For example, if you are testing ballistics on Earth then it may be appropriate to use Newtons laws even though they are known to be inexact in some extreme conditions. Now, this does not mean that optimization should never be done. Sometimes that extra 1% certainty is actually extremely important. What it does mean is that you should be spending your resources wisely. The beliefs that you do make should lead to increased abilities to anticipate accurately. You should also remember occams Razor. If you are committing yourself to a decision procedure that is accurate, but slow and wasteful then you will probably be outcompeted by others who spend their resources on more suitable and worthy activities.

Scenario 2: emotionally distressing truth

Assume for a moment that you have a child and that you have just finished watching that child fail horribly at a school performance. If your child then asks you, while crying, how the performance was. Do you tell them the truth in full or not? Most people would choose not to and would instead attempt to calm and comfort the child. To do otherwise is not seen as rational, but is instead seen as situationally unaware, rude and impolite. Obviously, some ways of telling the truth are worse than others. But, overall telling the full truth is probably not going to be the most prudent thing to do in this situation. This is because the child is not in an emotional state that will allow them to handle the truth well. The truth in this situation is unlikely to lead to improvement and will instead lead to further stress and trauma which will often cause future performance anxiety, premature optimization and other issues. For these reasons, I think that the truth should not be expressed in this situation. This does not mean that I think the rational person should forget about what has happened. They should instead remember it so that they can bring it up when the child is in an emotional state that would allow them to be better able to implement any advice that is given. For example, when practicing in a safe environment.

I want to point out that avoiding the truth is not what I am advocating. I am instead saying that we should be strategic about telling potentially face-threatening or emotionally distressing truths. I do believe that repression and avoidance of issues that have a persistent nature most often tends to lead to exacerbation or resignation of those issues. Hiding from the truth rarely improves the situation. Consider the child if you don't ever mention the performance because you don't want to cause the child pain then they are still probably going to get picked on at school. Knowing this, we can say that the best thing to do is to bring up the truth and frame it in a particular situation where the child can find it useful and come to be able to better handle it.

Scenario 3: psychologically exhausting truth

If we remember that truth seeking involves costs, then we are more likely to be aware of how we can reduce this cost when we are trying to change the beliefs of others. If you are trying to convince someone and they do not agree with you, this may not be because your arguments are weak or that the other person is stupid. It may just be that there is a significant cost involved for them to either understand your argument or to update their beliefs. If you want to convince someone and also avoid the illusion of transparency, then it is best to take into account the following:

• You should try to end arguments well and to avoid vitriol - the emotional contagion heuristic leads people to avoid contact with people or objects viewed as "contaminated" by previous contact with someone or something viewed as bad—or, less often, to seek contact with objects that have been in contact with people or things considered good. If someone gets emotional when you are in an argument, then you are going to be less likely to change their minds about that topic in the future. It is also a good idea to consider the peak-end rule which basically means that you should try to end your arguments well.
• If you find that someone is already closed off due to emotional contagion, then you should try a surprising strategy so that your arguments aren't stereotyped and avoided. As elizer said here: 

• The first rule of persuading a negatively disposed audience - rationally or otherwise - is not to say the things they expect you to say. The expected just gets filtered out, or treated as confirmation of pre-existing beliefs regardless of its content.

• Processing fluency - is the ease with which information is processed. You should ask yourself if your argument worded in such a way that it is fluent and easy to understand?
• Cognitive dissonance - is a measure of how much your argument conflicts with the other persons pre-existing beliefs? Perhaps, you need to convince them of a few other points first before your argument will work. People will try to avoid cognitive dissonance. Therefore, it is better to start off with shared arguments and in general to minimise disagreements. It is often the sounder strategy to modify and extend pre-existing beliefs.
• Inferential distance - is about how much background information that they need access to in order for them to understand your argument?
• Leave a line of retreat - think about whether they can admit that they were wrong without also looking stupid or foolish? In winning arguments there are generally two ways that you can go about it. The first is to totally demolish the other persons position. The second is to actually change their minds. The first leaves them feeling wrong, stupid and foolish which is often going to make them start rationalizing. The second just makes them feel wrong. You win arguments the second way by seeming to be reasonable and non face threatening. A good way to do this is through empathy and understanding the argument from the other persons position. It is important to see things as others would see them because we don't see the world as it is; we see the world as we are. The other person is not stupid or lying they might just in the middle of what I call an 'epistemic contamination cascade' (perhaps there is already a better name for this). It is when false beliefs lead to filters, framing effects and other false beliefs. Another potential benefit from viewing the argument from the other persons perspective is that it is possible that you may come to realise that your own is not as steadfast as you once believed.
• Maximise the cost of holding a false belief - ask yourself if there are any costs to them if they continue to hold a belief that you believe is false? One way to cause some cost is to convince their friends and associates of your position. The extra social pressure may help in getting them to change their minds.
• Give it time and get them inspecting their maps rather than information that has been filtered through their map. It is possible that there are filtering and framing effects which mean that your arguments are being distorted by the other person? Consider a depressed person: you can argue with them, but this is not likely to be overly helpful. THis is because it is likely that while arguing you will need to contradict them and this will probably lead to them blocking out what you are saying. I think that in these kinds of situations what you really need to do is to get them to inspect their own maps. This can be done by asking "what" or "how does that make you" type of questions. For example,“What are you feeling?”,“What’s going on?” and“What can I do to help?”. There are two main benefits to these types of questions over arguments. The first is that it gets them inspecting their maps and the second is that it is much harder for them to block out the responses since they are the ones providing them. This is a related quote from Sarah Silverman's book:

• My stepfather, John O'Hara, was the goodest man there was. He was not a man of many words, but of carefully chosen ones. He was the one parent who didn't try to fix me. One night I sat on his lap in his chair by the woodstove, sobbing. He just held me quietly and then asked only, 'What does it feel like?' It was the first time I was prompted to articulate it. I thought about it, then said, "I feel homesick." That still feels like the most accurate description--I felt homesick, but I was home. - Sarah Silverman

• Remember the other-optimizing bias and that perspectival types of issues need to be resolved by the individual facing them. If you have a goal to change another persons minds, then it often pays dividends to not only understand why they are wrong, but also why they think they are right or at least unaware that they are wrong. This kind of understanding can only come from empathy. Sometimes it is impossible to truly understand what another person is going through, but you should always try, without condoning or condemning, to see things as they are from the other persons perspective. Remember that hatred blinds and so does love. You should always be curious and seek to understand things as they are, not as you wish them, fear them or desire them to be. It is only when you can do this that you can truly understand the costs involved for someone else to change their minds.

If you take the point of view that changing beliefs is costly. Then you are less likely to be surprised when others don't want to change their beliefs. You are also more likely to think about how you can make the process of changing their beliefs easier for them.

Some other examples of when seeking the truth is not necessarily valuable are:

• Fiction writing and the cinematic experience
• When the pragmatic meaning does not need truth, but the semantic meaning does. An example is "Hi. How are you?" and other similar greetings which are peculiar because they look the same as questions or adjacency pairs, but function slightly differently. They are like a kind of ritualised question in which the answer is normally pre-specified or at least the detail of the answer is. If someone asks: "How are you" it is seen as aberrant to answer the question in full detail with the truth rather than simply with fine, which may be a lie. If they actually do want to know how you are, then they will probably ask a follow up question after the greeting like "so, is everything good with the kids".
• Evolutionary biases which cause delusions, but may help in perspectival and self confidence issues. For example, the sexual over perception bias from men. From a truth-maximization perspective young men who assume that all women want them are showing severe social-cognitive inaccuracies, judgment biases, and probably narcissistic personality disorder. However, from an evolutionary perspective, the same young men are behaving more optimally. That is, the bias is an adaptive bias one which has consistently maximized the reproductive success of their male ancestors. Other examples are the women's underestimation of men's commitment bias and positively biased perceptions of partners

tldr: this post posits that truth seeking should be viewed as an optimization process. This means that it may not always be worth it.

Personal Statement

I like to think about big questions from time to time. A fancy that quite possibly causes me more harm than good. Every once in a while I come up with some idea and wonder "hey, this seems pretty good, I wonder if anyone is taking it seriously?" Usually, answering that results at worst in me wasting a couple days on google and blowing $50 on amazon before I find someone who’s going down the same path and can tell myself. "Well, someone's got that covered". This particular idea is a little more stubborn and the amazon bill is starting to get a little heavy. So I cobbled together this “paper” to get this idea out there and see where it goes.  

I've been quite selective here and have only submitted it on two other places Vixra, and FXQI forum.  Vixra for posterity in the bizarre case that it's actually right.  FXQI because they play with some similar ideas (but the forum turned out to be not really vibrant for such things).  I'm now posting it on Less Wrong because you guys seem to be the right balance of badass skeptics and open minded geeks.  In addition I see a lot of cool work on Anthropic Reasoning and the like so it seems to go along with your theme.

Any and all feedback is welcome, I'm a good sport!

Abstract

A popular objection to the Many-worlds interpretation of Quantum Mechanics is that it allows for quantum suicide where an experimenter creates a device that instantly kills him or leaves him be depending the output of a quantum measurement, since he has no experience of the device killing him he experiences quantum immortality. This is considered counter-intuitive and absurd. Presented here is a speculative argument that accepts counter-intuitiveness and proposes it as a new approach to physical theory without accepting some of the absurd conclusions of the thought experiment. The approach is based on the idea that the Universe is Fragile in that only a fraction of the time evolved versions retain the familiar structures of people and planets, but the fractions that do not occur are not observed. This presents to us as a skewed view of physics and only by accounting for this fact (which I propose calling the Continual Anthropic Principle) can we understand the true fundamental laws.

Preliminary reasoning

Will a supercollider destroy the Earth?

A fringe objection to the latest generation of high energy supercolliders was they might trigger some quantum event that would destroy the earth such as by turning it to strangelets (merely an example). To assuage those fears it has been noted that since Cosmic Rays have been observed with higher energies then the collisions these supercolliders produce that if a supercollider were able to create such Earth-destroying events cosmic rays would have already destroyed the Earth. Since that hasn't happened physics must not work that way and we thus must be safe.

A false application of the anthropic principle

One may try to cite the anthropic principle as an appeal against the conclusion that physics disallows Earth-destruction by said mechanism. If the Earth were converted to strangelets, there would be no observers on it. If the right sort of multiverse exists, some Earths will be lucky enough to escape this mode of destruction. Thus physics may still allow for strangelet destruction and supercolliders may still destroy the world. We can reject that objection by noting that if that were the case, it is far more probable that our planet would be alone in a sea of strangelet balls that were already converted by highenergy cosmic rays. Since we observe other worlds made of ordinary matter, we can be sure physics doesn't allow for the Earth to be converted into strange matter by interactions at Earth’s energy level.

Will a supercollider destroy the universe?

Among the ideas on how supercolliders will destroy the world there are some that destroy not just the Earth but entire universe as well. A proposed mechanism is in triggering vacuum energy to collapse to a new lower energy state. By that mechanism the destructive event spreads out from the nucleation site at the speed of light and shreds the universe to something completely unrecognizable. In the same way cosmic rays rule out an Earth-destroying event it has said that this rules out a universe destroying event.

Quantum immortality and suicide

Quantum suicide is a thought experiment there is a device that measures a random quantum event, and kills an experimenter instantly upon one outcome, and leaves him alive upon the other. If Everett multiple worlds is true, then no matter how matter how many times an experiment is performed, the experimenter will only experience the outcome where he is not killed thus experiencing subjective immortality. There are some pretty nutty ideas about the quantum suicide and immortality, and this has been used as an argument against many-worlds. I find the idea of finding oneself for example perpetually avoiding fatal accidents or living naturally well beyond any reasonable time to be mistaken (see objections). I do however think that Max Tegmark came up with a good system of rules on his "crazy" page for how it might work: http://space.mit.edu/home/tegmark/crazy.html

The rules he outlines are: "I think a successful quantum suicide experiment needs to satisfy three criteria:

1. The random number generator must be quantum, not classical (deterministic), so that you really enter a superposition of dead and alive.

2. It must kill you (at least make you unconscious) on a timescale shorter than that on which you can become aware of the outcome of the quantum coin-toss - otherwise you'll have a very unhappy version of yourself for a second or more who knows he's about to die for sure, and the whole effect gets spoiled.

3. It must be virtually certain to really kill you, not just injure you.”

Have supercolliders destroyed the universe? 

Let's say that given experiment has a certain "probability" (by a probabilistic interpretation of QM) of producing said universe destructive event. This satisfies all 3 of Tegmark's conditions for a successful quantum suicide experiment. As such the experimenter might conclude that said event cannot happen. However, he would be mistaken, and a corresponding percentage of successor states would in fact be ones where the event occurred. If the rules of physics are such that an event is allowed then we have a fundamentally skewed perceptions of what physics are.

It's not a bug it's a feature!

If we presume such events could occur, we have no idea how frequent they are. There's no necessary reason why they need to be confined to rare high energy experiments and cosmic rays. Perhaps it dictates more basic and fundamental interactions. For instance certain events within an ordinary atomic nucleus could create a universe-destroying event. Even if these events occur at an astonishing rate, so long as there's a situation where the event doesn't occur (or is "undone" before the runaway effect can occur), it would not be contradictory with our observation. The presumption that these events don't occur may be preventing us from understanding a simpler law that describes physics in a certain situation in favor of more complex theories that limit behavior to that which we can observe.

Fragile Universe Hypothesis

Introduction

Because of this preliminary reasoning I am postulating what I call the "Fragile Universe Hypothesis". The core idea is that our universe is constantly being annihilated by various runaway events initiated by quantum phenomena. However, because for any such event there's always a possible path where such event does not occur, and since all possible paths are realized we are presented with an illusion of stability. What we see as persistent structures in the universe (chairs, planets, galaxies) are so only because events that destroy them by and large destroy us as well. What we may think are fundamental laws of our universe, are merely descriptions of the nature of possible futures consistent with our continued existence.

Core theory

The hypothesis can be summarized as postulating the following:

1. For a given event at Time T there are multiple largely non-interacting future successor events at T + ε (i.e. Everett Many Worlds is either correct or at least on the right track)

2. There are some events where some (but not all) successor events trigger runaway interactions that destroy the universe as we know it. Such events expand from the origin at C and immediately disrupt the consciousness of any being it encounters.

3. We experience only a subset of possible futures and thus have a skewed perspective of the laws of physics.

4. To describe the outcome of an experiment we must first calculate possible outcomes then filter out those that result in observer destruction (call it the "continual anthropic principle")

Possible Objections

"If I get destroyed I die and will no longer have experiences. This is at face value absurd"

I'm sympathetic, and I'd say this requires a stretch of imagination to consider. But do note that under this hypothesis, no one will ever have an experience that isn't followed by a successive experience (see quantum immortality for discussion of death). So from our perspective our existence will go on unimpeded. As an example, consider a video game save. The game file can be saved, copied, compressed, decompressed, moved from medium to medium (with some files being deleted after being copied to a new location). We say that the game continues so long as someone plays at least one copy of the file. Likewise for us, we say life (or the universe as we know it) goes on so long as at least one successor continues.

"This sort of reasoning would result in having to accept absurdities like quantum immortality"

I don't think so. Quantum immortality (the idea that many worlds guarantees one immortality as there will always be some future state in which one continues to exist) presumes that personhood is an all-ornothing thing. In reality a person is more of a fragmented collection of mental processes. We don't suddenly stop having experiences as we die, rather the fragments unbind, some live on in the memory of others or in those experiencing the products of our expression, while others fade out. A destructive event of the kind proposed would absolutely be an all-or-nothing affair. Either everything goes, or nothing goes.

"This isn't science. What testable predictions are you making? Heck you don't even have a solid theory" 

Point taken! This is, at this point, speculation, but I think at this point it might have the sort of elegance that good theories have. The questions that I have are:

1. Has this ever been seriously considered? (I’ve done some homework but undoubtedly not enough).

2. Are there any conceptual defeaters that make this a nonstarter?

3. Could some theories be made simpler by postulating a fragile universe and continual anthropic principle?

4. Could those hypothetical theories make testable predictions?

5. Have those tests been consistent with the theory.

My objective in writing this is to provide an argument against 2, and starting to look into 1 and 3. 4 and 5 are essential to good science as well too, but we’re simply not at that point yet.

Final Thoughts

The Copernican Principle for Many worlds

When we moved the Earth as the center of the solar system, the orbits of the other planets became simpler and clearer. Perhaps physical law can be made simpler and clearer when we move the futures we will experience away from the center of possible futures. And like the solar system's habitable zone, perhaps only a small portion of futures are habitable.

Why confine the Anthropic Principle to the past? 

Current models of cosmology limit the impact of the Anthropic selection on the cosmos to the past: string landscapes, bubble universes or cosmic branes, these things all got fixed at some set of values 13 billion years ago and the selection effect does no more work at the cosmic scale. Perhaps the selection effect is more fundamental then that. Could it be that instead 13 billion years ago is when the anthropic selection merely switched from being creative in sowing our cosmic seeds to conservative in allowing them to grow? 

I've talked earlier about integral and differential ethics, in the context of population ethics. The idea is that the argument for the repugnant conclusion (and its associate, the very repugnant conclusion) is dependent on a series of trillions of steps, each of which are intuitively acceptable (adding happy people, making happiness more equal), but reaching a conclusion that is intuitively bad - namely, that we can improve the world by creating trillions of people in torturous and unremitting agony, as long as balance it out by creating enough happy people as well.

Differential reasoning accepts each step, and concludes that the repugnant conclusions are actually acceptable, because each step is sound. Integral reasoning accepts that the repugnant conclusion is repugnant, and concludes that some step along the way must therefore be rejected.

Notice that key word, "therefore". Some intermediate step is rejected, but not for intrinsic reasons, but purely because of the consequence. There is nothing special about the step that is rejected, it's just a relatively arbitrary barrier to stop the process (compare with the paradox of the heap).

Indeed, things can go awry when people attempt to fix the repugnant conclusion (a conclusion they rejected through integral reasoning) using differential methods. Things like the "person-affecting view" have their own ridiculousness and paradoxes (it's ok to bring a baby into the world if it will have a miserable life; we don't need to care about future generations if we randomise conceptions, etc...) and I would posit that it's because they are trying to fix global/integral issues using local/differential tools.

The relevance of this? It seems that integral tools might be better suited to deal with the bad convergence of AI problem. We could set up plausibly intuitive differential criteria (such as self-consistency), but institute overriding integral criteria that can override these if they go too far. I think there may be some interesting ideas in that area, potentially. The cost is that integral ideas are generally seen as less elegant, or harder to justify.

Another month, another rationality quotes thread. The rules are:

• Please post all quotes separately, so that they can be upvoted or downvoted separately. (If they are strongly related, reply to your own comments. If strongly ordered, then go ahead and post them together.)
• Do not quote yourself.
• Do not quote from Less Wrong itself, HPMoR, Eliezer Yudkowsky, or Robin Hanson. If you'd like to revive an old quote from one of those sources, please do so here.
• No more than 5 quotes per person per monthly thread, please.
• Provide sufficient information (URL, title, date, page number, etc.) to enable a reader to find the place where you read the quote, or its original source if available. Do not quote with only a name.

This is the monthly thread for posting media of various types that you've found that you enjoy. Post what you're reading, listening to, watching, and your opinion of it. Post recommendations to blogs. Post whatever media you feel like discussing! To see previous recommendations, check out the older threads.

Rules:

• Please avoid downvoting recommendations just because you don't personally like the recommended material; remember that liking is a two-place word. If you can point out a specific flaw in a person's recommendation, consider posting a comment to that effect.
• If you want to post something that (you know) has been recommended before, but have another recommendation to add, please link to the original, so that the reader has both recommendations.
• Please post only under one of the already created subthreads, and never directly under the parent media thread.
• Use the "Other Media" thread if you believe the piece of media you want to discuss doesn't fit under any of the established categories.
• Use the "Meta" thread if you want to discuss about the monthly media thread itself (e.g. to propose adding/removing/splitting/merging subthreads, or to discuss the type of content properly belonging to each subthread) or for any other question or issue you may have about the thread or the rules.

This is the public group rationality diary for July 12th - August 1st, 2015. It's a place to record and chat about it if you have done, or are actively doing, things like:

• Established a useful new habit

• Obtained new evidence that made you change your mind about some belief

• Decided to behave in a different way in some set of situations

• Optimized some part of a common routine or cached behavior

• Consciously changed your emotions or affect with respect to something

• Consciously pursued new valuable information about something that could make a big difference in your life

• Learned something new about your beliefs, behavior, or life that surprised you

• Tried doing any of the above and failed

Or anything else interesting which you want to share, so that other people can think about it, and perhaps be inspired to take action themselves. Try to include enough details so that everyone can use each other's experiences to learn about what tends to work out, and what doesn't tend to work out.

Archive of previous rationality diaries

Note to future posters: no one is in charge of posting these threads. If it's time for a new thread, and you want a new thread, just create it. It should run for about two weeks, finish on a Saturday, and have the 'group_rationality_diary' tag.

Related: Diplomacy as a Game Theory Laboratory by Yvain.

I've been floating this idea around for a while, and there was enough interest to organize it.

Diplomacy is a board game of making and breaking alliances. It is a semi-iterative prisoner's dilemma with 7 prisoners. The rules are very simple, there is no luck factor and any tactical tricks can be learned quickly. You play as one of the great powers in pre-WW1 Europe, and your goal is to dominate over half of the board. To do this, you must negotiate alliances with the other players, and then stab them at the most opportune moment. But beware, if you are too stabby, no one will trust you. And if you are too trusting, you will get stabbed yourself.

If you have never played the game, don't worry. It is really quick to pick up. I explain the rules in detail here.

The game will (most likely) be played at webdiplomacy.net. You need an account, which requires a valid email. To play the game, you will need to spend at least 10 minutes every phase (3 days) to enter your orders. In the meantime, you will be negotiating with other players. That takes as much as you want it to, but I recommend setting away at least 30 minutes per day (in 5-minute quantums). A game usually lasts about 10 in-game years, which comes down to 30-something phases (60-90 days). A phase can progress early if everyone agrees. Likewise, the game can be paused indefinitely if everyone agrees (e.g. if a player will not have Internet access).

Joining a game is Serious Business, as missing a deadline can spoil it for the other 6 players. Please apply iff:

1. You will be able to access the game for 10 minutes every 3 days (90% certainty required)
2. If 1) changes, you will be able to let the others know at least 1 day in advance (95% certainty required)
3. You will be able to spend an average of 30 minutes per day (standard normal distribution)
4. You will not hold an out-of-game grudge against a player who stabbed you (adjusting for stabbyness in potential future games is okay)

If you still wish to play, please sign up in the comments. Please specify the earliest time it would suit you for the game to start. If we somehow get more than 7 players, we'll discuss our options (play a variant with more players, multiple games, etc).

See also: First game of LW Diplomacy

Well, the interest is there, so I've set up two games.

Game 1: http://webdiplomacy.net/board.php?gameID=164863  (started!)

Game 2: http://webdiplomacy.net/board.php?gameID=164912  (started! First phase will be extended to end on the 4th of August)

Password: clippy

Please note a couple important rules of the webdiplomacy.net website:

1. You can only have one account. If you are caught with multiple accounts, they will all be banned.
2. You may not blame your moves on the website bugs as a diplomacy tactic. This gives the site's mods extra work to do when someone actually reports the bug.
3. Should go without saying, but you are not allowed to illegally access another player's account (i.e. hacking).

This map shows that AI failure resulting in human extinction could happen on different levels of AI development, namely, before it starts self-improvement (which is unlikely but we still can envision several failure modes), during its take off, when it uses different instruments to break out from its initial confinement, and after its successful take over the world, when it starts to implement its goal system which could be plainly unfriendly or its friendliness may be flawed. 

AI also can halts on late stages of its development because of either technical problems or "philosophical" one.

I am sure that the map of AI failure levels is needed for the creation of Friendly AI theory as we should be aware of various risks. Most of ideas in the map came from "Artificial Intelligence as a Positive and Negative Factor in Global Risk" by Yudkowsky, from chapter 8 of "Superintelligence" by Bostrom, from Ben Goertzel blog and from hitthelimit blog, and some are mine.

I will now elaborate three ideas from the map which may need additional clarification.

The problem of the chicken or the egg

The question is what will happen first: AI begins to self-improve, or the AI got a malicious goal system. It is logical to assume that the goal system change will occur first, and this gives us a chance to protect ourselves from the risks of AI, because there will be a short period of time when AI already has bad goals, but has not developed enough to be able to hide them from us effectively. This line of reasoning comes from Ben Goertzel.

Unfortunately many goals are benign on a small scale, but became dangerous as the scale grows. 1000 paperclips are good, one trillion are useless, and 10 to the power of 30 paperclips are an existential risk. 

AI halting problem

Another interesting part of the map are the philosophical problems that must face any AI. Here I was inspired after this reading Russian-language blog hitthelimit

One of his ideas is that the Fermi paradox may be explained by the fact that any sufficiently complex AI halts. (I do not agree that it completely explains the Great Silence.)

After some  simplification, with which he is unlikely to agree, the idea is that as AI self-improves its ability to optimize grows rapidly, and as a result, it can solve problems of any complexity in a finite time. In particular, it will execute any goal system in a finite time. Once it has completed its tasks, it will stop.

The obvious objection to this theory is the fact that many of the goals (explicitly or implicitly) imply infinite time for their realization. But this does not remove the problem at its root, as this AI can find ways to ensure the feasibility of such purposes in the future after it stops. (But in this case it is not an existential risk if their goals are formulated correctly.)

For example, if we start from timeless physics, everything that is possible already exists and the number of paperclips in the universe is a) infinite b) unchangable. When the paperclip maximizer has understood this fact, it may halt. (Yes, this is a simplistic argument, it can be disproved, but it is presented solely to illustrate the approximate reasoning, that can lead to AI halting.) I think the AI halting problem is as complex as the halting problem for Turing Machine.

Vernor Vinge in his book Fire Upon the Deep described unfriendly AIs which halt any externally visible activity about 10 years after their inception, and I think that this intuition about the time of halting from the point of external observer is justified: this can happen very quickly. (Yes, I do not have a fear of fictional examples, as I think that they can be useful for explanation purposes.)

In the course of my arguments with “hitthelimit” a few other ideas were born, specifically about other philosophical problems that may result in AI halting.

One of my favorites is associated with modal logic. The bottom line is that from observing the facts, it is impossible to come to any conclusions about what to do, simply because oughtnesses are in a different modality. When I was 16 years old this thought nearly killed me.

It almost killed me, because I realized that it is mathematically impossible to come to any conclusions about what to do. (Do not think about it too long, it is a dangerous idea.) This is like awareness of the meaninglessness of everything, but worse.

Fortunately, the human brain was created through the evolutionary process and has bridges from the facts to oughtness, namely pain, instincts and emotions, which are out of the reach of logic.

But for the AI with access to its own source code these processes do not apply. For this AI, awareness of the arbitrariness of any set of goals may simply mean the end of its activities: the best optimization of a meaningless task is to stop its implementation. And if AI has access to the source code of its objectives, it can optimize it to maximum simplicity, namely to zero.

Lobstakle by Yudkowsky is also one of the problems of high level AI, and it's probably just the beginning of the list of such issues.

Existence uncertainty

If AI use the same logic as usually used to disprove existence of philosophical zombies, it may be uncertain if it really exists or it is only a possibility. (Again, then I was sixteen I spent unpleasant evening thinking about this possibility for my self.) In both cases the result of any calculations is the same. It is especially true in case if AI is philozombie itself, that is if it does not have qualia. Such doubts may result in its halting or in conversion of humans in philozombies. I think that AI that do not have qualia or do not believe in them can't be friendly. This topic is covered in the map in the bloc "Actuality".

The status of this map is a draft that I believe can be greatly improved. The experience of publishing other maps has resulted in almost a doubling of the amount of information. A companion to this map is a map of AI Safety Solutions which I will publish later.

The map was first presented to the public at a LessWrong meetup in Moscow in June 2015  (in Russian)

Pdf is here: http://immortality-roadmap.com/AIfails.pdf 

Previous posts:

A Roadmap: How to Survive the End of the Universe

A map: Typology of human extinction risks

Roadmap: Plan of Action to Prevent Human Extinction Risks

(scroll down to see the map)

I have noticed that the term 'nihilism' has quite a few different connotations. I do not know that it is a coincidence. Reputedly, the most popular connotation, and in my opinion, the least well-defined, is existential nihilism, 'the philosophical theory that life has no intrinsic meaning or value.' I think that most LessWrong users would agree that there is no intrinsic meaning or value, but also that they would argue that there is a contingent meaning or value, and that the absence of such intrinsic meaning or value is no justification to be a generally insufferable person.

There is also the slightly similar but perhaps more well-defined moral nihilism; epistemological nihilism; and the not-unrelated fatalism.

Here, it goes without saying that each of these positions is wrong.

In a sense, this plan needs to be perceived with irony because it is almost irrelevant: we have very small chances of surviving even next 1000 years and if we do, we have a lot of things to do before it becomes reality. And even afterwards, our successors will have completely different plans.

There is one important exception: there are suggestions that collider experiments may lead to a vacuum phase transition, which begins at one point and spreads across the visible universe. Then we can destroy ourselves and our universe in this century, but it would happen so quickly that we will not have time to notice it. (The term "universe" hereafter refers to the observable universe that is the three-dimensional world around us, resulting from the Big Bang.)

We can also solve this problem in next century if we create superintelligence.

The purpose of this plan is to show that actual immortality is possible: that we have an opportunity to live not just billions and trillions of years, but an unlimited duration. My hope is that the plan will encourage us to invest more in life extension and prevention of global catastrophic risks. Our life could be eternal and thus have meaning forever.

Anyway, the end of the observable universe is not an absolute end: it's just one more problem on which the future human race will be able to work. And even at the negligible level of knowledge about the universe that we have today, we are still able to offer more than 50 ideas on how to prevent its end.

In fact, to assemble and come up with these 50 ideas I spent about 200 working hours, and if I had spent more time on it, I'm sure I would have found many new ideas.  In the distant future we can find more ideas; choose the best of them; prove them, and prepare for their implementation.

First of all, we need to understand exactly what kind end to the universe we should expect in the natural course of things. There are many hypotheses on this subject, which can be divided into two large groups:

1. The universe is expected to have a relatively quick and abrupt end, known as the Big Crunch or Big Rip (accelerating expansion of the universe causes it to break apart), or the decay of the false vacuum. Vacuum decay can occur at any time; a Big Rip could happen in about 10-30 billion years, and the Big Crunch has hundreds of billions of years timescale.

2. Another scenario assumes an infinitely long existence of an empty, flat and cold universe which would experience so called "heat death" that is gradual halting of all processes and then disappearance of all matter.

The choice between these scenarios depends on the geometry of the universe, which is determined by the equations of general relativity and, – above all – the behavior of the almost unknown parameter: dark energy.

The recent discovery of dark energy has made Big Rip the most likely scenario, but it is clear that the picture of the end of the universe will change several times.

You can find more at: http://en.wikipedia.org/wiki/Ultimate_fate_of_the_universe

There are five general approaches to solve the end of the universe problem, each of them includes many subtypes shown in the map:

1.     Surf the Wave: Utilize the nature of the process which is ending the universe. (The most known of these type of solutions is Omega Point by Tippler, where the universe's energy collapse is used to make infinite calculations.)

2.     Go to parallel world

3.     Prevent the end of the universe

4.     Survive the end of the universe

5.     Dissolving the problem

 Some of the ideas are on the level of the wildest possible speculations and I hope you will enjoy them.

The new feature of this map is that in many cases mentioned, ideas are linked to corresponding wiki pages in the pdf. 

Download the pdf of the map here: http://immortality-roadmap.com/unideatheng.pdf

The inaugural New Hampshire Less Wrong meet-up will take place the week of June 29-July 4. I've created a Doodle poll to find out the best date for likely participants. If you are interested in attending, please fill out the poll here: http://doodle.com/4ypehfkvsm7cvf76

The first meeting will be in Manchester, but I'm open to rotating locations throughout NH in the future, especially if people want to host meetings in their homes.

I hope to coordinate crossover meetings with Boston LW, e.g. field trips to Sundays at the Citadel."

*****

I've posted this for Elizabeth Edwards-Appell-- she's confirmed her LW email, but still can't post, not even comments. I've notified tech, but meanwhile, if anyone can help with her posting problem, let me know.

(The following is armchair psychological speculation based on anecdotal evidence. If anyone can respond with relevant science, that'd be awesome, but otherwise responses in kind are quite welcome.)

I'm puzzled regarding the motivational effects of negative emotion, particularly shame, guilt, and regret--I'll just say 'regret' going forward, though there could be important differences between them. In particular, I've observed people being oddly unmotivated to avoid doing things that will very predictably cause them unhappiness, in a way that seems to go beyond garden variety akrasia. It recently occurred to me that homo hypocritus theories of regret predict this result.

In contrast to the naive theory, under which regret is inflicted by a brain on itself to teach it to change its behavior, the homo hypocritus theory holds that regret exist to convince other observers that its behavior will change. This allows someone to continually do antisocial things while convincing others that they won't do so in the future. Look at it from the perspective of a gene-selfish brain designer: self-inflicted regret, unlike externally-inflicted injuries, doesn't carry any intrinsic harms, so there's no reason to behave in a way that avoids it.

On closer inspection, neither theory makes a whole lot of sense on its own. Regret has distinctive displays that would be pointless if they were only for internal consumption. And there would be no reason for others to think regret would lead to changed behavior if this were never the case. So a revised theory combines the two as follows:

Naive 2.0) Regret originally evolved to act both as impetus for an individual to change behavior and signal to others that such a correction was taking place.

Homo Hypocritus 2.0) Sometimes brains exploit this by feeling regret and sending the signal while somehow blocking the behavior update--this is advantageous if the regretted action only harms the agent via others' disapproval and the emotional display allays that disapproval.

This could vary by individual, by situation, or both. And it comes with the standard evo-psych disclaimer that people with HH brains aren't faking their suffering. Rather, this might explain behavior we'd call 'compulsive' or 'self-destructive'--it could be that the compulsion to do regrettable things isn't extra-strong, but rather that the brain's motivation to avoid those behaviors is blocked. In many cases these individuals, subject to constant cycles of action and regret, would be the primary victims of their brain's cynical adaptation.

So what testable predictions would this yield? (We can worry about how to test these ethically later).

* There should be people and/or situations where concrete external punishment would be much more motivating than regret even if the latter causes much more suffering.

* If we could arrange for people to experience regret inside an MRI machine, we might observe variations in how much lasting change occurs, and observe those whose brains change more change their behavior more. This might also correlate with life outcomes or general tendency to do regrettable things.

* At least some humans should have innate defenses against evolutionarily-hypocritical but personally-sincere regret.

* There should be distinct and identifiable modes of compulsive/self-destructive behavior that only occur with behaviors whose harmful results are mediated by other humans' reactions, at least as far as the savannah-brain can tell.

Thoughts, all?

(Continuing the posting of select posts from Slate Star Codex for comment here, for the reasons discussed in this thread, and as Scott Alexander gave me - and anyone else - permission to do with some exceptions.)

Scott recently wrote a post called No Time Like The Present For AI Safety Work. It makes the argument for the importance of organisations like MIRI thus, and explores the last two premises:

1. If humanity doesn’t blow itself up, eventually we will create human-level AI.

2. If humanity creates human-level AI, technological progress will continue and eventually reach far-above-human-level AI

3. If far-above-human-level AI comes into existence, eventually it will so overpower humanity that our existence will depend on its goals being aligned with ours

4. It is possible to do useful research now which will improve our chances of getting the AI goal alignment problem right

5. Given that we can start research now we probably should, since leaving it until there is a clear and present need for it is unwise

I placed very high confidence (>95%) on each of the first three statements – they’re just saying that if trends continue moving towards a certain direction without stopping, eventually they’ll get there. I had lower confidence (around 50%) on the last two statements.

Commenters tended to agree with this assessment; nobody wanted to seriously challenge any of 1-3, but a lot of people said they just didn’t think there was any point in worrying about AI now. We ended up in an extended analogy about illegal computer hacking. It’s a big problem that we’ve never been able to fully address – but if Alan Turing had gotten it into his head to try to solve it in 1945, his ideas might have been along the lines of “Place your punch cards in a locked box where German spies can’t read them.” Wouldn’t trying to solve AI risk in 2015 end in something equally cringeworthy?

As always, it's worth reading the whole thing, but I'd be interested in the thoughts of the LessWrong community specifically.

Solomonoff Induction is a sort of mathematically ideal specification of machine learning. It works by trying every possible computer program and testing how likely they are to have produced the data. Then it weights them by their probability.

Obviously Solomonoff Induction is impossible to do in the real world. But it forms the basis of AIXI and other theoretical work in AI. It's a counterargument to the no free lunch theorem; that we don't care about the space of all possible datasets, but ones which are generated by some algorithm. It's even been proposed as a basis for a universal intelligence test.

Many people believe that trying to approximate Solomonoff Induction is the way forward in AI. And any machine learning algorithm that actually works, to some extent, must be an approximation of Solomonoff Induction.

But how do we go about trying to approximate true Solomonoff Induction? It's basically an impossible task. Even if you make restrictions to remove all the obvious problems like infinite loops/non-halting behavior. The space of possibilities is just too huge to reasonably search through. And it's discrete - you can't just flip a few bits in a program and find another similar program.

We can simplify the problem a great deal by searching through logic circuits. Some people disagree about whether logic circuits should be classified as Turing complete, but it's not really important. We still get the best property of Solomonoff Inducion; that it allows most interesting problems to be modelled much more naturally. In the worst case you have some overhead to specify the memory cells you need to emulate a Turing machine.

Logic circuits have some nicer properties compared to arbitrary computer programs, but they still are discrete and hard to do inference on. To fix this we can easily make continuous versions of logic circuits. Go back to analog. It's capable of doing all the same functions, but also working with real valued states instead of binary.

Instead of flipping between discrete states, we can slightly increase connections between circuits, and it will only slightly change the behavior. This is very nice, because we have algorithms like MCMC that can efficiently approximate true bayesian inference on continuous parameters.

And we are no longer restricted to boolean gates, we can use any function that takes real numbers. Like a function that takes a sum of all of it's inputs, or one that squishes a real number between 0 and 1.

We can also look at how much changing the input of a circuit slightly, changes the output. Then we can go to all the circuits that connect to it in the previous time step. And we can see how much changing each of their input changes their output, and therefore the output of the first logic gate.

And we can go to those gates' inputs, and so on, chaining it all the way through the whole circuit. Finding out how much a slight change to each connection will change the final output. This is called the gradient, and we can then do gradient descent. Basically change each parameter slightly in the direction that increases the output the way we want.

This is a very efficient optimization algorithm. With it we can rapidly find circuits that fit functions we want. Like predicting the price of a stock given the past history, or recognizing a number in an image, or something like that.

But this isn't quite Solomonoff Induction. Since we are finding the best single model, instead of testing the space of all possible models. This is important because essentially each model is like a hypothesis. There can be multiple hypotheses which also fit the data yet predict different things.

There are many tricks we can do to approximate this. For example, if you randomly turn off each gate with 50% probability and then optimize the whole circuit to deal with this. For some reason this somewhat approximates the results of true bayesian inference. You can also fit a distribution over each parameter, instead of a single value, and approximate bayesian inference that way.

Although I never said it, everything I've mentioned about continuous circuits is equivalent to Artificial Neural Networks. I've shown how they can be derived from first principles. My goal was to show that ANNs do approximate true Solomonoff Induction. I've found the Bayes-Structure.

It's worth mentioning that Solomonoff Induction has some problems. It's still an ideal way to do inference on data, it just has problems with self-reference. An AI based on SI might do bad things like believe in an afterlife, or replace it's reward signal with an artificial one (e.g. drugs.) It might not fully comprehend that it's just a computer, and exists inside the world that it is observing.

Interestingly, humans also have these problem to some degree.

Reposted from my blog here.

(Continuing the posting of select posts from Slate Star Codex for comment here, as discussed in this thread, and as Scott Alexander gave me - and anyone else - permission to do with some exceptions.)

Scott recently wrote a post called Bicameral Reasoning. It touches on epistemology and scope insensitivity. Here are some excerpts, though it's worth reading the whole thing:

Delaware has only one Representative, far less than New York’s twenty-seven. But both states have an equal number of Senators, even though New York has a population of twenty million and Delaware is uninhabited except by corporations looking for tax loopholes.

[...]

I tend to think something like “Well, I agree with this guy about the Iraq war and global warming, but I agree with that guy about election paper trails and gays in the military, so it’s kind of a toss-up.”

And this way of thinking is awful.

The Iraq War probably killed somewhere between 100,000 and 1,000,000 people. If you think that it was unnecessary, and that it was possible to know beforehand how poorly it would turn out, then killing a few hundred thousand people is a really big deal. I like having paper trails in elections as much as the next person, but if one guy isn’t going to keep a very good record of election results, and the other guy is going to kill a million people, that’s not a toss-up.

[...]

I was thinking about this again back in March when I had a brief crisis caused by worrying that the moral value of the world’s chickens vastly exceeded the moral value of the world’s humans. I ended up being trivially wrong – there are only about twenty billion chickens, as opposed to the hundreds of billions I originally thought. But I was contingently wrong – in other words, I got lucky. Honestly, I didn’t know whether there were twenty billion chickens or twenty trillion.

And honestly, 99% of me doesn’t care. I do want to improve chickens, and I do think that their suffering matters. But thanks to the miracle of scope insensitivity, I don’t particularly care more about twenty trillion chickens than twenty billion chickens.

Once again, chickens seem to get two seats to my moral Senate, no matter how many of them there are. Other groups that get two seats include “starving African children”, “homeless people”, “my patients in hospital”, “my immediate family”, and “my close friends”.

[...]

I’m tempted to say “The House is just plain right and the Senate is just plain wrong”, but I’ve got to admit that would clash with my own very strong inclinations on things like the chicken problem. The Senate view seems to sort of fit with a class of solutions to the dust specks problem where after the somethingth dust speck or so you just stop caring about more of them, with the sort of environmentalist perspective where biodiversity itself is valuable, and with the Leibnizian answer to Job.

But I’m pretty sure those only kick in at the extremes. Take it too far, and you’re just saying the life of a Delawarean is worth twenty-something New Yorkers.

Thoughts?

Your job, should you choose to accept it, is to comment on this thread explaining the most awesome thing you've done this month. You may be as blatantly proud of yourself as you feel. You may unabashedly consider yourself the coolest freaking person ever because of that awesome thing you're dying to tell everyone about. This is the place to do just that.

 I'm currently reading through some relevant literature for preparing my FLI grant proposal on the topic of concept learning and AI safety. I figured that I might as well write down the research ideas I get while doing so, so as to get some feedback and clarify my thoughts. I will posting these in a series of "Concept Safety"-titled articles.

The AI in the quantum box

In the previous post, I discussed the example of an AI whose concept space and goals were defined in terms of classical physics, which then learned about quantum mechanics. Let's elaborate on that scenario a little more.

I wish to zoom in on a certain assumption that I've noticed in previous discussions of these kinds of examples. Although I couldn't track down an exact citation right now, I'm pretty confident that I've heard the QM scenario framed as something like "the AI previously thought in terms of classical mechanics, but then it finds out that the world actually runs on quantum mechanics". The key assumption being that quantum mechanics is in some sense more real than classical mechanics.

This kind of an assumption is a natural one to make if someone is operating on an AIXI-inspired model of AI. Although AIXI considers an infinite amount of world-models, there's a sense in which AIXI always strives to only have one world-model. It's always looking for the simplest possible Turing machine that would produce all of the observations that it has seen so far, while ignoring the computational cost of actually running that machine. AIXI, upon finding out about quantum mechanics, would attempt to update its world-model into one that only contained QM primitives and to derive all macro-scale events right from first principles.

No sane design for a real-world AI would try to do this. Instead, a real-world AI would take advantage of scale separation. This refers to the fact that physical systems can be modeled on a variety of different scales, and it is in many cases sufficient to model them in terms of concepts that are defined in terms of higher-scale phenomena. In practice, the AI would have a number of different world-models, each of them being applied in different situations and for different purposes.

Here we get back to the view of concepts as tools, which I discussed in the previous post. An AI that was doing something akin to reinforcement learning would come to learn the kinds of world-models that gave it the highest rewards, and to selectively employ different world-models based on what was the best thing to do in each situation.

As a toy example, consider an AI that can choose to run a low-resolution or a high-resolution psychological model of someone it's interacting with, in order to predict their responses and please them. Say the low-resolution model takes a second to run and is 80% accurate; the high-resolution model takes five seconds to run and is 95% accurate. Which model will be chosen as the one to be used will depend on the cost matrix of making a correct prediction, making a false prediction, and the consequence of making the other person wait for an extra four seconds before the AI's each reply.

We can now see that a world-model being the most real, i.e. making the most accurate predictions, doesn't automatically mean that it will be used. It also needs to be fast enough to run, and the predictions need to be useful for achieving something that the AI cares about.

World-models as tools

From this point of view, world-models are literally tools just like any other. Traditionally in reinforcement learning, we would define the value of a policy  in state s as the expected reward given the state s and the policy ,

but under the "world-models are tools" perspective, we need to also condition on the world-model m,

 .

We are conditioning on the world-model in several distinct ways.

First, there is the expected behavior of the world as predicted by world-model m. A world-model over the laws of social interaction would do poorly at predicting the movement of celestial objects, if it could be applied to them at all. Different predictions of behavior may also lead to differing predictions of the value of a state. This is described by the equation above.

Second, there is the expected cost of using the world-model. Using a more detailed world-model may be more computationally expensive, for instance. One way of interpreting this in a classical RL framework would be that using a specific world-model will place the agent in a different state than using some other world-model. We might describe by saying that in addition to the agent choosing its next action a on each time-step, the agent also needs to choose the world-model m which it will use to analyze its next observations. This will be one of the inputs for the transition function  to the next state.

Third, there is the expected behavior of the agent using world-model m. An agent with different beliefs about the world will act differently in the future: this means that the future policy  actually depends on the chosen world-model.

Some very interesting questions pop up at this point. Your currently selected world-model is what you use to evaluate your best choices for the next step... including the choice of what world-model to use next. So whether or not you're going to switch to a different world-model for evaluating the next step depends on whether your current world-model says that a different world-model would be better in that step.

We have not fully defined what exactly we mean by "world-models" here. Previously I gave the example of a world-model over the laws of social interaction, versus a world-model over the laws of physics. But a world-model over the laws of social interaction, say, would not have an answer to the question of which world-model to use for things it couldn't predict. So one approach would be to say that we actually have some meta-model over world-models, telling us which is the best to use in what situation.

On the other hand, it does also seem like humans often use a specific world-model and its predictions to determine whether to choose another world-model. For example, in rationalist circles you often see arguments to the line of, "self-deception might give you extra confidence, but it introduces errors into your world-model, and in the long term those are going to be more harmful than the extra confidence is beneficial". Here you see an implicit appeal to a world-model which predicts an accumulation of false beliefs with some specific effects, as well as predicting the extra self-esteem with its effects. But this kind of an analysis incorporates very specific causal claims from various (e.g. psychological) models, which are models over the world rather than just being part of some general meta-model over models. Notice also that the example analysis takes into account the way that having a specific world-model affects the state transition function: it assumes that a self-deceptive model may land us in a state where we have a higher self-esteem.

It's possible to get stuck in one world-model: for example, a strongly non-reductionist model evaluating the claims of a highly reductionist one might think it obviously crazy, and vice versa. So it seems that we do need something like a meta-evaluation function. Otherwise it would be too easy to get stuck in one model which claimed that it was the best one in every possible situation, and never agreed to "give up control" in favor of another one.

One possibility for such a thing would be a relatively model-free learning mechanism, which just kept track of the rewards accumulated when using a particular model in a particular situation. It would then bias the selection of the model towards the direction of the model that had been the most successful so far.

Human neuroscience and meta-models

We might be able to identify something like this in humans, though this is currently very speculative on my part. Action selection is carried out in the basal ganglia: different brain systems send the basal ganglia "bids" for various actions. The basal ganglia then chooses which actions to inhibit or disinhibit (by default, everything is inhibited). The basal ganglia also implements reinforcement learning, selectively strengthening or weakening the connections associated with a particular bid and context when a chosen action leads to a higher or lower reward than was expected. It seems that in addition to choosing between motor actions, the basal ganglia also chooses between different cognitive behaviors, likely even thoughts: 

If action selection and reinforcement learning are normal functions of the basal ganglia, it should be possible to interpret many of the human basal ganglia-related disorders in terms of selection malfunctions. For example, the akinesia of Parkinson's disease may be seen as a failure to inhibit tonic inhibitory output signals on any of the sensorimotor channels. Aspects of schizophrenia, attention deficit disorder and Tourette's syndrome could reflect different forms of failure to maintain sufficient inhibitory output activity in non-selected channels. Conseqently, insufficiently inhibited signals in non-selected target structures could interfere with the output of selected targets (expressed as motor/verbal tics) and/or make the selection system vulnerable to interruption from distracting stimuli (schizophrenia, attention deficit disorder). The opposite situation would be where the selection of one functional channel is abnormally dominant thereby making it difficult for competing events to interrupt or cause a behavioural or attentional switch. Such circumstances could underlie addictive compulsions or obsessive compulsive disorder. (Redgrave 2007)

Although I haven't seen a paper presenting evidence for this particular claim, it seems plausible to assume that humans similarly come to employ new kinds of world-models based on the extent to which using a particular world-model in a particular situation gives them rewards. When a person is in a situation where they might think in terms of several different world-models, there will be neural bids associated with mental activities that recruit the different models. Over time, the bids associated with the most successful models will become increasingly favored. This is also compatible with what we know about e.g. happy death spirals and motivated stopping: people will tend to have the kinds of thoughts which are rewarding to them.

The physicist and the AI

In my previous post, when discussing the example of the physicist who doesn't jump out of the window when they learn about QM and find out that "location" is ill-defined:

The physicist cares about QM concepts to the extent that the said concepts are linked to things that the physicist values. Maybe the physicist finds it rewarding to develop a better understanding of QM, to gain social status by making important discoveries, and to pay their rent by understanding the concepts well enough to continue to do research. These are some of the things that the QM concepts are useful for. Likely the brain has some kind of causal model indicating that the QM concepts are relevant tools for achieving those particular rewards. At the same time, the physicist also has various other things they care about, like being healthy and hanging out with their friends. These are values that can be better furthered by modeling the world in terms of classical physics. [...]

A part of this comes from the fact that the physicist's reward function remains defined over immediate sensory experiences, as well as values which are linked to those. Even if you convince yourself that the location of food is ill-defined and you thus don't need to eat, you will still suffer the negative reward of being hungry. The physicist knows that no matter how they change their definition of the world, that won't affect their actual sensory experience and the rewards they get from that.

So to prevent the AI from leaving the box by suitably redefining reality, we have to somehow find a way for the same reasoning to apply to it. I haven't worked out a rigorous definition for this, but it needs to somehow learn to care about being in the box in classical terms, and realize that no redefinition of "location" or "space" is going to alter what happens in the classical model. Also, its rewards need to be defined over models to a sufficient extent to avoid wireheading (Hibbard 2011), so that it will think that trying to leave the box by redefining things would count as self-delusion, and not accomplish the things it really cared about. This way, the AI's concept for "being in the box" should remain firmly linked to the classical interpretation of physics, not the QM interpretation of physics, because it's acting in terms of the classical model that has always given it the most reward. 

There are several parts to this.

1. The "physicist's reward function remains defined over immediate sensory experiences". Them falling down and breaking their leg is still going to hurt, and they know that this won't be changed no matter how they try to redefine reality.

2. The physicist's value function also remains defined over immediate sensory experiences. They know that jumping out of a window and ending up with all the bones in their body being broken is going to be really inconvenient even if you disregarded the physical pain. They still cannot do the things they would like to do, and they have learned that being in such a state is non-desirable. Again, this won't be affected by how they try to define reality.

We now have a somewhat better understanding of what exactly this means. The physicist has spent their entire life living in the classical world, and obtained nearly all of their rewards by thinking in terms of the classical world. As a result, using the classical model for reasoning about life has become strongly selected for. Also, the physicist's classical world-model predicts that thinking in terms of that model is a very good thing for surviving, and that trying to switch to a QM model where location was ill-defined would be a very bad thing for the goal of surviving. On the other hand, thinking in terms of exotic world-models remains a rewarding thing for goals such as obtaining social status or making interesting discoveries, so the QM model does get more strongly reinforced in that context and for that purpose.

Getting back to the question of how to make the AI stay in the box, ideally we could mimic this process, so that the AI would initially come to care about staying in the box. Then when it learns about QM, it understands that thinking in QM terms is useful for some goals, but if it were to make itself think in purely QM terms, that would cause it to leave the box. Because it is thinking mostly in terms of a classical model, which says that leaving the box would be bad (analogous to the physicist thinking mostly in terms of the classical model which says that jumping out of the window would be bad), it wants to make sure that it will continue to think in terms of the classical model when it's reasoning about its location.

I'd appreciate feedback on optimizing a blog post that shares about my mental illness and popularizes future-oriented thinking to a broad audience. I'm using story-telling as the driver of the narrative, and sprinkling in elements of rational thinking, such as hyperbolic discounting, mental maps, and future-oriented thinking, in a strategic way. The target audience is college-age youth and young adults. Any suggestions for what works well, and what can be improved would be welcomed! The blog draft itself is below the line.

P.S. For context, the blog is part of a broader project, Intentional Insights, aimed at promoting rationality to a broad audience, as I described in this LW discussion post. To do so, we couch rationality in the language of self-improvement and present it in a narrative style.

_______________________________________________________________________________________________________________________

 Coming Out of the Mental Health Closet

My hand jerked back, as if the computer mouse had turned into a real mouse. I just couldn’t do it. Would they think I am crazy? Would they whisper behind my back? Would they never trust me again? These are the kinds of anxious thoughts that ran through my head as I was about to post on my Facebook profile revealing my mental illness to my Facebook friends, about 6 months after my condition began.

I really wanted to share much earlier about my mental illness, a mood disorder characterized by high anxiety, sudden and extreme fatigue, and panic attacks. It would have felt great to be genuinely authentic with people in my life, and not hide who I am. Plus, I would have been proud to contribute to overcoming the stigma against mental illness in our society, especially since this stigma impacts me on such a personal level.

Ironically, the very stigma against mental illness, combined with my own excessive anxiety response, made it very hard for me to share. I was really anxious about whether friends and acquaintances would turn away from me. I was also very concerned about the impact on my professional career of sharing publicly, due to the stigma in academia against mental illness, including at my workplace, Ohio State, as my colleague and fellow professor described in his article.

Whenever the thought of telling others entered my mind, I felt a wave of anxiety pass through me. My head began to pound, my heart sped up, my breathing became fast and shallow, almost like I was suffocating. If I didn’t catch it in time, the anxiety could lead to a full-blown panic attack, or sudden and extreme fatigue, with my body collapsing in place. Not a pretty picture.

Still, I did eventually start discussing my mental illness with some very close friends who I was very confident would support me. And one conversation really challenged my mental map, in other words how I perceive reality, about sharing my story of mental illness.

My friend told me something that really struck me, namely his perspective about how great would it be if all people who needed professional help with their mental health actually went to get such help. One of the main obstacles, as research shows, is the stigma against mental health. We discussed how one of the best ways to deal with such stigma is for well-functioning people with mental illness to come out of the closet about their condition.

Well, I am one of these well-functioning people. I have a great job and do it well, have wonderful relationships, and participate in all sorts of civic activities. The vast majority of people who know me don’t realize I suffer from a mental illness.

That conversation motivated me to think seriously through the roadblocks thrown up by the emotional part of my brain. Previously, I never sat down for a few minutes and forced myself to think what good things might happen if I pushed past all the anxiety and stress of telling people in my life about my mental illness.

I realized that I was just flinching away, scared of the short-term pain of rejection and not thinking about the long-term benefits to me and to others of sharing my story. I was falling for a thinking error that scientists call hyperbolic discounting, a reluctance to make short-term sacrifices for much higher long-term rewards.

To combat this problem, I imagined what world I wanted to live in a year from now – one where I shared about this situation now on my Facebook profile, or one where I did not. This approach is based on research showing that future-oriented thinking is very helpful for dealing with thinking errors associated with focusing on the present.

In the world where I would share right now about my condition, I would be very anxious about what people think of me. Anytime I saw someone who found out for the first time, I would be afraid about the impact on that person’s opinion of me. I would be watching her or his behavior closely for signs of distancing from me. And this would not only be my anxiety: I was quite confident that some people would not want to associate with me due to my mental illness. However, over time, this close watching and anxious thoughts would diminish. All the people who knew me previously would find out. All new people who met met would learn about my condition, since I would not keep it a secret. I would make the kind of difference I wanted to make in the world by fighting mental stigma in our society, and especially in academia. Just as important, it would be a huge burden off my back to not hide myself and be authentic with people in my life.

I imagined a second world. I would continue to hide my mental health condition from everyone but a few close friends. I would always have to keep this secret under wraps, and worry about people finding out about it. I would not be making the kind of impact on our society that I knew I would be able to make. And likely, people would find out about it anyway, whether if I chose to share about it or some other way, and I would get all the negative consequences later.

Based on this comparison, I saw that the first world was much more attractive to me. So I decided to take the plunge, and made a plan to share about the situation publicly. As part of doing so, I made that Facebook post. I had such a good reaction from my Facebook friends that I decided to make the post publicly available on my Facebook to all, not only my friends. Moreover, I decided to become an activist in talking about my mental condition publicly, as in this essay that you are reading.

What can you do?

So how can you apply this story to your life? Whether you want to come out of the closet to people in your life about some unpleasant news, or more broadly overcome the short-term emotional pain of taking an action that would help you achieve your long-term goals, here are some strategies.

1) Consider the world where you want to live a year from now. What would the world look like if you take the action? What would it look like if you did not take the action?

2) Evaluate all the important costs and benefits of each world. What world looks the most attractive a year from now?

3) Decide on the actions needed to get to that world, make a plan, and take the plunge. Be flexible about revising your plan based on new information such as reactions from others, as I did regarding sharing about my own condition.

What do you think?

• Do you ever experience a reluctance to tell others about something important to you because of your concern about their response? How have you dealt with this problem yourself?
• Is there any area of your life where an orientation to the short term undermines much higher long-term rewards? Do you have any effective strategies for addressing this challenge?
• Do you think the strategy of imagining the world you want to live in a year from now can be helpful in any area of your life? If so, where and how?

___________________________________________________________________________________________________________

Thanks in advance for your feedback and suggestions on optimizing the post!

Note: I'm terrible at making up titles, and I think that the one I gave may give the wrong impression. If anyone has a suggestion on what I should change it to, it would be much appreciated.

As I've been reading articles on less wrong, it seems to me that there are hints of an underlying belief which states that not only is capitalism a good economic paradigm, it shall remain so. Now, I don't mean to say anything like 'Capitalism is Evil!' I think that capitalism can, and has, done a lot of good for humanity. 

However, I don't think that capitalism will be the best economic paradigm going into the future. I used to view capitalism as an inherent part of the society we currently live in, with no real economic competition.

I recently changed my views as a result of a book someone recommended to me 'The zero marginal cost society' by Jeremy Rifkin. In it, the author states that we are in the midst of a third industrial revolution as a result of a new energy/production and communications matrix i.e. renewable energies, 3-D printing and the internet.

The author claims that these three things will eventually bring their respective sectors marginal costs to zero. This is significant because of a 'contradiction at the heart of capitalism' (I'm not sure how to phrase this, so excuse me if I butcher it): competition is at the heart of capitalism, with companies constantly undercutting each other as a result of new technologies. These technological improvement allow a company to produce goods/services at a more attractive price whilst retaining a reasonable profit margin. As a result, we get better and better at producing things, and it lets us produce goods at ever decreasing costs. But what happens when the costs of producing something hit rock bottom? That is, they can go no lower.

3D printing presents a situation like this for a huge amount of industries, as all you really need to do is get some designs, plug in some feedstock and have a power source ready. The internet allows people to share their designs for almost zero cost, and renewable energies are on the rise, presenting the avenue of virtually free power. All that's left is the feedstock, and the cost of this is due to the difficulty of producing it. Once we have better robotics, you won't need anyone to mine/cultivate anything, and the whole thing becomes basically free.

And when you can get your goods, energy and communications for basically free, doesn't that undermine the whole capitalist system? Of course, the arguments presented in the book are much more comprehensive, and it details an alternative economic paradigm called the Commons. I'm just paraphrasing here.

Since my knowledge of economics is woefully inadequate, I was wondering if I've made some ridiculous blunder which everyone knows about on this site. Is there some fundamental reason why Jeremy Rifkin's is a crackpot and I'm a fool for listening to him? Or is it more subtle than that? I ask because I felt the arguments in the book pretty compelling, and I want some opinions from people who are much better suited to critiquing this sort of thing than I.

Here is a link to the download page for the essay titled 'The comedy of the Commons' which provides some of the arguments which convinced me: 

http://digitalcommons.law.yale.edu/fss_papers/1828/

A lecture about the Commons itself:

http://www.nobelprize.org/nobel_prizes/economic-sciences/laureates/2009/ostrom_lecture.pdf

And a paper (?) about governing the commons: 

http://www.kuhlen.name/MATERIALIEN/eDok/governing_the_commons1.pdf

And here is a link to the author's page, along with some links to articles about the book:

http://www.thezeromarginalcostsociety.com/pages/Milestones.cfm

http://www.thezeromarginalcostsociety.com/pages/Press--Articles.cfm

An article displaying some of the sheer potential of 3D printers, and how it has the potential to change society in a major way:

http://singularityhub.com/2012/08/22/3d-printers-may-someday-construct-homes-in-less-than-a-day/

Edit: Drat! I forgot about the stupid questions thread. Should I delete this and repost it there? I mean, I hope to discuss this topic with others, so it seems suitable for the DISCUSSION board, but it may also be very stupid. Advice would be appreciated.

Are there any LessWrong Skype groups, or active live chatrooms? I've been looking around and found nothing. Well, with the exception of the LW Study Hall, but it doesn't quite fit since it is primarily for social work/study facilitation purposes with only minor breaks. This would fulfill a primarily social function.

But you ask, wouldn't a regular Skype chat reduce effectiveness by distracting people from their work? A little bit, but I'd rather the distracting thing be increasing my rationality by getting me engaging in the ideas with other people who are actively trying to do the same. I expect it to have an overall positive effect on productivity since I am bound to encounter one or two ideas to do so.

Thus, the value of such a group for me would be to discuss topics pertinent to rationality, and increase increase the shininess and entertainment value of LessWrong's ideas- it is already pretty interesting, and I've had fun thinking while sitting around reading the Sequences (finished How To Actually Change Your Mind not too long ago). There are no meetups near me, and I'd rather engage via online interactions anyway.

If there is no such group already, I'd be happy to start one. Feel free to either leave your Skype name in the comments or send me a PM if you're interested.

edit: My Skype id is bluevertro

This isn't a trick question, nor do I have a particular answer in mind. 

Tomorrow, all of your memories are going to be wiped. There is a crucial piece of information that you need to make sure you remember, and more specifically, you need to be very confident you were the one that sent this message and not a third party pretending to be you.

How do you go about transmitting, "signing", and verifying such a message*?

--edit: I should have clarified that one of the assumptions is that some malicious third party can/will be attempting to send you false information from "yourself" and you need to distinguish between that and what's really you.

--edit2: this may be formally impossible, I don't actually know. If anyone can demonstrate this I'd be very appreciative. 

--edit3: I don't have a particular universal definition for the term "memory wipe" in mind, mainly because I didn't want to pigeonhole the discussion. I think this pretty closely mimics reality. So I think it's totally fine to say, "If you retain this type of memory, then I'd do X." 

A putative new idea for AI control; index here.

Part of the problem with a reduced impact AI is that it will, by definition, only have a reduced impact.

Some of the designs try and get around the problem by allowing a special "output channel" on which impact can be large. But that feels like cheating. Here is a design that accomplishes the same without using that kind of hack.

Imagine there is an asteroid that will hit the Earth, and we have a laser that could destroy it. But we need to aim the laser properly, so need coordinates. There is a reduced impact AI that is motivated to give the coordinates correctly, but also motivated to have reduced impact - and saving the planet from an asteroid with certainty is not reduced impact.

Now imagine that instead there are two AIs, X and Y. By abuse of notation, let ¬X refer to the event that the output signal from X is scrambled away from the the original output.

Then we ask X to give us the x-coordinates for the laser, under the assumption of ¬Y (that AI Y's signal will be scrambled). Similarly, we Y to give us the y-coordinates of the laser, under the assumption ¬X.

Then X will reason "since ¬Y, the laser will certainly miss its target, as the y-coordinates will be wrong. Therefore it is reduced impact to output the correct x-coordinates, so I shall." Similarly, Y will output the right y-coordinates, the laser will fire and destroy the asteroid, having a huge impact, hooray!

The approach is not fully general yet, because we can have "subagent problems". X could create an agent that behave nicely given ¬Y (the assumption it was given), but completely crazily given Y (the reality). But it shows how we could get high impact from slight tweaks to reduced impact.

EDIT: For those worried about lying to the AIs, do recall http://lesswrong.com/r/discussion/lw/lyh/utility_vs_probability_idea_synthesis/ and http://lesswrong.com/lw/ltf/false_thermodynamic_miracles/

A putative new idea for AI control; index here.

This post contains no new insights; it just puts together some old insights in a format I hope is clearer.

Most satisficers are unoptimised (above the satisficing level): they have a limited drive to optimise and transform the universe. They may still end up optimising the universe anyway: they have no penalty for doing so (and sometimes it's a good idea for them). But if they can lazily achieve their goal, then they're ok with that too. So they simply have low optimisation pressure.

A safe "satisficer" design (or a reduced impact AI design) needs to be not only un-optimised, but specifically anti-optimised. It has to be setup so that "go out and optimise the universe" scores worse that "be lazy and achieve your goal". The problem is that these terms are undefined (as usual), that there are many minor actions that can optimise the universe (such as creating a subagent), and the approach has to be safe against all possible ways of optimising the universe - not just the "maximise u" for a specific and known u.

That's why the reduced impact/safe satisficer/anti-optimised designs are so hard: you have to add a very precise yet general (anti-)optimising pressure, rather than simply removing the current optimising pressure.

What do you think about these pairs of statements?

1. People's misfortunes result from the mistakes they make
2. Many of the unhappy things in people's lives are partly due to bad luck

1. In the long run, people get the respect they deserve in this world.
2. Unfortunately, an individual's worth often passes unrecognized no matter how hard he tries.

1. Becoming a success is a matter of hard work; luck has little or nothing to do with it.
2. Getting a good job mainly depends on being in the right place at the right time.

They have a similar theme: the first statement suggests that an outcome (misfortune, respect, or a good job) for a person are the result of their own action or volition. The second assigns the outcome to some external factor like bad luck.(1)

People who tend to think their own attitudes or efforts can control what happens to them are said to have an internal locus of control, those who don't, an external locus of control. (Call them 'internals' and 'externals' for short).

Internals seem to do better at life, pace obvious confounding: maybe instead of internals doing better by virtue of their internal locus of control, being successful inclines you to attribute success internal factors and so become more internal, and vice versa if you fail.(2) If you don't think the relationship is wholly confounded, then there is some prudential benefit for becoming more internal.

Yet internal versus external is not just a matter of taste, but a factual claim about the world. Do people, in general, get what their actions deserve, or is it generally thanks to matters outside their control?

Why the external view is right

Here are some reasons in favour of an external view:(3)

1. Global income inequality is marked (e.g. someone in the bottom 10% of the US population by income is still richer than two thirds of the population - more here). The main predictor of your income is country of birth, it is thought to explain around 60% of the variance: not only more important than any other factor, but more important than all other factors put together.
2. Of course, the 'remaining' 40% might not be solely internal factors either. Another external factor we could put in would be parental class. Include that, and the two factors explain 80% of variance in income.
3. Even conditional on being born in the right country (and to the right class), success may still not be a matter of personal volition. One robust predictor of success (grades in school, job performance, income, and so on) is IQ. The precise determinants of IQ remain controversial, it is known to be highly heritable, and the 'non-genetic' factors of IQ proposed (early childhood environment, intra-uterine environment, etc.) are similarly outside one's locus of control.

On cursory examination the contours of how our lives are turned out are set by factors outside our control, merely by where we are born and who our parents are. Even after this we know various predictors, similarly outside (or mostly outside) of our control, that exert their effects on how our lives turn out: IQ is one, but we could throw in personality traits, mental health, height, attractiveness, etc.

So the answer to 'What determined how I turned out, compared to everyone else on the planet?', the answer surely has to by primarily about external factors, and our internal drive or will is relegated a long way down the list. Even if we want to look at narrower questions, like "What has made me turn out the way I am, versus all the other people who were likewise born in rich countries in comfortable circumstances?" It is still unclear whether the locus of control resides within our will: perhaps a combination of our IQ, height, gender, race, risk of mental illness and so on will still do the bulk of the explanatory work.(4)

Bringing the true and the prudentially rational together again

If it is the case that folks with an internal locus of control succeed more, yet also the external view being generally closer to the truth of the matter, this is unfortunate. What is true and what is prudentially rational seem to be diverging, such that it might be in your interests not to know about the evidence in support of an external locus of control view, as deluding yourself about an internal locus of control view would lead to your greater success.

Yet it is generally better not to believe falsehoods. Further, the internal view may have some costs. One possibility is fueling a just world fallacy: if one thinks that outcomes are generally internally controlled, then a corollary is when bad things happen to someone or they fail at something, it was primarily their fault rather than them being a victim of circumstance.

So what next? Perhaps the right view is to say that: although most important things are outside our control, not everything is. Insofar as we do the best with what things we can control, we make our lives go better. And the scope of internal factors - albeit conditional on being a rich westerner etc. - may be quite large: it might determine whether you get through medical school, publish a paper, or put in enough work to do justice to your talents. All are worth doing.

Acknowledgements

Inspired by Amanda MacAskill's remarks, and in partial response of Peter McIntyre. Neither are responsible for what I've written, and the former's agreement or the latter's disagreement with this post shouldn't be assumed.

1) Some ground-clearing: free will can begin to loom large here - after all, maybe my actions are just a result of my brain's particular physical state, and my brain's particular physical state at t depends on it's state at t-1, and so on and so forth all the way to the big bang. If so, there is no 'internal willer' for my internal locus of control to reside.

However, even if that is so, we can parse things in a compatibilist way: 'internal' factors are those which my choices can affect; external factors are those which my choices cannot affect. "Time spent training" is an internal factor as to how fast I can run, as (borrowing Hume), if I wanted to spend more time training, I could spend more time training, and vice versa. In contrast, "Hemiparesis secondary to birth injury" is an external factor, as I had no control over whether it happened to me, and no means of reversing it now. So the first set of answers imply support for the results of our choices being more important; whilst the second set assign more weight to things 'outside our control'.

2) In fairness, there's a pretty good story as to why there should be 'forward action': in the cases where outcome is a mix of 'luck' factors (which are a given to anyone), and 'volitional ones' (which are malleable), people inclined to think the internal ones matter a lot will work hard at them, and so will do better when this is mixed in with the external determinants.

3) This ignores edge cases where we can clearly see the external factors dominate - e.g. getting childhood leukaemia, getting struck by lightning etc. - I guess sensible proponents of an internal locus of control would say that there will be cases like this, but for most people, in most cases, their destiny is in their hands. Hence I focus on population level factors.

4) Ironically, one may wonder to what extent having an internal versus external view is itself an external factor.

http://www.engadget.com/2015/04/01/ex-machina-alex-garland-interview/

The title says he "embraces the rise of superintelligent AI", but that isn't really supported by the text.

What struck me about this was that he seems to just take good habits of thought for granted.

I instinctively disagreed with this, but I didn't have the sort of armory to disagree with it on his terms, so I started reading as much as I could.

What other sorts of AI books have you read?

I pretty much would read everything I could. I tried to read people like Penrose, who were arguing against what I instinctively believed.

That's a good way to solidify your argument.

I don't want to dignify it from my point of view, because I can't stress enough I'm a real layman. So I can understand the principles of an argument, but when it comes to the actuality [...] I really don't understand it.

Being clear about these things is important. Otherwise, we're very quick to conflate stuff, and suddenly you'll be talking about the sentience of Siri. And Siri doesn't have any fucking sentience. AI is probably too broad of a term to be useful at the moment.

This is the monthly thread for posting media of various types that you've found that you enjoy. Post what you're reading, listening to, watching, and your opinion of it. Post recommendations to blogs. Post whatever media you feel like discussing! To see previous recommendations, check out the older threads.

Rules:

• Please avoid downvoting recommendations just because you don't personally like the recommended material; remember that liking is a two-place word. If you can point out a specific flaw in a person's recommendation, consider posting a comment to that effect.
• If you want to post something that (you know) has been recommended before, but have another recommendation to add, please link to the original, so that the reader has both recommendations.
• Please post only under one of the already created subthreads, and never directly under the parent media thread.
• Use the "Other Media" thread if you believe the piece of media you want to discuss doesn't fit under any of the established categories.
• Use the "Meta" thread if you want to discuss about the monthly media thread itself (e.g. to propose adding/removing/splitting/merging subthreads, or to discuss the type of content properly belonging to each subthread) or for any other question or issue you may have about the thread or the rules.

https://docs.google.com/document/d/1jDGpIT3gKZQZByO6A036dojRKMv62KEDEfEz87VuDoY/

Abstract: Discussion of how we might want to define human preferences, particularly in the context of building an AI intended to learn and implement those preferences. Starts with actual arguments about the applicability of the VNM utility theorem, then towards the end gets into hypotheses that are less well defended but possibly more important. At the very end, suggests that current hypothesizing about AI safety might be overemphasizing “discovering our preferences” over “creating our preferences”.

A putative new idea for AI control; index here.

The insight this post comes from is a simple one: defining concepts such as “human” and “happy” is hard. A superintelligent AI will probably create good definitions of these, while attempting to achieve its goals: a good definition of “human” because it needs to control them, and of “happy” because it needs to converse convincingly with us. It is annoying that these definitions exist, but that we won’t have access to them.

Modelling and defining

Imagine a game of football (or, as you Americans should call it, football). And now imagine a computer game version of it. How would you say that the computer game version (which is nothing more than an algorithm) is also a game of football?

Well, you can start listing features that they have in common. They both involve two “teams” fielding eleven “players” each, that “kick” a “ball” that obeys certain equations, aiming to stay within the “field”, which has different “zones” with different properties, etc...

As you list more and more properties, you refine your model of football. There are some properties that distinguish real from simulated football (fine details about the human body, for instance), but most of the properties that people care about are the same in both games.

My idea is that once you have a sufficiently complex model of football that applies to both the real game and a (good) simulated version, you can use that as the definition of football. And compare it with other putative examples of football: maybe in some places people play on the street rather than on fields, or maybe there are more players, or maybe some other games simulate different aspects to different degrees. You could try and analyse this with information theoretic considerations (ie given two model of two different examples, how much information is needed to turn one into the other).

Now, this resembles the “suggestively labelled lisp tokens” approach to AI, or the Cyc approach of just listing lots of syntax stuff and their relationships. Certainly you can’t keep an AI safe by using such a model of football: if you try an contain the AI by saying “make sure that there is a ‘Football World Cup’ played every four years”, the AI will still optimise the universe and then play out something that technically fits the model every four years, without any humans around.

However, it seems to me that ‘technically fitting the model of football’ is essentially playing football. The model might include such things as a certain number of fouls expected; an uncertainty about the result; competitive elements among the players; etc... It seems that something that fits a good model of football would be something that we would recognise as football (possibly needing some translation software to interpret what was going on). Unlike the traditional approach which involves humans listing stuff they think is important and giving them suggestive names, this involves the AI establishing what is important to predict all the features of the game.

We might even combine such a model with the Turing test, by motivating the AI to produce a good enough model that it could a) have conversations with many aficionados about all features of the game, b) train a team to expect to win the world cup, and c) use it to program successful football computer game. Any model of football that allowed the AI to do this – or, better still, that a football-model module that, when plugged into another, ignorant AI, allowed that AI to do this – would be an excellent definition of the game.

It’s also one that could cross ontological crises, as you move from reality, to simulation, to possibly something else entirely, with a new physics: the essential features will still be there, as they are the essential features of the model. For instance, we can define football in Newtonian physics, but still expect that this would result in something recognisably ‘football’ in our world of relativity.

Notice that this approach deals with edge cases mainly by forbidding them. In our world, we might struggle on how to respond to a football player with weird artificial limbs; however, since this was never a feature in the model, the AI will simply classify that as “not football” (or “similar to, but not exactly football”), since the model’s performance starts to degrade in this novel situation. This is what helps it cross ontological crises: in a relativistic football game based on a Newtonian model, the ball would be forbidden from moving at speeds where the differences in the physics become noticeable, which is perfectly compatible with the game as its currently played.

Being human

Now we take the next step, and have the AI create a model of humans. All our thought processes, our emotions, our foibles, our reactions, our weaknesses, our expectations, the features of our social interactions, the statistical distribution of personality traits in our population, how we see ourselves and change ourselves. As a side effect, this model of humanity should include almost every human definition of human, simply because this is something that might come up in a human conversation that the model should be able to predict.

Then simply use this model as the definition of human for an AI’s motivation.

What could possibly go wrong?

I would recommend first having an AI motivated to define “human” in the best possible way, most useful for making accurate predictions, keeping the definition in a separate module. Then the AI is turned off safely and the module is plugged into another AI and used as part of its definition of human in its motivation. We may also use human guidance at several points in the process (either in making, testing, or using the module), especially on unusual edge cases. We might want to have humans correcting certain assumptions the AI makes in the model, up until the AI can use the model to predict what corrections humans would suggest. But that’s not the focus of this post.

There are several obvious ways this approach could fail, and several ways of making it safer. The main problem is if the predictive model fails to define human in a way that preserves value. This could happen if the model is too general (some simple statistical rules) or too specific (a detailed list of all currently existing humans, atom position specified).

This could be combated by making the first AI generate lots of different models, with many different requirements of specificity, complexity, and predictive accuracy. We might require some models make excellent local predictions (what is the human about to say?), others excellent global predictions (what is that human going to decide to do with their life?). 

Then everything defined as “human” in any of the models counts as human. This results in some wasted effort on things that are not human, but this is simply wasted resources, rather than a pathological outcome (the exception being if some of the models define humans in an actively pernicious way – negative value rather than zero – similarly to the false-friendly AIs’ preferences in this post).

The other problem is a potentially extreme conservatism. Modelling humans involves modelling all the humans in the world today, which is a very narrow space in the range of all potential humans. To prevent the AI lobotomising everyone to a simple model (after all, there does exist some lobotomised humans today), we would want the AI to maintain the range of cultures and mind-types that exist today, making things even more unchanging.

To combat that, we might try and identify certain specific features of society that the AI is allowed to change. Political beliefs, certain aspects of culture, beliefs, geographical location (including being on a planet), death rates etc... are all things we could plausibly identify (via sub-sub-modules, possibly) as things that are allowed to change. It might be safer to allow them to change in a particular range, rather than just changing altogether (removing all sadness might be a good thing, but there are many more ways this could go wrong, than if we eg just reduced the probability of sadness). 

Another option is to keep these modelled humans little changing, but allow them to define allowable changes themselves (“yes, that’s a transhuman, consider it also a moral agent.”). The risk there is that the modelled humans get hacked or seduced, and that the AI fools our limited brains with a “transhuman” that is one in appearance only.

We also have to beware of not sacrificing seldom used values. For instance, one could argue that current social and technological constraints mean that no one has today has anything approaching true freedom. We wouldn’t want the AI to allow us to improve technology and social structures, but never get more freedom than we have today, because it’s “not in the model”. Again, this is something we could look out for, if the AI has separate models of “freedom” we could assess and permit to change in certain directions.

Note: The following is a draft post I've had since 2009, and it's not great but it's worth posting for discussion. I do like the way that it prefigures some of the problems of Quirrell Points when traitors are allowed...

Need to see if this can be easily gamed, but...

Step 1. Introduce Prisoner's Dilemma.  Set up computer system so that they can log in and play it in partners with investments of points (caution them: this is their actual grade at stake).  Let them know that they currently don't have enough points for a passing grade on that part of the course, but that maximum investment and mutual cooperation will result in A's for (almost) everyone on it (with high probability); also that points are converted to grades on a logarithmic scale.  Let them know that creating institutions and alliances is a good strategy in such games.

Initially, each student is allowed to play once per day, with 1 partner.  Students log in, enter the name of their requested PD partner, enter how much they're willing to invest, and enter C or D.  They'll get a "bank statement" daily as well.

If both enter C, they each get 1.2 points back (per initial point invested).  If one enters C and the other D, the cooperator gets nothing back and the defector gets 2 points (per point invested). If both enter D, then each gets 0.5 points back.

Once they've had some practice with this, we move to

Step 2: Bigger investments, luck, observation.

Introduce larger group investments with higher rates of return.  E.g. a five-person opportunity that pays each C player 0.2 guaranteed plus 0.4 for each C (not counting themselves), and each D player gets 0.5 guaranteed plus 0.5 for each C.  (Set these up to be balanced in some way.)

Add a factor of luck, so that people can't just (be forced to) show one another their bank statements as proof of their cooperation.  People should average the proper amounts, but have enough variation that it's often difficult to tell whether they cooperated or defected.  (Or is there another way to handle this?)

Finally, allow a third option of observation within a deal.  One idea: if you choose to observe, then you take up a spot in the investment, keep your own point, and don't count as a C for others, but you get to see some or all of the other players' choices.  Make sure that information is proportionately expensive.

Step 3: Keep scaling it up, adjust (only) as necessary.

If things get too unbalanced, some progressive taxation and welfare might be in order; but disrupting the system too much will tend to destroy the things they need to learn.

Ideally, larger and larger opportunities should be offered, with the kicker being a gigantic one-shot Prisoner's Dilemma that the whole class can partake in.  C might get guaranteed 0.2 plus 0.2 for every other C, while D gets guaranteed 1 plus 0.3 for every C.

Implementation:

Need a non-hackable computer program to work off of.  Also, have students (for real points) give critiques and suggestions for improvement.

This is an analysis of the Yom Kippur war (Egypt vs. Israel, 1973)-- the Israelis were interested in how Egypt managed a surprise attack, and it turned out that too many Israelis believed that the Egyptians would only attack if they had rockets which could reach deep into Israel. The Egyptians didn't have those rockets, so the Israeli government ignored evidence that the Egyptians were massing military forces on the border.

The rest of the article is analysis of the recent Israeli election, but to put it mildly, an election has much less in the way of well-defined factors than a surprise military attack, so it's much harder to say whether any explanation is correct. 

I'm sure there are many examples of plausible theories keeping people from getting to the correct explanation for a long time. Any suggestions? Also, is there a standard name for this mistake?

If it's worth saying, but not worth its own post (even in Discussion), then it goes here.

Notes for future OT posters:

1. Please add the 'open_thread' tag.

2. Check if there is an active Open Thread before posting a new one. (Immediately before; refresh the list-of-threads page before posting.)

3. Open Threads should be posted in Discussion, and not Main.

4. Open Threads should start on Monday, and end on Sunday.

LessWrongers who like GiveWell recommended charities may be interested in taking part in or donating to a new fundraiser called Experience Poverty. As well as deworming hundreds of children, it's an unusually good opportunity to find people who are interested in cost-effective charity and start conversations with them - something it's rare to find an excuse to do. Dozens of people and many effective altruist groups around the world are going to take part. Here are the details:

What: Spend only $2.50 a day on food for three days. Get sponsored to raise money to fight poverty.
Why: Half of the world spends $2.50 or less on food each day. This reflects income levels at which people often can’t afford basic health care. All money raised goes to buy medicines that cost only 50 cents. It is so cheap to us, but people still cannot afford it.
Who: People who want to do something to help the global poor and get a sense of what poverty's like.
When: April 22-24, 2015. If those dates don’t work for you you can set your own. For example, if you're at a US university you can choose to do it on April 6-8, when some American college groups are doing it.
What you can vary: Any of this: the amount you spend, the number of days and the date. For example, see below for the ‘challenge mode’ of spending only $1.50.
How: Sign up via these links for the US, Canada, the UK, Australia, Switzerland, Sweden and the Eurozone. Then ask friends and family to sponsor you. We’ll send you guides, pointers, and we can even do a one-on-one Skype to help you help the most people possible. Just contact us at info@charityscience.com with “Experience Poverty Skype” as the subject line.


Left: What a typical American family eats in a week. Right: What a typical family in Chad (an African country) eats in a week.

Where does the money go? SCI

The money raised goes to the Schistosomiasis Control Initiative (SCI for short). Experience Poverty is a partnership between Charity Science and SCI. SCI gives out medicine that cures children who have parasitic worms (including those that cause the disease schistosomiasis). These parasites live inside your body and suck your blood. The effects include fatigue, bloody diarrhoea, fever, chills, and other such lovely symptoms. Rather understandably, being so ill leads to many sick days from school and work. In fact, this is why deworming is arguably one of the most cost-effective ways to give children an education, among other things (Miguel and Kremer 2004 found that it led to a 25% reduction in school absenteeism, although this is a difficult effect to pin down; for more details, see this from GiveWell).

There isn't really a single figure which gives a good sense of SCI's strength - the closest would an estimate of its cost per QALY, but I believe GiveWell sees extant estimates of this as problematic. (In our own marketing we use the figure for the cost of the drugs - around 50 cents - but this is purely to maximise the amount raised.) By far the strongest reason to raise money for SCI is that it's been picked as one of the world's most effective charities by GiveWell, the most reliable and empirical charity evaluator by a long way. For more information see GiveWell's detailed review, or Charity Science's simpler summary. 

How do you join? - Click Here

Click here then follow the prompts. You set your goal for how much you’d like to raise (we suggest setting it as $500 or more), write about why you’re doing it, and then it’s as simple as asking your friends and family to sponsor you. Most people will be touched by your dedication to helping those less fortunate and will be happy to help you out.

Or you can just donate to the campaign! 50 cents gives a treatment to a child, so just $20 treats 40 children.

If you'd like to use a currency other than US dollars, you can use our pages for Canada, the UK, Australia, Switzerland, Sweden and the Eurozone.

What could you possibly eat for only $2.50 a day?

Not much. But that’s kind of the point. It’s to get a sense of how limiting $2.50 is. It’s like running a marathon for charity, but a lot more related.

However, it’s not all abstinence. (That’s reserved for the people taking the challenge mode who will spend only $1.50 a day.) Here’s a few cheap meals people have tried in the past:

• Rice, beans, and spices or soy sauce
• Butternut squash pasta
• Ramen noodles (always a guilty pleasure anyway)
• Oatmeal with chopped banana

Things you’ll have to give up:

• Starbucks or buying coffee out
• Restaurants
• Alcoholic drinks

What are the rules?

The spirit of the event involves getting a (very rough) experience of poverty, and to ultimately make it so that fewer people have to have similar experiences. As long as it is done in this spirit, do what works for you. If you need to spend a bit more money, you can set your own amount. If you can’t do it on April 22-24, do a different time. If you lapse during your three days, it’s OK. You might even want to mention it on Facebook, talking about how difficult it was. It will help people understand how hard poverty is.

Why $2.50?

Half of the world lives on less than $1,368 a year. Around 65% of that is spent on food, which means $2.43 a day. We rounded to $2.50 a day because round numbers are nicer, but you can spend only $2.43 if you’d like.

You might have traveled to a poor country and know that you can buy more for $2.50 overseas. This is a good point, which is why the $2.43 figure is adjusted to how much $2.43 could buy in the USA in 2005. That’s not much. Now imagine that you have to use the remaining $1.75 per day on shelter, transport, healthcare, and entertainment. That’s why we’re running this campaign. Because that’s just too little and we want to change it.

Challenge Mode: live on $1.50

Yawn. $2.50? You already did that in college. Well worry not my frugal friend, there is a challenge mode! $1.50 a day is roughly the international poverty line, so why not try to live on only that for 3 days?

Why do the challenge mode?

• People raise more when they suffer more. Yes, they did a study on this, and yes, people are strange creatures.
• You’ll get a better sense of what extreme poverty is like.
• You’ll get an e-high five from the Experience Poverty team.

What if I live outside the US?

We have separate pages with different currencies for the following countries:

• The US
• Canada
• The UK
• Australia
• Switzerland
• Sweden
• The Eurozone

If you don’t live in any of those countries then you can either join the US page which will let your friends donate in dollars or you can email us to see if we can add your country. Donations are tax-deductible in the US and Canada and SCI will get GiftAid in the UK.

If it's worth saying, but not worth its own post (even in Discussion), then it goes here.

Notes for future OT posters:

1. Please add the 'open_thread' tag.

2. Check if there is an active Open Thread before posting a new one. (Immediately before; refresh the list-of-threads page before posting.)

3. Open Threads should be posted in Discussion, and not Main.

4. Open Threads should start on Monday, and end on Sunday.

I forgot about it until after were were past the Hall's birthday, so the survey is late this year and won't run for as long. Nevertheless, for those of you that use the Less Wrong Study Hall, here is this year's census:

2015 LWSH Census and Survey

It will close on April 7th and I'll post the results a few weeks after that. I'll be advertising it during breaks in the Hall; I encourage others to do the same to maximize turnout.