I feel safe saying that nearly everyone reading this will agree that, given sufficient technology, a perfect replica or simulation could be made of the structure and function of a human brain, producing an exact copy of an individual mind including a consciousness.  Upon coming into existence, this consciousness will have a separate but baseline-identical subjective experience to the consciousness from which it was copied, as it was at the moment of copying.  The original consciousness will continue its own existence/ subjective experience.  If the brain containing the original consciousness is destroyed, the consciousness within ceases to be.  The existence or non- of a copy is irrelevant to this fact.

With this in mind, I fail to see the attraction of the many transhuman options for extra-meat existence, and I see no meaningful immortality therein, if that's what you came for.

Consciousness is notoriously difficult to define and analyze and I am far from an expert in it's study.  I define it as an awareness: the sense organ which perceives the activity of the mind.  It is not thought.  It is not memory or emotion.  It is the thing that experiences or senses these things.  Memories will be gained and lost, thoughts and emotions come and go, the sense of self remains even as the self changes.  There exists a system of anatomical structures in your brain which, by means of electrochemical activity, produces the experience of consciousness.  If a brain injury wiped out major cognitive functions but left those structures involved in the sense of consciousness unharmed, you would, I believe, have the same central awareness of Self as Self, despite perhaps lacking all language or even the ability to form thoughts or understand to world around you.  Consciousness, this awareness, is, I believe, the most accurate definition of Self, Me, You.  I realize this sort of terminology has the potential to sound like mystical woo.  I believe this is due to the twin effects of the inherent difficulty in defining and discussing consciousness, and of our socialization wherein these sorts of discussions are more often than not heard from Buddhists or Sufis, whose philosophical traditions have looked into the matter with greater rigor for a longer time than Western philosophy, and Hippies and Druggies who introduced these traditions to our popular culture.  I am not speaking of a magical soul.  I am speaking of a central feature of the human experience which is a product of the anatomy and physiology of the brain.

Consider the cryonic head-freeze. Ideally, the scanned dead brain, cloned, remade and restarted (or whatever) will be capable of generating a perfectly functional consciousness, and it will feel as if it is the same consciousness which observes the mind which is, for instance, reading these words; but it will not be. The consciousness which is experiencing awareness of the mind which is reading these words will no longer exist. To disagree with this statement is to say that a scanned living brain, cloned, remade and started will contain the exact same consciousness, not similar, the exact same thing itself, that simultaneously exists in the still-living original.  If consciousness has an anatomical location, and therefore is tied to matter, then it would follow that this matter here is the exact matter as that separate matter there. This is an absurd proposition.  If consciousness does not have an anatomical / physical location then it is the stuff of magic and woo.

*Aside: I believe that consciousness, mind, thought, and memory are products not only of anatomy but of physiology, that is to say the ongoing electrochemical state of the brain, the constant flux of charge in and across neurons.  In perfect cryonic storage, the anatomy (hardware) might be maintained, but I doubt the physiology (software), in the form of exact moment-in-time membrane electrical potentials and intra-and extra-cellular ion concentrations for every neuron, will be.  Therefore I hold no faith in its utility, in addition to my indifference to the existence of a me-like being in the future.

Consider the Back-Up. Before lava rafting on your orbital, you have your brain scanned by your local AI so that a copy of your mind at that moment is saved.  In your fiery death in an unforeseen accident, will the mind observed by the consciousness on the raft experience anything differently than if it were not backed up? I doubt I would feel much consolation, other than knowing my loved ones were being cared for.  Not unlike a life insurance policy: not for one's own benefit.  I image the experience would be one of coming to the conclusion of a cruel joke at one's own expense.  Death in the shadow of a promise of immortality.  In any event, the consciousness that left the brain scanner and got on the raft is destroyed when the brain is destroyed, it benefits not at all from the reboot.

Consider the Upload.  You plug in for a brain scan, a digital-world copy of your consciousness is made, and then you are still just you.  You know there is a digital copy of you, that feels as if it is you, feels exactly as you would feel were it you who had travelled to the digital-world, and it is having a wonderful time, but there you still are. You are still just you in your meat brain.  The alternative, of course, is that your brain is destroyed in the scan in which case you are dead and something that feels as if it is you is having a wonderful time.  It would be a mercy killing.

If the consciousness that is me is perfectly analyzed and a copy created, in any medium, that process is external to the consciousness that is me.  The consciousness that is me, that is you reading this, will have no experience of being that copy, although that copy will have a perfect memory of having been the consciousness that is you reading this.  Personally, I don't know that I care about that copy.  I suppose he could be my ally in life.  He could work to achieve any altruistic goals I think I have, perhaps better than I think that you could.  He might want to fuck my wife, though.  And might be jealous of the time she spends with me rather than him, and he'd probably feel entitled to all my stuff, as would I be vice versa. The Doppelganger and the Changeling have never been considered friendly beasts.

I have no firm idea where lines can be drawn on this.  Certainly consciousness can be said to be an intermittent phenomenon which the mind pieces together into the illusion of continuity.  I do not fear going to sleep at night, despite the "loss of consciousness" associated. If I were to wake up tomorrow and Omega assures me that I am a freshly made copy of the original, it wouldn't trouble me as to my sense of self, only to the set of problems associated with living in a world with a copy of myself.  I wouldn't mourn a dead original me any more than I'd care about a copy of me living on after I'm dead, I don't imagine.  

Would a slow cell by cell, or thought by thought / byte by byte, transfer of my mind to another medium: one at a time every new neural action potential is received by a parallel processing medium which takes over?  I want to say the resulting transfer would be the same consciousness as is typing this but then what if the same slow process were done to make a copy and not a transfer?  Once a consciousness is virtual, is every transfer from one medium or location to another not essentially a copy and therefore representing a death of the originating version?

It almost makes a materialist argument (self is tied to matter) seem like a spiritualist one (meat consciousness is soul is tied to human body at birth) which, of course, is weird place to be intellectually.

I am not addressing the utility or ethics or inevitability of the projection of the self-like-copy into some transhuman state of being, but I don't see any way around the conclusion that that the consciousness that is so immortalized will not be the consciousness that is writing these words, although it would feel exactly as if it were.  I don't think I care about that guy.  And I see no reason for him to be created. And if he were created, I, in my meat brain's death bed, would gain no solace from knowing he, a being which started out it's existence exactly like me, will live on.

EDIT: Lots of great responses, thank you all and keep them coming.  I want to bring up some of my responses so far to better define what I am talking about when I talk about consciousness.

I define consciousness as a passively aware thing, totally independent of memory, thoughts, feelings, and unconscious hardwired or conditioned responses. It is the hard-to-get-at thing inside the mind which is aware of the activity of the mind without itself thinking, feeling, remembering, or responding. The demented, the delirious, the brain damaged all have (unless those brain structures performing the function of consciousness are damaged, which is not a given) the same consciousness, the same Self, the same I and You, as I define it, as they did when their brains were intact. Dream Self is the same Self as Waking Self to my thinking. I assume consciousness arises at some point in infancy. From that moment on it is Self, to my thinking.

If I lose every memory slowly and my personality changes because of this and I die senile in a hospital bed, I believe that it will be the same consciousness experiencing those events as is experiencing me writing these words. That is why many people choose suicide at some point on the path to dementia.

I recognize that not everyone reading this will agree that such a thing exists or has the primacy of existential value that I ascribe to it.

And an addendum:
Sophie Pascal's Choice (hoping it hasn't already been coined): Would any reward given to the surviving copy induce you to step onto David Bowie Tesla's Prestige Duplication Machine, knowing that your meat body and brain will be the one which falls into the drowning pool while an identical copy of you materializes 100m away, believing itself to be the same meat that walked into the machine and ready to accept the reward?

Background

In a recent Slate Star Codex Post (http://slatestarcodex.com/2016/01/02/2015-predictions-calibration-results/), Scott Alexander made a number of predictions and presented associated confidence levels, and then at the end of the year, scored his predictions in order to determine how well-calibrated he is. In the comments, however, there arose a controversy over how to deal with 50% confidence predictions. As an example, Scott has these predictions at 50% confidence, among his others:

Scott goes on to score himself as having made 0/3 correct predictions at the 50% confidence interval, which looks like significant overconfidence. He addresses this by noting that with only 3 data points it’s not much data to go by, and could easily have been correct if any of those results had turned out differently. His resulting calibration curve is this:

However, the commenters had other objections about the anomaly at 50%. After all, P(A) = 50% implies P(~A) = 50%, so the choice of “I will not get any new girlfriends: 50% confidence”  is logically equivalent to “I will get at least 1 new girlfriend: 50% confidence”, except that one results as true and the other false. Therefore, the question seems sensitive only to the particular phrasing chosen, independent of the outcome.

One commenter suggests that close to perfect calibration at 50% confidence can be achieved by choosing whether to represent propositions as positive or negative statements by flipping a fair coin. Another suggests replacing 50% confidence with 50.1% or some other number arbitrarily close to 50%, but not equal to it. Others suggest getting rid of the 50% confidence bin altogether.

Scott recognizes that predicting A and predicting ~A are logically equivalent, and choosing to use one or the other is arbitrary. But by choosing to only include A in his data set rather than ~A, he creates a problem that occurs when P(A) = 50%, where the arbitrary choice of making a prediction phrased as ~A would have changed the calibration results despite being the same prediction.

Symmetry

This conundrum illustrates an important point about these calibration exercises. Scott chooses all of his propositions to be in the form of statements to which he assigns greater or equal to 50% probability, by convention, recognizing that he doesn’t need to also do a calibration of probabilities less than 50%, as the upper-half of the calibration curve captures all the relevant information about his calibration.

This is because the calibration curve has a property of symmetry about the 50% mark, as implied by the mathematical relation P(X) = 1- P(~X) and of course P(~X) = 1 –P(X).

We can enforce that symmetry by recognizing that when we make the claim that proposition X has probability P(X), we are also simultaneously making the claim that proposition ~X has probability 1-P(X). So we add those to the list of predictions and do the bookkeeping on them too. Since we are making both claims, why not be clear about it in our bookkeeping?

When we do this, we get the full calibration curve, and the confusion about what to do about 50% probability disappears. Scott’s list of predictions looks like this:

You will by now have noticed that there will always be an even number of predictions, and that half of the predictions always are true and half are always false. In most cases, like with E and ~E, that means you get a 95% likely prediction that is true and a 5%-likely prediction that is false, which is what you would expect. However, with 50%-likely predictions, they are always accompanied by another 50% prediction, one of which is true and one of which is false. As a result, it is actually not possible to make a binary prediction at 50% confidence that is out of calibration.

The resulting calibration curve, applied to Scott’s predictions, looks like this:

Sensitivity

By the way, this graph doesn’t tell the whole calibration story; as Scott noted it’s still sensitive to how many predictions were made in each bucket. We can add “error bars” that show what would have resulted if Scott had made one more prediction in each bucket, and whether the result of that prediction had been true or false. The result is the following graph:

Note that the error bars are zero about the point of 0.5. That’s because even if one additional prediction had been added to that bucket, it would have had no effect. That point is fixed by the inherent symmetry.

I believe that this kind of graph does a better job of showing someone’s true calibration. But it's not the whole story.

Ramifications for scoring calibration (updated)

Clearly, it is not possible to make a binary prediction with 50% confidence that is poorly calibrated. This shouldn’t come as a surprise; a prediction at 50% between two choices represents the correct prior for the case where you have no information that discriminates between X and ~X. However, that doesn’t mean that you can improve your ability to make correct predictions just by giving them all 50% confidence and claiming impeccable calibration! An easy way to "cheat" your way into apparently good calibration is to take a large number of predictions that you are highly (>99%) confident about, negate a fraction of them, and falsely record a lower confidence for those. If we're going to measure calibration, we need a scoring method that will encourage people to write down the true probabilities they believe, rather than faking low confidence and ignoring their data. We want people to only claim 50% confidence when they genuinely have 50% confidence, and we need to make sure our scoring method encourages that.

A first guess would be to look at that graph and do the classic assessment of fit: sum of squared errors. We can sum the squared error of our predictions against the ideal linear calibration curve. If we did this, we would want to make sure we summed all the individual predictions, rather than the averages of the bins, so that the binning process itself doesn’t bias our score.

If we do this, then our overall prediction score can be summarized by one number:

Here P(X_i) is the assigned confidence of the truth of X_i, and X_i is the ith proposition and has a value of 1 if it is True and 0 if it is False. S is the prediction score, and lower is better. Note that because these are binary predictions, the sum of squared errors gives an optimal score if you assign the probabilities you actually believe (ie, there is no way to "cheat" your way to a better score by giving false confidence).

In this case, Scott's score is S=0.139, much of this comes from the 0.4/0.6 bracket. The worst score possible would be S=1, and the best score possible is S=0. Attempting to fake a perfect calibration by everything by claiming 50% confidence for every prediction, regardless of the information you actually have available, yields S=0.25 and therefore isn't a particularly good strategy (at least, it won't make you look better-calibrated than Scott).

Several of the commenters pointed out that log scoring is another scoring rule that works better in the general case. Before posting this I ran the calculus to confirm that the least-squares error did encourage an optimal strategy of honest reporting of confidence, but I did have a feeling that it was an ad-hoc scoring rule and that there must be better ones out there.

The logarithmic scoring rule looks like this:

Here again X_i is the ith proposition and has a value of 1 if it is True and 0 if it is False. The base of the logarithm is arbitrary so I've chosen base "e" as it makes it easier to take derivatives. This scoring method gives a negative number and the closer to zero the better. The log scoring rule has the same honesty-encouraging properties as the sum-of-squared-errors, plus the additional nice property that it penalizes wrong predictions of 100% or 0% confidence with an appropriate score of minus-infinity. When you claim 100% confidence and are wrong, you are infinitely wrong. Don't claim 100% confidence!

In this case, Scott's score is calculated to be S=-0.42. For reference, the worst possible score would be minus-infinity, and claiming nothing but 50% confidence for every prediction results in a score of S=-0.69. This just goes to show that you can't win by cheating.

Example: Pretend underconfidence to fake good calibration

In an attempt to appear like I have better calibration than Scott Alexander, I am going to make the following predictions. For clarity I have included the inverse propositions in the list (as those are also predictions that I am making), but at the end of the list so you can see the point I am getting at a bit better.

Look carefully at this table. I've thrown in a particular mix of predictions that I will or will not win the lottery on certain days, in order to use my extreme certainty about the result to generate a particular mix of correct and incorrect predictions.

To make things even easier for me, I’m not even planning to buy any lottery tickets. Knowing this information, an honest estimate of the odds of me winning the lottery are astronomically small. The odds of winning the lottery are about  1 in 14 million (for the Canadian 6/49 lottery). I’d have to win by accident (one of my relatives buying me a lottery ticket?). Not only that, but since the lottery is only held on Wednesday and Saturday, that makes most of these scenarios even more implausible since the lottery corporation would have to hold the draw by mistake.

I am confident I could make at least 1 billion similar statements of this exact nature and get them all right, so my true confidence must be upwards of (100% - 0.0000001%).

If I assemble 50 of these types of strategically-underconfident predictions (and their 50 opposites) and plot them on a graph, here’s what I get:

You can see that the problem with cheating doesn’t occur only at 50%. It can occur anywhere!

But here’s the trick: The log scoring algorithm rates me -0.37. If I had made the same 100 predictions all at my true confidence (99.9999999%), then my score would have been -0.000000001. A much better score! My attempt to cheat in order to make a pretty graph has only sabotaged my score.

By the way, what if I had gotten one of those wrong, and actually won the lottery one of those times without even buying a ticket? In that case my score is -0.41 (the wrong prediction had a probability of 1 in 10^9 which is about 1 in e^21, so it’s worth -21 points, but then that averages down to -0.41 due to the 49 correct predictions that are collectively worth a negligible fraction of a point).* Not terrible! The log scoring rule is pretty gentle about being very badly wrong sometimes, just as long as you aren’t infinitely wrong. However, if I had been a little less confident and said the chance of winning each time was only 1 in a million, rather than 1 in a billion, my score would have improved to -0.28, and if I had expressed only 98% confidence I would have scored -0.098, the best possible score for someone who is wrong one in every fifty times.

This has another important ramification: If you're going to honestly test your calibration, you shouldn't pick the predictions you'll make. It is easy to improve your score by throwing in a couple predictions that you are very certain about, like that you won't win the lottery, and by making few predictions that you are genuinely uncertain about. It is fairer to use a list of propositions that is generated by somebody else, and then pick your probabilities. Scott demonstrates his honesty by making public predictions about a mix of things he was genuinely uncertain about, but if he wanted to cook his way to a better score in the future, he would avoid making any predictions at the 50% category that he wasn't forced to.

Input and comments are welcome! Let me know what you think!

* This result surprises me enough that I would appreciate if someone in the comments can double-check it on their own. What is the proper score for being right 49 times with 1-1 in a billion certainty, but wrong once?

This is intended to be a pretty broad discussion of sports. I have some thoughts, but feel free to start your own threads.

tl;dr - My impression is that people here aren't very interested in sports. My impression¹ is that most people have something to gain by both competitive and recreational sports. With competitive sports you have to be careful not to overdo it. With recreational sports, the circumstances have to be right for it to be enjoyable. I also think that sports get a bad rep for being simple and dull. In actuality, there's a lot of complexity. 

1 - Why does this have to sound bad?! I have two statements I want to make. And for each of them, I want to qualify it by saying that it as an impression that I have. What is a better way to say this? 

Me

I love sports. Particularly basketball. I was extremely extremely dedicated to it back in middle/high school. Actually, it was pretty much all I cared about (not an exaggeration). This may or not be crazy... but I wanted to be the best player who's ever lived. That was what I genuinely aspired and was working towards (~7th-11th grade).

My thinking: the pros practice, what, 5-6 hours a day? I don't care about anything other than basketball. I'm willing to practice 14 hours a day! I just need time to eat and sleep, but other than that, I value basketball above all else (friends, school...). Plus, I will work so much smarter than they do! The norm is to mindlessly do push ups and eat McDonalds. I will read the scientific literature and figure out what the most effective ways to improve are. I'm short and not too athletic, so I knew I was starting at a disadvantage, but I saw a mismatch between what the norm is and what my rate of improvement could be. I thought I could do it.

In some ways I succeeded, but ultimately I didn't come close to my goal of greatness. In short, I spent too much time on high level actions such as researching training methods and not enough time on object level work; and with school and homework, I simply didn't have enough time to put in the 14 hour days I envisioned. I was a solid high school player, but was no where near good enough to play college ball.

Take Aways

Intense work. I've gone through some pretty intense physical exercise. Ex. running suicides until you collapse. And then getting up to do more until you collapse again. It takes a lot of willpower to do that. I think willpower is like a muscle, and you have to train yourself to be able to work at such intensities. I haven't experienced anything intellectual that has required such intensity. Knowing that I am capable of working at high intensities has given me confidence that "I could do anything".

Ambition. The culture in athletic circles is often one where, "I'm not content being where I am". There's someone above you, and you want to beat them out. I guess that sort of exists in academic and career circles as well, but I don't think it's the same (in the average case; there's certainly exceptions). What explains this? Maybe there's something very visceral about lining up across from someone, getting physically and unambiguously beaten, and letting your teammates and yourself down.

Confidence. Often times, confidence is something you learn because you have to. Often times, if you're not confident, you won't perform, so you need to learn to be confident. But it's not just that; there's something else about the culture that promotes confidence (perhaps cockiness). Think: "I don't care who the opponent is, no one can stop me!".

Group Bonds. When you spend so much time with a group of people, go through exhausting practices together, and work as a team to experience wins and losses, you develop a certain bond that is enjoyable. It reminds me a bit of putting in long hours on a project and eventually meeting the deadline, but it isn't the same.

Other: There's certainly other things I'm forgetting.

All of that said, there are downsides that correspond with all of these benefits. My overarching opinion is "all things in moderation". Ambition can be poison. So can the habitual productivity that often comes with ambition. Sometimes the atmosphere can backfire and make you less confident. And sometimes teammates can bully and be cruel. I've experienced the good and bad extremes along all of these axes.

Honestly, I'm not quite sure when it's worth it and when it isn't. I think it often depends on the person and the situation, but I think that in moderation, most people have a decent amount to gain (in aggregate) by experiencing these things.

Recreational

So far I've really only talked about competitive sports. Now I want to talk about recreational sports. With competitive sports, as I mention above, I think there's a somewhat fine line between underdoing it and overdoing it. But I think that line is a lot wider for recreational sports. I think it's wide enough such that recreational sports are very often a good choice.

One huge benefit of recreational sports is that it's a fun way to get exercise. You do/should exercise anyway; why not make a game out of it?

Part of me feels like sports are just inherently fun! I know that calling them inherently fun is too strong a statement, but I think that under the right circumstances, they often are fun (I think the same point can be applied to most other things as well).

In practice, what goes wrong?

• You aren't in shape. You're playing a pick up basketball game where everyone else is running up and down the court and you're too winded to breathe. That's no fun.
• Physical bumps and bruises. You're playing football and get knocked around, or perhaps injured.
• Lack of involvement.
  • You're playing baseball. You only get to hit 1/18th of the time. And you are playing right field and no one ever hits it to you (for these reasons, I don't like baseball).
  • You're playing soccer with people who don't know how to space the field and move the ball, and you happen to get excluded.
  • You're playing basketball where each team has a ball hog who brings up the ball and shoots it every possession.
• Difficulty-skill mismatch. You're playing with people who are way too good for you, so it isn't fun. Alternatively, maybe you're way better than the people you're playing with and aren't being challenged.
• Other. Again, I'm sure there are things I'm not thinking of.

Complexity

I sense that sports get a bit of an unfair rep for being simple and dull games. Maybe some are, but I think that most aren't.

Perhaps it's because of the way most people experience the game. Take basketball as an example. A lot of people just like to watch to see whether the ball goes in the hoop or not and cheer. Ie. they experience the game in a very binary way. Observing this, it may be tempting to think, "Ugh, what a stupid game." But what happens when you steelman?

I happen to know a lot about basketball, so I experience the game very differently. Here's an example:

Iguodala has the ball and is being guarded by LeBron. LeBron is playing close and is in a staggered stance. He's vulnerable and Iguodala should attack his lead foot. People (even NBA players) don't look at this enough! Actually no, he shouldn't attack: the weak side help defense looks like it's in position, and LeBron is great at recovery. Plus, you have to think about the opportunity cost. Curry has Dellavedova and could definitely take him. Meaning, if Delly plays off, Curry can take a shot, but if Delly plays him more tightly, Curry could penetrate and either score or set someone else up, depending on how the help defense reacts. That approach has a pretty high expected value. But actually, Draymond Green looks like he has JR Smith on him (who is much smaller), which probably has an even higher expected value than Curry taking Delly. But to get Green the ball they'd have to reverse it to the weak side, and they'd have to keep the court spaced such that the Cavs won't have an opportunity to switch a bigger defender on to Green. All of this is in contrast with running a motion offense or some set plays. And you also have to take into account the stamina of the other team. Maybe you want to attack LeBron on defense to make him work, get him tired, and make him less effective on offense (I think this is a great approach to take against Curry and the Warriors, because Curry isn't a good defender and is lethal on offense).

Hopefully you could see that the amount of information there is to process in any given second is extremely high! If you know what to look for. Personally, I've never played organized football. But after playing the video game Madden (and doing some further research), I've learned a good amount about how the game works. Now when I watch football, I know the intricacies of the game and am watching for them. The density of information + the excitement, skill and physicality makes these ports extremely enjoyable for me to watch. Alternatively, I don't know too much about golf and don't enjoy watching it. All I see when I watch golf is, "The ball was hit closer to the hole... the ball was hit closer to the hole... the ball was it in the hole. This was a par 3, so that must have been an average performance."

Hi, I'm an entrepreneur looking for a startup idea.

In my experience, the reason most startups fail is because they never actually solve anyone's problem. So I'm cheating and starting out by identifying a specific person with a specific problem.

So I'm asking you, what's the most annoying part of your life/job? Also, how much would you pay for a solution?

MIRI is running an AMA on the Effective Altruism Forum tomorrow (Wednesday, Oct. 11): Ask MIRI Anything. Questions are welcome in the interim!

Nate also recently posted a more detailed version of our 2016 fundraising pitch to the EA Forum. One of the additions is about our first funding target:

We feel reasonably good about our chance of hitting target 1, but it isn't a sure thing; we'll probably need to see support from new donors in order to hit our target, to offset the fact that a few of our regular donors are giving less than usual this year.

The Why MIRI's Approach? section also touches on new topics that we haven't talked about in much detail in the past, but plan to write up some blog posts about in the future. In particular:

Loosely speaking, we can imagine the space of all smarter-than-human AI systems as an extremely wide and heterogeneous space, in which "alignable AI designs" is a small and narrow target (and "aligned AI designs" smaller and narrower still). I think that the most important thing a marginal alignment researcher can do today is help ensure that the first generally intelligent systems humans design are in the “alignable” region. I think that this is unlikely to happen unless researchers have a fairly principled understanding of how the systems they're developing reason, and how that reasoning connects to the intended objectives.

Most of our work is therefore aimed at seeding the field with ideas that may inspire more AI research in the vicinity of (what we expect to be) alignable AI designs. When the first general reasoning machines are developed, we want the developers to be sampling from a space of designs and techniques that are more understandable and reliable than what’s possible in AI today.

In other news, we've uploaded a new intro talk on our most recent result, "Logical Induction," that goes into more of the technical details than our previous talk.

See also Shtetl-Optimized and n-Category Café for recent discussions of the paper.

"A bit over four years ago I wrote a glowing review of Daniel Kahneman’s Thinking, Fast and Slow. I described it as a “magnificent book” and “one of the best books I have read”. I praised the way Kahneman threaded his story around the System 1 / System 2 dichotomy, and the coherence provided  by prospect theory.

What a difference four years makes. I will still describe Thinking, Fast and Slow as an excellent book – possibly the best behavioural science book available. But during that time a combination of my learning path and additional research in the behavioural sciences has led me to see Thinking, Fast and Slow as a book with many flaws."

Continued here: https://jasoncollins.org/2016/06/29/re-reading-kahnemans-thinking-fast-and-slow/

Many people are probably aware of the hack at DAO, using a bug in their smart contract system to steal millions of dollars worth of the crypto currency Ethereum.

There's various arguments as to whether this theft was technically allowed or not, and what should be done about it, and so on. Many people are arguing that the code is the contract, and that therefore no-one should be allowed to interfere with it - DAO just made a coding mistake, and are now being (deservedly?) punished for it.

That got me wondering whether its ever possible to make a smart contract without a full AI of some sort. For instance, if the contract is triggered by the delivery of physical goods - how can you define what the goods are, what constitutes delivery, what constitutes possession of them, and so on. You could have a human confirm delivery - but that's precisely the kind of judgement call you want to avoid. You could have an automated delivery confirmation system - but what happens if someone hacks or triggers that? You could connect it automatically with scanning headlines of media reports, but again, this is relying on aggregated human judgement, which could be hacked or influenced.

Digital goods seem more secure, as you can automate confirmation of delivery/services rendered, and so on. But, again, this leaves the confirmation process open to hacking. Which would be illegal, if you're going to profit from the hack. Hum...

This seems the most promising avenue for smart contracts that doesn't involve full AI: clear out the bugs in the code, then ground the confirmation procedure in such a way that it can only be hacked in a way that's already illegal. Sort of use the standard legal system as a backstop, fixing the basic assumptions, and then setting up the smart contracts on top of them (which is not the same as using the standard legal system within the contract).

Outline: I will discuss my background and how I prepared for the workshop, and then how I would have prepared differently if I could go back and have the chance to do it again; I will then discuss my experience at the CFAR workshop, and what I would have done differently if I had the chance to do it again; I will then discuss what my take-aways were from the workshop, and what I am doing to integrate CFAR strategies into my life; finally, I will give my assessment of its benefits and what other folks might expect to get who attend the workshop.

Acknowledgments: Thanks to fellow CFAR alumni and CFAR staff for feedback on earlier versions of this post

Introduction

Many aspiring rationalists have heard about the Center for Applied Rationality, an organization devoted to teaching applied rationality skills to help people improve their thinking, feeling, and behavior patterns. This nonprofit does so primarily through its intense workshops, and is funded by donations and revenue from its workshop. It fulfills its social mission through conducting rationality research and through giving discounted or free workshops to those people its staff judge as likely to help make the world a better place, mainly those associated with various Effective Altruist cause areas, especially existential risk.

To be fully transparent: even before attending the workshop, I already had a strong belief that CFAR is a great organization and have been a monthly donor to CFAR for years. So keep that in mind as you read my description of my experience (you can become a donor here).

Preparation

First, some background about myself, so you know where I’m coming from in attending the workshop. I’m a professor specializing in the intersection of history, psychology, behavioral economics, sociology, and cognitive neuroscience. I discovered the rationality movement several years ago through a combination of my research and attending a LessWrong meetup in Columbus, OH, and so come from a background of both academic and LW-style rationality. Since discovering the movement, I have become an activist in the movement as the President of Intentional Insights, a nonprofit devoted to popularizing rationality and effective altruism (see here for our EA work). So I came to the workshop with some training and knowledge of rationality, including some CFAR techniques.

To help myself prepare for the workshop, I reviewed existing posts about CFAR materials, with an eye toward being careful not to assume that the actual techniques match their actual descriptions in the posts.

I also delayed a number of tasks for after the workshop, tying up loose ends. In retrospect, I wish I did not leave myself some ongoing tasks to do during the workshop. As part of my leadership of InIn, I coordinate about 50ish volunteers, and I wish I had placed those responsibilities on someone else during the workshop.

Before the workshop, I worked intensely on finishing up some projects. In retrospect, it would have been better to get some rest and come to the workshop as fresh as possible.

There were some communication snafus with logistics details before the workshop. It all worked out in the end, but I would have told myself in retrospect to get the logistics hammered out in advance to not experience anxiety before the workshop about how to get there.

Experience

The classes were well put together, had interesting examples, and provided useful techniques. FYI, my experience in the workshop was that reading these techniques in advance was not harmful, but that the techniques in the CFAR classes were quite a bit better than the existing posts about them, so don’t assume you can get the same benefits from reading posts as attending the workshop. So while I was aware of the techniques, the ones in the classes definitely had optimized versions of them - maybe because of the “broken telephone” effect or maybe because CFAR optimized them from previous workshops, not sure. I was glad to learn that CFAR considers the workshop they gave us in May as satisfactory enough to scale up their workshops, while still improving their content over time.

Just as useful as the classes were the conversations held in between and after the official classes ended. Talking about them with fellow aspiring rationalists and seeing how they were thinking about applying these to their lives was helpful for sparking ideas about how to apply them to my life. The latter half of the CFAR workshop was especially great, as it focused on pairing off people and helping others figure out how to apply CFAR techniques to themselves and how to address various problems in their lives. It was especially helpful to have conversations with CFAR staff and trained volunteers, of whom there were plenty - probably about 20 volunteers/staff for the 50ish workshop attendees.

Another super-helpful aspect of the conversations was networking and community building. Now, this may have been more useful to some participants than others, so YMMV. As an activist in the moment, I talked to many folks in the CFAR workshop about promoting EA and rationality to a broad audience. I was happy to introduce some people to EA, with my most positive conversation there being to encourage someone to switch his efforts regarding x-risk from addressing nuclear disarmament to AI safety research as a means of addressing long/medium-term risk, and promoting rationality as a means of addressing short/medium-term risk. Others who were already familiar with EA were interested in ways of promoting it broadly, while some aspiring rationalists expressed enthusiasm over becoming rationality communicators.

Looking back at my experience, I wish I was more aware of the benefits of these conversations. I went to sleep early the first couple of nights, and I would have taken supplements to enable myself to stay awake and have conversations instead.

Take-Aways and Integration

The aspects of the workshop that I think will help me most were what CFAR staff called “5-second” strategies - brief tactics and techniques that could be executed in 5 seconds or less and address various problems. The stuff that we learned at the workshops that I was already familiar with required some time to learn and practice, such as Trigger Action Plans, Goal Factoring, Murphyjitsu, Pre-Hindsight, often with pen and paper as part of the work. However, with sufficient practice, one can develop brief techniques that mimic various aspects of the more thorough techniques, and apply them quickly to in-the-moment decision-making.

Now, this doesn’t mean that the longer techniques are not helpful. They are very important, but they are things I was already generally familiar with, and already practice. The 5-second versions were more of a revelation for me, and I anticipate will be more helpful for me as I did not know about them previously.

Now, CFAR does a very nice job of helping people integrate the techniques into daily life, as this is a common failure mode of CFAR attendees, with them going home and not practicing the techniques. So they have 6 Google Hangouts with CFAR staff and all attendees who want to participate, 4 one-on-one sessions with CFAR trained volunteers or staff, and they also pair you with one attendee for post-workshop conversations. I plan to take advantage of all these, although my pairing did not work out.

For integration of CFAR techniques into my life, I found the CFAR strategy of “Overlearning” especially helpful. Overlearning refers to trying to apply a single technique intensely for a while to all aspect of one’s activities, so that it gets internalized thoroughly. I will first focus on overlearning Trigger Action Plans, following the advice of CFAR.

I also plan to teach CFAR techniques in my local rationality dojo, as teaching is a great way to learn, naturally.

Finally, I plan to integrate some CFAR techniques into Intentional Insights content, at least the more simple techniques that are a good fit for the broad audience with which InIn is communicating.

Benefits

I have a strong probabilistic belief that having attended the workshop will improve my capacity to be a person who achieves my goals for doing good in the world. I anticipate I will be able to figure out better whether the projects I am taking on are the best uses of my time and energy. I will be more capable of avoiding procrastination and other forms of akrasia. I believe I will be more capable of making better plans, and acting on them well. I will also be more in touch with my emotions and intuitions, and be able to trust them more, as I will have more alignment among different components of my mind.

Another benefit is meeting the many other people at CFAR who have similar mindsets. Here in Columbus, we have a flourishing rationality community, but it’s still relatively small. Getting to know 70ish people, attendees and staff/volunteers, passionate about rationality was a blast. It was especially great to see people who were involved in creating new rationality strategies, something that I am engaged in myself in addition to popularizing rationality - it’s really heartening to envision how the rationality movement is growing.

These benefits should resonate strongly with those who are aspiring rationalists, but they are really important for EA participants as well. I think one of the best things that EA movement members can do is studying rationality, and it’s something we promote to the EA movement as part of InIn’s work. What we offer is articles and videos, but coming to a CFAR workshop is a much more intense and cohesive way of getting these benefits. Imagine all the good you can do for the world if you are better at planning, organizing, and enacting EA-related tasks. Rationality is what has helped me and other InIn participants make the major impact that we have been able to make, and there are a number of EA movement members who have rationality training and who reported similar benefits. Remember, as an EA participant, you can likely get a scholarship with a partial or full coverage of the regular $3900 price of the workshop, as I did myself when attending it, and you are highly likely to be able to save more lives as a result of attending the workshop over time, even if you have to pay some costs upfront.

Hope these thoughts prove helpful to you all, and please contact me at gleb@intentionalinsights.org if you want to chat with me about my experience.