Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Comment author: Unnamed 14 April 2017 09:01:13PM 0 points [-]

I'd like to see some changes to the CFAR-related questions; I've sent a PM with details.

Comment author: gjm 15 March 2017 11:11:04PM 3 points [-]

On #2, I've seen it claimed -- but have no idea how good the science behind it is -- that better than visualizing positive or negative outcomes alone is doing both and paying attention to the contrast. "If I do X, then the result will look like Y. If I don't do X, the result will look like Z. Wow, Y is much better than Z: better get on with doing X".

Comment author: Unnamed 15 March 2017 11:40:48PM 3 points [-]

The keyword for that research is mental contrasting. It was previously discussed on LW here.

My impression is that the quality of the science is relatively good, compared to other psychology research that was done in 2000-2012. But as far as I know it has not yet been tested with the improved research methods that have come out of the replication crisis (e.g., I don't know of any large sample size, preregistered studies of mental contrasting).

Comment author: Bound_up 04 March 2017 02:34:25AM 1 point [-]

Double Crux was largely a re-invention of Street Epistemology

You can find people to practice it with at the Street Epistemology Facebook group. They're having role-play sessions, making how-to videos, etc.

streetepistemology.com

Comment author: Unnamed 07 March 2017 05:51:15AM 1 point [-]

(This is Dan from CFAR)

This is the first I've heard of Street Epistemology, or Boghossian's book A Manual for Creating Atheists where it was apparently introduced. A key difference between it and Double Crux:

From their guide, it looks like Street Epistemology is intended to be an asymmetric game. Only player A knows about Street Epistemology, player A chooses to start the conversation about a topic where player A is confident that they are right and player B is wrong, and the conversation is about the reasons for player B's beliefs. Player A attempts to change player B's mind by improving player B's epistemology. Player A needn't talk about their own beliefs; there is a short subsection in the guide which addresses this topic, beginning "If asked about your own beliefs you should be prepared to answer." The guide describes Street Epistemology as being "most useful for extraordinary claims, such as miracles and supernatural phenomena."

Double Crux is intended to be a symmetric game, where both players know what kind of conversation they're getting into and both players put their beliefs (and the reasons for their beliefs) on the table in an attempt to improve their models. The object of the game (as its name suggests) is to find a crux that is shared by both players, where either of them would change their mind about the original disagreement if they changed their mind about the crucial point. I previously described Double Crux as being most useful for tricky, important-to-you questions where "digging into your own thinking and the other person's thinking on the topic is one of the more promising options available for making progress towards figuring out something that you care about."

Comment author: ChristianKl 28 February 2017 08:23:45AM 0 points [-]

Do you know how many people who participate in the CFAR focusing workshop got Focusing enough to fell a felt shift?

Comment author: Unnamed 28 February 2017 10:14:03AM 5 points [-]

About 60%.

More specifically: At the February workshop, 65% of participants filled out the optional data collection handout at the end of the hour-long Focusing class. Of the participants who filled it out, 60% circled 6 or higher in response to the question Did you experience a "felt shift"? (0 = not at all, 10 = yes, definitely).

(This is Dan from CFAR.)

In response to The Semiotic Fallacy
Comment author: Unnamed 22 February 2017 08:23:10PM 1 point [-]
Comment author: Unnamed 20 February 2017 07:38:33AM 4 points [-]

I'd bet on 6. I have information that the market doesn't have, and my information points to 6 as the answer, so the market is underpricing 6 (compared to how it would price 6 if it had all the information).

Another way to think of it: suppose that, instead of a market, there was just a single person looking at all the die rolls and updating using Bayes's Rule. There have been n rolls and that person has assigned the appropriate probabilities to each of the possible die weightings. Then the n+1th roll is a 6. The person then updates their probability assignments to give 6 a higher chance of being the favored side.

If the prediction market is efficient, then it should be analogous to this situation. The market price reflects the first n rolls, and now I know that the n+1th roll was a 6, so I get to profit (in expectation) by updating the market's probabilities to take that new piece of information into account.

It may be possible to give a more precise answer, but this is what I have for now.

Comment author: Unnamed 20 February 2017 09:47:54PM 1 point [-]

It may be possible to give a more precise answer, but this is what I have for now.

AlexMennen and Oscar_Cunningham have run the numbers and gotten that more precise answer. I did some calculations myself and agree with them. If the market has been efficiently incorporating information, then the prior die rolls included k+1 rolls of 3, and k rolls of each of the other numbers (this gives 1:1:5:1:1:1 odds regardless of k). My roll brings it up to k+1 6's, so the odds should now be 1:1:5:1:1:5 (i.e., 1/14 for most numbers and 5/14 for 3 and 6).

This is assuming that the market is basically just doing Bayesian updating; it's possible that there are some more complicated things happening with the market which make it a bad idea to make this assumption.

Comment author: Unnamed 20 February 2017 07:38:33AM 4 points [-]

I'd bet on 6. I have information that the market doesn't have, and my information points to 6 as the answer, so the market is underpricing 6 (compared to how it would price 6 if it had all the information).

Another way to think of it: suppose that, instead of a market, there was just a single person looking at all the die rolls and updating using Bayes's Rule. There have been n rolls and that person has assigned the appropriate probabilities to each of the possible die weightings. Then the n+1th roll is a 6. The person then updates their probability assignments to give 6 a higher chance of being the favored side.

If the prediction market is efficient, then it should be analogous to this situation. The market price reflects the first n rolls, and now I know that the n+1th roll was a 6, so I get to profit (in expectation) by updating the market's probabilities to take that new piece of information into account.

It may be possible to give a more precise answer, but this is what I have for now.

[Link] Case Studies Highlighting CFAR’s Impact on Existential Risk

4 Unnamed 10 January 2017 06:51PM
Comment author: Douglas_Knight 01 January 2017 11:42:19PM 0 points [-]

Most questions don't have a preferred direction. Look at Scott's predictions. Which direction should you point each one?

Most people don't make enough predictions to get a statistically significant difference between the two sides of the scale. And even if they do, their bias to the extremes ("overconfidence") swamps the effect.

Comment author: Unnamed 02 January 2017 12:57:47AM *  0 points [-]

Just looking at the 50% questions, here is how I would score 1) if either direction is an event rather than the default and 2) if either direction is probably preferred by Scott:

US unemployment to be lower at end of year than beginning: 50%

Neither direction is an event, Yes is preferred.

SpaceX successfully launches a reused rocket: 50%

Yes is an event, Yes is preferred.

California’s drought not officially declared over: 50%

No is an event, No is preferred.

At least one SSC post > 100,000 hits: 50%

Yes is an event, Yes is preferred.

UNSONG will get > 1,000,000 hits: 50%

Yes is an event, Yes is preferred.

UNSONG will not miss any updates: 50%

No is an event, Yes is preferred.

I will be involved in at least one published/accepted-to-publish research paper by the end of 2016: 50%

Yes is an event, Yes is preferred.

[Over] 10,000 Twitter followers by end of this year: 50%

Yes is an event, Yes is preferred.

I will not get any new girlfriends: 50%

No is an event, perhaps No is preferred.

I will score 95th percentile or above in next year’s PRITE: 50%

Yes is an event, Yes is preferred.

I will not have any inpatient rotations: 50%

No is an event, perhaps Yes is preferred.

I get at least one article published on a major site like Huffington Post or Vox or New Statesman or something: 50%

Yes is an event, Yes is preferred.

I don’t attend any weddings this year: 50%

No is an event, perhaps No is preferred.

Scott would know better than I do, and he also could have marked a subset that he actually cared about.

Including the "perhaps"es, I count that 7/12 happened in the preferred direction, and 5/11 of the events happened. With this small sample there's no sign of optimism bias, and he's also well-calibrated on whether a non-default event will happen. Obviously you'd want to do this with the full set of questions and not just the 50% ones to get a more meaningful sample size.

Comment author: Douglas_Knight 01 January 2017 06:27:04PM *  1 point [-]

You can do calibration and accuracy. You can start with predictions of arbitrary granularity and then force them into whatever boxes you want.

For calibration, it isn't very useful to score events at 50%. Instead of making boxes of 50, 60, 70, 80, 90, 95, 99%, you should instead do something like 55, 70, 80, 90, 95, 99%. Taking an event that you "really" think is 50/50 and forcing yourself to choose a side to make it 45/55 is no worse than taking an event that you think is 45/55 and forcing it to be either 50 or 60%.

Also, the jump from 95 to 99 is pretty big. Better to add an intermediate category of 97 or 98. Or just replace 99 with 98.


I think 60, 80, 90, 95, 98 would be a good set of bins for beginners.

Comment author: Unnamed 01 January 2017 07:45:24PM 2 points [-]

50% predictions can be useful if you are systematic about which option you count as "yes". e.g., "I estimate a 50% chance that I will finish writing my book this year" is a meaningful prediction. If I am subject to standard biases, then we would expect this to have less than a 50% chance of happening, so the outcomes of predictions like this provide a meaningful test of my prediction ability.

2 conventions you could use for 50% predictions: 1) pose the question such that "yes" means an event happened and "no" is the default, or 2) pose the question such that "yes" is your preferred outcome and "no" is the less desirable outcome.

Actually, it is probably better to pick one of these conventions and use it for all predictions (so you'd use the whole range from 0-100, rather than just the top half of 50-100). "70% chance I will finish my book" is meaningfully different than "70% chance I will not finish my book"; we are throwing away information about possible miscalibrated by treating them both merely as 70% predictions.

Even better, you could pose the question however you like and also note when you make your prediction 1) which outcome (if either) is an event rather than the default and 2) which outcome (if either) you prefer. Then at the end of the year you could look at 3 graphs, one which looks at whether the outcome that you considered more likely occurred, one that looks at whether the (non-default) event occurred, and one which looks at whether your preferred outcome occurred.

View more: Next