LESSWRONG
LW

Comment Permalink
VincentYu50

Is there any information on how well-calibrated the community predictions are on Metaculus?

Great question! Yes. There was a post on the official Metaculus blog that addressed this, though this was back in Oct 2016. In the past, they've also sent to subscribed users a few emails that looked at community calibration.

I've actually done my own analysis on this around two months ago, in private communication. Let me just copy two of the plots I created and what I said there. You might want to ignore the plots and details, and just skip to the "brief summary" at the end.

(Questions on Metaculus go through an 'open' phase then a 'closed' phase; predictions can only be made and updated while the question is open. After a question closes, it gets resolved either positive or negative once the outcome is known. I based my analysis on the 71 questions that have been resolved as of 2 months ago; there are around 100 resolved questions now.)

First, here's a plot for the 71 final median predictions. The elements of this plot:

  • Of all monotonic functions, the black line is the one that, when applied to this set of median predictions, performs the best (in mean score) under every proper scoring rule given the realized outcomes. This can be interpreted as a histogram with adaptive bin widths. So for instance, the figure shows that, binned together, predictions from 14% to 45% resolved positive around 0.11 of the time. This is also the maximum-likelihood monotonic function.

  • The confidence bands are for the null hypothesis that the 71 predictions are all perfectly calibrated and independent, so that we can sample the distribution of counterfactual outcomes simply by treating the outcome of each prediction with credence p as an independent coin flip with probability p of positive resolution. I sampled 80,000 sets of these 71 outcomes, and built the confidence bands by computing the corresponding maximum-likelihood monotonic function for each set. The inner band is pointwise 1 sigma, whereas the outer is familywise 2 sigma. So the corner of the black line that exceeds the outer band around predictions of 45% is a p < 0.05 event under perfect calibration, and it looks to me that predictions around 30% to 40% are miscalibrated (underconfident).

  • The two rows of tick marks below the x-axis show the 71 predictions, with the upper green row comprising positive resolutions, and the lower red row comprising negatives.

  • The dotted blue line is a rough estimate of the proportion of questions resolving positive along the range of predictions, based on kernel density estimates of the distributions of predictions giving positive and negative resolutions.

Now, a plot of all 3723 final predictions on the 71 questions.

  • The black line is again the monotonic function that minimizes mean proper score, but with the 1% and 99% predictions removed because—as I expected—they were especially miscalibrated (overconfident) compared to nearby predictions.

  • The two black dots indicate the proportion of question resolving positive for 1% and 99% predictions (around 0.4 and 0.8).

  • I don't have any bands indicating dispersion here because these predictions are a correlated mess that I can't deal with. But for predictions below 20%, the deviation from the diagonal looks large enough that I think it shows miscalibration (overconfidence).

  • Along the x-axis I've plotted kernel density estimates of the predictions resolving positive (green, solid line) and negative (red, dotted line). Kernel densities were computed under log-odds with Gaussian kernels, then converted back to probabilities in [0, 1].

  • The blue dotted line is again a rough estimate of the proportion resolving positive, using these two density estimates.

Brief summary:

  • Median predictions around 30% to 40% occur less often than claimed.
  • User predictions below around 20% occur more often than claimed.
  • User predictions at 1% and 99% are obviously overconfident.
  • Other than these, calibration seems okay everywhere else; at least, they aren't obviously off.
  • I'm very surprised that user predictions look fairly accurate around 90% and 95% (resolving positive around 0.85 and 0.90 of the time). I expected strong overconfidence like that shown by the predictions below 20%.

Also, if one wanted to get into it, could you describe what your process is?

Is there anything in particular that you want to hear about? Or would you rather have a general description of 1) how I'd suggest starting out on Metaculus, and/or 2) how I approach making and updating predictions on the site, and/or 3) something else?

(The FAQ is handy for questions about the site. It's linked to by the 'help' button at the button of every page.)

Reply
See in context
by JenniferRM
1 min read
64

21

I'm interested from hearing from everyone who reads this.

Who is checking LW's Discussion area and how often?

1. When you check, how much voting or commenting do you do compared to reading?

2. Do bother clicking through to links?

3. Do you check using a desktop or a smart phone?  Do you just visit the website in browser or use an RSS something-or-other?

4. Also, do you know of other places that have more schellingness for the topics you think this place is centered on? (Or used to be centered on?) (Or should be centered on?)

I would ask this in the current open thread except that structurally it seems like it needs to be more prominent than that in order to do its job.

If you have very very little time to respond or even think about the questions, I'd appreciate it if you just respond with "Ping" rather than click away.

New to LessWrong?

Getting Started

FAQ

Library

Personal Blog

21

How often do you check this forum?
New Comment


64 comments, sorted by
top scoring
Click to highlight new comments since:
[-]gjm130

Here, have a poll.

How often do you check the LW discussion forum?

[pollid:1190]

Do you just read, or also vote or comment?

[pollid:1191]

Do you bother clicking through to links?

[pollid:1192]

How do you mainly read LW discussion?

[pollid:1193]

Do you know of other places that make better Schelling points for LW-ish topics?

[pollid:1194]

Reply
[-]JenniferRM40

I appreciate the poll, but a large part of my goal was to just get a lot of comments, hopefully at the "Ping" level, because I want to see how many people are here with at least that amount of "social oomph" when the topic is themselves.

For people responding to this poll, please also give a very small overall comment that you used the poll.

Reply
[-]gjm60

For the avoidance of doubt, I did fill in my own poll. (I'm not quite sure what the point is of getting a lot of comments, but never mind.)

Reply
[-]maxjmartin20

I did the poll

Reply
[-]John_Maxwell20

Did the poll.

Reply
[-]Dagon20

used poll.

Reply
[-]Sabiola20

Used the poll

Reply
[-]mindspillage00

Took the poll.

Reply
[-]satt00

I was polled.

Reply
[-]Alex K. Chen (parrot)60

You know, I barely checked LW from 2015 to 2020, and now I check it like, every time I feel like I need some novelty refresh, almost as much as I do Twitter... It has definitely improved since a few years ago

Reply
[-]RHollerith60

when I visit LW, I am almost always using a desktop computer. sometimes whole months go by without my visiting LW. to see what is new on LW, I scan http://lesswrong.com/r/all/recentposts/ (which mixes discussion and main). I never look at Open Thread posts. added: I took the poll.

Reply
[-]Gunnar_Zarncke60

Polled. And I think the polling has a higher response rate than even a simple "ping" because of peoples curiosity what others may have answered. This is already confirmed by the rate of votes in the poll.

I'm here since 2013 and used to write a fair number of polls and comments. Haven't written much lately (quite typical in that regard during the decline phase of LW). But I like the link aggregator revival. I think it works. I posted a few links myself. And continue to check about once a day.

I never used RSS readers so the current mode is very useful for me as it provides access to high quality posts from all over the rationalverse (I check OB manually as there seems to be no links to those posts).

Reply
[-]paulfchristiano60

I visit /discussion every 1-2 days on my laptop, typically spend a few seconds and then move on to something else. Sometimes I click through links or read posts if they look interesting, probably averaging ~1/day. I'm more likely to read posts with comment threads, and more likely to read posts than links. I rarely comment or vote.

Reply
[-][anonymous]60

I used to just view.

Now, I'm making an intentional effort to engage more with things. I think a community norm of voicing even simple agreement / appreciation can make this place a more enjoyable place to be.

Reply
[-]JenniferRM40

Do you have a causal theory as to what changed your intention? If so, I'm curious to know how it seemed to work.

Reply
[-][anonymous]60

Hmm. Well, I figure a community tends to be more fun when people are actively responding to things. I know I'd show up here more often if I knew that my ideas would get bounced off of by others.

At the same time, I'd been writing my own thoughts up on rationality, and I'd gotten to a point where I thought these ideas might be interesting to the LW audience.

Then the Anna post on LW becoming a central locus showed up, and i thought, "hey, this is cool, things are lining up well." (Which also triggered more thoughts about the community)

So basically a combination of the above tree things.

Reply
[-]9eB150

I use a feed reader, so I check out almost all the posts and links. I click through to the comments on almost all of them as well, since that's the real point.

The reddit /r/slatestarcodex community is very active too, and I like that.

Reply
[-]JenniferRM40

If you don't mind more questions... Which feed reader? Also do you look mostly for activity? Do you know of places with good quality/noise ratios, even if they are not very active?

Reply
[-]9eB160

I use Inoreader which is paid and web-based, but has browser plugins. I wouldn't say I "mostly" look for activity, since time is limited. The Open Thread on slatestarcodex.com is extremely active, I think it gets about 200 comments per day, but I don't frequently browse it. Slate Star Codex and Less Wrong are the only actual "communities" in my feed reader. The others that are directly or tangentially related are blogs, and most blogs don't really engender that much discussion.

Reply
[-]Zack_M_Davis50

ping (glance at site occasionally, rarely log in these days)

Reply
[-]sarahconstantin50

I check, using desktop, vote and comment occasionally, click through some links.

Reply
[-]username240

I read Discussion daily, by desktop and smartphone using browser (desktop mode on smartphone). I check Main regularly as well, just to ascertain there's nothing new there. I never click through link posts, will regularly follow interesting-sounding links with descriptive context in posts and comments.

I only read and do not comment except VERY rarely using this community account to comment on meta issues when they are raised by others who are taking active role in trying to improve LW. I used to have a personal account with a moderate amount of karma (500 < k < 1000) but removed it, now I only read except for comments like this. Perhaps I should not have a voice in this discussion but I feel like there are many other such members who have effectively checked out in similar way, yet continue to lurk.

Other places: None. Perhaps because there are no longer coherent themes remaining at LW (despite the momentary attempt in Nov/Dec to bring in high-quality posts on "classic" LW topics), and different subtopics are variously served by combinations of other sorts of networks (including increasingly, engagement with academia).

Reply
[-]jimmy40

Polled.

I don't read much here anymore, and comment less (7 comment threads in the last year). I'm commenting now mostly because you specifically mentioned appreciating any response, and because of having known and respecting you personally, not because I'd normally read and comment on something like this.

I'm not aware of any other place with more shellingness on this stuff than here, but I am also skeptical of recent efforts to bring more discussion back to LW.

Reply
[-]JenniferRM60

I do appreciate your response. Thank you. Also, I share your skepticism to some degree... hence this post as a data collection effort.

Reply
[-]katydee40

Once every few days or so.

Reply
[-]whpearson40

I check it a fair bit at the moment. I suspect I'll give it up in a bit.

Reply
[-]whpearson40

Because participating in a community that isn't helpful for what I am trying to achieve is a significant drain on my mental and temporal resources :)

I'm currently seeing how helpful it is.

Reply
[-]JenniferRM40

That seems like a setup for some interesting questions!

What are you trying to achieve? How do you think the community could be helpful towards this goal? Also, how is the assessment of helpfulness going?

Reply
[-]whpearson20

My path to LW was to Sl4 -> overcoming bias -> LW. This was because interested in systems that can change themselves in how they handle problems ( how they manipulate facets I didn't have a good language for it back then though).

I'm now getting close-ish (1-3 years) out of having something that I think can manipulate facets in a flexible way (using an internal market place based on user feedback). I mentioned it here. My rough strategy for what to do with it long term is here.

This requires different long term strategies. With the way I view things I think it more likely we will build/become demi-gods slowly. My over-arching strategy is to try and figure out what problem-solving is then figure out what to do about it. I have come across stuff like this which suggests world views on this forum at odds with mine. I've asked for back up plans. But got no response.

Most recently I've been trying to argue that we as rationalists need a place to work together on experiments that may end up wrong or not have necessarily the highest predicted value, because exploring via experiments gives us information about the world and may surprise us. The community could provide this sort of space for me and others..

I think it likely my work will fail, just because I am trying something complicated and exploring an unexplored area.

My assessment of the helpfulness is not going well. I put lots of effort into discussions. I should probably be putting more effort into posts which get more visibility. But I think I get a taste of the likely engagement. I got some people clicking through my links on my project, one person watching my video to the end. But no comments and one upvote.

Reply
[-]Lumifer20

experiments that may end up wrong or not have necessarily the highest predicted value, because exploring via experiments gives us information about the world and may surprise us

I want to express support for this idea. We need to do experiments where the expected answer is "Hell if I know" -- where you might discover something and not just fine-tune the precision of the answer you already know.

Reply
[-][anonymous]20

That's fair. If a lot of your goals seem orthogonal to most of ours, this might not be worth your time.

However, I suspect that if you verbalize your goal somewhere (assuming you haven't already), you can probably see if anyone else has similar aspirations.

Just a thought.

Reply
[-]Huluk40

Used the poll. Mostly check discussion in my feed reader (both smart phone and desktop), read easily digestible stuff right away, put off the harder stuff for later (and read about 1/3 of the things I intend for later consumption). Do vote, rarely comment.

Reply
[-]VincentYu30

Polled.

  1. I generally do only a quick skim of post titles and open threads (edit: maybe twice a month on average; I'll try visiting more often). I used to check LW compulsively prior to 2013, but now I think both LW and I have changed a lot and diverged from each other. No hard feelings, though.

  2. I rarely click link posts on LW. I seldom find them interesting, but I don't mind them as long as other LWers like them.

  3. I mostly check LW through a desktop browser. Back in 2011–2012, I used Wei Dai's "Power Reader" script to read all comments. I also used to rely on Dbaupp's "scroll to new comments" script after they posted it in 2011, but these days I use Bakkot's "comment highlight" script. (Thanks to all three of you!)

  4. I've been on Metaculus a lot over the past year. It's a prediction website focusing on science and tech (the site's been mentioned a few times on LW, and in fact that's how I heard of it). It's sort of like a gamified and moderated PredictionBook. (Edit: It's also similar to GJ Open, but IMO, Metaculus has way better questions and scoring.) It's a more-work-less-talk kind of website, so it's definitely not a site for general discussions.

    I've been meaning to write an introductory post about Metaculus… I'll get to that sometime.

    Given that one of LW's past focus was on biases, heuristics, and the Bayesian interpretation of probability, I think some of you might find it worthwhile and fun to do some real-world practice on manipulating subjective probabilities based on finding evidence. Metaculus is all about that sort of stuff, so join us! (My username there is 'v'. I recognize a few of you, especially WhySpace, over there.) The site itself is under continual improvement and work, and I know that the admins have high ambitions for it.

Edit: By the way, this is a great post and idea. Thanks!

Reply
[-]9eB120

Is there any information on how well-calibrated the community predictions are on Metaculus? I couldn't find anything on the site. Also, if one wanted to get into it, could you describe what your process is?

Reply
[-]VincentYu50

Is there any information on how well-calibrated the community predictions are on Metaculus?

Great question! Yes. There was a post on the official Metaculus blog that addressed this, though this was back in Oct 2016. In the past, they've also sent to subscribed users a few emails that looked at community calibration.

I've actually done my own analysis on this around two months ago, in private communication. Let me just copy two of the plots I created and what I said there. You might want to ignore the plots and details, and just skip to the "brief summary" at the end.

(Questions on Metaculus go through an 'open' phase then a 'closed' phase; predictions can only be made and updated while the question is open. After a question closes, it gets resolved either positive or negative once the outcome is known. I based my analysis on the 71 questions that have been resolved as of 2 months ago; there are around 100 resolved questions now.)

First, here's a plot for the 71 final median predictions. The elements of this plot:

  • Of all monotonic functions, the black line is the one that, when applied to this set of median predictions, performs the best (in mean score) under every proper scoring rule given the realized outcomes. This can be interpreted as a histogram with adaptive bin widths. So for instance, the figure shows that, binned together, predictions from 14% to 45% resolved positive around 0.11 of the time. This is also the maximum-likelihood monotonic function.

  • The confidence bands are for the null hypothesis that the 71 predictions are all perfectly calibrated and independent, so that we can sample the distribution of counterfactual outcomes simply by treating the outcome of each prediction with credence p as an independent coin flip with probability p of positive resolution. I sampled 80,000 sets of these 71 outcomes, and built the confidence bands by computing the corresponding maximum-likelihood monotonic function for each set. The inner band is pointwise 1 sigma, whereas the outer is familywise 2 sigma. So the corner of the black line that exceeds the outer band around predictions of 45% is a p < 0.05 event under perfect calibration, and it looks to me that predictions around 30% to 40% are miscalibrated (underconfident).

  • The two rows of tick marks below the x-axis show the 71 predictions, with the upper green row comprising positive resolutions, and the lower red row comprising negatives.

  • The dotted blue line is a rough estimate of the proportion of questions resolving positive along the range of predictions, based on kernel density estimates of the distributions of predictions giving positive and negative resolutions.

Now, a plot of all 3723 final predictions on the 71 questions.

  • The black line is again the monotonic function that minimizes mean proper score, but with the 1% and 99% predictions removed because—as I expected—they were especially miscalibrated (overconfident) compared to nearby predictions.

  • The two black dots indicate the proportion of question resolving positive for 1% and 99% predictions (around 0.4 and 0.8).

  • I don't have any bands indicating dispersion here because these predictions are a correlated mess that I can't deal with. But for predictions below 20%, the deviation from the diagonal looks large enough that I think it shows miscalibration (overconfidence).

  • Along the x-axis I've plotted kernel density estimates of the predictions resolving positive (green, solid line) and negative (red, dotted line). Kernel densities were computed under log-odds with Gaussian kernels, then converted back to probabilities in [0, 1].

  • The blue dotted line is again a rough estimate of the proportion resolving positive, using these two density estimates.

Brief summary:

  • Median predictions around 30% to 40% occur less often than claimed.
  • User predictions below around 20% occur more often than claimed.
  • User predictions at 1% and 99% are obviously overconfident.
  • Other than these, calibration seems okay everywhere else; at least, they aren't obviously off.
  • I'm very surprised that user predictions look fairly accurate around 90% and 95% (resolving positive around 0.85 and 0.90 of the time). I expected strong overconfidence like that shown by the predictions below 20%.

Also, if one wanted to get into it, could you describe what your process is?

Is there anything in particular that you want to hear about? Or would you rather have a general description of 1) how I'd suggest starting out on Metaculus, and/or 2) how I approach making and updating predictions on the site, and/or 3) something else?

(The FAQ is handy for questions about the site. It's linked to by the 'help' button at the button of every page.)

Reply
[-]ShardPhoenix30

I used to come here daily or more, now more like once a week, as the amount of interesting discussion has gradually declined.

Reply
[-]Richard Korzekwa 30

I check once or twice most days.

  1. On average I'll maybe read two or three posts per day, comment once per week (though it comes in spurts), and vote once per day.
  2. Maybe 1/3-1/2 of link posts. Oftentimes, they're links to things I've already seen posted elsewhere (like Facebook)
  3. Browser on a desktop. Occasionally smartphone.
  4. SSC is the obvious answer. Also a few Facebook groups*, or people I follow on Facebook.

Much of the time, I'll just look at the recent comments to see what's getting activity. But when I look at the actual discussion posts, I usually find things worth clicking that I wouldn't have found by only looking at recent comments. I don't know why I do this. Maybe commenting about it here will make me stop.

Also, I feel like the LW meetup in Austin meets some of my need for the sorts of discussions that might otherwise be met by the website.

*Optimal Memes for Cosmopolitan Teens has become an unexpectedly useful place to hang out.

Reply
[-]dogiv30

Polled. I check Discussion almost every day on a desktop, but almost never comment (though I plan to do so more often). I occasionally click through links but I prefer LW posts because of the community aspect. Ideas reliably get a meaningful response from intelligent readers with a common background, and the author of the post is always part of the discussion. For the same reason, I don't think any other rationalist discussion site has the potential of LW. Although I wasn't here back before the Diaspora, I do think a revival is in order--centralization may have its drawbacks, but the existence of this website, even now, gives us a baseline both for what has been explored in the past and what topics are currently open/unresolved.

Reply
[-]MrMind30
  1. I come here daily, mostly 10 read to 1 comment to 2 upvote, perhaps.
  2. Yes
  3. Desktop, web browser
  4. Was: Omnilibrium; Kinda but nefarious to use: SSC; is currently my favorite besides LW: putanumonit
Reply
[-]erratio30

I come here probably a few times a week, almost entirely just for reading or clicking through on particularly interesting links. I always use a desktop.

Reply
[-]dglukhov30

Used the poll, though I'll elaborate.

I'd probably spend more time if I wasn't busy reading the archives. I imagine this will change to more use over time.

Reply
[-]JenniferRM60

Do you have heuristics for finding good stuff in the archives?

Reply
[-]dglukhov40

Well, I started off with the Sequences, and the discussion flows that follow those essays. I find myself branching off from there, mostly, since I'm still relatively new here.

Reply
[-]scarcegreengrass20

1-2 visits per week.

I read 1-2 posts per visit.

50% link click rate.

Desktop via browser, sometimes RSS (Feedly).

I generally prefer the Less Wrong slack to this site. But this site has a karma system, which i like. (Contact Elo here for more info).

I might start reading the similar effective-altruism.com too.

Reply
[-]simple_name20

Used the poll. I've been reading mostly old posts for the last year but never registered as I feel I can't contribute much to the discussions. I check the discussion forum regularly but read/click only what seems interesting or is highly upvoted.

Reply
[-]tut20

I have it in my RSS reader. I read almost everything, but often more than a week after it's posted when I happen to want something to read. I vote more often than I comment, but not nearly a quarter of everything.

Reply
[-]philh20

Taken the poll.

I read most weekdays, but not usually on weekends; almost always on a desktop browser. Occasionally I check my inbox on my phone. I probably click through to less than half of the articles posted and less than half of the links, and don't read all of what I click through to.

Reply
[-]Dustin00

I read almost every post and not very many comments...because i subscribe to the RSS feeds.

Reply
Moderation Log
64Comments