GWS's Shortform
Feb 14, 20223
Stephen Bennett
594
1
142
Congratulations! I wish we could have collaborated while I was in school, but I don't think we were researching at the same time. I haven't read your actual papers, so feel free to answer "you should check out the paper" to my comments.
For chapter 4: From the high level summary here it sounds like you're offloading the task of aggregation to the forecasters themselves. It's odd to me that you're describing this as arbitrage. Also, I have frequently seen the scoring rule be used with some intermediary function to determine monetary rewards. For example, when I worked with IARPA on geopolitical forecasting, our forecasters would get financial rewards depending on what percentile... (read more)
What do you think Metz did that was unethical here?
Replying toIf you weren't such an idiot...
Soft downvoted for encouraging self-talk that I think will be harmful for most of the people here. Some people might be able to jest at themselves well, but I suspect most will have their self image slightly negatively affected by thinking of themselves as an idiot.
Most of the individual things you recommend considering are indeed worth considering.
Interesting work, congrats on achieving human-ish performance!
I expect your model would look relatively better under other proper scoring rules. For example, logarithmic scoring would punish the human crowd for giving >1% probabilities to events that even sometimes happen. Under the Brier score, the worst possible score is either a 1 or a 2 depending on how it's formulated (from skimming your paper, it looks like 1 to me). Under a logarithmic score, such forecasts would be severely punished. I don't think this is something you should lead with, since Brier scores are the more common scoring rule in the literature, but it seems like an easy win and would highlight the possible... (read 614 more words →)
The second thing that I find surprising is that a lie detector based on ambiguous elicitation questions works. Again, this is not something I would have predicted before doing the experiments, but it doesn’t seem outrageous, either.
I think we can broadly put our ambiguous questions into 4 categories (although it would be easy to find more questions from more categories):
Somewhat interestingly, humans who answer nonsensical questions (rather than skipping them) generally do worse at tasks: pdf. There's some other citations in there of nonsensical/impossible questions if you're interested ("A number of previous studies have utilized impossible questions...").
It seems plausible to me that this is a trend in human writing more broadly and that... (read more)
See this comment.
You edited your parent comment significantly in such a way that my response no longer makes sense. In particular, you had said that Elizabeth summarizing this comment thread as someone else being misleading was itself misleading.
In my opinion, editing your own content in this way without indicating that this is what you have done is dishonest and a breach of internet etiquette. If you wanted to do this in a more appropriate way, you might say something like "Whoops, I meant X. I'll edit the parent comment to say so." and then edit the parent comment to say X and include some disclaimer like "Edited to address Y"
Okay, onto your actual comment. That link does indicate that you have read Elizabeth's comment, although I remain confused about why your unedited parent comment expressed disbelief about Elizabeth's summary of that thread as claiming that someone else was misleading.
I took Tristan to be using "sustainability" in the sense of "lessened environmental impact", not "requiring little willpower"
The section "Frame control" does not link to the conversation you had with wilkox, but I believe you intended for there to be one (you encourage readers to read the exchange). The link is here: https://www.lesswrong.com/posts/Wiz4eKi5fsomRsMbx/change-my-mind-veganism-entails-trade-offs-and-health-is-one?commentId=uh8w6JeLAfuZF2sxQ
In the comment thread you linked, Elizabeth stated outright what she found misleading: https://forum.effectivealtruism.org/posts/3Lv4NyFm2aohRKJCH/change-my-mind-veganism-entails-trade-offs-and-health-is-one?commentId=mYwzeJijWdzZw2aAg
Getting the paper author on EAF did seem like an unreasonable stroke of good luck.
I wrote out my full thoughts here, before I saw your response, but the above captures a lot of it. The data in the paper is very different than what you described. I think it was especially misleading to give all the caveats you did without mentioning that pescetarianism tied with veganism in men, and surpassed it for women.
I expect people to read the threads that they are linking to if they are claiming someone is misguided, and I do not think that you did that.
testing footnotes: ^1
This is a special post for quick takes (aka "shortform"). Only the owner can create top-level comments.
I'm pretty sure that this is incorrect compared to healthcare more broadly, although the best I can come up with is this meta-analysis: https://journals.plos.org/plosone/article/file?id=10.1371/journal.pone.0226361&type=printable
Which has this to say: