[Survey Taken Thread]
By ancient tradition, if you take the survey you may comment saying you have done so here, and people will upvote you and you will get karma.
Let's make these comments a reply to this post. That way we continue the tradition, but keep the discussion a bit cleaner.
The question "How Long Since You Last Posted On LessWrong?" is ambiguous--I don't know if posting includes comments or just top-level posts.
It's not silly. I still find these newer comments useful.
Can you imagine a Hollywood movie in which the hero did that, instead of coming up with some amazing clever way to save the civilians on the ship?
Jack Bauer might do it.
I have been in a happy, mutually satisfying romantic/sexual relationship once in my life. We had one good year together, and it was The. Best. Year. Of. My. Life. I know people say that when something good happens to you, you soon adjust, and you wind up as happy or as sad as you were before, but that was simply not my experience. I'd give just about anything to have that again. Such is my utility function, and I do not intend to tamper with it.
This is really remarkable to read six years later, since, although I don't know you personally, I know your reputation as That Guy Who Has Really Awesome Idyllic Relationships.
But what if you're hallucinating the increase in mental capacity and resulting discernment?
It may be theoretically possible to increase my mental capacity in some way such that I can distinguish mental capacity from hallucination. I cannot conceive of how that would be done, but it may be possible.
P.S. I love when people reply to comments that are two and a half years old. It feels like we're talking to the past.
It's also completely ridiculous, with a sample size of ~10 questions, to give the success rate and probability of being well calibrated as percentages with 12 decimals. Since the uncertainty in such a small sample is on the order of several percent, just round to the nearest percentage.
It probably just computes it as a float and then prints the whole float.
(I do recognize the silliness of replying to a three-year old comment that itself is replying to a six-year old comment.)
Calibration Score
Using a log scoring rule, I calculated a total accuracy+calibration score for the ten questions together. There's an issue that this assumes the questions are binary when they're not- someone who is 0% sure that Thor is the right answer to the mythology question gets the same score (0) as the person who is 100% sure that Odin is the right answer to the mythology question. I ignored infinitely low scores for the correlation part.
I replicated the MWI correlation, but I noticed something weird- all of the really low scorers gave really low probabilities to MWI. The worst scorer had a score of -18, which corresponds to giving about 1.6% probability to the right answer. What appears to have happened is they misunderstood the survey, and answered in fractions instead of percents- they got 9 out of 10 questions right, but lost 2 points every time they assigned 1% or slightly less than 1% to the right answer (i.e. they mean to express almost certainty by saying 1 or 0.99) and only lost 0.0013 points when they assigned 0.3% probability to a wrong answer.
When I drop the 30 lowest scorers, the direction of the relationship flips- now, people with better log scores (i.e. closer to 0) give lower probabilities for MWI (with a text answer counting as a probability of 0, as most were complaints that asking for a number didn't make sense).
What about Tragic Mistakes? These are people that assign 0% probability to a correct answer, or 100% probability to a wrong one, and under a log scoring rule lose infinite points. Checking those showed both errors, as well as highlighting that several of the 'wrong' answers were spelling mistakes- I probably would have accepted "Oden" and "Mitocondria."
(Amusingly, the person with the most tragic mistakes- 9 of them- supplied a probability for their answers instead of an answer, so they were 100% sure that the battle of Trafalgar was fought off the coast of 100, which was the state where Obama was born.)
There's a tiny decline in tragic mistakes as P(MWI) increases, but I don't think I'd be confident in drawing conclusions from this data.
Sort-of related question: How do you compute calibration scores?
This definitely belongs on the next survey!
Why do you read LessWrong? [ ] Rationality improvement [ ] Insight Porn [ ] Geek Social Fuzzies [ ] Self-Help Fuzzies [ ] Self-Help Utilons [ ] I enjoy reading the posts
And then check if the "rationality improvement" people do better on calibration. (I'm guessing they don't.)
View more: Next
= f037147d6e6c911a85753b9abdedda8d){kind=logo}
= 27d567bc2d48d9f40e9ccc20ea370b15)
= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
I took the survey 2 days ago. It was fun. I think I was well calibrated for those calibration questions, but sadly there was no "results" section.
Is it possible to self-consistently believe you're poorly calibrated? If you believe you're overconfident then you would start making less confident predictions right?