Unnamed comments on 2014 Survey Results - Less Wrong

87 Post author: Yvain 05 January 2015 07:36PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (279)

Sort By: Controversial

You are viewing a single comment's thread.

Comment author: Unnamed 05 January 2015 10:10:13PM 6 points [-]

I think that there are better analyses of calibration which could be done than the ones that are posted here.

For example, I think it's better to combine all 10 questions into a single graph rather than looking at each one separately.

The pattern of overconfidence on hard questions and underconfidence on easy questions is actually what you'd expect to find, even if people are well-calibrated. One thing that makes a question easy is if the obvious guess is the correct answer (like a question about Confederate Civil War generals where the correct answer is Robert E. Lee). On those sorts of easy questions, a bunch of people who made the obvious guess with moderate confidence will turn out to be correct, and they'll look underconfident. Whereas on questions where the correct answer is more obscure or counterintuitive, those guesses will be wrong and they'll look overconfident.

I'll see what other analyses I can do with the data. First I'll need to make some not-entirely-straightforward decisions about what to do with misspellings and other interesting responses. This may be fairly important, since those account for a decent chunk of all responses. For example, here are the most common responses to question 8 (in descending order of frequency):

Mitochondria
mitochondria
Mitochondrion
mitochondrion
Nucleus
Mitocondria
mitochondrium
nucleus
mitocondria
Mitochondrium
Mithocondria
Mitochondira
Ribosome

Comment author: RobbBB 06 January 2015 04:08:16AM *  0 points [-]

I would count minor typos (like 'mitochondira') and spelling errors (like 'mitocondria'), and trivial variants (like 'mitochondrium') as correct. I'd count major typos, where the pronunciation would be majorly different from the correct name, as neither correct nor incorrect -- ditch the data, since it may require too many judgment calls about whether someone's zeroing in on the correct name, and since the respondents themselves weren't told how much to make their calibration correct-spelling-sensitive. So my system says:

Heisenberg

Also correct: Werner Heisenberg, Heisenburg, Heissenburg

Incorrect: Heinsperbg

Neither: Herzinberg

Hawai'i

Also correct: Hawai`i, Hawaii, Hawii, Hawai, HI

Neither: HA, Hawaii or Arkansas

Odin

Also correct: Oden, Wudin, Wodin, Wodan, Wotan, Oưinn, Woden

Incorrect: Oris, Alduin

Jacob

Also correct: Yaakov, Jacob aka. Israel, Israel, Jocob, Jakob, Jacob/Israel, Jacob aka Israel

Incorrect: Joab, Jesus, Isai

Indonesia

Also correct: Indonisia, Indonessia

Comment author: bbleeker 06 January 2015 02:42:15PM 1 point [-]

Hawaii is both in the 'Also correct' and the 'Neither' list.

Comment author: RobbBB 07 January 2015 04:05:19AM 0 points [-]

One uses a typewriter apostrophe ('), the other doesn't (`).

Comment author: arundelo 07 January 2015 04:49:04AM 1 point [-]

No, bbleeker is saying that "Hawaii" (no apostrophe) is in both lists.

Comment author: RobbBB 07 January 2015 09:19:43AM 3 points [-]

Ah. "Hawaii or Arkansas" is its own entry, and was typed in as an answer.