[Summary: Trying to use new ideas is more productive than trying to evaluate them.]
I haven't posted to LessWrong in a long time. I have a fan-fiction blog where I post theories about writing and literature. Topics don't overlap at all between the two websites (so far), but I prioritize posting there much higher than posting here, because responses seem more productive there.
The key difference, I think, is that people who read posts on LessWrong ask whether they're "true" or "false", while the writers who read my posts on writing want to write. If I say something that doesn't ring true to one of them, he's likely to say, "I don't think that's quite right; try changing X to Y," or, "When I'm in that situation, I find Z more helpful", or, "That doesn't cover all the cases, but if we expand your idea in this way..."
Whereas on LessWrong a more typical response would be, "Aha, I've found a case for which your step 7 fails! GOTCHA!"
It's always clear from the context of a writing blog why a piece of information might be useful. It often isn't clear how a LessWrong post might be useful. You could blame the author for not providing you with that context. Or, you could be pro-active and provide that context yourself, by thinking as you read a post about how it fits into the bigger framework of questions about rationality, utility, philosophy, ethics, and the future, and thinking about what questions and goals you have that it might be relevant to.
Oh man, you're not doing yourself any favors in trying to shift my understanding of you. Not that I doubt that your algorithm worked well! Let me explain.
You've used a multilevel modelling scheme in which the estimands are the eight proportions. In general, in any multilevel model, the parameters at a given level determine the prior probabilities for the variables at the level immediately below. In your specific context, i.e., estimating these proportions, a fully Bayesian multilevel model would also have a prior distribution on those proportions (a so-called "hyperprior", terrible name).
If you didn't use one, your algorithm can be viewed as a fully Bayesian analysis that implicitly used a constant prior density for the proportions, and this will indeed work well given enough information in the data. Alternatively, one could view the algorithm as a (randomized) type II maximum likelihood estimator, also known as "empirical Bayes".
In a fully Bayesian analysis, there will always be a top-level prior that is chosen only on the basis of prior information, not data. Any approach that uses the data to set the prior at the top level is an empirical Bayes approach. (These are definitions, by the way.) When you speak of "estimating the prior probabilities", you're taking an empirical Bayes point of view, but you're not well-informed enough to be aware that "Bayesian" and "empirical Bayes" are not the same thing.
The kinds of prior distributions with which I was concerned in my posts are those top-level prior distributions that don't come from data. Now, my pair of posts were terrible -- they basically dropped all of the readers into the inferential gap. But smart mathy guy cousin_it was intrigued enough to do his own reading and wrote some follow-up posts, and these serve as an existence proof that it was possible for someone with enough background to understand what I was talking about.
On the other hand, you didn't know what I was talking about, but you thought you did, and you offered questions and comments that apparently you still believe are relevant to the topic I addressed in my posts. To me, it really does look like -- in this context, at least -- you are laboring under a "cognitive bias in which unskilled individuals suffer from illusory superiority, mistakenly rating their ability much higher than is accurate".
So now I'll review my understanding of you:
To claim evidence that I'm overconfident, you have to show me asserting something that is wrong, and then failing to update when you provide evidence that it's wrong.
In the thread which you referenced, I asked you questions, and the only thing I asserted was that EM and Gibbs sampling find priors which will result in computed posteriors being well-calibrated to the data. You did not provide, and still have not provided, evidence that that statement was wrong. Therefore I did not exhibit a failure to update
I might be using different terminology than you--by... (read more)