Bayesian Collaborative Filtering

I'm the founder of TakeOnIt, so let me add a little here.

JGWeissman's algorithm, or an evolution of it based on the feedback here, will be replacing the crude algorithm that's already online. You can see the results of that algorithm here:

Predicting Eliezer Yudkowsky's Opinions

If anyone else needs access to the database let me know.

Part of the vision here is to help people choose their beliefs in areas where they don't have domain expertise. This concept was described here.

In addition, this technique can also be used to detect correct contrarian opinions, as described here. The algorithm will predict that person S should believe in a minority opinion I if S has similar opinions to the set of people T, where T hold the opinion I.

[-]fburnaby16y60

An extremely minor quibble with your site (I'm just trying to be helpful):

You ask for an opinion using a question: "Does god exist?", but then you allow people to provide answers as if they're agreeing or disagreeing with an affirmative statement (options: agree, neutral, disagree). The grammatical disconnect caused me some confusion when I first looked at the site. I think changing this to be either:

(1) "God exists" [agree, neutral disagree]

(2) "Does god exist?" [yes, maybe, no]

would make this more clear.

I haven't gotten to really read the above article yet, but do you think the proposed method would perform substantially better than a simple cluster analysis?

[-]BenAlbahari16y00

That's a good point. I've added it to the user voice feature suggestions:

http://takeonit.uservoice.com/forums/44305-general

[-]JGWeissman16y00

do you think the proposed method would perform substantially better than a simple cluster analysis?

Can you describe an algorithm that uses "simple cluster analysis" to predict what position a person S will take on an issue I?

[-]Johnicholas16y10

There's a system (I think maintained by NASA) called AutoClass, which is fairly easy to use. As I understand it, it accepts input "points" (who here would be people) and outputs clusters of similar points (people).

In order to predict using AutoClass, I think you would model unanswered questions as "missing values", and then predict based on observed frequencies from the same cluster.

There's some ad-hoc-ish-ness about the way AutoClass decides how many clusters there should be, but it's a solid, existing technology that has been used successfully in many applications.

[-]BenAlbahari16y40

There's some ad-hoc-ish-ness about the way AutoClass decides how many clusters there should be...

If a collaborative filter algorithm is accurate, that's all that really matters to the consumers of the algorithm. It's primarily the designers of the algorithm who care about the scientific basis as to why the algorithm works.

A decent overview of the various CF algorithms:
http://www.hindawi.com/journals/aai/2009/421425.html

I find it both amusing and disturbing how pioneers in this field have been trying to optimize guessing movie preferences (recall the famous Netflix $1 million prize), when we can use these techniques to actually predict stuff that IMBO "really matters". It's yet another interesting data point as to what people really care about.

[-]orthonormal16y10

Rather, it's a reminder that more effort is spent on projects that can be immediately profitable, however trivial they may be in scope. If there were a pay market for intellectual content as robust as the one for movies currently is, we'd have seen this done already. (Alas, I don't see how that could happen in the near future.)

[-]BenAlbahari16y10

I've been considering the question: why has using Collaborative Filtering to predict the "trivial" opinions about movies been prioritized over predicting the "important" opinions about political/social/economic/etc. issues?

On reflection, I don't actually think it's because people care more about the former than the latter. Would you rather have a prediction regarding the opinion as to whether the movie Titanic was good, or a prediction regarding the opinion as to whether there's a housing bubble?

I think the answer is that opinions about products are naturally schematized, and hence easy to collate. Products are already tracked everywhere in databases, so it's pretty easy to extend that model to add opinions about those products. In contrast, opinions about issues, although often even more passionate than opinions about products, are not as naturally schematizable, hence they're harder to collate. Even in terms of representing the identity of an issue, it's not like we have the equivalent of an ISBN number for each issue. So opinions about issues are not adequately schematized and hence we can't collate those opinions into the nice big datasets we'd want to make predictions. Obviously websites like TakeOnIt are trying to change that. Each question ID is analogous to an ISBN number for that issue, if you will.

Yes, I agree with you that opinions of products can help sell products, so predicting opinions about products has the incentive of an immediately obvious monetization strategy. But if there's money to be made depending on correctly predicting the answer to a question, then the potential for monetization of the prediction of those opinions is also there.

[-]BenAlbahari16y00

Well there's a subset of questions on TakeOnIt where the correct answer has a financial reward/impact. An example of such a question was Is there a housing bubble in the United States?. These type of questions overlap with the kind of questions seen on prediction markets (which is a nice model for monetizing intellectual content). I'd be curious as to the relative accuracy between prediction markets and using collaborative filtering on expert predictions.

[-]fburnaby16y00

I had vaguely what Jonicholas mentions in mind.

[-]wnoise16y00

I've finally taken a look at this site. I'm strongly tempted to add H.P. Lovecraft quotes to many issues, but of course he's not an expert in the relevant senses, and the easily findable quotes are from fiction which should not generally be taken as his own.

[-]BenAlbahari16y30

Based on your idea and the discussion that followed, I've added the feature to flag a quote as fictional.

On the question pages, fictional quotes are put in their own group:
Is information-theoretic death the most real interpretation of death?

On the expert pages, fictional quotes are flagged per quote:
H.P.Lovercraft's Opinions

Fictional quotes are discounted from the prediction analysis.

[-]BenAlbahari16y00

Thinking out loud (this could be a terrible idea, "green hat" thinking alert!): I wonder if it would be interesting to be able to tag a quote as "fiction". There's so many insightful quotes that are spoken through the fictional characters of great authors. It seems a shame that such quotes are "illegitimate". Better to perhaps allow the quotes but tag them appropriately so they can be filtered out of prediction analysis. Thoughts?

[-]khafra16y10

How would they be attributed? Valentine Michael's opinions are substantially different from Lazarus Long's.

[-]BenAlbahari16y10

Simple model: a flag on a quote, present if it's a fictional character, with text preceding the quote explaining the source.

Complex model: Each fictional character is on par with an expert/influencer, with an extra field referencing back to the expert/influencer who's the author. E.g. you could look up all the quotes of "Sherlock Holmes" or all the fictional quotes of characters written by Arthur Conan Doyle.

[-]wnoise16y00

It seems worth trying, if you want to code it up. While it certainly doesn't make much sense to base predictions about others based on quite possibly incoherent groupings of characters, predicting the other way could be interesting.

But it does occur to me that I could just create user account and post them there, though that wouldn't let others add quotes.

[-]RobinZ16y00

I generally include the name of the character along with the rest.

[-]wnoise16y00

That doesn't always work -- sometimes it's from an impersonal narrator.

[-]RobinZ16y00

Point - "Narrator", perhaps?

[-]JGWeissman16y00

Fun Fact: My algorithm's most confident prediction about Ben is that with 67% probability, he Disagrees on the issue Is the unconscious philosophical zombie possible?.

(Ben has registered his position on 7 issues.)

[-]Daniel_Burfoot16y70

Your updating method is more properly called "Naive Bayes". Naive Bayes works well if the events you are conditioning on (Tj take Yj on I) are independent. It's not clear how reasonable that assumption is for your application.

To see the problem, let's say I agree with 80% probability with Eliezer and Robin. Now consider some new issue on which both agree. What's the probability I also agree? Naive Bayes predicts much higher than 80%; it effectively double-updates, one for both influences. But that's probably wrong - since Eliezer's opinions overlap strongly with Robin's, the conditioning events are not independent.

To do better, you need a method that can capture this kind of conditional independence effect (my opinion is probably independent of Eliezer's given Robin's). I would try a boosting method like AdaBoost.

As a meta-level note, you shouldn't imagine that any one technique is "correct" and some other incorrect. The only criterion you should use is empirical performance. Specifically, you can calculate the negative log likelihood (compression rate) each method achieves for the database and select the one that provides the lowest number.

[-]JGWeissman16y00

One way of looking at this problem, that fits well with an expansion I am trying to figure out, is that when I know Robin's position on the question, I should update my probability distribution for Eliezer's position, which should affect the likelihood ratios for Eliezer's observed position with respect to the positions of the person whose opinion is being predicted.

[-]JGWeissman16y00

One feature of this method, which clustering does not have, is that if person S consistently Disagrees on issues that person T Mostly Disagrees on, then when person T Mostly Disagrees on an issue, it is evidence that person S will Disagree on that same issue.

LESSWRONG
LW

LESSWRONG
LW

19

Bayesian Collaborative Filtering

19

19