Evaluating Multiple Metrics (where not all are required)

-3 imbatman 20 February 2012 07:38PM

This is my first article, and I'm submitting it in the discussion forum, so hopefully I've done this correctly and we can discuss!

Anyway, I have a group of friends who are really interested in movies, and they feel very strongly about them.  I find their convictions interesting.  Specifically the way they adamantly argue that, for instance, Midnight in Paris is a "better" movie than Bridesmaids or whatever.  I got to thinking about how one would create metrics by which you could evaluate any movie.

First attempt: A simple scale by which you give rankings (1-10) to a list of movie attributes (the metrics), sum up the total, and highest number is the best movie.

Some metrics might be:

Plot/Story

Acting

Effects, Costumes, Editing, etc.

Script/Dialogue

Humor

Drama/Passion

Suspense

So we can argue about what the metrics should be and how many we need, but since we're not worried about justifying our system objectively we can include whatever criteria we want.  We could even add a weighting component so some metrics are worth more than others.  My system can even be different than yours.  The problem, though, is that in reality movies don't need to excel at all metrics to be perfect for what they are.  Would Schindler's List be a better movie if they were cracking jokes the whole time?  Would 12 Angry Men be better if it had more special effects?  And it's a little weird to evaluate the acting in Up or Toy Story 3.  (No offense to voice actors.)

The idea of ranking movies is really about the challenge of comparing things that are the same class (movies) but very different types (comedy, horror, drama, etc) -- in content, goal, method, etc.  Is it possible to come up with metrics by which to compare anything in the class regardless of type?  Assuming you can come up with which metrics you find valuable/relevant, some of them will apply to one type but not another.  But you also can't completely disregard metrics that are not common between all types, because you've just said you find them valuable/relevant (in this case, to your enjoyment of a movie).

These thoughts led me to the question which I will pose here: How do you evaluate items in a class based on multiple metrics when not all metrics are ALWAYS relevant?

Some brainstorming to try to answer that question (modifying the system proposed above):

Allow "N/A" for a metric and then divide the total points by the total possible based on applicable metrics.  But this ignores, for example, humorless movies that could have used some humor.

Ok, so maybe give a movie with no humor a 10/10 in the humor metric IF it was perfect without it, or some other X/10 if it needed some humor.  But that seems to inflate the movie's rating by giving some amount of credit for an attribute that it didn't actually have.

I briefly considered having flexible weightings assigned subjectively to the metrics for each movie rated.  But the whole point of this is to have standard criteria for all movies -- not different scales.

 

Anyway, any ideas?  Are there already systems for this sort of thing in different arenas of which I'm not aware?  Could you develop a system for this sort of evaluation that could also be used to evaluate businesses, school classes, marketing techniques, or just about anything else?

Comment author: imbatman 11 January 2012 03:19:36PM 4 points [-]

"A man's gotta know his limitations." - Dirty Harry

Comment author: imbatman 10 January 2012 11:21:33PM 11 points [-]

"A Confucian has stolen my hairbrush! Down with Confucianism!"

-GK Chesterton (on ad hominems)

Comment author: gwern 13 December 2011 09:27:49PM *  6 points [-]

"If it ever turns out that Bayes fails - receives systematically lower rewards on some problem, relative to a superior alternative, in virtue of its mere decisions - then Bayes has to go out the window."

This is such an important concept.

Yes, but like falsifiability, dangerous. This also goes for 'rationalists win', too.

'We' (Bayesians) face the Duhem-Quine thesis with a vengeance: we have often found situations where Bayes failed. And then we rescued it (we think) by either coming up with novel theses (TDT) or carefully analyzing the problem or a related problem and saying that is the real answer and so Bayes works after all (Jaynes again and again). Have we corrected ourselves or just added epicycles and special pleading? Should we just have tossed Bayes out the window at that point except in the limited areas we already proved it to be optimal or useful?

This can't really be answered.

Comment author: imbatman 13 December 2011 11:18:09PM 0 points [-]

I liked the quote not because of any notion that Bayes will or should "go out the window," but because, coming from a devout (can I use that word?) Bayesian, it's akin to a mathematician saying that if 2+2 ceases to be 4, that equation goes out the window. I just like what this says about one's epistemology -- we don't claim to know with dogmatic certainty, but in varying degrees of certainty, which, to bring things full circle, is what Bayes seems to be all about (at least to me, a novice).

More concisely, I like the quote because it draws a line. We can rail against the crazy strict Empiricism that denies rationality, but we won't hold to a rationality so devoutly that it becomes faith.

Comment author: imbatman 13 December 2011 06:01:53PM 2 points [-]

Upvoted for this sentence:

"If it ever turns out that Bayes fails - receives systematically lower rewards on some problem, relative to a superior alternative, in virtue of its mere decisions - then Bayes has to go out the window."

This is such an important concept.

I will say this declaratively: The correct choice is to take only box two. If you disagree, check your premises.

"But it is agreed even among causal decision theorists that if you have the power to precommit yourself to take one box, in Newcomb's Problem, then you should do so. If you can precommit yourself before Omega examines you; then you are directly causing box B to be filled."

Is this your objection? The problem is, you don't know if the superintelligent alien is basing anything on "precommital." Maybe the superintelligent alien has some technology or understanding that allows him to actually see the end result of your future contemplation. Maybe he's solved time travel and has seen what you pick.

Unless you understand not only the alien's mode of operation but also his method, you really are just guessing at how he'll decide what to put in box two. And your record on guesses is not as good as his.

There's nothing mystical about it. You do it because it works. Not because you know how it works.

Comment author: imbatman 08 December 2011 05:20:23AM 6 points [-]

Hello All. I came across Less Wrong via Common Sense Atheism a few weeks ago. I have enjoyed it so far, but I have yet to put in the time to get up to speed on the sequences. Plan to, though.

I'm a Financial Accountant in Birmingham, AL. I'm not sure I would (yet) identify myself as a rationalist, but as for what I value, I value truth above all. And if I'm not mistaken, valuing truth seems a big step toward becoming a rationalist. I also value life, liberty, happiness, fun, music, pizza, and many other things.

Here's a little more about me:

Height: 6'0" Shoe Size: 12 Favorite Sport: Basketball Favorite Philosophers: Calvin & Hobbes Greatest Weakness: Distinguishing between reality and fantasy Greatest Strength: I'm Batman

In response to comment by [deleted] on More "Personal" Introductions
Comment author: NancyLebovitz 02 December 2011 05:48:50AM 3 points [-]

For people who use pseudonyms, would you care to explain why you chose yours? I don't necessarily mean why you're using a pseudonym at all, I'm more interested in why you chose the particular one you've got.

Comment author: imbatman 07 December 2011 05:03:45PM 5 points [-]

I'm not using a pseudonym.

View more: Prev