= 27d567bc2d48d9f40e9ccc20ea370b15)
sixes_and_sevens comments on Open Thread, June 2-15, 2013 - Less Wrong Discussion
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (433)
Blame this comment.
Hah, okay.
You're not distinguishing original from quoted text, then?
It's not obvious to me that I should. TF-IDF is about identifying key terms in a document. Quoted text counts towards that.
That depends on what "the document" is. Everything appearing in a posting by a given author, or all of the text written by a given author?
"The document" is my wild sample that I've gone out and caught. TF-IDF tells me what it's broadly about. For this purpose, quoted text provides useful information.
If I want to infer personal facts about the author (beyond "what are the key terms in the posts they write"), it would make sense to weight original text higher than quoted text, but it would also make sense to use something other than TF-IDF for that purpose.