Regarding reference management: I use Wikipedia as my personal citation tool. Whenever I come across an interesting idea, fact or study, I add it to the relevant Wikipedia article, with reference. That way, all I have to remember is which article I used, and that is not a problem for me. If someone else deletes the material, which is exceedingly rare, I can still find it in the article's history by going to the last version that was edited by me. On a few occasions, people have helpfully updated the article with newer research, pointing out that my reference was outdated.
Zotero is an excellent Firefox extension that allows you to gather and organize citations and to add them to Wikipedia in an (almost) painless manner.
Do you add the articles to your watchlist, or just count on memories/searches turning them up again when needed?
I don't use the watchlist; I found that it sucks up way too much time once it contains a couple hundred items. If I vaguely remember a fact I added to Wikipedia, I can usually quickly come up with an approximate title of the article or with a search that will bring me there. (Instead of Wikipedia's internal search engine, I use Google with "site:en.wikipedia.org" which is faster and better.)
As an academic let me tell you that (1) it takes a long time to do your footnotes, (2) working on footnotes is really boring, (3) almost none of the readers look at your footnotes, and (4) there are lots of errors in footnotes.
Academics use footnotes as a guide to build on other peoples' work not (usually) as a means of verifying the accuracy of others' writings.
Academics use footnotes as a guide to build on other peoples' work not (usually) as a means of verifying the accuracy of others' writings.
That's very true; so I guess footnotes aren't a good way of doing it for us - LW discussions are more akin to Wikipedia pages (where the references are supposed to allow the claims to be verified) than academic work.
So if footnotes won't do the trick, and personal authority obviously isn't workable, then what? The only suggestion I have is extensive quoting from the work in question. This allows one to find it later by googling and allows a more direct assessment of the quality, to say nothing of making the material accessible at all (I never can forget how much ancient philosophy - or ancient anything - we know only from quotations in other works).
I assume you mean what to do in conversation and in comments on Less Wrong. The standards for things you put in print are that everything be supportable (this is never achieved throughout an entire book, but it is the goal).
This is difficult for me, because my memory is so poor that I can rarely remember the reasons for my conclusions, let alone the data behind those reasons, let alone the sources of that data. If I can remember my conclusions, I think I'm doing pretty well.
The best that I can do is be upfront about it when I can't remember the data. This is harder to do than it sounds. It's easy to blurt out a statement without checking whether you can support it. I did this just 2 days ago, when someone asked me about the precision and recall figures being claimed for a piece of software I had worked on. They sounded too high to me, and I said I didn't believe them. Hours later, I realized that although I had a feeling those figures were too high, I had no memory for any specific precision and recall figures. So tomorrow I need to talk with that person again and give them a disclaimer.
My approach is to say what I think, but give disclaimers when I can't remember supporting information. The amount of disclaiming depends on the audience. If I'm telling a child to eat his vegetables, I'm not going to give him a disclaimer saying that I can't actually remember the nutritional content of broccoli.
I do a lot of inferential reconstruction. For instance, the Steve Jobs anecdote is from Steve Jobs: The Journey is the Reward. I say this not because I specifically remember it being in that book, but because that is the only biography of Steve Jobs that I remember reading.
I say just post it, and if you're wrong then there will be plenty of industrious people to come up with counterexamples. When I remember something interesting, I can usually just post what I remember, and someone else will quickly come up with the correct source. Virginia Postrel calls this relying on local, distributed knowledge - it's what markets do well, including the marketplace of ideas.
Bayesians must condition their beliefs on all available evidence; it is not cheating to use less than ideal sources of information. However, this process also requires conditioning on the evidence for your evidence. Outside of academic journals, evidence is often difficult to trace back to the source and is dependent on our notoriously faulty memory. Given the consequences of low-fidelity copying, should rationalists trust evidence they can't remember the source of, even if they remember reading the primary source themselves? Should community members be expected to produce citations on demand?
This issue came to mind while trying to find a study I vaguely remembered about how the increased happiness of the religious could be explained by increased community involvement and while trying to factcheck PhilGoetz's now infamous anecdote about Steve Jobs. I started contemplating the standards for relaying highly relevant, but potentially wrong or distorted information.
Luckily factchecking is much easier in the age of the internet. Wikipedia serves as a universally accessible standard reference, and Google serves well for everything else. But sometimes my google-fu is not strong enough. So, I'll put this to the community: how should rationalists balance the tradeoff between neglecting evidence and propogating bad information?
Hygienic practices have been touched on before, but I haven't seen any consensus on this issue. Are the standards for what you personally condition on and what you share in discussion different? What needs a citation and what doesn't? Does anyone have recommendations for ways to better track the sources of evidence, i.e. reference management software?