I have spent a lot of time with Quantified Self and in that community. From there my impressions:
1) Structured experimentation doesn't come naturally. It's hard work for anybody in that community. At the beginning I would have thought that a lot more people get into it, but it seems to be a really hard habit to pick up. There seem to be strong cognitive barriers that you have to overcome.
2) Getting claims specific enough to be testable is hard.
3) Reflecting about what you learn in writing is useful even if you can't put numbers on things. Having questions that you care about in writing in Evernote is useful.
I want to make regular experimentation a part of my life and don't really know how. I thought that I should associate with and assist people who do run experiments (I'm interning with a psych lab and a paranormal investigator, and hope to work with some behavioral economists who run field-experiments), but I relied that I haven't taken the time to consider if that is actually a good approach or if there is somthign else I should be doing in addition.
How do I gain proficiency with experimental methods and build the habit of running simple experiments regularly? I suppose that there's a certain kind of phenomenon that to the educated mind is automatically flagged as ripe for experimentation (I'm thinking of Feynman's curiosity about the ants in his room, from Surely You're Joking, or Harry James Potter-Evans-Verres testing with his army to find out what the optimal way to fight is, prior to the first of Quirrell’s battles), but I don't have that intuition, yet.
What are the key insights, procedures, or guidelines that I need to know in order to experiment fruitfully? How do I build that intuition?
I’m looking either for recommendations or critiques. Perhaps personal experimentation is not as useful as my veneration of science in general leads me to believe? It seems to me beneficial that when faced with a problem, confusion, or dispute, one of my go-to approaches is to run an experiment.
Suggestions?