@Alicorn, have you managed to fix this problem yet?

I read that a while back and I've been thinking about it for a while. After reading the discussion post in this thread, I thought I'd post this suggestion since the community doesn't appear to have a consensus approach to the problem.

Alicorn, have you considered adding more dimensions than diet to your self-tracking? And @Algon, have you considered self-tracking. If I had migraines constantly, it would be well worth my time to figure out the problem. I've been brainstorming dimensions to assess over the last few days and came up with a spreadsheet. I've come up with the broadest possible set of dimensions. Realistically, it might be hard to enter in values into the sheet at regular intervals, but depending on how important this is to you I'd narrow the scope from all to some.

Right now the variables I have in it are:

Anxiety level Emotional state Energy level Thoughts People Temperature Mood Comfort Memories Sensations Awareness Attention Beliefs Time Visualisation Identity Sequence of introspection attribution style confidence self esteem empathy risk perception diet cognitive distortions exercise attachment style expectations subjective wellbeing rating personality arousal

If you have any mental disorder symptoms, I'd recommend adding a variable for that too. For instance: psychosis, or obsessive thoughts.

Now, the next step up would be to go from subjective ratings of these categories, to more 'objective' ratings. I don't know what literature exists on the execution of psychometric tests, so I'm going to assume its irrelevant and the formalisation of the occupation of psychometrics is more a protectionist measure than something inherently dire.

The way psychometricians make assessment of variables formal is to ask specific questions.

Does anyone know, re there question banks for psychometricians? Psychometrics is about assessing particular psychological variables through structures tests and interviews. With such a tool, psychometricians designing new scales, inventories and constructs can simply use that search engine to find relevant questions. I suppose the traditional way to do things is just to trawl through the literature for related domains and just to individually extract items. Say, a question that assesses mood in a depression inventory might be taken from a paper constructing the depression inventory then brought into a novel inventory along with other dimensions of interest from different inventories. However, I'm not a psychometrician and haven't read the seminal works in the field nor the textbooks, only Wikipedia, a couple of guides to parts of the field on university websites, and individual papers of interest. Can someone enlighten me?

Edit: Beware, rationalists. Quantified self seems like a good option only after exhausting population wide studies - which are the source of other people's recommendations here and in Alicorn's thread. Take quote Gwern:

Confirmation bias is an issue in self-experimentation/Quantified Self because one is already at a disadvantage in evaluating the results - humans don’t weigh them very well; satt points out that (via the Bienaymé formula) “An RCT with a sample size of e.g. 400 would still be 10 times better than 4 self-experiments by this metric.” If one encountered 4 immaculately run self-experiments, I suspect they would feel like more evidence than 1/10th that RCT. When you toss in any selection effects (due to confirmation bias), the value of those 4 trials plunges even further.

Fortunately, just as there is a somewhat easy way to test for status quo bias, there’s also a somewhat easy way to test for confirmation bias: simply present a high-quality result - with the reverse of the true outcome. Or present the initial data about the setup and whatnot, but hide the results. (The savvy will recognize this as similar to Robin Hanson’s proposal for result-blind peer review.) Or better yet, present the same study with both egosyntonic and egodystonic. If the subject rates them differently, well, the only varying factor was the outcome… (One of the tests on YourMorals.org does just this by rewording a ‘study’ on gun control.) This is a strategy similar but not identical to a Sokal affair, since the subjects in the Sokal affair could plausibly claim that they didn’t understand the pseudo-physics and were trusting the word of a physics & mathematics professor in good standing - in a confirmation bias test, the subject must understand the material and reject it because it conflicts with his prior beliefs.