That probably where there's something I don't understand. I don't understand why the analysis took ~1500 hours. Spending that much time with a dataset also instinctively triggers "fishing expedition" in my head. I don't know to what extend that's warranted.
The issue of multiple hypothesis testing is precisely why it took 1500 hours :-). I was dealing with the general question "how can you find the most interesting generalizable patterns in a human interpretable data set?" It'll take me a long time to externalize what I learned.
For now I'll just remark that dimensionality reduction reduces concerns around multiple hypothesis testing. If you have a cluster of variables A and a cluster of features B and you suspect that there's some relationship between the variables A and the variables B, you can do PCA on the two clusters separately, then look at correlations between the first few principal components rather than looking at all pairwise correlations between variables in A and variables in B.
A more interesting project would be to explore LW's ideological landscape. It would be very interested in how various rationalist beliefs interact with each other. Does seeing yourself as an "aspiring rationalist" correlates to beliefs on UFAI risk?
There is the 2014 LW survey data, which is interesting, even if less substantive than what you have in mind. I have an unfinished project that I'm doing with it (got bogged down in cleaning it to make it nicely readable).
tl;dr: If you struggle with motivational problems, it's likely that the problem is not intrinsic to you, but instead that you haven't yet found work that you find very interesting.
How I discovered how to do great work
Last winter I did something that I had never done before. I spent ~1500 hours working on genuinely original scientific research.
I had done research for my PhD in pure math, but faced squarely, the problems that I worked on were of very little interest to anyone outside of the fields, and I was not very engaged with my research. Pure math is very heavily stacked with talent, and the low hanging fruit has been plucked, so unless you're one of the most talented people in the world, your prospects for doing anything other than derivative work are very poor.
What I did last fall was entirely different. As I trained to be a data scientist, I found that there's far more low hanging fruit in the field than there is in pure math, and found myself working on novel problems that are of broad interest almost immediately.
Having very high intrinsic motivation made a huge difference. I found myself spending all waking hours (~90 hours / week) working on it obsessively, almost involuntarily. Once I emerged, I realized that what I had done over the past ~3.5 months was far more significant than all of the other work that I had done over the span of my entire life combined. I was astonished to find myself having ascended to the pantheon of those who have made major contributions to human knowledge, something that I hadn't imagined possible in my wildest dreams.
The problem isn't "laziness"
Many of the most interesting people who I know are achieving at a level far below their potential. They often have major procrastination problems, and believe this to correspond to them having a character flaw of "laziness". I've become convinced that these people's problems don't come from them being insufficiently disciplined.
Their problems come from them spending their time trying to do work that they find boring. If you find your work boring, it's very likely that you should be doing something else.
References
My position is not unique to me: it's common to extremely high functioning people.
[1] Steve Jobs created Apple, which owns ~0.1%+ of the world's wealth. In his 2005 Stanford commencement address he said:
[2] Bill Thurston is one of the greatest mathematicians of the 20th century. He formulated the geometrization conjecture, which subsumes the 100 year old Poincare conjecture, considered one of the ~7 most important unsolved problems. He describes his own character as follows:
[3] Scott Alexander / Yvain is widely regarded as a great writer. Political celebrity Ezra Klein characterized his blog as fantastic. Scott wrote:
[4] Paul Graham is the co-founder of Y-Combinator, a seed funder with a portfolio of combined value exceeding $30 billion (with investees including Dropbox, AirBnB and Stripe). In What You'll Wish You Had Known he wrote