AnnaSalamon comments on Help Fund Lukeprog at SIAI - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (276)
So, no plans for providing any substantiation of the mini-camp's purported success? (Some want to know.) Or of people who have increased their level of life success as a result of the winning at life guides?
Among the ultimate criteria for the minicamps is their impact on long-term life success. To assess this, both minicamp participants and a control group completed a long, anonymous survey containing many indicators of life success (income, self-reported happiness and anxiety levels, many questions about degree of social connectedness and satisfaction with relationships, etc.); we plan to give it again to both groups a year after mini-camp, to see whether minicampers improved more than controls. I’m eager to see and update from those results, but we’re only a couple months into the year’s waiting period. (The reason we decided ahead of time to wait a year is that minicamp aimed to give participants tools for personal change; and, for example, it takes time for improved social skills, strategicness, and career plans to translate into income.)
Meanwhile, we’re working with self-report measures because they are what we have. But they are more positive than I anticipated, and that can’t be a bad sign. I was also positively surprised by the number of rationality, productivity, and social effectiveness habits that participants reported using regularly, in response to my email asking, two months out. To quote a significant fraction of the numerical data from the exit survey (from the last day of minicamp), for those who haven’t seen participants’ ratings:
Some excerpts from the survey, about about Luke’s sessions in particular:
I worked on mini-camp with Luke, and I can honestly say that it’s only because of Luke that we were able to hold minicamp at all, and also that he was a phenomenal work partner in organizing the camp, getting all the logistics together, and generally making it a positive and, for many, life-changing experience.
More generally: In minicamp and other SingInst projects, Luke combines energy, reliable ability to carry projects to completion, and strategicness as to which projects make sense and which aspects of those projects are most worth the extra effort; if you’re looking to reduce existential risk, making it possible for SingInst to stably hire Luke seems to me to offer unusually good bang for your buck.
I have a real hard time deciding how seriously I should take this survey.
The halo effect for doing anything around awesome people like are found in a selected group of Lesswrongians is probably pretty strong. I fear at least some of the participants may have mixed up being with awesome people with becoming awesome. Don't get me wrong being with awesome people in of itself will work ... for a while, until you leave that group.
I'm not that sceptical of the claims, but from the outside its hard to tell the difference between this scenario and the rationality camps working as intended.
You're right to suspect that this could have happened. That said: I was a mini-camp participant, and I actually became more awesome as a result. Since mini-camp, I've:
I stuck around in California for the summer, and gained a lot from long conversations with other SIAI-related people. The vigor and insight of the community was a major factor in showing me how much more was possible and helping me stick to plans I initiated.
But, that said - the points listed above appear to be a direct result of the specific things I learned at mini-camp.
I suspect that it's precisely because of concerns like these that they didn't present these numbers until now. It's hard to see what other evidence they could have for the efficacy of the "minicamp" at this stage.
(Edited to replace "bootcamp" with "minicamp" as per wedrified's correction)
Indeed. SIAI is conducting a year-later follow up which should provide the information needed to differentiate. Answering that question now is probably not possible to the degree of certainty required.
That's exactly the complaint though -- many people have described it as a success, before the data is available.
I think people are seeing drastically different things in the word 'success'.
Yes; what I meant by "success" was more like a successful party or conference; Luke pulled off an event that nearly all the attendees were extremely glad they came to, gave presentations that held interest and influenced behavior for at least the upcoming weeks, etc. It was successful enough that, when combined with Luke's other accomplishments, I know we want Luke, for his project-completion, social effectiveness, strategicness, fast learning curves, and ability to fit all these qualities into SingInst in a manner that boosts our overall effectiveness. I don't mean "Minicamp definitely successfully created new uber-rationalists"; that would be a weird call from this data, given priors.
Sure, but Konkvistador's post is about how the survey might be contaminated by awesome-people-halo-effect, not that we shouldn't be calling it a success. That's a separate concern addressed elsewhere. My post was addressing how we would tell the difference between "working" and "near awesome people".
How was the control group selected? Did you select a pool of candidates larger than you could accept then randomly take a subset of these as a control? If not then calling it a 'control group' is borderline at best.
The prior expectation of the influence of one week of training on personal success over a year is far lower than that of various personal and environmental qualities in the individual. This being the case it is more reasonable to attribute differences in progress between the groups to the higher potential for growth in the chosen minicampers. This primarily reflects well on the ability of the Singinst rationality trainers to identify indicators of future success - a rather important skill in its own right!
A good point. The control group was of folks who made it through the initial screening but not the final screening, so, yes, there are differences. We explicitly discussed the possibility of randomizing admissions, but, for our first go, elected to admit the 25 people we most wanted, and to try randomizing some future events if the first worked well enough to warrant follow-ups (which it did). It is a bit of a hit to the data-gathering, but it wasn't growth potential as such that we were selecting for -- for example, younger applicants were less likely to have cool accomplishments and therefore less likely to get in, although they probably have more growth potential -- so there should still be evidence in the results.
Also, we marked down which not-admitted applicants were closest to the cut-off line (there were a number of close calls; I really wished we had more than 25 spaces), so we can gain a bit of data by seeing if they were similar to the minicamp group or to the rest of the controls.
Thanks for putting that together, Anna!
Since you're using self-reporting anyway, it would have been good if you had a 'how invested do you feel in minicamp's success?' question. Of course I say that having seen the results already.