SilasBarta comments on Help Fund Lukeprog at SIAI - Less Wrong

40 Post author: Eliezer_Yudkowsky 24 August 2011 07:16AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (276)

You are viewing a single comment's thread.

Comment author: SilasBarta 24 August 2011 02:03:22PM 24 points [-]

So, no plans for providing any substantiation of the mini-camp's purported success? (Some want to know.) Or of people who have increased their level of life success as a result of the winning at life guides?

Comment author: AnnaSalamon 25 August 2011 02:08:08AM *  39 points [-]

Among the ultimate criteria for the minicamps is their impact on long-term life success. To assess this, both minicamp participants and a control group completed a long, anonymous survey containing many indicators of life success (income, self-reported happiness and anxiety levels, many questions about degree of social connectedness and satisfaction with relationships, etc.); we plan to give it again to both groups a year after mini-camp, to see whether minicampers improved more than controls. I’m eager to see and update from those results, but we’re only a couple months into the year’s waiting period. (The reason we decided ahead of time to wait a year is that minicamp aimed to give participants tools for personal change; and, for example, it takes time for improved social skills, strategicness, and career plans to translate into income.)

Meanwhile, we’re working with self-report measures because they are what we have. But they are more positive than I anticipated, and that can’t be a bad sign. I was also positively surprised by the number of rationality, productivity, and social effectiveness habits that participants reported using regularly, in response to my email asking, two months out. To quote a significant fraction of the numerical data from the exit survey (from the last day of minicamp), for those who haven’t seen participants’ ratings:

  • In answer to “Zero to ten, are you glad you came?”, the median answer was 10 (mean was 9.3).
  • In answer to “Zero to ten, will your life go significantly differently because you came to mini-camp?” the median answer was 7.5 (the mean was 6.9). [This was the response that was most positively surprising to me.]
  • In answer to “Zero to ten, has your epistemic rationality improved?”, the median answer was 7 (mean 6.9)
  • In answer to “Zero to ten, are you more motivated to learn epistemic rationality, than you were when you came?”, the median answer was 8.5 (mean 8.1)
  • In answer to “Zero to ten, have you become more skilled at modifying your emotions and dispositions?”, the median answer was 7 (mean 6.3).
  • In answer to “Zero to ten, are you more motivated to modify your emotions and dispositions, than you were when you came?”, the median answer was 9 (mean 8.3).
  • In answer to “Zero to ten, are you more motivated to gain social skills than you were when you came?”, the median answer was 8 (mean 7.7).
  • In answer to “Zero to ten, have you gained social skills since coming?”, the median answer was 7.5 (mean 7.2).
  • In answer to “zero to ten, did you like Luke’s sessions?”, the median answer was 9 (mean answer 8.7).

Some excerpts from the survey, about about Luke’s sessions in particular:

  • “Luke is an excellent presenter. These sessions exceeded my expectations: I am convinced I have under-valued social interaction and techniques and that I can accelerate my success curve by aggressively adopting them. ”
  • “I really liked Luke's sessions. They were fun and interactive and well put together. There is an effect of being a bit more personally interested in the material.”
  • “Very useful content. Great presentation of it. Very good at handling the practical camp-issues and also useful fashion tips.”
  • “Luke’s sessions were concise, and well structured. Good PPT templates!”
  • “The social effectiveness and fashion sessions were very useful for me. ”
  • “Some parts of some sessions i felt went too slowly... but mostly extremely valuable information. wish we could have more social skills sessions - i would take another camp just for these super low-hanging fruit.”
  • “Luke gave concrete examples and advice. It was very helpful.”
  • “Luke was great as a session leader. His sessions were very clearly, cleanly organized, and discussions in his sessions were handled very well. Luke has, by far, the presence to lead a discussion among 16 people. :)”
  • “Luke was great. His sessions hit the relevant points in an effective manner.”
  • “Luke was very helpful and knowledgeable. The pace of his sessions was really good, and there was a lot of room for discussion. Luke also gave some helpful and specific fashion advice. ”
  • “Pretty much everything with Luke was phenomenal... Luke really made this whole camp worthwhile. I know this is more praise than constructive feedback, but I legitimately can't think of anything!”

I worked on mini-camp with Luke, and I can honestly say that it’s only because of Luke that we were able to hold minicamp at all, and also that he was a phenomenal work partner in organizing the camp, getting all the logistics together, and generally making it a positive and, for many, life-changing experience.

More generally: In minicamp and other SingInst projects, Luke combines energy, reliable ability to carry projects to completion, and strategicness as to which projects make sense and which aspects of those projects are most worth the extra effort; if you’re looking to reduce existential risk, making it possible for SingInst to stably hire Luke seems to me to offer unusually good bang for your buck.

Comment author: [deleted] 25 August 2011 10:47:27PM *  11 points [-]

I have a real hard time deciding how seriously I should take this survey.

The halo effect for doing anything around awesome people like are found in a selected group of Lesswrongians is probably pretty strong. I fear at least some of the participants may have mixed up being with awesome people with becoming awesome. Don't get me wrong being with awesome people in of itself will work ... for a while, until you leave that group.

I'm not that sceptical of the claims, but from the outside its hard to tell the difference between this scenario and the rationality camps working as intended.

Comment author: michaelcurzi 27 August 2011 10:28:19PM *  9 points [-]

You're right to suspect that this could have happened. That said: I was a mini-camp participant, and I actually became more awesome as a result. Since mini-camp, I've:

  • used Fermi calculations (something we practiced) to decide to graduate from school early.
  • started making more money than I had before.
  • started negotiating for things, which saved me over $1000 this summer.
  • begun the incredibly fucking useful practice of rejection therapy, which multiplied my confidence and caused the above two points.
  • rapidly improved my social abilities, including the easily measurable 'success with women' factor. This was mostly caused by a session about physical contact by Will Ryan, and from two major improvements in wardrobe caused by the great and eminent lukeprog (in whose name I just donated). I wasn't bad at social stuff before - this was a step from good to great.
  • resolved my feelings about a bad relationship, mostly as a result of boosted confidence from increased social success.

I stuck around in California for the summer, and gained a lot from long conversations with other SIAI-related people. The vigor and insight of the community was a major factor in showing me how much more was possible and helping me stick to plans I initiated.

But, that said - the points listed above appear to be a direct result of the specific things I learned at mini-camp.

Comment author: ciphergoth 26 August 2011 07:38:38AM *  4 points [-]

I suspect that it's precisely because of concerns like these that they didn't present these numbers until now. It's hard to see what other evidence they could have for the efficacy of the "minicamp" at this stage.

(Edited to replace "bootcamp" with "minicamp" as per wedrified's correction)

Comment author: shokwave 26 August 2011 12:24:24AM *  3 points [-]

Don't get me wrong being with awesome people in of itself will work ... for a while, until you leave that group.

I'm not that sceptical of the claims, but from the outside its hard to tell the difference between this scenario and the rationality camps working as intended.

Indeed. SIAI is conducting a year-later follow up which should provide the information needed to differentiate. Answering that question now is probably not possible to the degree of certainty required.

Comment author: [deleted] 26 August 2011 12:28:37AM 6 points [-]

Answering that question now is probably not possible to the degree of certainty required.

That's exactly the complaint though -- many people have described it as a success, before the data is available.

Comment author: jsalvatier 27 August 2011 03:16:04PM 6 points [-]

I think people are seeing drastically different things in the word 'success'.

Comment author: AnnaSalamon 27 August 2011 04:07:42PM 13 points [-]

Yes; what I meant by "success" was more like a successful party or conference; Luke pulled off an event that nearly all the attendees were extremely glad they came to, gave presentations that held interest and influenced behavior for at least the upcoming weeks, etc. It was successful enough that, when combined with Luke's other accomplishments, I know we want Luke, for his project-completion, social effectiveness, strategicness, fast learning curves, and ability to fit all these qualities into SingInst in a manner that boosts our overall effectiveness. I don't mean "Minicamp definitely successfully created new uber-rationalists"; that would be a weird call from this data, given priors.

Comment author: shokwave 26 August 2011 02:51:36PM 1 point [-]

Sure, but Konkvistador's post is about how the survey might be contaminated by awesome-people-halo-effect, not that we shouldn't be calling it a success. That's a separate concern addressed elsewhere. My post was addressing how we would tell the difference between "working" and "near awesome people".

Comment author: wedrifid 25 August 2011 03:55:17AM 12 points [-]

both minicamp participants and a control group

How was the control group selected? Did you select a pool of candidates larger than you could accept then randomly take a subset of these as a control? If not then calling it a 'control group' is borderline at best.

The prior expectation of the influence of one week of training on personal success over a year is far lower than that of various personal and environmental qualities in the individual. This being the case it is more reasonable to attribute differences in progress between the groups to the higher potential for growth in the chosen minicampers. This primarily reflects well on the ability of the Singinst rationality trainers to identify indicators of future success - a rather important skill in its own right!

Comment author: AnnaSalamon 25 August 2011 04:14:34AM *  14 points [-]

A good point. The control group was of folks who made it through the initial screening but not the final screening, so, yes, there are differences. We explicitly discussed the possibility of randomizing admissions, but, for our first go, elected to admit the 25 people we most wanted, and to try randomizing some future events if the first worked well enough to warrant follow-ups (which it did). It is a bit of a hit to the data-gathering, but it wasn't growth potential as such that we were selecting for -- for example, younger applicants were less likely to have cool accomplishments and therefore less likely to get in, although they probably have more growth potential -- so there should still be evidence in the results.

Also, we marked down which not-admitted applicants were closest to the cut-off line (there were a number of close calls; I really wished we had more than 25 spaces), so we can gain a bit of data by seeing if they were similar to the minicamp group or to the rest of the controls.

Comment author: lukeprog 25 August 2011 09:03:28AM 3 points [-]

Thanks for putting that together, Anna!

Comment author: Alexandros 26 October 2011 07:11:47AM *  1 point [-]

Since you're using self-reporting anyway, it would have been good if you had a 'how invested do you feel in minicamp's success?' question. Of course I say that having seen the results already.

Comment author: GuySrinivasan 24 August 2011 03:09:20PM 7 points [-]

At the very least I would like to see some kind of plan for evaluating future ventures... it may be too late now for anything but post-hoc qualitative for the mini-camp (or mega-camp?). Except that we did take a survey just before the camp, whose general idea I remember but not the specific questions, and I think there are plans to send it out again 10 months from now. Unfortunately that survey is available online IIRC, but I won't go look at it.

Comment author: WrongBot 25 August 2011 07:47:16AM 13 points [-]

One of the many things I updated on as a result of the 9-week bootcamp is the importance of tone. I'm sympathetic to your data-crusade, but the way in which you're prosecuting it is leading me to dislike you.

You've made a number of posts indicating that you place a high priority on finding and joining a rationalist community. That will be more difficult if you are generally perceived by rationalists as a hostile conversationalist; you should be more strategic about achieving your goals.

Comment author: XFrequentist 24 August 2011 03:02:52PM 12 points [-]

Upvoted, and props for sticking to your guns on this.

I appreciate that substantiating this sort of thing is non-trivial, but I would like to see at least an effort at some sort of evaluation.

Comment author: RobertLumley 24 August 2011 03:19:12PM 2 points [-]

Agreed. And to clarify, my repeated upvotes of Silas's comments on this matter are meant as a seconding of his request...

Comment author: jsalvatier 24 August 2011 03:36:48PM *  14 points [-]

I agree with 'more testing and evidence, please', but you often come across as adversarial and I think that generally makes it harder for you to convince the people you want to convince.

As an aside, remember that the minicamp was a relatively unplanned event; it came about because SIAI had extra time and space. I will be more concerned if the megacamp has a similar lack of testing.

Comment author: bentarm 25 August 2011 01:13:14AM 3 points [-]

remember that the minicamp was a relatively unplanned event; it came about because SIAI had extra time and space. I will be more concerned if the megacamp has a similar lack of testing.

As far as I can tell, the mega-camp actually had even less testing than the mini-camp. I did leave before the last week though, so I can't be sure quite what was done then. We had a discussion at the beginning about how we would decide if the camp had been a success, but I don't think we came to any very satisfactory conclusions.

Comment author: SilasBarta 24 August 2011 04:05:24PM *  7 points [-]

I agree with 'more testing and evidence, please', but you often come across as adversarial and I think that generally makes it harder for you to convince the people you want to convince.

Well, I hope they're not relying on "Silas is a meanie" as their intellectual "covering fire" for not substantiating this claim. And it's not that I want more testing and evidence, I just want to see what they think proves its success.

As an aside, remember that the minicamp was a relatively unplanned event; it came about because SIAI had extra time and space.

True, but I wouldn't be asking for any of this if leaders didn't try to paint it afterwards as a major success. If they want to take a risky venture, fine. If they want to play, "I meant to do that", let's see what it accomplished.

Comment author: jsalvatier 24 August 2011 04:25:32PM *  21 points [-]

I'm not talking about 'covering fire'. If your goal is to win an argument or appear righteous, then your strategy is alright. If your goal is to actually get SIAI to change their behavior, then your language is hurting your cause. You want to make it as easy as possible for them to change their behavior, and it's psychologically much easier to do something because an ally asks than because an adversary asks.

You have seen evidence: both Guy (link) and I (link) posted 'lessons learned' for the minicamp. You are right to say this is not especially strong evidence, but it is evidence. I think it would have been good to video tape some of the sessions and post them and post the exit surveys (they took testimonials too).

Comment author: wedrifid 26 August 2011 06:59:51AM 2 points [-]

If your goal is to win an argument or appear righteous, then your strategy is alright.

No, it clearly isn't. He left himself wide open to this sort of attack.

More fundamentally he justified and explained himself. He did it reasonably well and a good justification can work but it is almost never the optimal strategy.

Comment author: [deleted] 24 August 2011 03:53:55PM -1 points [-]

It's been almost a month, and nothing. I think a bit of contrarianism is warranted.

Comment author: Douglas_Knight 24 August 2011 04:20:48PM 0 points [-]

I don't think that number is correct.

Comment author: [deleted] 24 August 2011 08:50:04PM *  0 points [-]

Silas' original comment was August 1, and it's nearly the end of August.

Perhaps I want payday to come too dearly. Eh.

Comment author: [deleted] 24 August 2011 10:00:46PM 4 points [-]

Did paper-machine get downvoted for admitting he said something hastily, and correcting it to be accurate and precise? People are weird.

Comment author: SilasBarta 24 August 2011 10:06:23PM *  4 points [-]

Head honcho promotes SIAI insider. Annoying guy asks for evidence of insider's successes. Imagine how some people are going to vote on that.

Comment author: [deleted] 24 August 2011 10:08:54PM *  -1 points [-]

I really wish I had my PGP key here. You've earned yourself some papermachine points.

EDIT:

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Today (25/8/11) User:SilasBarta
earned five paper-machine points
by successfully arguing for the release of mini-camp data
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
iQEcBAEBAgAGBQJOVmChAAoJEDjQ6lJtxNBEEHIIALmWPoAYHBYFh1+PFxxnLA41
CM90FZxEVQEbXVSCEGtU4P0PIfJn0sQ1aweXIoQE10imRsNF8RMUwT5C+ImrJx5O
SL8KrtGqYARoMho+H017TwXyX3tg3wId2ZS6j2wxmkYiX/SOWK5rIityoF2d2pV5
SrVma3My24oQlFYiXOPloIGDWMUn+DzBTJH646qrZHqTIGbu+hXfV9zTDK9uGHBb
u5a3BLfQNXe+2LvXcuLiAJ+nGfYDTztZ98OehFq4BdChPug3GnXpKWKfaZ9zlcOb
zyP6MKxLfqtyVJeUBVyJYQUeTFWw8ROG0z8pzK5mey9u6eZuaglDqG4qv16pC3Q=
=TdF7
-----END PGP SIGNATURE-----
Comment author: lukeprog 24 August 2011 04:20:07PM *  19 points [-]

SilasBarta,

We collected lots of data before and during minicamp. We are waiting for some time to pass before collecting followup data, because it takes time for people's lives to change, if they're going to change. Minicamp was only a couple months ago.

Minicampers are generally still in contact, and indeed we are still gathering data. For example, several minicampers sent me before and after photos concerning their fashion (which was part of the social effectiveness section of the minicamp) and I'm going to show them to people on the street and ask them to choose which look they prefer (without indicating which is 'before' and which one is 'after').

So yes: by all qualitative measures, minicamp seems to have been a success. The early quantitative measures have been taken, but before-and-after results will have to wait a while.

As for future rationality training, we are taking the data gathered from minicamp and boot camp and also from some market research we did and trying to build a solid curriculum. To my knowledge, four people are seriously working on this project, and Eliezer is one of them.

Cheers,

Luke

Comment author: jsalvatier 24 August 2011 04:34:54PM *  5 points [-]

I'm curious, why you guys didn't post the testimonials or surveys you gathered at the end of the camp? Obviously these should be accompanied with appropriate caveats, but I think this would help explain to people why you are pleased with the results and think 'we're on to something'.

Comment author: lukeprog 24 August 2011 06:41:12PM 6 points [-]

Largely, lack of available staff hours.

Comment author: AnnaSalamon 25 August 2011 04:08:01AM 19 points [-]

Here are excerpts from the Minicamp testimonials (which were written to be shown to the public), with a link to the full list at the end:

“The week I spent in minicamp had by far the highest density of fun and learning I have ever experienced. It's like taking two years of college and condensing it to a week: you learn just as much and you have just as much fun. The skills I've learned will help me set and achieve my own life goal, and the friends I've made will help me get there.” --Alexei

“This was an intensely positive experience. This was easily the most powerful change self-modification I've ever made, in all of the social, intellectual, and emotional spheres. I'm now a more powerful person than I was a week ago -- and I can explain exactly how and why this is true.

At mini-camp, I've learned techniques for effective self-modification -- that is, I have a much deeper understanding of how to change my desires, gather my willpower, channel my time and cognitive resources, and model and handle previously confusing situations. What's more, I have a fairly clear map of how to build these skills henceforth, and how to inculcate them in others. And all this was presented in such a way that any sufficiently analytical folk -- anyone who has understood a few of the LW sequences, say -- can gain in extreme measures.” --Matt Elder / Fiddlemath

“I expected a week of interesting things and some useful tools to take away. What I got was 8 days of constant, deep learning, challenges to my limits that helped me grow. I finally grokked that I can and should optimize myself on every dimension I care about, that practice and reinforcement can make me a better thinker, and that I can change very quickly when I'm not constrained by artificial barriers or stress.

I would not recommend doing something like this right before another super-busy week, because I was learning at 100% of capacity and will need a lot of time to unpack all the things I learned and apply them to my life, but I came away with a clear plan for becoming better. It is now a normal and easy thing for me to try things out, test my beliefs, and self-improve. And I'm likely to be much more effective at making the world a better place as well, by prioritizing without fear.

The material was all soundly-researched and effectively taught, with extremely helpful supplemental exercises and activities. The instructors were very helpful in and out of session. The other participants were excited, engaged, challenging, and supportive.

I look forward to sharing what I've learned with my local Lesswrong meetup and others in the area. If that's even 1/4 as awesome as my time at the Mini-Camp, it will make our lives much better.” --Ben Hoffman / Benquo

“I really can't recommend this camp enough! This workshop broke down a complex and intertwined set of skills labelled in my brain as "common sense" and distinguished each part so that I could work on them separately. Sessions on motivation, cognition, and what habits to build to not fool yourself were particularly helpful. This camp was also the first example that I've seen of people taking current cognitive science and other research, decoding it, and showing people what's been documented to work so that they can use it too. It feels to me now as though the coolest parts of the sequences have been given specific exercises and habits to build off of. This camp, and the people in it, have changed my path for the better.” --David Jones / TheDave

You can now also read the full testimonials, from everyone who chose to give one.

Comment author: jsalvatier 24 August 2011 07:47:48PM 6 points [-]

Is this something one of the minicampers might be willing and able to do?

Comment author: handoflixue 25 August 2011 12:54:08AM 5 points [-]

I'm genuinely interested in seeing this data published, because I think it's something that a lot of people can build off of. If the only obstacle is really hours, I am happy to contribute.

I would be happy to show up in person while I'm in the area, pick up any paper notes you have available, transcribe them, and mail the originals back once finished. I have professional experience with data entry (including specifically product surveys) and market research in general. I'll be in San Francisco the afternoon of Monday, September 5th, hopefully around noon. I leave early Tuesday morning.

Comment author: lukeprog 25 August 2011 01:20:20AM 8 points [-]

Great! Much of the minicamp data is private and anonymous, so I can't share that with you, but I definitely have tasks for volunteers to do that will free uo time for me to write up a minicamp report - some of those tasks are even directly relevant to minicamp. Please email me at lukeprog at gmail if you'd like to help.

Comment author: SilasBarta 24 August 2011 07:07:44PM *  3 points [-]

Is there a post requesting volunteer help with this administrative task?

Comment author: Eliezer_Yudkowsky 24 August 2011 09:23:15PM 7 points [-]

Administrating volunteers also requires staff hours. Sometimes more than the original task. Why, are you volunteering to administrate them?

Comment author: SilasBarta 24 August 2011 09:43:18PM *  0 points [-]

Sure, I'd love to! (I thought I didn't qualify to volunteer for SIAI?) Hand over whatever you have and I'll make sure they do it right! (I thought this administration has to be done by someone in the loop on this, but whatever.)

Oh, you were just hoping I'd drop it, and the issue of actual substantiation of the mini-camp's success (which lies at the top of your reasons for wanting to fund Luke) would die off? Can't help there.

Comment author: lukeprog 25 August 2011 12:40:41AM 11 points [-]

SilasBarta,

I need to write up the results myself because I personally ran the minicamp with Anna Salamon and Andrew Critch. But I have tons of stuff I could have you do that would free up more of my time to get around to writing up results of minicamp data. If you're interested in helping, that would be awesome. You can contact me at lukeprog [at] gmail.

Comment author: AuthorityFigure 25 August 2011 03:29:24PM 12 points [-]

"Sure, I'd love to! (I thought I didn't qualify to volunteer for SIAI?)"

LOL. Way to play up the role of the passive-aggressive outsider.

Comment author: OptimalFAI 25 August 2011 06:59:45PM 6 points [-]

Very true.

In the interest of optimizing our rationality I think that we need to continue to call out instances "community distancing" such as the one exhibited by Silas above.

The reason for doing so? It lets the dissenters know that a community can tolerate and appreciate criticism but not the creation of a lone wolf character. Lone wolves do not contribute to a community and instead impede our advances in rationality by drawing conversations back to their status. As such, their status seeking should be pointed out and skepticism should be attached to their future postings.

Passive-aggressive comments in particular are troublesome because these types eventually find ways to disrupt substantive threads by reminding others of their loner status and their unacknowledged genius. Their resentment then leads to them mocking key figures in a community (note Silas' comments to both Eliezer and Luke).

Perhaps LW needs a mini-sequence on acceptable and non-acceptable signaling within a rational community.

Comment author: wedrifid 26 August 2011 06:51:24AM *  6 points [-]

As such, their status seeking should be pointed out and skepticism should be attached to their future postings.

I object. "Lone wolves", and Silas in particular, are not more status seeking than average. Luke's contributions are far more status seeking than Silas's are. Luke is good at status seeking while Silas's biggest weakness is that he fails to status seek when it would clearly be in his interests to do so.

Comment author: [deleted] 25 August 2011 07:16:53PM 2 points [-]

It lets the dissenters know that a community can tolerate and appreciate criticism but not the creation of a lone wolf character. Lone wolves do not contribute to a community and instead impede our advances in rationality by drawing conversations back to their status.

Let's taboo "lone wolf" and see what you actually mean by it, because I don't see Silas as a lone wolf figure in this debacle. For example, most of his comments have positive karma -- what I would consider a lone wolf wouldn't have such support.

Comment author: lessdazed 25 August 2011 08:31:22PM *  1 point [-]

Lone wolves do not contribute to a community

You think they don't contribute at all?

Comment author: handoflixue 25 August 2011 12:54:46AM 0 points [-]

http://lesswrong.com/lw/78s/help_fund_lukeprog_at_siai/4oxp I can administer myself when it comes to basic data transcription.

Comment author: SilasBarta 24 August 2011 04:54:19PM *  11 points [-]

So it's early enough to call it an unqualified success, but too soon for evidence to exist that it was was a success? If I have to be patient for the evidence to come back, shouldn't you be a little more patient about judging it a success?

Edit: I gave a list of information you could post. The fashion part isn't suprising enough to count as strong evidence, and was a relatively small part of the course that, in any case, you previously claimed could be accomplished by looking at a few fashion magazines.

Comment author: lukeprog 24 August 2011 06:38:35PM *  9 points [-]

I mean 'evidence' in the Bayesian sense, not the scientific sense. I have significant Bayesian evidence that minicamp was a success on several measures, but I can't know more until we collect more data.

Thanks for providing a list of information we could post. One reason for not posting more information is that doing so requires lots of staff hours, and we don't have enough of those available. We're also trying to, for example, develop a rationality curriculum and write a document of open problems in FAI theory.

If you're anxious to learn more about the rationality camps before SI has time to publish about that data, you're welcome to contact the people who attended; many of them have identified themselves on Less Wrong.

I'm fairly confident that campers got more out of my fashion sessions than what they can learn only from looking at a few fashion magazines.

Cheers,

Luke

Comment author: handoflixue 25 August 2011 12:31:50AM 7 points [-]

This comes off very strongly as the typical bureaucratic protectiveness - a business doesn't want to share raw data, because raw data is a valuable resource. If you came out and said this was the reason, I'd be more understanding, but it would still feel like a major violation of community norms to be so secretive.

If simple secrecy is indeed the case, I would urge you, please, be honest about this motive and say so explicitly! At least then we are having an honest discussion, and the rest of this comment can be disregarded.

We collected lots of data before and during minicamp

In short, what is the reason you can't share this RAW data, which you state you collected, and which you've presumably found sufficient for your own preliminary conclusions? I don't think Silas is asking for or expecting an elegant power-point presentation or a concise statistical analysis - I know I would personally love to simply see raw data.

Is there truly not a single spreadsheet or writeup that you could drop up for us to study while you collect the rest of the data?

Comment author: Mass_Driver 25 August 2011 07:29:17AM 12 points [-]

Good grief, people. There are conspiracies that need ferreting out, but they do not revolve around generating fake data about the effectiveness of an alpha version of a rationality training camp that was offered for free to a grateful public.

I went to the minicamp, I had a great time, I learned a lot, and I saw shedloads of anecdotal evidence that the teachers are striving to become as effective as possible. I'm sure they will publish their data if and when they have something to say.

Meanwhile, consider re-directing your laudable passion for transparency toward a publicly traded company or a medium-sized city or a research university. Fighting conspiracies is an inherently high-risk activity, both because you might be wrong about the conspiracies' existence, and because even if the conspiracy exists, you might be defeated by its shadowy and awful powers. Try to make sure the risks you run are justified by an even bigger payoff at the end of the tunnel.

Comment author: Kaj_Sotala 25 August 2011 09:34:58AM 19 points [-]

There are conspiracies that need ferreting out, but they do not revolve around generating fake data about the effectiveness of an alpha version of a rationality training camp that was offered for free to a grateful public.

I don't think anybody is accusing the minicamp folks of anything of the kind. But public criticism and analysis of conclusions is the only reliable way to defend against overconfidence and wishful thinking.

When I ended my term as an SIAI Visiting Fellow, I too felt like the experience would really change my life. In reality, most of the effects faded away within some months, though a number of factors combined to permanently increase my average long-term happiness level.

Back then the rationality exercises were still being worked out and Luke wasn't around, so it's very plausible that the minicamp is a lot more effective than the Visiting Fellow program was for me. But the prior for any given self-help program having a permanent effect is small, even if participants give glowing self-reports at first, so deep skepticism is warranted. No conspiracies are necessary, just standard wishful thinking biases.

Though I think this was the third time that Silas raised the question before finally getting a reply, despite his comment being highly upvoted each time. If some people are harboring suspicions of SIAI covering up information, well, I can't really say I'd blame them after that.

Comment author: lukeprog 25 August 2011 11:02:11AM *  6 points [-]

For the record, I for one don't recall reading any of SilasBarta's earlier comments on this topic.

Comment author: katydee 25 August 2011 10:28:03AM 1 point [-]

It seems rather unlikely to me that being a mini-camp participant would have more of an effect on someone's life than being a Visiting Fellow, new techniques or not-- and if I am wrong, I would very much want to encounter these new techniques!

Comment author: Kaj_Sotala 25 August 2011 12:54:31PM *  7 points [-]

I wouldn't be that surprised. Explicit rationality exercises were only starting to be developed during the last month of my stay, and at that point they mostly fell into the category of "entertaining, but probably not hugely useful". The main rationality boost came from being around others with a strong commitment to rationality, but as situationist psychology would have it, the effect faded once I was out of that environment.

Comment author: handoflixue 25 August 2011 08:08:38AM 8 points [-]

The positive endorphin rush from you and lukeprog sends signals that loook just like the enthusiastic gushing I see from any week-long "how to fix your life in five easy steps!" seminar. Smart people get caught up in biased thinking all the time. I had a good friend quit AI research to sell a self-help book, so I may be particularly sensitive to this :)

Objective data means I can upgrade this from "oh bunnies, another self-help meme" to "oooh, fascinating and awesome thing that I want to steal for myself." As long as it signals like a self-help meme, I'm going to shoot it down just like I'd shoot down any similar meme that tried to sell itself here on LessWrong.

Comment author: Mass_Driver 25 August 2011 08:38:13AM 7 points [-]

All right, but there's a fine line between shooting down self-help memes and unnecessarily discouraging project-builders from getting excited about their work. It's not fun or helpful for a pioneer to have his or her every first step be met with boundless skepticism. Your concerns sound real enough to me, but even an honest concern can be rude, and even a rationalist can validly trade off a tiny little bit of honesty for a whole lot of politeness and sympathy.

Why do I say "a tiny little bit of honesty?" Well, if the minicamp were being billed as "finished," "polished," "complete," "famous," "proven," or "demonstrably successful," as many self-help programs are, then it would make sense to demand data supporting those claims.

Instead, the PR blurb says that "Starting on May 28th, the Singularity Institute ran a one-week Rationality Training Camp. Our exit survey shows that the camp was a smashing success, surpassing the expectations of the organizers and the participants."

Leaving aside the colorful language that can and should characterize most press releases, this is a pretty weak claim: the camp beat expectations. Do you really need to see data to back that up?

Comment author: handoflixue 25 August 2011 08:49:31AM 7 points [-]

this is a pretty weak claim: the camp beat expectations

"Please give us money" and "Co-organized and taught sessions for a highly successful one-week Rationality Minicamp" are stronger claims.

Comment author: SilasBarta 29 August 2011 05:03:37PM 3 points [-]

For me, this isn't about making SIAI transparent; it does quite enough in that regard. It's about stopping an information cascade genie that's already out of the bottle.

Let me put it this way: right now the ratio of "relying on the assumption of mini-camp's success for decision making" to "available evidence for its success" is about 20-to-1. As I warned before, it's quickly becoming something "everyone knows" despite the lack of evidence (and major suspicions of many people that it wouldn't succeed going in). And that believe will keep feeding on itself unless someone traces it back to its original evidence.

It doesn't reassure me that I'm told I have to keep waiting before anything's conclusive, yet they can declare it a success now.

I just want the reliable evidence they claim to have, rather than just dime-a-dozen self-help testimonials. They collected hard data, and I gave them a list of things they could provide that are easy to gather and don't compromise privacy, and are much more likely to be present if the success were real than if it were not. Even after AnnaSalamon's circling of the wagons I don't see that.

Comment author: jsalvatier 29 August 2011 08:15:47PM 5 points [-]

I think this is largely a case of people reading different things into 'success'.

Comment author: lukeprog 25 August 2011 12:49:44AM *  8 points [-]

Oh, sure. The reason is easy to communicate. We explicitly told minicampers that their feedback on the exit survey would be private and anonymous, for maximal incentive to be direct and honest. We are not going to violate that agreement. The testimonials were given via a separate form with permission granted to publish THAT data publically.

Comment author: handoflixue 25 August 2011 12:57:28AM 1 point [-]

Oh, sure. The reason is easy to communicate. We explicitly told minicampers that their feedback on the exit survey would be private and anonymous, for maximal incentive to be direct and honest.

I'm unclear, then, why you are citing a lack of staff hours if the data cannot be published at all.

The testimonials were given via a separate form with permission granted to publish THAT data publically.

Has the raw testimonial data been published?

I'm assuming you have data beyond exit surveys and testimonials...?

Comment author: lukeprog 25 August 2011 01:16:34AM 9 points [-]

A summary of the data can be published, for example median scores for measured values. But the data can't be published in raw form.

Not sure if raw testimonial data has been published yet. We do have data beyond testimonials and exit surveys, but that, too, requires precious staff hours to compile and write up, and it is still in the process of being collected.

Typing this stuff from a phone, pardon the brevity...

Comment author: SilasBarta 24 August 2011 07:05:40PM *  6 points [-]

I mean 'evidence' in the Bayesian sense, not the scientific sense.

Great, so did I! Now communicate that evidence. If it can't be communicated, I don't think you should be so confident in it.

One reason for not posting more information is that doing so requires lots of staff hours

I find that hard to believe. It may take time for the participants to report back, but not for you to tabulate the results.

We're also trying to, for example, develop a rationality curriculum and write a document of open problems in FAI theory.

I'm sorry, but this just sounds like excuse-making. Do you want your audience to be people who just take your word on something like this? I've asked several times for some very simple checks. This claim that you're too busy just doesn't fly.

I'm fairly confident that campers got more out of my fashion sessions than what they can learn only from looking at a few fashion magazines.

Then why don't you mention this in your "how to be happy" post, which is also being used as evidence of your productivity? Do you know a single person who has improved fashion to an acceptable level as a result of those magazines?

Cheers,

Luke

Not necessary.

Comment author: lukeprog 25 August 2011 12:38:42AM *  11 points [-]

Great, so did I! Now communicate that evidence. If it can't be communicated, I don't think you should be so confident in it.

Here:

  1. Surprisingly positive reviews in both qualitative and quantitative forms on our exit surveys.
  2. Follow-ups with many individual minicampers who report that several of the things we taught have stuck with them and improved their lives.
  3. People telling me to my face during minicamp that they were getting lots of value out of it.
  4. Enthusiastic testimonials.

It may take time for the participants to report back, but not for you to tabulate the results.

No, it definitely takes time to tabluate the results and write a presentable post about the results. I've personally spent 3 hours on it already but the project is unfinished.

Do you want your audience to be people who just take your word on something like this?

Ah. I may not have communicate this clearly: I think your skepticism concerning the success of the minicamp is warranted because almost no evidence is available to you. You're welcome to not take my word for it. When I have another 5-10 hours to finish putting together the results and write a post with more details about minicamp, I will, but I'm mostly waiting to invest that time until I can do it most profitably, for example when we've gathered more 'after minicamp' data.

Then why don't you mention this in your "how to be happy" post, which is also being used as evidence of your productivity?

I don't understand. The 'How to Be Happy' post was written before I helped run minicamp. And, there are tons of things not mentioned in that post. That post barely scratches the surface of my thoughts on happiness, let alone research on happiness in general.

Do you know a single person who has improved fashion to an acceptable level as a result of those magazines?

I doubt magazines is ever the sole input on someone's fashion sense, but yes I know people who have improved their fashion as a result of following magazines (or fashion blogs; same thing basically). Ask Peter Scheyer about this, for example.

Comment author: JoshuaZ 25 August 2011 08:46:33PM 11 points [-]

Given the large amount of effort it took to get to the miny camps, all four of these could be easily explained by cognitive dissonance.

Comment author: ciphergoth 26 August 2011 07:39:58AM *  2 points [-]

What evidence would you expect them to have if the "minicamp" was a genuine success? (Edited - thanks for the correction, wedrified!)

Comment author: wedrifid 26 August 2011 08:02:01AM 8 points [-]

What evidence would you expect them to have if the "bootcamp" was a genuine success?

Bootcamp? I found the wording Eliezer used fascinating:

Co-organized and taught sessions for a highly successful one-week Rationality Minicamp, and taught sessions for the nine-week Rationality Boot Camp.

Have they actually claimed anywhere here that the bootcamp was successful?

Comment author: ciphergoth 26 August 2011 11:54:00AM 0 points [-]

Oops, fixed - thanks!

Comment author: JoshuaZ 26 August 2011 12:56:09PM 4 points [-]

Honestly, I'm not sure. Having a randomized control group and then looking at actual success would be nice. Even without a good control, one obvious thing to do would have been to do before and after tests of similar questions that test for rational behavior (e.g. whether they can recognize they are engaging in the sunk cost fallacy and things like that). It may be that given the circumstances the best evidence we have is self-reporting like this. If so, it is evidence for the success of the minicamps. But, it is not very strong evidence precisely because it is consistent with a variety of other not implausible hypotheses. This thread has made me more inclined to believe that the minicamps were successful, but had not strongly increased my confidence.

Comment author: Vladimir_Nesov 24 August 2011 08:10:54PM 16 points [-]

(I disapprove of downvoting the parent (which I just found at '-2'). It continues the same conversation as the previous Silas's posts, pointing out what does look like rationalization. If raising a possibility of interlocutor's rationalizing in defense of their position is considered too rude to tolerate, we'll never fix such problems.)

Comment author: Spurlock 24 August 2011 09:00:33PM 14 points [-]

I suspect that most of the downvotes came from the very last sentence, which struck me as more than a little snarky. "Cheers" might not be necessary, but it is a gesture of politeness and was probably added in an attempt to convey a positive tone (which is important but somewhat tricky in text). I wouldn't say "not necessary" if someone held the door for me, even if it is obviously true.

Agree with you that the actual substance of the post was in no way downvote-worthy.

Comment author: komponisto 24 August 2011 09:59:43PM 21 points [-]

Because "signing" comments is not customary here, doing so signals a certain aloofness or distance from the community, and thus can easily be interpreted as a passive-aggressive assertion of high status. (Especially coming from Luke, who I find emits such signals rather often -- he may want to be aware of this in case it's not his intention.)

I interpret Silas's "Not necessary" as roughly "Excuse me, but you're not on Mount Olympus writing an epistle to the unwashed masses on LW down below".

Comment author: handoflixue 25 August 2011 12:24:19AM 4 points [-]

Second this.

Comment author: Miller 25 August 2011 04:13:24AM 6 points [-]

Because "signing" comments is not customary here, doing so signals a certain aloofness or distance from the community

No. I am very confident the intention was to signal that Luke was not being emotionally affected by the intense criticism for the purpose of appearing to be leader type material, which is substantially not aloofness from the community.

It's not a convincing signal primarily because it's idiosyncracy highlights it for analysis, but I still think the above holds.

Comment author: komponisto 25 August 2011 02:40:15PM 7 points [-]

I am very confident the intention was to signal that Luke was not being emotionally affected by the intense criticism for the purpose of appearing to be leader type material

Or, in another words, signaling high status -- just like I said.

which is substantially not aloofness from the community.

It may not be aloofness, but it certainly is distance (I used two words for a reason); a leader is, necessarily, separated in some way from those who are led.

Comment author: NancyLebovitz 26 August 2011 11:23:18AM 1 point [-]

I saw lukeprog's signing messages as minor noise and possibly a finger macro developed long ago, so I stopped seeing the signature.

I'd be dubious about assuming one can be certain (where's the Bayesianism?) about what someone else is intending to signal, especially considering that it's doctrine here that one can't be certain of even one's own motivations. How much less certain should one be about other people's?

I would add some further uncertainty if one feels very sure about the motivation driving a behavior that's annoying.

Comment author: saturn 26 August 2011 04:20:52PM 11 points [-]

I just looked through several pages of lukeprog's most recent comments, and the only ones that were signed were direct replies to SilasBarta.

Comment author: lessdazed 26 August 2011 11:34:18AM *  6 points [-]

doing so signals...a passive-aggressive assertion... emits such signals rather often -- he may want to be aware of this in case it's not his intention.

vs.

I'd be dubious about assuming one can be certain (where's the Bayesianism?) about what someone else is intending to signal

I didn't read komponisto as necessarily or primarily talking about consciously intended signals.

Comment author: SilasBarta 24 August 2011 09:45:46PM *  8 points [-]

I get annoyed by people who "sign" posts in the text like that, especially when they do it specifically on replies to me. It really isn't necessary. I'm interested in substance, not pleasantries, as I was three months ago when I asked how the mini-camp was a success.

Comment author: lukeprog 25 August 2011 12:30:49AM 6 points [-]

Hmmm. Fair enough. Didn't mean anything by it.

Comment author: AuthorityFigure 25 August 2011 03:18:29PM 1 point [-]

" I'm interested in substance, not pleasantries"

You are so right. Fuck being nice as it is merely a tool for the irrational.

Comment author: [deleted] 25 August 2011 11:59:26PM 3 points [-]

a tool for the irrational

Sign me up.

Comment author: SilasBarta 24 August 2011 08:13:44PM *  0 points [-]

Thanks, Vladimir.

Comment author: timtyler 24 August 2011 10:14:03PM *  0 points [-]

I mean 'evidence' in the Bayesian sense, not the scientific sense.

Shouldn't these be the same? Bayesian evidence is surely scientific evidence - and visa versa. I don't see much point in multiplying definitions of "evidence". Let's just have one notion of "evidence", please. Promoting multiple "evdience" concepts seems to be undesirable terminology - unless there's a really good reason for doing so.

Comment author: Vladimir_Nesov 24 August 2011 10:21:57PM *  8 points [-]

There is a good reason. A lot of things people know can't contribute to forming reliable public knowledge for all sorts of practical reasons. And you know the reference for the arguments about this question: Scientific Evidence, Legal Evidence, Rational Evidence.

Comment author: timtyler 24 August 2011 11:43:11PM *  7 points [-]

Actually, that helps. As a teenager, I noticed that most of the scientific method, including the key concept of experimentation, extended to personal knowledge and understanding. So, I did what seemed to be the obvious thing: I expanded my conception of science to include that domain. The public-only conception of science wasn't really much of a natural kind - since eventually technology would gain access to people's minds.

That explains why I don't get very much out of the Science vs Bayes material on this site. To me it just looks as though the true nature of science has not been properly grokked.

I must say, I still like my way: expanding the definition of science a teeny bit has a number of virtues over trying to stage a rationality revolution.