= 27d567bc2d48d9f40e9ccc20ea370b15)
SilasBarta comments on Help Fund Lukeprog at SIAI - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (276)
SilasBarta,
We collected lots of data before and during minicamp. We are waiting for some time to pass before collecting followup data, because it takes time for people's lives to change, if they're going to change. Minicamp was only a couple months ago.
Minicampers are generally still in contact, and indeed we are still gathering data. For example, several minicampers sent me before and after photos concerning their fashion (which was part of the social effectiveness section of the minicamp) and I'm going to show them to people on the street and ask them to choose which look they prefer (without indicating which is 'before' and which one is 'after').
So yes: by all qualitative measures, minicamp seems to have been a success. The early quantitative measures have been taken, but before-and-after results will have to wait a while.
As for future rationality training, we are taking the data gathered from minicamp and boot camp and also from some market research we did and trying to build a solid curriculum. To my knowledge, four people are seriously working on this project, and Eliezer is one of them.
Cheers,
Luke
So it's early enough to call it an unqualified success, but too soon for evidence to exist that it was was a success? If I have to be patient for the evidence to come back, shouldn't you be a little more patient about judging it a success?
Edit: I gave a list of information you could post. The fashion part isn't suprising enough to count as strong evidence, and was a relatively small part of the course that, in any case, you previously claimed could be accomplished by looking at a few fashion magazines.
I mean 'evidence' in the Bayesian sense, not the scientific sense. I have significant Bayesian evidence that minicamp was a success on several measures, but I can't know more until we collect more data.
Thanks for providing a list of information we could post. One reason for not posting more information is that doing so requires lots of staff hours, and we don't have enough of those available. We're also trying to, for example, develop a rationality curriculum and write a document of open problems in FAI theory.
If you're anxious to learn more about the rationality camps before SI has time to publish about that data, you're welcome to contact the people who attended; many of them have identified themselves on Less Wrong.
I'm fairly confident that campers got more out of my fashion sessions than what they can learn only from looking at a few fashion magazines.
Cheers,
Luke
This comes off very strongly as the typical bureaucratic protectiveness - a business doesn't want to share raw data, because raw data is a valuable resource. If you came out and said this was the reason, I'd be more understanding, but it would still feel like a major violation of community norms to be so secretive.
If simple secrecy is indeed the case, I would urge you, please, be honest about this motive and say so explicitly! At least then we are having an honest discussion, and the rest of this comment can be disregarded.
In short, what is the reason you can't share this RAW data, which you state you collected, and which you've presumably found sufficient for your own preliminary conclusions? I don't think Silas is asking for or expecting an elegant power-point presentation or a concise statistical analysis - I know I would personally love to simply see raw data.
Is there truly not a single spreadsheet or writeup that you could drop up for us to study while you collect the rest of the data?
Good grief, people. There are conspiracies that need ferreting out, but they do not revolve around generating fake data about the effectiveness of an alpha version of a rationality training camp that was offered for free to a grateful public.
I went to the minicamp, I had a great time, I learned a lot, and I saw shedloads of anecdotal evidence that the teachers are striving to become as effective as possible. I'm sure they will publish their data if and when they have something to say.
Meanwhile, consider re-directing your laudable passion for transparency toward a publicly traded company or a medium-sized city or a research university. Fighting conspiracies is an inherently high-risk activity, both because you might be wrong about the conspiracies' existence, and because even if the conspiracy exists, you might be defeated by its shadowy and awful powers. Try to make sure the risks you run are justified by an even bigger payoff at the end of the tunnel.
I don't think anybody is accusing the minicamp folks of anything of the kind. But public criticism and analysis of conclusions is the only reliable way to defend against overconfidence and wishful thinking.
When I ended my term as an SIAI Visiting Fellow, I too felt like the experience would really change my life. In reality, most of the effects faded away within some months, though a number of factors combined to permanently increase my average long-term happiness level.
Back then the rationality exercises were still being worked out and Luke wasn't around, so it's very plausible that the minicamp is a lot more effective than the Visiting Fellow program was for me. But the prior for any given self-help program having a permanent effect is small, even if participants give glowing self-reports at first, so deep skepticism is warranted. No conspiracies are necessary, just standard wishful thinking biases.
Though I think this was the third time that Silas raised the question before finally getting a reply, despite his comment being highly upvoted each time. If some people are harboring suspicions of SIAI covering up information, well, I can't really say I'd blame them after that.
For the record, I for one don't recall reading any of SilasBarta's earlier comments on this topic.
It seems rather unlikely to me that being a mini-camp participant would have more of an effect on someone's life than being a Visiting Fellow, new techniques or not-- and if I am wrong, I would very much want to encounter these new techniques!
I wouldn't be that surprised. Explicit rationality exercises were only starting to be developed during the last month of my stay, and at that point they mostly fell into the category of "entertaining, but probably not hugely useful". The main rationality boost came from being around others with a strong commitment to rationality, but as situationist psychology would have it, the effect faded once I was out of that environment.
The positive endorphin rush from you and lukeprog sends signals that loook just like the enthusiastic gushing I see from any week-long "how to fix your life in five easy steps!" seminar. Smart people get caught up in biased thinking all the time. I had a good friend quit AI research to sell a self-help book, so I may be particularly sensitive to this :)
Objective data means I can upgrade this from "oh bunnies, another self-help meme" to "oooh, fascinating and awesome thing that I want to steal for myself." As long as it signals like a self-help meme, I'm going to shoot it down just like I'd shoot down any similar meme that tried to sell itself here on LessWrong.
All right, but there's a fine line between shooting down self-help memes and unnecessarily discouraging project-builders from getting excited about their work. It's not fun or helpful for a pioneer to have his or her every first step be met with boundless skepticism. Your concerns sound real enough to me, but even an honest concern can be rude, and even a rationalist can validly trade off a tiny little bit of honesty for a whole lot of politeness and sympathy.
Why do I say "a tiny little bit of honesty?" Well, if the minicamp were being billed as "finished," "polished," "complete," "famous," "proven," or "demonstrably successful," as many self-help programs are, then it would make sense to demand data supporting those claims.
Instead, the PR blurb says that "Starting on May 28th, the Singularity Institute ran a one-week Rationality Training Camp. Our exit survey shows that the camp was a smashing success, surpassing the expectations of the organizers and the participants."
Leaving aside the colorful language that can and should characterize most press releases, this is a pretty weak claim: the camp beat expectations. Do you really need to see data to back that up?
"Please give us money" and "Co-organized and taught sessions for a highly successful one-week Rationality Minicamp" are stronger claims.
For me, this isn't about making SIAI transparent; it does quite enough in that regard. It's about stopping an information cascade genie that's already out of the bottle.
Let me put it this way: right now the ratio of "relying on the assumption of mini-camp's success for decision making" to "available evidence for its success" is about 20-to-1. As I warned before, it's quickly becoming something "everyone knows" despite the lack of evidence (and major suspicions of many people that it wouldn't succeed going in). And that believe will keep feeding on itself unless someone traces it back to its original evidence.
It doesn't reassure me that I'm told I have to keep waiting before anything's conclusive, yet they can declare it a success now.
I just want the reliable evidence they claim to have, rather than just dime-a-dozen self-help testimonials. They collected hard data, and I gave them a list of things they could provide that are easy to gather and don't compromise privacy, and are much more likely to be present if the success were real than if it were not. Even after AnnaSalamon's circling of the wagons I don't see that.
I think this is largely a case of people reading different things into 'success'.
Oh, sure. The reason is easy to communicate. We explicitly told minicampers that their feedback on the exit survey would be private and anonymous, for maximal incentive to be direct and honest. We are not going to violate that agreement. The testimonials were given via a separate form with permission granted to publish THAT data publically.
I'm unclear, then, why you are citing a lack of staff hours if the data cannot be published at all.
Has the raw testimonial data been published?
I'm assuming you have data beyond exit surveys and testimonials...?
A summary of the data can be published, for example median scores for measured values. But the data can't be published in raw form.
Not sure if raw testimonial data has been published yet. We do have data beyond testimonials and exit surveys, but that, too, requires precious staff hours to compile and write up, and it is still in the process of being collected.
Typing this stuff from a phone, pardon the brevity...
Great, so did I! Now communicate that evidence. If it can't be communicated, I don't think you should be so confident in it.
I find that hard to believe. It may take time for the participants to report back, but not for you to tabulate the results.
I'm sorry, but this just sounds like excuse-making. Do you want your audience to be people who just take your word on something like this? I've asked several times for some very simple checks. This claim that you're too busy just doesn't fly.
Then why don't you mention this in your "how to be happy" post, which is also being used as evidence of your productivity? Do you know a single person who has improved fashion to an acceptable level as a result of those magazines?
Not necessary.
Here:
No, it definitely takes time to tabluate the results and write a presentable post about the results. I've personally spent 3 hours on it already but the project is unfinished.
Ah. I may not have communicate this clearly: I think your skepticism concerning the success of the minicamp is warranted because almost no evidence is available to you. You're welcome to not take my word for it. When I have another 5-10 hours to finish putting together the results and write a post with more details about minicamp, I will, but I'm mostly waiting to invest that time until I can do it most profitably, for example when we've gathered more 'after minicamp' data.
I don't understand. The 'How to Be Happy' post was written before I helped run minicamp. And, there are tons of things not mentioned in that post. That post barely scratches the surface of my thoughts on happiness, let alone research on happiness in general.
I doubt magazines is ever the sole input on someone's fashion sense, but yes I know people who have improved their fashion as a result of following magazines (or fashion blogs; same thing basically). Ask Peter Scheyer about this, for example.
Given the large amount of effort it took to get to the miny camps, all four of these could be easily explained by cognitive dissonance.
What evidence would you expect them to have if the "minicamp" was a genuine success? (Edited - thanks for the correction, wedrified!)
Bootcamp? I found the wording Eliezer used fascinating:
Have they actually claimed anywhere here that the bootcamp was successful?
Oops, fixed - thanks!
Honestly, I'm not sure. Having a randomized control group and then looking at actual success would be nice. Even without a good control, one obvious thing to do would have been to do before and after tests of similar questions that test for rational behavior (e.g. whether they can recognize they are engaging in the sunk cost fallacy and things like that). It may be that given the circumstances the best evidence we have is self-reporting like this. If so, it is evidence for the success of the minicamps. But, it is not very strong evidence precisely because it is consistent with a variety of other not implausible hypotheses. This thread has made me more inclined to believe that the minicamps were successful, but had not strongly increased my confidence.
(I disapprove of downvoting the parent (which I just found at '-2'). It continues the same conversation as the previous Silas's posts, pointing out what does look like rationalization. If raising a possibility of interlocutor's rationalizing in defense of their position is considered too rude to tolerate, we'll never fix such problems.)
I suspect that most of the downvotes came from the very last sentence, which struck me as more than a little snarky. "Cheers" might not be necessary, but it is a gesture of politeness and was probably added in an attempt to convey a positive tone (which is important but somewhat tricky in text). I wouldn't say "not necessary" if someone held the door for me, even if it is obviously true.
Agree with you that the actual substance of the post was in no way downvote-worthy.
Because "signing" comments is not customary here, doing so signals a certain aloofness or distance from the community, and thus can easily be interpreted as a passive-aggressive assertion of high status. (Especially coming from Luke, who I find emits such signals rather often -- he may want to be aware of this in case it's not his intention.)
I interpret Silas's "Not necessary" as roughly "Excuse me, but you're not on Mount Olympus writing an epistle to the unwashed masses on LW down below".
Second this.
No. I am very confident the intention was to signal that Luke was not being emotionally affected by the intense criticism for the purpose of appearing to be leader type material, which is substantially not aloofness from the community.
It's not a convincing signal primarily because it's idiosyncracy highlights it for analysis, but I still think the above holds.
Or, in another words, signaling high status -- just like I said.
It may not be aloofness, but it certainly is distance (I used two words for a reason); a leader is, necessarily, separated in some way from those who are led.
I saw lukeprog's signing messages as minor noise and possibly a finger macro developed long ago, so I stopped seeing the signature.
I'd be dubious about assuming one can be certain (where's the Bayesianism?) about what someone else is intending to signal, especially considering that it's doctrine here that one can't be certain of even one's own motivations. How much less certain should one be about other people's?
I would add some further uncertainty if one feels very sure about the motivation driving a behavior that's annoying.
I just looked through several pages of lukeprog's most recent comments, and the only ones that were signed were direct replies to SilasBarta.
Which could just mean that he feels the need to counter hostility with extra friendliess.
That's interesting. Thanks for checking. I still have no idea what lukeprog intended, but my finger macro theory is clearly wrong.
vs.
I didn't read komponisto as necessarily or primarily talking about consciously intended signals.
I get annoyed by people who "sign" posts in the text like that, especially when they do it specifically on replies to me. It really isn't necessary. I'm interested in substance, not pleasantries, as I was three months ago when I asked how the mini-camp was a success.
Hmmm. Fair enough. Didn't mean anything by it.
" I'm interested in substance, not pleasantries"
You are so right. Fuck being nice as it is merely a tool for the irrational.
Sign me up.
Thanks, Vladimir.
Shouldn't these be the same? Bayesian evidence is surely scientific evidence - and visa versa. I don't see much point in multiplying definitions of "evidence". Let's just have one notion of "evidence", please. Promoting multiple "evdience" concepts seems to be undesirable terminology - unless there's a really good reason for doing so.
There is a good reason. A lot of things people know can't contribute to forming reliable public knowledge for all sorts of practical reasons. And you know the reference for the arguments about this question: Scientific Evidence, Legal Evidence, Rational Evidence.
Actually, that helps. As a teenager, I noticed that most of the scientific method, including the key concept of experimentation, extended to personal knowledge and understanding. So, I did what seemed to be the obvious thing: I expanded my conception of science to include that domain. The public-only conception of science wasn't really much of a natural kind - since eventually technology would gain access to people's minds.
That explains why I don't get very much out of the Science vs Bayes material on this site. To me it just looks as though the true nature of science has not been properly grokked.
I must say, I still like my way: expanding the definition of science a teeny bit has a number of virtues over trying to stage a rationality revolution.