Louie comments on Experiment Idea Thread - Spring 2011 - Less Wrong

28 Post author: Psychohistorian 06 May 2011 06:10PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Article Navigation

Sort By: Best

You are viewing a single comment's thread. Show more comments above.

Comment author: Louie 08 May 2011 06:41:49PM 0 points [-]

A few of us at Singularity Institute tested Dual N-Back last year. For 1 week, 13 people were tested on dissimilar metrics of intelligence while some of them performed the same kind of Dual N-Back done in the original Jaeggi study.

Conclusion: It doesn't make you smarter.

Bonus: You get better at Dual N-Back though!

Interestingly, at around the same time as we were doing our tests last last year, the original research "replicated" her own results and published them again using new data. I'm sort of confused. I don't want to say Jaeggi doesn't understand training and practice effects... but I'm struggling to understand how else to explain this.

That said, it would still be cool to see LW folks test IA interventions. I just recommend exploring more promising ones. Perhaps seeking to confirm the results of these studies instead?

Comment author: AnnaSalamon 08 May 2011 11:42:57PM * 12 points [-]

Louie, I don't remember the details of this. I thought folks ended up with a very small and not-very-powerful study (such that the effect would have had to be very large to show up anyhow), with the main goal of the "study", such as it was, being to test our procedures for running future potential experiments?

Could you refresh my memory on what tests were run?

Also, speaking about how "the folks at SIAI believe X" based on small-scale attempt run within the visiting fellows program last summer seems misleading; it may inaccurately seem to folks as though you're speaking for Eliezer, Michael Vassar, or others. I was there (unlike Eliezer or Michael) and I don't recall having the beliefs you mention.

Comment author: Will_Newsome 10 May 2011 11:47:20AM * 1 point [-]

This definitely agrees with my memory, too...

I mostly felt compelled to actually sign in and comment because I wanted to point out that designating something as "[t]he opinion of folks at Singularity Institute" or the like is often an annoying simplification/inaccuracy that allows people to thenceforth feel justified in using their social cognition to model a group roughly as they would a person.

Comment author: gwern 11 May 2011 02:34:28PM 0 points [-]

When I asked back in September 2010, you said

They're ad hoc, we've used one for a dual n-back study which ended up yielding insufficient data....We didn't study long enough to get any statistically significant data. Like, not even close....really, there's no information there, no matter how much Bayes magic you use

So at least your memory has been consistent over the last 9 months.

Comment author: cousin_it 08 May 2011 06:49:05PM * 8 points [-]

Hmm. If your replication attempt was good science, you could help the world by publishing it. If it wasn't good science, you probably shouldn't update on it very strongly.

Comment author: Louie comment score below threshold [+] (10 children)

Comment author: cousin_it 08 May 2011 08:57:28PM * 6 points [-]

I'm not sure what the opinions of folks at SIAI have to do with this (just mentioning them doesn't constitute an valid argument, I'm not at SIAI, I was speaking seriously), but I can recall a quote from Eliezer expressing a sentiment that's pretty close to my second one:

"What happened to me personally is only anecdotal evidence," Harry explained to her. "It doesn't carry the same weight as a replicated, peer-reviewed journal article about a controlled study with random assignment, many subjects, large effect sizes and strong statistical significance."

-- HP:MOR, Ch.6

But that's not very relevant. What's relevant is this: if I believed in the efficacy ot the DNB task, it would be wrong for me to change my opinion substantially after reading your comment on LW. 13 people, 1 week, methodology and results unpublished and unverified?

Comment author: Louie comment score below threshold [+] (2 children)

Comment author: cousin_it 09 May 2011 09:04:16PM * 2 points [-]

Re your edit: yes, I was referring to the theoretical ideal of Science. (Don't care about Journals.) If a lot of published science is bogus, IMO the right response is to try to do better and nail down our results with more precision, not less. Especially in a topic as important as intelligence amplification.

That said, I wasn't very convinced by the evidence in favor of the DNB task in the first place. In my mind the jury's still out.

Comment author: Louie 09 May 2011 09:49:18PM 0 points [-]

the right response is to try to do better

Agreed. Better is better.

and nail down our results with more precision, not less

I disagree. For example, you can rule out Dual N-Back as a possible Intelligence Amplification intervention with less precision than Jaeggi used to repeatedly mis-prove it as one. Depends on what you mean by precision I suppose. If you mean more time, effort, people, and statistical significance then precision is not needed. If by precision, you just mean being more right... well, I agree, we should be more right.

Most bogus science is very precise: That's why it looks stronger than it is. Poor methodology and experimental design will still allow someone to prove any correlation with p < 0.05 significance. If I want to disprove someone who published an incorrect result, should I have to expend more time, people, and resources than they used just to over-prove the counter-claim with "more precision" -- even though their claim was never wrong due to "lack of precision" in the first place?

Calling for "more precision" is like calling for "more preparation". It has 100% applause appeal and costs nothing for people to call for. But it costs people actually doing research a lot of time. When you advocate for smarter people to use "more precision", you're also advocating for smarter people to do "less research"... the extra precision comes from somewhere.

Are you actually in favor of smarter people doing less research than they currently do?

Comment author: Desrtopa 11 May 2011 04:10:21AM 2 points [-]

Are you actually in favor of smarter people doing less research than they currently do?

Considering how much research, given the low levels of confidence warranted by its methodology, is essentially worthless, yes, I am willing to say without reservation that there is dead wood to cut away.

Comment author: cousin_it 10 May 2011 08:56:07AM * 3 points [-]

Let's agree on the interpretation "we should be more right" and skip over the issues of time and costs.

Sometimes a published result can indeed be overturned by a small amount of Bayesian evidence. But that's only possible if you also prove that your methodology was much more right than the original paper's methodology. Right now I have no way of knowing that from your comments. If you add a critique of Jaeggi's study and an explanation why your study was better, that will work for me.

Comment author: Vladimir_Nesov 10 May 2011 09:48:46AM 3 points [-]

Are you actually in favor of smarter people doing less research than they currently do?

Downvoted for this piece of empty rhetoric.

Comment author: Will_Newsome 10 May 2011 11:55:45AM 0 points [-]

Upvoted; I think this is a good downvoting policy but hope that whoever uses it takes the time to point out what they perceive as empty rhetoric. (I think the habit of spouting such rhetoric is particularly poisonous and particularly easy to stop, making it rather worth the effort of correcting.)

Comment author: [deleted] 08 May 2011 07:08:50PM * 4 points [-]

For 1 week, 13 people were tested on dissimilar metrics of intelligence

One week seems very short compared to the studies. I didn't check all, but one mentioned in Wikipedia was 5 weeks and another whose abstract I found was 4 weeks. It seems probable that any effects on the brain would be slow to accumulate. As a point of comparison, it takes a lot longer than one week to assess the value of a particular muscle training program.

Louie comments on Experiment Idea Thread - Spring 2011 - Less Wrong

Article Navigation

Comments (53)

Nearest Meetups

Virtual Study Room

Recent Comments

Recent Posts

Recent Rationality Quotes

Recent Wiki Edits

Top Contributors, 30 Days

Recent Karma Awards

Comments (53)

Top Contributors, 30 Days

You'll need to login or register to do that

(Don't worry, it only takes a few seconds)

Create

Login