Scientifically optimizing education: Hard problem, or solved problem? Introducing the Theory of Direct Instruction

[-]Alicorn14y590

I have only skimmed your post, but now feel motivated to leave feedback as requested. It is possible that some of my objections are misplaced, addressed somewhere in the depths of this article that my eyes glazedly passed over. In fact, my first complaint is that:

it is too long. LW tolerates long articles under limited circumstances and this doesn't meet any of them (you're not an established poster, don't have fifty footnotes with sources, don't apologize off the bat for length, and have missed many obvious opportunities for compression/excision). You should have made it much shorter (500 words about what the hell Direct Instruction consists of) or much much shorter (a two-sentence blurb with a link to more information).
It's sales-y. Full of applause lights (counted five instances of the string "rational" in your text). You claim that your intent is to pique interest, but that is not done by saying "This thing is interesting! This thing is interesting!" repeatedly in the local idiom.
It is badly structured. Rambles all over the place. If you laid out the contents of your article in conceptspace and made me walk from point to point in the order you present them, my feet would get tired and I would become dizzy. You have definitely not convinced me that you have learned a secret of how to teach things, on a meta as well as object level.
It makes you look like a crank. If DI needs this much fluff and meandering and enthusiastic pitching, it's probably not interesting. Oops.

In fact, the only reason I am bothering to think about this article ever again, having successfully scrolled all the way down to the unnecessary signature, is that you do repeatedly ask for feedback. If you're sincere about that: I invite you to post, as a reply to this comment, a 1-3 sentence description of what DI is, plus one sentence about whatever evidence (beyond your enthusiasm about it) which exists for its splendidness. (Last sentence but not the first 1-3 can be/consist primarily of linkage.)

[-]NancyLebovitz14y220

I'm in strong agreement, with the one addition that it would be interesting to see an explanation of DI using DI principles, and then an explanation of how the principles were used to shape the explanation.

[-][anonymous]14y20

I'd like to see this too. The example given of DI was way too contrived.

[-]fubarobfusco14y150

Yes.

I felt like I was slogging through this article to glean the relevant bits, which turned out to be the axioms and principles and the history and study evidence for the efficacy of the system.

To the author:

Keep those relevant bits. Expand on them with examples and ramifications. Cut everything else. Aggressively.

Assume that you're talking to people who already care about the effectiveness of instructional methods; who are already basically aware of issues such as neurotypicality/neurodiversity even; and who are aware of the difference between saying you care about a goal, and actually acting to maximize that goal.

Assume that we already have knowledge and skills that we desperately want to teach, because we highly value ensuring that humanity retains that knowledge and those skills. Give us tools to accomplish our existing goals.

[-]Owen_Richardson14y120

Ouch ouch ouch, but thank you for helping me.

I'll have to think about a good "1-3 sentence description of what DI is" (in terms of what you can do with it, or in terms of its internal structure, or...?) while I'm at work, but as for evidence, if I said:

"Look at those two graphs from Project Follow-Through, because the meta-analysis says this stark difference between DI and other models of education keeps showing up in experiments comparing them."

Is that at all useful?

[-]Vaniver14y60

How about:

Direct Instruction is an educational theory which extensively used experiments during its creation, and seeks to explain educational concepts in a sensible and efficient way. The amount of experience individual teachers can acquire pales in comparison to the amount of experience curriculum-builders can acquire, which DI takes advantage of by giving teachers heavily researched, and thus effective, scripts and practices to follow. Experiments dramatically verify the superiority of DI over other instruction methods.

In three sentences, you can't explain anything about the method besides its inputs and its outputs, which is what other people are interested in. If they've got a use for your method, then they'll start asking about its moving parts.

[-]prase14y30

That explains DI's origins, but doesn't say anything about how it looks like and how it is different from other approaches. (Why it is called Direct Instruction, after all?)

[-]Vaniver14y40

Right- that's because I don't know anything about what it looks like and how it's different from other approaches. (I haven't dug any deeper than this article and wikipedia).

[-]FiftyTwo14y80

Agreed, the main problem is that you don' seem to explain what the theory is until deep into the text. the section 'what it looks like in practice' doesn't actually tell us anything other than that scripts are maybe involved, and that teachers don't like it (personally i found the tone unnecessarily harsh there).

The next sections with graphs seem persuasive, but again at this point we don't know what it is they are proposing, or where this data is coming from. Perhaps better would be just to link to the study with a quick (one or two sentence) summary of its findings.

Finally we get an explanation of what it is we have been talking about all this time in 'a quick sketch of the basic theory.' First paragraph is why we should like it, then an explanation of how you are explaining it.

As far as I can tell what its saying is:

Use examples to illustrate the concept, they should be similar in only the important ways, and counter examples that are different in relevant ways. Information should be arranged in a logical way,

All I get from the example is that its better to teach via rules than via memorisation.

At the risk of being harsh or flippant is that it? That doesn't seem like a massive educational innovation to me (though I should say I don't know much about the US system which seems the example here).

If there is something more to it (which I would hope given your enthusiasm) you need to make it very clear what that is. All the stuff about evidence, teachers and examples is unhelpful until we know that.

[Hope this helps, apologies if it sounds harsh.]

[-]Vaniver14y90

that teachers don't like it (personally i found the tone unnecessarily harsh there).

From my understanding of the alternate education literature, pretty much everyone is bitter at teachers as a group and has pretty good reason to be. The harshness didn't even register until now because I see it so often.

[-]FiftyTwo14y40

Is the thesis that the majority of teachers and education employees are engaged in an active conspiracy to prevent the introduction of new teaching methods? I find that difficult to believe without substantial evidence. (It sets off my warning lights by similarities to claims made by 'alternative' medicine advocates about the biased scienetifc establishment and general conspiracy theories).

Even if a conspiracy existed, given the incentives individual teachers/schools/local governments have to improve performance of students why wouldn't they defect and reap the benefits?

[-]Mercurial14y240

Is the thesis that the majority of teachers and education employees are engaged in an active conspiracy to prevent the introduction of new teaching methods?

I can't speak in sweeping generality here, but I can speak from my own experience. (I'm finishing a dissertation in educational research and bump into this pretty frequently.)

Teachers who have been "in the trenches" for any appreciable length of time (say, 5+ years) have developed some sense of what is necessary from a pragmatic point of view. They're usually extremely well-intentioned toward the children, and often for that reason they resist a lot of the suggestions brought to them from educational research. There's this sense that education researchers are too ivory-tower to know how things really happen in a classroom. There are exceptions, but they're actually quite rare.

However, not all education researchers are naive about this issue. I know of many who were teachers in public schools for a decade or so before they moved on to research to do something about the mess they personally encountered. They're able to build face-to-face rapport more readily with teachers. But that's simply not good enough to convince teachers as a collective to try something new, even if the pragmatics of how to do it are spelled out in excruciating detail. They seem to resist change purely because it's change, especially if it interferes with their personal emotional impressions of what it means to teach (which, unfortunately, were instilled more by accidental impressions than by training or rational scrutiny).

In addition to this challenge, a painfully large portion of education researchers don't apply reasonable scientific methods. The majority of articles I've encountered in this field amount to philosophical commentary, which is often based on spotty evidence that to me would constitute preliminary research. So surprise surprise, a lot of the reform suggestions being proposed are saturated in cognitive biases that no one seems to be at all aware of let alone make any effort to account for. And, thus, most "well-researched" reform suggestions really don't work in classrooms! So on the rare occasions that teachers are willing to try something radically new based on a new theory (or are required to by fiat), in most cases it doesn't make much of a difference at best.

And this is ignoring the fact that researchers frequently forget about (or never consider!) the inferential distance teachers have to traverse to even understand what they're supposed to do let alone do it well enough to teach. The "New Math" of the 1960s was a spectacular failure both because the theory behind it was psychologically and neurologically bankrupt and also because the teachers weren't able to implement the original idea anyway.

So no, I don't think it's a conspiracy. I think it's more that teachers are tired of having people try to mechanize their profession as though it didn't require any skill to interact with children, especially since the vast majority of attempts to do so that they've encountered failed dramatically - often because the reform wouldn't have worked anyway, but often because it was implemented very poorly due to undertraining the teachers in the first place.

[-]Owen_Richardson14y20

Of course it's not a "conspiracy" per se. It's just yer standard lost purposes at work, in a field where the population really doesn't have a strong grasp of the basic philosophy of science, leaning very much to "romanticism" rather than "enlightenment", I mean.

Does the Rosenhan experiment imply any 'conspiracies' in psychiatry?

[-]Owen_Richardson14y50

Thanks for your help. It's more... well, look: We at LessWrong are very familiar with the idea that if you really want to pin some hypothesis down, there's a certain minimum amount of work you have to do. You can't just magically jump there without certain bits of information, and processing the bits together in the right way. And most of the work goes into just locating the general area of the correct hypothesis.

Okay, that's obvious. We get that. The big thing DI does (in the stimulus-locus analysis) is turn that sideways and apply it to teaching.

There are certain bits of information without which the naive student can not just magically figure out what we're trying to communicate, so we need to sequence the introduction of those bits of information, make each logically unambiguous, and prompt the proper processing of the bits together in a manageable context that lets us ensure each step has gotten across properly before moving on to the next bit. The teaching communication should be designed to at least strongly imply the correct conclusion as early as possible.

That make any sense?

[-]hamnox14y60

Aha!

This is a very useful and relevant explanation, which would have been unutterably more useful to read in the article itself.

[-]Owen_Richardson14y00

Thank you, I'm glad to know I'm at least making some progress (I sweated over my first attempt for ages and it ended up terrible, but then just a little time of feedback and back and forth discussion seems to really be tightening up my understanding of what I need to communicate and how! That's probably a highly generalizable principle :P - actually reminds me of some "just make an attempt already if failure is low cost!" post I saw some months back in main... can't find it right now, but maybe you remember the one I mean. [Edit: it was "Just Try It")

Anyway, is this also helpful? (Even 'unutterably' so? :P)

[-]Scott Alexander14y170

Agree that the article has serious structure and style problems, but a quick check at Wikipedia confirms that DI seems to be approximately as awesome as you say it is, and the biases of the educational establishment certainly seem worth investigating. Upvoted.

[-]Owen_Richardson14y00

I agree it has serious problems. I knew that when I posted it! I just thought the harder bits of content studded here and there would anchor the rest better until people got a better idea what I was talking about from their own research. Clearly I made some miscalculations.

Do you think the notes I added to the beginning as a replacement are a good start at straightening out the mess of confusion I've made?

ADD-ON: Are this comment and this comment helpful? (Reading those as a conversation with FiftyTwo, I mean).

Oh, and by the way, I wanted to tell you that your post on "dissolving diseased questions" holds a special place in my heart. I found it soon after finding LessWrong itself (through HPMOR, through David Brin's blog), and it was just wonderful, the feeling of, "There are other people in the world who think like this! And understand it better than I do, so not only can I communicate with them, I can learn from them!"

So, uh, yeah. Thanks =]

[-]prase14y130

Downvoted. The information is too diluted, the post is too long and badly formatted, and after reading it I don't know what Direct Instruction is and how it differs from other approaches. The post is written in the same style as all spamish advertisements recommending products whose only description is that they are amazing and revolutionary. In fact, I am very surprised that this post gained so much upvotes, if it hadn't, I would probably quit reading after the first paragraph.

To improve the structure, I'd suggest:

Explain as concisely as possible what DI is and how it is different from other approaches. You have said that DI somehow uses tested algorithms. Is that it? Are there any other postulates or specific characteristics of the method? Why is it called Direct?
Give few actual examples. All I have found is the fictional example with teaching numbers (did somebody try your method to teach numbers in practice?) and an anecdote about your experience with learning languages (different people like different methods, why do you think your taste is universal here?), and there is no clear way to see how these examples correspond to DI.
Since your graphs are the most salient evidence of effectivity of DI, they should be accompanied by a link to their source.
A slightly less emotional and more technical style would improve apparent trustworthiness of the post. Specifically, don't replace evidence by strong words ("whosoever adopts this crazy ‘teach my kindergarteners 1-20’ goal is going to horribly slow down and confuse their kids").

[-]Owen_Richardson14y10

Well... yeah, you pretty much nailed it with "too long and badly formatted". NOT my best piece of writing ever. Yeesh did I ever drop the ball on that one.

Do the notes added at the beginning as a replacement for the whole long thing help to start clearing stuff up any?

[-]prase14y30

The notes are somewhat helpful, yes. They directed me at the Athabasca University page where is a sort of ... description. Well, to my dismay even they don't say what DI is (except six vague features - that isn't too much given that they say it is the only theory of instruction) before they give the readers an excercise whether they can recognise DI or not. But perhaps that's DI in practice.

Now, let me ask few questions about the Athabasca University presentation. They say

In Direct Instruction the learners are usually responding at least 10 times per minute. This very high pace probably contributes to the heightened attention and reduced boredom observed during Direct Instruction.

It means that the teacher has on average 6 seconds to (a) convey new information (b) formulate the question (c) wait for the students' replies (d) tell them whether they are right. This seems impossible even if we allow all learners responding at once and if the taught material needs only crude memorisation, with no students' questions, no explanations, no writing, reading, looking at graphs... So, is the claim that each learner responds 10 times per minute correct?

There is another suspect claim:

Q: If faultless instruction fails to result in appropriate learning: A: Faultless instruction, if truly faultless, does not need revision.

This is almost a tautology (with great potential for equivocation in appropriate, fail, faultless, need etc.), but can be also viewed as a claim that DI doesn't need revision (the answer I chose, the instruction has a problem and needs revision, has been claimed incorrect) no matter what evidence we get. Do the DI proponents in general think that when DI fails, it's never a sign of errors in theory, but rather imperfections in implementation of the method?

And, more generally, all examples given may be used for teaching categorization of objects. How do you teach algorithms (such as multiplication)? How do you teach history and geography? How do you teach calculus? How do you teach scientific method? Not every knowledge can be reduced to questions of form "does X have property Y" taught by presenting series of objects which either are or aren't Y. In the whole presentation there was not a single practically applicable example. Children don't need to go to school to learn what "is longer than" or "not horizontally aligned" means.

[-]Owen_Richardson14y10

{continued from last comment because of character limit}

(Again, thank you so much for working with me so patiently to get through such a big inferential distance!)

You said:

There is another suspect claim:

Q: If faultless instruction fails to result in appropriate learning: A: Faultless instruction, if truly faultless, does not need revision.

This is almost a tautology (with great potential for equivocation in appropriate, fail, faultless, need etc.), but can be also viewed as a claim that DI doesn't need revision (the answer I chose, the instruction has a problem and needs revision, has been claimed incorrect) no matter what evidence we get. Do the DI proponents in general think that when DI fails, it's never a sign of errors in theory, but rather imperfections in implementation of the method?

“Faultless communication” is the basis of the stimulus-locus analysis branch of the theory. If faultless communication fails with a particular learner, that gives you specific information about how the learner is not using the two-attribute learning mechanism. That tells you to shift to the response-locus analysis branch of the theory to figure out how to modify the learner so that they do use it (and the stimulus-locus analysis is more just an application of normal behavioural analysis to the situations encountered around the context of DI). For instance, it could quite possibly be that the learner is able respond, but there is a compliance issue (that's usually relatively easy to diagnose and correct in one step). Or it could be that the learner is missing at least one logically necessary concept underlying the task, in which case the stim-loc tells you what to probe for, and the resp-loc tells you how. Once you find it, you shift back to the stim-loc to figure out how to teach the missing background bits and integrate them. Or if the learner simply can't produce the response, you shape it (or apply context-shaping if they can produce it but in the wrong context).

Once a learner has been correctly placed in a full DI program, they're in a context where the probability of compliance problems is drastically cut down, and continuously receiving positive reinforcement for compliance. And in the DI program the way in which later, more complex concepts logically depend on earlier, simpler concepts has already been accounted for, and students are brought to mastery on the logical pre-reqs before the dependent task is introduced, so that kind of problem is pretty much ruled-out. (Although understanding in detail how this is so requires more understanding of the knowledge-systems analysis portion of the stim-loc, and in the AthabascaU module you've only been shown how the communications-analysis applies to the first of the 'basic form' concepts in that hierarchy [that is, single-dimensional non-comparatives - 'non-comparative' meaning that the value of an example as positive or negative is absolute rather than relative to the preceding example])

Still, the two parts of the stim-loc and and resp-loc do interplay a lot in practice, of course.

“How do you teach algorithms (such as multiplication)? How do you teach history and geography? How do you teach calculus? How do you teach scientific method?”

You teach algorithms through 'cognitive routines' (a classification in the knowledge-systems analysis), if they can't be sufficiently communicated as basic or... hold on, I should lay out a quick sketch of the hierarchy:

Basic forms:

single-dimensional non-comparatives
single-dimensional comparatives
multi-dimensional non-comparative ('nouns', and the reason why LW familiarity with “thingspace” should help with understanding DI)
[multi-dimensional comparatives seem to be implied to me, but Theory of Instruction doesn't even mention them. I can see how they'd be a lot harder to construct sequences for, and would in practice be already 'naturally' generalized by the learner once they've got enough examples of basic forms of the other three types]

Joining forms:

transformations
correlated-features concepts

Joining forms are the two ways in which basic forms can be related to each other.

Transformations being generalizable systems of relating various examples of the same 'type' to corresponding regularities in the response [like grammar rules, spelling and reading, equivalent notations, and...I think I'm actually stipulating with those examples a much narrower range of variation in what transformations can cover than I should, but you know]

And correlated-features being communications about empirical relationships between two basic forms ("if the grade gets steeper, the stream runs faster" An example of a steeper grade is shown. "Did the stream run faster?" (not shown in the example, although they understand the verbal reference to the unshown sensory discrimination of 'runs faster' from a previous sequence). Learner: 'yes'. "How do you know?" Learner: "Because the grade got steeper").

Complex forms:

communications about events ('fact-systems')
cognitive routines

Complex forms being just that, complex systems of basic and joining forms.

Communications about events are kind of a systematic way of designing and teaching mind-maps (which applies to a lot of things from history and geography).

Cognitive routines are algorithms, overtized so that they can be treated as physical operations. (You get any of the factors wrong in a physical operation like opening a door - unlocking it first if necessary, how you turn the handle, direction of applied force - and the environment gives you feedback: the door stays closed! But you try to read a word the wrong way and the environment does nothing to prevent you from saying the incorrect word! Independent practice on cognitive routines new to the learner is a logically insane idea, and experiments can prove it!)

Any concept that can be classified as joining can also be treated as basic, and anything that can be classified as complex can be treated through joining or basic, if the learner is already familiar. Like, you could do a non-comparative sequence "is this calculus? yes/no", but you couldn't teach the discrimination of 'red' through a cognitive routine or fact system or one of the joining forms.

But yeah, calculus as an unfamiliar topic could of course be largely approached as a body of inter-related cognitive routines (and their inter-relations mean the teaching of the whole body can be much simplified by applying single- and double-transformations).

Cognitive strategies like 'scientific method'... well, read this comment. Like I say there, all the concepts represented by your brain on an idea like 'reductionism' must themselves by reducible somehow. We might not be practically able to reduce the whole huge thing in vaguely the same way we can't calculate the exact aerodynamics of various shapes, but we can apply basic principles to do a lot better than just throwing something together (that analogy feels a bit looser than my other physics ones, but you get the point).

And I realize most of that was probably ridiculously hard to follow and pretty much most useful to me as practice reviewing the material, but unless you have some reason to think that the book Theory of Instruction is just 376 pages (not counting index and references) of crank techno-babble by two Ph.D.'s (fine, Zig Engelmann's is honorary from Western Michigan University, but whatever, he's also a recipient of a Council of Scientific Society Presidents award) who are respected by multiple other Ph.D.'s they've collaborated with on books and papers and the DI programs themselves... and that the contents of the book have nothing to do with the reason that the DI programs they designed actually manage to achieve success in experiments like nothing else in the field of education has...

Just get your hands on the book! Because as much as I wish I could I'm not gonna be able to repost everything in it as a series of blog posts any time soon! Check a local university library, or just order it from ADI if you can't find a copy! (It's forty bucks, not exactly a huge expense!)

Again, thank you thank you thank you SO much for being patient and working with me so well through such a huge inferential distance!

[-]Owen_Richardson14y00

Thank you so much for going to the trouble of writing such long and thoughtful feedback! Especially since it's obvious that I still have a lot of this unclear for you.

Actually, I should ask while I'm here: why are you being so diligent about pursuing this?

For me, the most obvious direction of thought would seem to be as follows (but I'm not too sure about my judgement on what would be obvious to someone who doesn't yet understand the theory, hence why I'm checking with you):

The differences between DI and anything else ('normal' education and competing models together) as shown on those graphs from Project F-T are really impressive
And it did say in the meta-analysis that "DI's consistent achievement of such scores is unique in educational research", so the F-T results aren't likely to be a random fluke
So there must be some explanation for that, and the possibility that the people who make this DI stuff actually know what they're talking about - something complex and non-obvious that I don't yet understand - at least deserves some serious consideration (...despite the fact that this idiot has so far done a terrible job at explaining what that might be :P)

If that does seem to accurately reflect what's going on in your mind, please do tell me, because that seems like it would be of great use in fixing up my "DI to LW" communication problems.

Anyway, I'm gonna use an analogy to explain to your what this challenge of communication feels like from my perspective, and then I'll try to give you some meatier replies to your questions.

Analogy: Imagine that you were trying to explain physics to someone who had never even heard of it. Why it's exciting in and of itself, and the amazing engineering feats it allows you to accomplish. You gave them an introductory module on Newton's three laws and they came back and said, "Honestly, it seemed pretty vague. And the axiom 'moving objects stay moving unless they're made still, and still objects stay still unless they're made to move' seems almost like a tautology. And how on Earth does this allow us to create 'amazingly faster transportation'?" (Note it does not occur to them to ask the last question: "How does this allow us to engineer trains and bridges etc?")

(Please remember, this isn't meant as an argument by analogy! I just think it could help you to understand what I say better if you have some idea what it feels like for me trying to find good explanations. On to meatier bits.)

So you asked:

is the claim that each learner responds 10 times per minute correct?

I dunno. Offhand, it sounds plausible as something a good presenter could achieve for many sections of the programs, but it's not like it's mandated by the theory or empirically shown to be necessary. Get your hands on the Michel Thomas lessons for a personal experience with how this is actually not too implausible. And I could try to scan a lesson from a kindergarten reading program or something for you some time, too. (I'm from Canada, and I'm staying with a homestay during my internship in Baltimore, so I'd have to ask them if their scanner is working).

Anyway, before I go any further, have you read this short comment yet? (Just because I want you to have that background and I wasn't able to integrate it below.)

{goes to next comment because it hit the character limit per comment}

[-]prase14y20

why are you being so diligent about pursuing this?

I am not much diligent, but even if I were, I doubt my ability to state true reasons for my participation in online discussions.

The differences between DI and anything else ('normal' education and competing models together) as shown on those graphs from Project F-T are really impressive.

If it wasn't clear, I didn't mean differences in results, but differences in method. That's still what I was complaining about: I have read several times how magnificent DI is, but still haven't learned what the hell DI consists of. Well, I have a rough idea now, but it isn't based on unambiguous statements.

Analogy: Imagine that you were trying to explain physics ...

This was getting interesting, but was interrupted exactly at the moment when I expected you to write the most important part: how does a DI teacher explain Newton's laws? Can you show?

From the continuation comment:

“Faultless communication” is the basis of the stimulus-locus analysis branch of the theory. If faultless communication fails with a particular learner, that gives you specific information about how the learner is not using the two-attribute learning mechanism. That tells you to shift to the response-locus analysis branch of the theory to figure out how to modify the learner so that they do use it (and the stimulus-locus analysis is more just an application of normal behavioural analysis to the situations encountered around the context of DI)....

This sound extremely vague (much vaguer than Newton's laws ever sounded to me). Faultless communication is, as far as I understand, a technical term with some precise meaning. What's its meaning? How is it defined? What are the basics of the stimulus-locus theory? I assume majority of LW readers aren't familiar with the theory and if it is a key component of DI, you should give at least a brief explanation of its basics.

You teach algorithms through 'cognitive routines' ...

Once more, nine paragraphs or so and I am not able to make sense of it (probably because I don't know the specialised vocabulary). Somewhere in your original post you said that DI is based on algorithms which teachers apply and this doesn't need the teacher to understand DI on theoretical level. So, consider me such a teacher who wants to teach multiplication and give me an algorithm to follow.

[-]Owen_Richardson14y00

I... find myself quite surprised at the way my understanding of your response to my question (round the first three bullets) doesn't seem to address what I meant to ask. Was I not clear enough, or were you just skimming around there (not that I don't understand you skimming occasionally at this point).

Man, I just read the first sentence of this comment back to myself, and...

Well, I've been working on less than four hours of sleep a night for the past three days. I'ma try to keep this short by giving only a limited treatment of one point you asked about, go to bed, and give you something more detailed later.

You teach algorithms through 'cognitive routines' ...

Once more, nine paragraphs or so and I am not able to make sense of it (probably because I don't know the specialised vocabulary). Somewhere in your original post you said that DI is based on algorithms which teachers apply and this doesn't need the teacher to understand DI on theoretical level. So, consider me such a teacher who wants to teach multiplication and give me an algorithm to follow.

All right, I'll ask in the DI community for advice on good examples of places in programs that teach cognitive routines (well, places that review the whole routine at once, since the initial teaching of all the components is distributed over long sections of the script, of course). (I'll also ask if they can give me the reference to the experimental evidence on the 1-20 vs 1-99 thing, and so on.)

But yeah, the section of Theory of Instruction on "Constructing Cognitive Routines" begins on page 191 of the text, so you being a bit confused after only nine paragraphs written by a student pretty much reciting an outline of their own mental notes is not that odd.

If you could possibly find the time to check the online catalogs of any university libraries near you to see if they have the book... because if you could easily get your hands on a copy, it wouldn't be too hard to just try skimming the section and chapter summaries.

[-]wedrifid14y130

I look forward to your feedback.

Too many serifs.

[-]Risto_Saarelma14y50

It sounds minor, but yes, this. It's jarring to see a long article that doesn't use the same text style as the rest of the site.

[-]Owen_Richardson14y00

Thanks, THAT at least is easily fixable. Didn't even notice it with all the other formatting headaches from copy-pasting it from an open office file.

[-]RolfAndreassen14y90

Downvoted for the extremely high ratio of enthusiasm to crunchy bits. In fact for all I know there are zero crunchy bits; I gave up about halfway down the page, when the benefits of the method were being iterated for the third time with no hint of any specifics.

[-]Owen_Richardson14y10

Sorry, yeah, I see what you mean now. The crunchy bits were meant to be a few studs of hard content holding down the rest until people filled in more of the whole hard structure of the subject with their own research, with the excitement serving as, well, yeah, a sales pitch. To study. It's okay to try to talk people into studying, right? :P

But yeah, that turned out to be a serious miscalculation on my part. Do the notes added at the beginning as a replacement for the whole long thing help to start clearing stuff up any?

[-]RolfAndreassen14y40

Your notes seem to mainly consist of a long-winded apology for being long-winded. So, no. However, the link to the correct DI article on la Wik was helpful. Suggest you move it to the top. You may also want to steal the one shining jewel of that Wiki article, the summarising sentence: "Direct Instruction (DI) is an instructional method that is focused on systematic curriculum design and skillful implementation of a prescribed behavioral script." Thank you, anonymous Wiki editor, although "skillful implementation of" is superfluous.

[-]Owen_Richardson14y10

...although "skillful implementation of" is superfluous.

Man, if you only knew how wrong that is. Just something like the timing of the teacher's signal for the student to respond can be crucial. It's not that complicated to train a teacher to do it right, but if you don't, hoh-boy.

To explain, imagine you have a group of very young, very naive children, and you need to induce knowledge of the basic-form concept "getting wider" (a single-dimensional comparative ['comparative' meaning the value of each example is relative to the previous one rather than absolute]).

You have for this a logically unambiguous communication, made of a series of positive and negative examples that is sufficient to zero in on that concept in conceptspace by showing what it is and is not, with the ordering and juxtaposition of minimal differences (between negatives and positives, to show difference) and large differences (between positives, to show sameness) tightly controlled, and a consistent prompt for the learner to properly process each example, and a test that they have done so integrated right in to the sequence. (Again, guided induction.)

The teacher is presenting examples through continuous conversion (moving their hands farther apart and closer together, or leaving them the same), and saying "watch the space between my hands. I'll tell you if it gets wider or doesn't get wider." and then "it got wider/didn't get wider" (for the first maybe five examples, which are modeled) and then "did it get wider?" for the rest, to which the kids go "yes/no" (or any clear response they can reliably produce; it can be transformed later anyway).

This works great if done properly, but imagine if the teacher messes up just the timing by asking the question while their hand is still moving? The kids will be lost!

Also see this and this

If you then still can't find anything worthwhile in what I have to say, I'm sorry for wasting your time right now, and I'll work with the people who are already at a closer inferential distance to the topic until I can drill back far enough to communicate clearly with people at a greater distance like you.

Thanks.

[-]RolfAndreassen14y10

The point I'm making is that any method whatsoever requires some degree of skillful implementation; consequently, the phrase does not convey any information. All you're doing is giving an example of what a skillful implementation looks like in this case, namely, attention to timing.

[-]Owen_Richardson14y00

Okay. I was thinking 'not superfluous' as in a non-trivial detail in actually turning around a failed school serving a disadvantage population. But saying that it 'conveys NO information' is technically correct, by a very narrow definition. Although I think the wiki poster made the right choice in including it, since there are often logically implied things that none-the-less should be pointed out to the reader.

Anyway, did you find anything worthwhile in those two comments I linked? Or should I stop bothering you for now?

[-]RolfAndreassen14y10

Ok, ok, I didn't mean to be quite that harsh. Your later comments did give a bit more useful detail. :)

[-]Owen_Richardson14y10

I... thank you =]

I like to think that I'm pretty good at taking things in the best way possible. I'm happy to have been told by one person in a private message, "good on ya for having a thick skin", and I'm trying to live up to that, but I have never had the pleasure of dealing so extensively with people who are just like me in regards to this quality I'm vaguely gesturing towards right now.

I mean, I knew intellectually that the people on less wrong are unusually awesome in that way like you, but it had still never really been my personal experience before, so my emotional belief was much weaker, and I was starting to feel like, "Oh, maybe everyone really is offended that I had the gall to show them such a terrible mess and ask for feedback. Maybe I should have kept trying to work it over by myself until it was perfect rather than just making an attempt..."

Oh, and if you found my later comments to have more useful detail, am I doing a good job continuing that improvement with this (first half and second half) or is that a step in the wrong direction?

[-]RolfAndreassen14y20

You seem to be moving in the right direction: In the comments you linked, you are laying out some jargon of the DI and explaining what it means in simpler words. If at all possible, you might also supply examples taken from actual teaching; for example, when you say

Or it could be that the learner is missing at least one logically necessary concept underlying the task, in which case the stim-loc tells you what to probe for, and the resp-loc tells you how.

it would be helpful if you could descend from the abstraction for a moment and say "For example, one time I was trying to teach trig using DI; the student was not getting why the double-angle formulas work, so the stim-loc told me I should look for a faulty understanding of [something], and I checked that by [something else]..." Failing real anecdotes, a fictional one (clearly marked, of course!) using a real DI locus schema could also be helpful. But at any rate you're now explaining what sort of techniques are involved and what the 'detailed curriculum' is to consist of, and my desire for anecdotes is more about how to present it rather than what to say.

[-]Owen_Richardson14y10

Sorry I promised I'd type that section out yesterday, but didn't. Honestly, I've been juggling so many things I'd need to kage bunshin myself with my computer to handle them all.

(Yes that's right, I just made a "Nartuto" reference. :P

Can you imagine what a "Naruto" equivalent of HPMOR would be like?

...I can't. Other than "awesome".)

Anyway, rather than typing out the section, I found a scanner and signed up at photobucket.

Here's the page. The section I was referring to starts at Prescriptive Applications of Programs ["programs" meaning the task analysis], and ends at the Summary.

[-]Owen_Richardson14y10

Okay, I'll type out the section on "Prescriptive Applications of [Task Analysis]" from page 143 of Theory of Instruction and the accompanying figures tomorrow (been working on less than four hours a night of sleep for the past three days, so I'ma keep this short right now).

The concrete example there is based off:

We suspect that the students do not understand the explanation ... "When the water-laden air rises over the mountains, it cools and can no longer hold all the water. The result is rain."

And the two examples of probed items are both correlated-features concepts.

But yeah, if you could possibly find the time to check the online catalogs of any university libraries near you to see if they have the book... because if you could easily get your hands on a copy, it wouldn't be too hard to just try skimming the section and chapter summaries.

I'll also ask in the DI community for advice on good examples of places in programs that teach cognitive routines, and ask if they can give me the reference to the experimental evidence on the 1-20 vs 1-99 thing, and so on.

[-]Peterdjones14y70

How was DI tested? I can see it being a good fit for traditional "what is the answer to this question" exams, but less so for problem solving and creative thinking.

The material quoted relates DI to elementary education. Is it supposed to be good all the way up to postrgraduate level?

[-][anonymous]14y70

My general feeling is "How can this possibly work?" I can't imagine anyone teaching, say, calculus who doesn't understand calculus thoroughly enough to answer every possible question. If this Direct Instruction thing has mechanics for dealing with that, I don't see them. Maybe the lesson plans involved are so wonderful that every student gets it straight off and doesn't have questions? Then the trick is in the wonderful lesson plans that I don't see how anyone could come up with without a 10-year experience of teaching calculus.

I suppose having better-than-average lesson plans and making teachers stick to them would yield an improvement when the teachers don't really know what they're doing. I don't know if that constitutes a novel education method. It certainly doesn't seem to be the way that Direct Instruction claims to succeed, so if that's what's good about it, that's a problem.

I also don't see any other feature of Direct Instruction that would actually make it good for teaching. But it's possible that by reading this post and the Wikipedia article, I haven't actually discovered the key insights of DI, and now I'm trying to figure out how the hell a horse runs so fast just by having a tail.

In that case, OP, the ball is in your court. What are the things that make DI work?

Edit: after looking at Vaniver's post I think maybe I misunderstood the idea; maybe the point is to have really really good lesson plans. This doesn't address the issue of answering student questions, but I can imagine a way it potentially gets addressed. Maybe students who are confused by something (and we hope there are not too many of them) are just left alone until they do poorly on the next assessment, at which point the lesson plan targeted at them tries to answer all possible sources of confusion.

Is this how it's done? If it is, I suppose I'm not confident that it won't work well, and I'd be open to considering those pretty graphs as evidence that it might work well. My main worry is that this method seems like it would do worse and worse as the material gets more advanced, and it might not work at all past middle school level material.

[-]Owen_Richardson14y00

Thanks for your feedback. Does this and this help?

And the rewrite added to the beginning of the original post as a replacement for the whole long thing?

[-]FiftyTwo14y60

From the wikipedia article:

Direct instruction is a general term for the explicit teaching of a skill-set using lectures or demonstrations of the material, rather than exploratory models such as inquiry-based learning.

I can see how this would be useful in certain areas like learning the names of numbers, where obviously there is not benefit to innovation. However I would be very cautious about applying it to wider areas, as much of what school systems are meant to teach is methods of researching and acquiring beliefs, not mere memorisation. Doesn't this just make it easier to memorise the teachers password?

I would be especially cautious of trying to teach rationality this way.

[This is based on my admittedly limited understanding of the subject, so if there are details I have missed that tackle these concerns please explain.]

[-]Owen_Richardson14y10

Seems our conversation's getting a bit chopped up. Continuing from this (and forgetting the mixed up di/DI confusion from wikipedia)...

I'm going to quote "Theory of Instruction":

"If the goal of instruction is to teach the learner to discover a particular relationship, actual practice in discovery is imperative. Such practice can be provided through a cognitive routine, however. The routine is demonstrated with some examples. The learner then applies the routine to other examples. By encouraging the learner to make up problems that may be solved by the routine and then testing them to see if the routine works, we provide the learner with a framework for discovery." (pp 191-192)

Cognitive strategies are not mysteriously complete wholes. They reduce to generalizations and cross-fertilizations in pools of good cognitive routines, and cognitive routines are made up of moving parts themselves.

These moving parts are joining form concepts - transformation and correlated-feature relationships - which boil down to basic form discriminations of single-dimensional comparatives/non-comparatives and 'nouns' (multi-dimensional non-comparatives).

All the concepts represented by your brain on the idea of 'reductionism' must themselves by reducible somehow.

This making any sense? Some of the terminology related to the hierarchies in the knowledge system analysis might be slightly opaque...

[-]Eneasz14y40

Upvoted in anticipation of the DI Sequence that will be forthcoming over the next dozen months or so. Yes? :)

[-]Owen_Richardson14y20

Well, one of the things I intend to do once I master the application of DI theory myself is create a DI course covering the material in Theory of Instruction itself.

But I don't think I'm going to blog the entire contents of 376 page text-book in the next year, what with the huge amount of studying and practice I have to do myself on a more advanced level, and my full-time internship as an elementary teacher.

My intention here is simply to interest some LWers in joining me in my studies, so that they have the opportunity to catch up to me as soon as possible (and given the intelligence distribution on LW, hopefully have some of them surpass me!).

Do I still get to keep your upvote?

PS: Just occurred to me that there's a terminology mismatch I may have to remember to explain at some point. In the context of DI theory, "sequence" is usually used to mean one of the shortest units of instruction, like, directed at a single basic-form or joining-form concept (a simple joining form concept! Not a transformation concept with many sub-types, which would also be split into multiple sequences).

A 'program' refers to a relatively short series of sequences directed at bringing the learner to mastery of a "task" for which those concepts are logically necessary.

How an entire course like "Reading Mastery" for the entire grade-level of kindergarten is unambigiously referred to I'm honestly not sure offhand. I think it's usually clear from context.

[-][anonymous]14y30

Very interesting. Upvoted, despite structural problems, but that has all been said already.

I have a minor interest in education (my mother is a teacher and I'm kinda her outside adviser), but I'm mostly interested in how to apply this to self-teaching. I can see how this would be awesome if you had a competent teacher to design a course for you, but how can I use DI if no such course exists?

As far as I can tell, one key idea is to construct a series of minimal examples to illustrate one (and only one) new concept. I've been using this for language learning very successfully so far (and it works even better combined with spaced repetition), but how do I use this for more abstract or complex concepts?

Mathematical proofs are something that I'm still struggling with in general (both constructing and understanding them). Let's take the relatively simple proof that sqrt(2) is irrational. The presentation is fairly typical: it's terse, no motivation for any step is provided, and the whole setup is confusing. Even worse, I don't even see how I can apply the idea of example cases to this proof. It's not a general property, so I can't look at other cases. How do I now transform this into something not-confusing using DI? What would a DI teacher do here? (And ideally, how can I do this on my own without understanding it first?)

(As an aspiring polyglot, I also picked up the French Michel Thomas course and will test it this week. I had previously given up on French because of how awful the learning material for it was, compared to Japanese or Latin, and because it's only a minor language for me, so I can't be bothered to design my own material. Maybe this will work better.)

[-]Owen_Richardson14y10

Cool, I look forward to discussing the French lessons with you (although honestly I've lately been practicing Spanish a lot more).

Remember to ask me for some small charts I made that will help you immensely in properly producing all the standard French phonemes that differ from the standard (American) English set at some point.

DI could probably be somewhat adapted to help in self-teaching, since it would at least give you a useful classification system for what possible logical structure the ideas you're looking for might have... but that's the possible structures of the most basic components, which for an advanced subject are arranged in relationships which make the whole thing exponentially more complex. Although different complex ideas can often differ in small ways, but...

Yeah, I dunno. It would have at least a little use I'm sure, but I would bet it couldn't produce anywhere near the level of "magic"-seeming results that a good DI program designed by someone who already understands the material can.

But really, the basic answer is, "No. DI is simply the application of practical epistemology to teaching. When you are learning something on your own, you have to apply practical epistemology in the normal direction."

Oh, and If you're interested in Japanese but aren't yet at a very high level, look for the book "Japanese Verbs & Essentials of Grammar" by Rita L. Lampkin. I took Japanese in high school, and the school only had a grade 11 class. I came into it below the level of the other students, but the teacher bumped me up to being the only grade 12 Japanese student in the school during that year. I believe that book was the single most significant factor in that, and any enabling traits I had on the side, I'm sure you already have too.

Actually, even though I still achieved a very weak grasp of the language at my peak and quickly lost most of that when I stopped practicing after grade 12, I think I would be able to apply DI theory to use that reference work to produce some great instruction on using the language expressively in a conversational context once I master the use of DI theory more.

(Essentially I would be using the book as a 'prosthetic' understanding of the material while I designed the instruction. I haven't given it much thought, but I think this may not be possible in the same way with mathematical proofs [cuz that's more cognitive routines than just transformation concepts, and cognitive routines are higher in the hierarchy of the knowledge-system analysis because they incorporate transformations as components.])

And that's actually a project I've been thinking about how I might possibly do it eventually for a while.

You sound like you could make an awesome collaborator on something like that, if you got your hands on a copy of Theory of Instruction.

[Edit: note that language instruction will generally be much easier to design (for any student who already has a good adult grasp of their native language).

This is because it's largely just subtype analysis of single- and double-transformations that you need to understand, which means you really don't need the sections of Theory of Instruction that deal with cognitive routines, diagnosis and corrections, the response-locus analysis, and philosophical and research issues, and that's the entire last half of the book!

Well, a bit of the response-locus analysis would be useful for teaching the production of new phonemes, but not that much theoretical detail.

The major practical difficulty would be tracking the schedule of the integrated review, simply because of the sheer number of distinct entires (vocabulary words and grammatical patterns), but it would be relatively easy to design a Computer Assisted Design program to help with that.

Oh, and the subtype analysis, which is the most complicated theoretical part, needed for ordering the introduction of the transformation concepts, is only needed for teaching all the basic grammatical structures of the language.

Once you're done with that, all that's left is just vocabulary and idioms, which pretty much just follow the same logical template over and over again.

(The only difference is that you'd start to be able to provide more and more of the definitions and directions of the instruction entirely within the target language.)]

[-]Richard_Kennaway14y00

Mathematical proofs are something that I'm still struggling with in general (both constructing and understanding them). Let's take the relatively simple proof that sqrt(2) is irrational. The presentation is fairly typical: it's terse, no motivation for any step is provided, and the whole setup is confusing. Even worse, I don't even see how I can apply the idea of example cases to this proof. It's not a general property, so I can't look at other cases.

It's a particular property, which you can apply to other cases by substituting some other particularity -- for example, replace 2 by any other number throughout and see whether the proof still goes through. Doing this sort of thing will tell you how the proof works and why it works.

As an aspiring polyglot, I also picked up the French Michel Thomas course and will test it this week.

I'm interested in knowing how this goes. I've never got very far when learning other languages, although I have to say I'm not impressed by the Michel Thomas web site or the Amazon reviews. I suspect that his method would drive me up the wall.

[-][anonymous]14y30

Feedback because you asked.

This article took so long to say much of anything about Direct Instruction I wondered if it might be a sales pitch. My eyes glazed over halfway through. I can see from your participation in the comments thread that you are sincere, and having trouble articulating what DI is in an accessible way.

So I hopped on Wikipedia and looked up "Direct Instruction." The first paragraph told me everything I need to know:

"Direct instruction is a general term for the explicit teaching of a skill-set using lectures or demonstrations of the material, rather than exploratory models such as inquiry-based learning.

"This method is often contrasted with tutorials, participatory laboratory classes, discussion, recitation, seminars, workshops, observation, case study, active learning, practica or internships. Usually it involves some explication of the skill or subject matter to be taught and may or may not include an opportunity for student participation or individual practice. Some direct instruction is usually part of other methodologies, such as athletic coaching."

Okay, that does sound pretty cool. As someone with teaching experience, I've made use of it without knowing it was a thing, many times. And apparently, studies suggest it's exceptionally great at getting the point across -- awesome! My perception that the most notable incidences of using that method were especially effective with my students may not be off the mark!

I suggest you try to find a way to state, in your own words, what the bit I quoted is saying -- or, failing that, just explicitly quote Wikipedia within the first or second paragraph, and keep your whole essay to about 500 words or so pending the community's reaction and desire to learn more.

[-]RolfAndreassen14y50

Maybe I'm being stupid here, but I find it hard to see the difference between Wiki's "explicit teaching of a skill-set using lectures or demonstrations of the material" and, say, "traditional blackboard-and-chalk instruction". Possibly I'm missing a joke? If not, what is the difference between "teaching [using] lectures" and, um, lectures?

[-][anonymous]14y00

That's...not what's being contrasted here...

[-]RolfAndreassen14y30

Glad to hear it. In that case, it seems that the Wiki description is rather a bad one, since it could give such a misleading impression, and some different summary is needed.

[-]Owen_Richardson14y00

Thanks for your feedback... and patience. The wikipedia articles (there are two that differ only in the capitalization of one letter!) are confused.

The one with the lowercase "instruction" fluidly mixes up "little-di" and "big-DI".

“Little-di” is merely instruction that is somehow more direct than some other supposed norm. “Big-DI” is not “little-di” in rather the same way that linear algebra is not algebra that has been put in a line.

Do the notes added at the beginning as a replacement for the whole long thing help to start clearing stuff up any?

And does this help?

[-]amcknight14y30

I think this is a good subject to write about and that you can work this into a nice article if you continue to boil it down. Others have given some good (and maybe a bit harsh?) feedback. All I want to add is that for a post this long, it would be much more useful to have reference numbers (or better yet, links) within your article. I found myself initially having questions and wanting to look at references and then lost that motivation once I finally got to them. Also, I think your conclusion has too much new content.

[-]Owen_Richardson14y20

There are plenty of very valid criticisms of my horribly writing here that have been repeated a few times each, but the one about my 'conclusion' adding too much new content is spot-on and original.

Does the added notes at the beginning as a replacement for the whole long thing help? Like with the shorter more immediate list of references (and the clarification of why I wanted to put that information about Michel Thomas in there, despite having to shoehorn it in)?

[-]scientism14y20

I think if you keep the same structure (introduction, evidence, explanation, example), tighten it up and briefly summarise DI at the beginning it would be more effective. It looks like some people have interpreted your informality as a sales pitch so I'd tone it down a bit. Personally I found the post compelling and will look into DI more in the future. As far as I can tell, it ties with my own belief that knowledge is a set of skills and should be taught by breaking down those skills into components and training them (as you would any other skill).

[-]Douglas_Knight14y20

What is the meaning of the two graphs? How are they consistent?
ETA: the graphs come from here, but that doesn't help. My guess is that they have different baselines, the first using the previous measurement of 20% and the second using the control group, if it regressed to 30%. But they still don't look consistent to me.

When I looked at Project Follow-Through, about 10 years ago, when DI was considerably less popular than it is now, the graphs looked very different. No single method beat out all the others, though three or four were clearly better than the others.

[-]Armok_GoB14y10

I don't see the problems others are talking about at all, to me this seems entirely awesome and I were surprised when I got to the comments and people didn't agree with me.

[-]Richard_Kennaway14y80

There's too much of the interesting-if-true about it. The quoted statistics say that in aggregate it's awesomely successful, but the article only gives an imaginary example of how DI is done, with strident assertions that this obviously must logically work and no claim that this method of teaching numbers has ever actually been used. There's also the claim that because this obviously must work, if it doesn't it's the teachers' fault for not doing what they're told, which is pretty much a standard rationality failure.

So there may be something awesome here but if so, it doesn't come through very well from the posting.

[-]Vaniver14y30

I don't see the problems others are talking about at all, to me this seems entirely awesome and I were surprised when I got to the comments and people didn't agree with me.

I found the article interesting and exciting, and emailed the article to a few people that I thought would be interested, and am just now getting to the comments. I am unsurprised that the people who went directly to the comments and left one had a different impression.

LESSWRONG
LW

LESSWRONG
LW

22

Scientifically optimizing education: Hard problem, or solved problem? Introducing the Theory of Direct Instruction

22

22