US schools perform extremely well on international tests

US students do well; Massachusetts students far outperform Finns
This is worth noting, as a counter to the usual scare stories about the failure of American schools and teachers: US students did really, really well on the recent TIMSS (math) and PIRLS (reading) tests.  Many individual states did even better. In our state, Massachusetts, the average score on the 8th grade math test was far better than that of Finland, a country with a comparable population. In fact, despite a continuing gap between the scores of white students and black students, the average math score for African-American 8th graders in Massachusetts was better than that of the average for all 8th graders in Finland.  It's very likely that if reading scores were available for Massachusetts, our students would have done better overall than Finnish students (a state-level reading score was only available for Florida, and Floridians scored as well as the Finns).

Hooray!

Junkets to Leafstrewn?
Perhaps the most important thing about this news is the way it has been reported in the mainstream press (1).  As usual, it is very important to read press reports about education, and everything else, with a skeptical eye. Some papers, like the DEtroit Free Press that I linked to before, covered the story reasonably well. Others, like the Times and the Post, stuck largely to the "School Failure" storyline.  Here's the headline in the Washington Post, for instance: "U.S. students continue to trail Asian students in math, reading, science."  A better headline might be: "U.S. students perform extremely well on international tests".

You might think that the press might switch the narrative from "Our schools are failing, so we need reform" to "Look, reform is working"--but a meme as well-established as the "failure" of US schools has so much momentum that it is, like the Titanic, almost impossible to turn.  If enough icebergs like these results come in, maybe the old meme will finally sink, and education experts can take junkets to Leafstrewn instead of junkets to Finland.

Now if we can just eliminate poverty...
On the other hand, these scores also remind us that there are wide gaps between rich kids and poor kids, and the best thing we could do to improve our students' academic achievement is reduce inequality.  If the US had the poverty levels of Finland, just think how well we'd do then!

***************************************************************************
Footnotes:
1. Bob Somerby at the Daily Howler has done an excellent job of deconstructing the press coverage of these recent scores. 

An interesting question in regards to teaching

News flash: new research shows that if you run, you get better at running
To me it was news that this was ever in doubt, but a recent post on a New York Times blog (okay, not so recent, but a smart education professor just sent it to me) tells us there is a hot debate over whether running should be taught, and that scientists in England have done a study showing that runners develop more efficient form "just by running more."

Apparently the runners in the study increased their "running efficiency" (oxygen used at a given pace) by nearly ten percent over a ten week period in which they were given no instruction, they just ran.  The scientists say that the runners all learned to bend their knees and ankles more at certain appropriate points in their strides, and all learned to be less "wobbly in the rear foot," whatever that means. 

The post made me think, naturally, about reading; perhaps, like running, reading can be improved just by doing it? Hm...

An interesting question
The most eye-opening thing in the post was not the (somewhat predictable) result that regular running made people better at running, but a little paragraph near the end of the post in which the researcher who led the study said that the results ""raise an interesting question in regards to teaching people to run."

What was this interesting question?  The next sentence in the article began: "'If runners can self­-optimize,' as the women in this study seemed to do, then..."

Okay, how would you finish that sentence?  What interesting question does the study raise in regards to teaching people to run?  I myself thought the question was going to be: "Why bother?"

But no!  The interesting question about teaching running that this study raises for the researcher is not, "Why bother?" but instead: "maybe we should teach runners to learn to understand how the movement feels to them."

Ha!  This woman has done a ten week study showing that people can improve on their own, without instruction, and her conclusion is that maybe we should teach people to learn to understand how the movement feels to them?!

I mean, I am not against mindfulness.  Far from it.  I recently spent a weekend at a Buddhist monastery, and it was great!  Pay attention to the way things feel to you, by all means.  It will help you in many ways, and it might even help your running, and your reading.  But if non-teaching works well for some things, why not just leave it at that?

Interestingly, non-teaching doesn't work well for everything...
The runners in the experiment all learned to bend their joints more efficiently, and all learned to be less wobbly--so maybe those aspects of running aren't worth teaching. None of the runners in the study, however, learned to land on the middle or front of the foot, which some scientists (and, if I remember correctly, the running tribe in the book Born to Run) say makes the stride more efficient and the runner less injury-prone.  So maybe people do need to be taught that?

This result goes along with what I've been thinking about "Visible Learning" and "Direct Instruction" and so on.  There are very likely some things (math?) that require more explicit instruction, and some things (reading?) that require less explicit instruction, and even within a discipline, like reading or running, there may be some things that require more or less explicit instruction. 

One might think that this conclusion--use direct instruction when it's helpful and don't use it when it's not necessary--would be obvious to everybody, but it's not obvious.  We are, I hope, nearing the end of a counterrevolution against the progressive education of the seventies and eighties, and we have gone so far that people like the researcher in the running study cannot imagine anyone learning anything that is not taught--even when she has herself just completed a scientific study showing exactly the opposite.

Teaching or no teaching, you can't run better if you don't run at all
Again, no matter how we are teaching, we need to make sure kids are reading enough.  So I'll close with this old chestnut (table from Allington, numbers originally from here):

Reading Volume of Fifth Grade Students at Different Levels of Achievement

Achievement                         Minutes of                                       Words per

Percentile                            Reading per day                                     year

90th                                           40.4                                             2, 357, 000

50th                                           12.9                                                601, 000

10th                                             1.6                                                  51, 000

 

Rappelling and reading: When is "direct instruction" not appropriate?

Learning Targets: "Isn't this stupid?"
At my son's school, every Monday his ELA teacher hands out a syllabus with the daily learning targets for the week, and at the beginning of every class the teacher writes the day's Learning Target on the board.  This past week the Learning Targets were:

• I can write engaging introductions to draw my readers in.
• I can write strong topic sentences to guide my readers through my writing.
• I can reread my own writing to make choices for revision.
• I can eliminate unnecessary parts of my writing and add transitions to support my readers.
• I can add information and adjust my word choice to improve meaning and precision of language.

On the one hand, these seem like perfectly reasonable things for students to be working on. On the other hand, when my son just handed me the sheet, he said, "Isn't this stupid?"

True, my son is a twelve-year old to whom almost everything seems either totally stupid or totally amazing, and yes, he has imbibed some of my own skepticism about pretending that reading and writing can be broken down into discrete and teachable "skills"--but some of the most mature, sophisticated and enthusiastic of the high school juniors in my "Honors" American Literature class also seem to think "Learning Targets" are ridiculous. When I wrote a "Learning Target" on the board of my own classroom one day a couple of weeks ago, a few of the top students in the class, the kids who are taking three AP classes and are sons and daughters of fancy doctors or professors at Perfect University, said, "Are you kidding?"

That my son and my students are so critical and even dismissive of these Learning Targets is interesting, because Learning Targets are central to the way education is conceived of these days. A  lot of the current discourse about education says that teaching and learning  should always have specific goals ("Learning Targets"), that it should always be "visible" to teacher and student, and that student progress should be assessable along the way. This is often a good idea, but it has its limits. The reaction my son and my students had is largely related, I think, to the fact that it was in English class.  My guess is that if their AP Math teacher had put up learning targets (e.g. "I will be able to calculate the mean, the variance, and the standard deviation for a population or for a sample"), my students would have found it less risible.

"The heart of the model of successful teaching and learning"?
A prominent book on education research offers the following as an illustration of its central argument:

Consider the following: I am going to take you to the top of a three-storey building and teach you to rappel down the outside of this building.  Typically, I would demonstrate to you how to put on a safety harness, tie the rope in a bowline, and then show you how to lean backwards to commence the descent.  In line with the principles of of good teaching, I would then ask you, the student, to implement this learning.  Typically, such a learning situation lead to much care by the students, an enhanced level of interest in what peers are doing, and high levels of help-seeking behaviors to ensure the knowledge of rope-work is correct and harnesses are correctly positioned.  The goals are challenging, specific, and visible, and the learners are committed to them!  The learning is actively visible and there are high levels of feedback and monitoring.  The learner typically "seeks" the feedback.  When a novice first gets to the edge, there is a remarkably high level of peer teaching and learning: it is not natural to fall backwards when descending as it is more typical for the feet to precede the head.  When finally the student reaches the bottom there is a surge of excitement appreciating that the challenging goal has been reached (it is abundantly clear what the success criteria are!), the experience was exhilarating, and the learning absorbed in the experience itself.  Most want to repeat the experience and continue to enjoy meeting the challenging goals.  Moreover, all these acts and most of the "thinking" about the task are visible to the teacher and to the learner.  This is the heart of the model of successful teaching and learning advocated in this book.

The first time I read this passage, I laughed out loud.  The notion that rappelling down the face of a three story building is in any way comparable to what students do in school is just ridiculous. Nevertheless, I think looking at the passage a bit more closely may shed light on when this model of teaching and learning is appropriate and when it is not.

In the spirit of being visible and explicit, let me state the obvious:

(1) Rappelling can be taught in a day.  Even the basics of reading usually take a year or more.

(2) Rappelling is (as presented here) a new skill that the "novice" is trying for the first time.  Reading is for nearly all students not a brand new skill (see #1), and most English teachers, like me, are teaching kids who have been studying reading for more than five years.  I'd like to see the education researcher try to teach rappelling to people who have been doing it nearly every day for five years.  He might find that his model didn't seem quite as snazzy.

(3) Rappelling is, especially for the novice, hugely exciting.  Reading is pleasant, but not exciting.

(4) Rappelling is potentially extremely dangerous to your health.  Reading is fairly safe.

(5) If it is "abundantly clear what the success criteria are" in rappelling, the same is much less true in reading.  Even in an hour-long assessment, it is difficult to measure reading skill with much precision on an individual level, and in daily practice the success criteria are far from "abundantly clear."

(6) Rappelling has certain discrete and fairly straightforward skills and conventions (how to tie a bowline and put on a harness, etc.) that are necessary to master before you begin the difficult part.  In this, it is actually similar to reading.  Reading requires knowing some facts (the sounds the letters represent), knowing a few conventions (you read from left to right, etc.).  Mastering this knowledge is necessary but is, in some sense, the easy part.  But with both reading and rappelling, and perhaps with every important skill, the really important part is the practice--actually doing it.


Actually doing it
Although I have not rappeled, I have both learned and attempted to teach a very similar, and perhaps more difficult task: "dropping in" to a halfpipe on a skateboard.  Learning to drop in is quite similar to learning to rappel: there is the same frightening moment at the top of a vertical face when all of your instincts are telling you to lean back, not to lean forward, and telling you that if you are going to fall, you had better fall feet first.  As the passage says, "it is more typical for the feet to precede the head."  In some ways, dropping in may be even more scary, since there's no rope.  It's quite thrilling, even after you've done it thousands of times.  These photos can't convey the thrill, but the one on the right gives some idea of the scale that's often involved:

Now, what inevitably happens to beginners is what happened to me the first time I tried to drop in: you don't lean forward enough, and you slide down the ramp, kind of like this--except that this guy is leaning so far forward, he is clearly not a beginner:

When I dropped in for the first time, at a ramp in Canton, Massachusetts, in 1986 or so, I sprained my ankle quite badly. Would better instruction have made it easier for me?  I don't think so; my friend Jon had told me everything I needed to know, had modeled it for me, and had offered lots of encouragement. I have since tried instructing other people to drop in, and there's not much to say; the steps are quite simple:

(1) put your back foot, and all your weight, on the tail of the skateboard, which is perched on the coping of the ramp;

(2) put your front foot lightly on the front of the skateboard;

(3) simultaneously push down hard with your front foot and throw all your weight forward and down.

That's it.  There is no knot-tying, no complicated safety harness (everyone knows how to put on a helmet and knee-pads).  It's all in the doing. That last step, pushing down hard with your front foot and throwing all of your weight forward and down into the ramp, is both the most important and the most terrifying.  And it famously cannot be taught. An online how-to has this to say about dropping in:

It can be scary to stomp down and lean into open air. There is no turning back once you've started the stomp, and I would say at least 80% of the problems people have when dropping in is not being committed enough to this part. You have to trust that you and your skateboard will make this work. You have to invest in dropping in 100%. It's all or nothing. Be committed to the drop in. Once you do it, it will get easier and easier every time.

Here's a secret about skateboarding - skill is very important, but even more important than skill is self confidence. It's all in your head. This is what separates something like skateboarding from other "sports". Your strongest opponent is yourself. So when you face something like dropping in, and you do it, you are taking a huge step toward self control.

That was a little deep, but it's true. The point is, if you are going to try and learn to drop in, then just do it. It's like Yoda says, "Do or do not, there is no try."

Just do it.  When my brother and I had a half-pipe in our backyard, and when I was the advisor of the skateboarding club at Leafstrewn, I noticed this about skateboarding: there was a strong culture, not only of "just doing it", but of encouragement, support and collaborative learning.  The important things were not the direct instruction or the "clear learning goal," but the culture, the engagement, the modeling, feedback and encouragement from peers, and the endless practice.

 Direct Instruction? Yes, in small doses.
"Direct instruction" is a pretty good way to teach people a new skill.  You make it clear what you're about to teach and why, you teach it and model it, you have students practice it under your supervision, you review, and then students practice it on their own.  These steps seem like common sense.  If I were going to teach people how to, say, fix a flat tire on a bicycle, this is pretty much how I would do it.  Nevertheless, this is not to say that all of school should consist of "guided instruction."

In learning rappelling, or dropping in, or reading, direct instruction should play a role--but only a small role.  There may be subjects, like math or science, in which there should be a new learning target every day, and there should be new instruction every day, but English is mostly not like that.  The most striking thing about the learning targets my son's 7th grade teacher put on the board this past week is that they might serve just as well for my 11th grade students.  In reading, the learning goals are even less "challenging, specific, and visible" than they are in writing, and the success criteria are even less "abundantly clear."

Therefore, even though I am trying to be clearer about what the learning goals are in my classes ("I will become more comfortable reading colonial-era literature"; etc.), and even though I am trying to be clearer and more deliberate in my instruction, I still think the most important thing is to foster a culture of engaged reading, humane discussion and intellectual inquiry.  There is a mania in education today for direct instruction, visible learning, abundantly clear learning targets, accountability, testing, and so on.  These things are important, but there is a great danger that we will go too far and think that every minute of every schoolday must be specifically targeted, and that the only worthwhile learning targets are those with abundantly clear success criteria.

As always, we need to think about the long term.  After trying it over and over, I did finally learn to drop in, and it was fun--as learning should be.

Can we trust educational research? ("Visible Learning": Problems with the evidence)

I've been reading several books about education, trying to figure out what education research can tell me about how to teach high school English. I was initially impressed by the thoroughness and thoughtfulness of John Hattie's book, Visible Learning, and I can understand why the view of Hattie and others has been so influential in recent years.  That said, I'm not ready to say, as Hattie does, that we must make all learning visible, and in particular that "practice at reading" is "minimally" associated with reading gains.  I discussed a couple of conceptual issues I have with Hattie's take in an earlier post--I worry that Visible Learning might be too short-term, too simplistic, and less well-suited to English than to other disciplines.  Those arguments, however, are not aimed at Hattie's apparent strength, which is the sweep and heft of his empirical data.  Today, then, I want to address a couple of the statistical weaknesses in Hattie's work.  These weaknesses, and the fact that they seem to have been largely unnoticed by the many educational researchers around the world who have read Hattie's book, only strengthen my doubts about the trustworthiness of educational research.  I agree with Hattie that education is an unscientific field, perhaps analogous to what medicine was like a hundred and fifty years ago, but while Hattie blames this on teachers, whom he characterizes as "the devil in this story" because we ignore the great scientific work of people like him, I would ask him to look in the mirror first.  Visible Learning is just not good science.

Hattie's data
Visible Learning attempts to be both encyclopedia and synthesis. The book categorizes and describes over 800 meta-analyses of educational research (altogether, those 800 meta-analyses included over 50,000 separate studies), and it puts the results of those meta-analyses onto a single scale, so that we can compare the effectiveness of the very different approaches.  After categorizing the meta-analyses, into, for instance, "Vocabulary Programs", "Exposure to Reading", "Outdoor Programs", or "Use of Calculator", Hattie then determines the average effect that the constituent meta-analyses show for that educational approach.  By these measures, exposure to reading seems to make more of a difference than the use of calculators, but less of a difference than outdoor programs, and much less of a difference than vocabulary programs. (There are some odd results: "Direct Instruction," according to Hattie's rank-ordering, makes more of a difference than "Socioeconomic Status.")

Like other teaching gurus and meta-meta-analyzers (for instance, Robert Marzano, whose 2000 monograph, A New Era of School Reform, makes the case very explicitly), Hattie believes that good teaching can be codified and taught (that sounds partly true to me), that good teaching involves having very clear and specific learning objectives (I'm somewhat doubtful about that), and that good teaching can overcome, at the school level, the effects of poverty and inequality (I don't believe that).  Hattie uses a fair amount of data to back up his argument, but the data and his use of it are somewhat problematic.

First, questions about the statistical competence of Hattie in particular
I am not sure whether we can trust education research, and I am not alone.  John Hattie seems to be a leading figure in the field, and while he seems to be a decent fellow, and while most of his recommendations seem somewhat reasonable, his magnum opus, Visible Learning, has such significant issues that my one friend who's a professional statistician believes, after reading my copy of the book, that Hattie is incompetent.

The most blatant errors in Hattie's book have to do with something called "CLE" (Common Language Effect size), which is the probability that a random kid in a "treatment group" will outperform a random kid in a control group.  The CLEs in Hattie's book are wrong pretty much throughout.  He seems to have written a computer program to calculate them, and the computer program was poorly written.  This might be understandable (all programming has bugs), and it might not have meant that Hattie was statistically incompetent, except that the CLEs Hattie cites are dramatically wrong.  For instance, the CLE for homework, which Hattie uses prominently (page 9) as an example to explain what CLE means,  is given as .21.  This would imply that it was much more likely that a student who did not have homework would do well than a student who did have homework.  This is ridiculous, and Hattie should have noticed it.  But even more egregious is when Hattie proposes CLEs that are less than 0.  Hattie has defined the CLE as a probability.  A probability cannot be less than 0.  There cannot be a less than zero chance of something happening (except perhaps in the language of hyperbolic seventh graders.)

As my statistician friend wrote me in an email, "People who think probabilities can be negative shouldn't write books about statistics."

Second, doubts about the trustworthiness of educational researchers in general
My statistician friend is not the first to have noticed the probabilities of less than zero. A year and a half ago a Norwegian researcher wrote an article called "Can We Trust The Use of Statistics in Educational Research" in which he raised questions about Hattie's statistical competence, and in follow-up correspondence with Hattie the Norwegian was not reassured.  (Hattie seems, understandably, not to want  to admit that his errors were anything more than minor technical details.  In a exchange of comments on an earlier post on this blog, as well, Hattie seems to ignore the CLE/negative probability problem.)

For me, the really interesting thing about Hattie's exchange with the Norwegians was that he seemed genuinely surprised, two years after his book had come out, by the fact that his calculations of CLE were wrong.  In his correspondence with the Norwegians, Hattie wrote, "Thanks for Arne Kåre Topphol for noting this error and it will be corrected in any update of Visible Learning."  This seems to imply that Hattie hadn't realized that there was any error in his calculations of CLE until it was pointed out by the Norwegians--which means, if I'm right, that no one in the world of education research noticed the CLE errors in between 2009 and 2011.

If it is true that the most prominent book on education to use statistical analysis (when I google "book meta-analysis education", Hattie's book is the first three results) was in print for two years, and not a single education researcher looked at it closely enough and had enough basic statistical sense to notice that a prominent example on page 9 of the book didn't make sense, or that the book was apparently proposing negative probabilities, then education research is in a sorry state.  Hattie suggests that the "devil" in education is the "average" teacher, who has "no idea of the damage he or she is doing," and Hattie approvingly quotes someone who calls teaching "an immature profession, one that lacks a solid scientific base and has less respect for evidence than for opinions and ideology" (258).  He essentially blames teachers for the fact that teaching is not more evidence-based, implying that if we hidebound practitioners would only do what the data-gurus like him suggest, then schools could educate all students to a very high standard.  There is no doubt that there is room for improvement in the practice of many teachers, as there is in the practice of just about everyone, but it is pretty galling to get preachy advice about science from a guy and a field who can't get their own house in order.

Another potential problem with Hattie's data
Aside from the CLE issue, I am troubled by the way Hattie presents his data.  He uses a "barometer" that is supposed to show how effective is the curricular program or pedagogical practice he is considering.  This is the central graphic tool in Hattie's book, the gauge by which he measures every curricular program, pedagogical practice and administrative shift:

Note that developmental and teacher effects are both above zero. What this implies is that the effect size represented by the arrow is not the effect as compared to a control group of students that got traditional schooling, nor even the effect size as compared to students who got no schooling but simply grew their brains over the course of the study, but the effect size as compared to the same students before the study began.

This would imply that offering homework, with a reported effect size of .29, is actually worse than having students just do normal school, or that multi-grade classes, with an effect size of .04, make kids learn nothing.

Now, that is obviously not what Hattie means.  The truth is that Hattie sometimes uses "effect size" to mean "as compared to a control group" and other times uses it to mean "as compared to the same students before the study started." He seems comfortable with this ambiguity, but I am not.  Not only is the "barometer" very confusing in cases like homework and multi-grade classrooms, where the graphic seems clearly to imply that those practices are less effective than just doing the regular thing (especially confusing in the case of homework, which is the regular thing), this confusion makes me very, very skeptical of the way Hattie compares these different effect sizes.  The comparison of these "effect sizes" is absolutely central to the book.  Comparing effect sizes (and he rank orders them in an appendix) is just not acceptable if the effects are being measured against dramatically different comparison groups.

Hattie, in a comment on an earlier post in which I expressed annoyance at this confusion, suggested that we should think of effect sizes as "yardsticks"--but in the same comment he says that effect size is the effect as compared to two different things.  In his words: "An effect size of 0 means that the experimental group didn't learn more than the control group and that neither group learned anything."  Now, I am an English teacher, so I know that words can mean different things in different contexts.  But that is exactly what a yardstick is not supposed to do!

Of course, it is possible that many of Hattie's conclusions are correct.  Some of them (like the idea that if you explicitly teach something and have kids practice it under your close observation, then they will get better at it more quickly than if you just ask them to try it out for themselves) are pretty obvious.  But it is very hard to have much confidence in the book as a whole as a "solid scientific base" when it contains so much slipperiness, confusion and error.

Beyond these broad issues with Hattie's work, I also have some deep qualms about the way he handles reading in particular.  Maybe one day I'll address those in another post.

David Coleman's beautiful fantasy: that students will read 7000 pages a year!

In a Boston Globe column cheering on the Common Core standards and their infamous call for 70% of student reading in high school to be non-fiction, David Coleman is quoted as saying that "non-fiction will mostly be read in science, history, and social studies classes, where students ought to be using primary sources and learning to decipher scientific research."

This is pretty interesting. I'd be happy if students read as much fiction as they are supposed to do now but read twice as much nonfiction, but that is very obviously not going to happen. The problem with reading a lot is that it takes time, and if student reading is really split 70-30, then Leafstrewn students who are now assigned 8 novels a year (2000+ pages) would then have to read the equivalent of close to 7000 pages--which would probably be great, but would not leave a lot of time to work on all the other stuff that's mentioned in the standards--let alone leave time for students to do a lot of independent reading--another thing Coleman has on occasion called for. 

Another possibility is that the CC people envision much more independent reading, and much less whole-class reading than is currently assigned, perhaps because they realize that a lot of students don't do the reading.  That's a vision I could respect, but I don't think David Coleman and his collaborators have thought it through that deeply. Instead, my sense is that, notwithstanding the Oxford degree the Globe columnist admiringly cites, Coleman has little idea what it's like in an actual school and little sense of how the document he was paid to prepare might actually be put into effect.

Student stories about reading

I had my 9^th graders write a little essay about reading.  I gave them a Malcolm X passage about how reading was liberating, and asked if reading was liberating or confining for them now, how it had changed, and what they thought it would be like in the future.  I encouraged them to include specific details, but I tried not to tell them what to write.  

One common theme was hating being forced to read:

“Up till I was in about third grade my parents forced me to read books like the ABC Mystery books.  I hated it because I didn’t pick them out of interest, I read them because I was forced to.  I also hated reading because it was hard for me growing up.  I can remember my second grade teacher Ms. D--- helping me read my first chapter books.  But as I got older I started to pick my own books and started reading and liked some books.”

“Unlike Malcolm X I don’t feel free when I read.  Every time I have to read I am being forced either by a teacher like Mr. C--- or by my parents.  I can’t recall one time where I have ever wanted to read voluntarily.  One of the reasons I hate reading is because I don’t picture the scenes in the book all I see are a bunch of words.  Reading maked me feel very trapped, almost as if being thrown in the Concord jail is a better place for me to be.  Reading also gives me a headache.  All of the makes me feel like reading is very confining.”

On the flip side, another common theme was the way other people—especially teachers—can influence kids’ reading in a very positive way:

“I think that overall I am a pretty strong reader and even though it can be boring at times I still think it’s fun.  Earlier this year in Ms. A---’s Honors class I read “The Odyssey, by Homer.  The book was long, confusing, and even boring for parts of it, but I still kind of enjoyed it.  I thought it was pretty interesting when Odysseus and his men got trapped in the Cyclops’ cave, and they had to trick him into getting drunk so they could stab him in his one eye to blind him, so that they could escape more easily."

"Reading is very relaxing because it allows me to step outside of everything I was doing before I started reading. […] I remember a couple of weeks ago when Ms. K--- showed me where And Then There Were None by Agatha Christie was in the mystery section of the library.”

“I remember when I never used to like reading.  Every time someone would ask me to pick up a book, I refused, or grabbed it and stared at it.  Just the sight of a book would make me sick, but not anymore.  It happened one day when I came back from school, I saw my older brother in the dining room reading a book.  He seemed genuinely interested in it.  The book happened to be Twilight, that was the book everybody was talking about then.  Before he even got to finish the page he was on I stole it from him and read chapter after chapter each passing day.  My fifth grade teachers had to beg me to put it down.  When I went home that night my oldest brother had brought home Twilight on Red Box and he said he would only keep it for one day.  So as the hours passed and the day turned to night, I was still struggling to finish the book so I could watch the movie.  Finally I just placed my bookmark in and set the book aside.  I popped the movie in the DVD player and settled back into the couch as the previews flashed by.”

“Right now I am reading Moneyball, by Michael Lewis, and it is one of my favorite books so far.  I saw the movie and I was recommended by my summer baseball coach, Mr. N---, to read it.  He told me that it was really good and he thought I would enjoy it because we both really liked baseball.  I remember that he recommended it to me at Eagle’s Deli after one of our summer baseball games at Warren Field.”

“Even though right now I say that I love reading, I used to hate reading.  It all changed when I met this person.  Until 5^th grade before I met this person, I didn’t really have any interest in reading.  I though it was very confining.  Once, when I was in the second grade, my mom almost spanked me because I didn’t read.  (Unlike the United States, many parents spank their children in Korea.)  It all changed when I met Ms. C----, my 5^th grade teacher.  When I first walked into her class, all the students stared at me because I was a new student.  I had difficulties with English.  She always helped me to improve my English.  I still remember the first time she handed me this book.  It was one day in winter, and we were released early from school because it was snowing so much.  I was waiting for my guardian to pick me up.  When I was doing my homework, she came up to me and smiled at me.  I didn’t know what she meant by smiling.  She said, “I have a present for you, A----. It is my Christmas present for you.”  I was very confused.  Then, she handed me a book.  It was a small book with a black cover, and the title of the boook was The Giver.  Looking at the old man in the front cover, I thought it would be boring.  When I got back home, I started reading the book.  That book with the old man on the cover changed everything about reading.  It made me feel liberated, unlike other hard books.  I felt like I was in the book with the characters.  I could even see their faces in my head.  Thanks to Mrs. C--- and a snow day, she opened a new world for me.”

What the heck does John Hattie mean by "effect size"?

OK, I am kind of annoyed.  I've been going through John Hattie's impressive book, Visible Learning, trying to make sense of it, and I'm confused on a basic point. Here I am reading this long book that is all about comparing effect sizes, and I realize that I don't know exactly what Hattie means by "effect size." If a program has an "effect size" of 0, does that mean that the the kids in the program didn't learn anything, or does it mean that they didn't learn anything more than the kids in the control group?  At different times in the book, it seems to mean different things. It is hugely annoying that Hattie doesn't make this clearer.  He claims to have spent 15 years writing the book, and he can't even be clear about the most basic possible element of his work?  Grrr.

Of course, part of what makes me so annoyed is that I am aware that the problem is probably partly mine as well as Hattie's.  If I were smarter, maybe I'd be able to figure it out.  But I can't help suspecting that maybe "effect size" means different things for different studies, and that this may have some effect on Hattie's conclusions.

Update (12/16): OK, it's not my problem, it's Hattie's.  More later, but this is just ridiculous.  The state of educational research is just pathetic.  This guy is some kind of big shot, and his book is supposedly a masterwork fifteen years in the making, and he's making various basic errors.  I didn't catch them at first because I'm an English teacher, not a statistician, but it didn't take me THAT long, either.  How is it that people continue to take him seriously?  I'll explain in a bit more detail in another post when I have a free half hour or so.

"Visible Learning," School Reform, and In-school Reading

(I wrote this really quickly; I apologize for typos.  I wanted to get it written while it was still fresh in my mind, but I have too much schoolwork to take a long time and write it well.)

In-school reading makes parents uncomfortable
At my son's middle school, and indeed throughout the district in the city my family lives in (slightly less leafstrewn than Leafstrewn, but graced with the Perfect University and the Institute of Machine Perfection), English teachers are having their students do a lot of in-class reading.  On Parents' Night, I told my son's teacher I was thrilled about this.  Last week, at a pot luck dinner for my daughter's class, a mom who also has a seventh grader told me she had wondered about what I said.  "You said, 'I think in-class reading is a great idea,'" she said. "I couldn't tell if you were serious,  or being sarcastic."

It turns out that she, and many other parents, see reading in class as a strange idea.  That same week, on a parent email forum, a couple of parents raised the issue: why, they said, does school have them doing silent reading, an activity that they could easily do at home?

I wrote a response in which I made the obvious arguments: that reading is (as even such a champion of whole-class direct instruction as Doug Lemov argues) always a productive use of class time; that it is, in fact (as Lemov says), the (very high) baseline against which any other use of class time should be measured; that many kids don't read at home and indeed hardly read at all; that in-class reading allows the teacher to do more one-on-one instruction, which is the best kind; and so on.

Of the parents who responded, most backed me up.  Some parents, however, were still not convinced. Most of these parents were willing to admit that reading in class might be good for other people's children--the implication I think was that good parents would have kids who read--but not for their own children, because their own children didn't need it.  At the pot luck dinner I talked to a couple of parents whose kids read a lot at home; they were confused as to why their kids would spend time in school reading when they could be doing something else.

What the Research Says
I spent several hours over the weekend immersed in John Hattie's magisterial 2009 overview, Visible Learning.  I've also spent some time looking back at Marzano's interesting 2000 monograph, A New Era of School Reform, which is, while less impressively encyclopedic, very much along the line that Hattie takes. Hattie and Marzano stress very clear and well-defined "learning objectives," very clear teacher control over where the lesson is going, and lots of "feedback" going both ways between teacher and student.  Most education reformers over the past twenty years or so have justified their efforts by pointing to research like that which Hattie and Marzano synthesize. For a different perspective, I also looked back at Stephen Krashen's summary of studies of in-class reading, The Power of Reading.  I've learned a lot, but I come away with more questions than clear answers.

Hattie and Marzano are very clear on what does work.  Hattie calls it "visible learning"--he says that it's very important that both teachers and students be able to see what is working and what is not working.  That requires very clear learning objectives, something Marzano stresses that seems to be taken to heart by all of today's reformers.  It also requires a lot of feedback, from teachers to students, but also, and according to Hattie even more importantly, from students to teachers. Teachers need to know what their students are "getting" and what they are not getting.   Teachers should be "activators" and not "facilitators."  Hattie argues strongly and explicitly against the classic "constructivist" conception of teaching, in which the teacher facilitates "authentic" learning that is open-ended, "student-centered," and aimed at "discovery"--that is, learning whose end points or goals neither the teacher nor the student know beforehand.  According to Hattie, this kind of "hands-off" teaching and "intrinsically motivated" learning is "almost directly opposite" to the kind of learning that the experimental data says is most successful.

Hattie and Marzano back up their arguments with vast amounts of experimental data.  Hattie's book is particularly sweeping: his book is a meta-meta-analysis--that is, a meta-analysis of over 800 meta-analyses, each of which included several individual studies.  Hattie's book synthesizes research on over a hundred million students, and with this vast body of evidence, a logical organization, a thoughtful, placid style but a forceful argument, it is certainly the most impressive analysis of educational data I've ever seen.  That said, I came away not fully convinced.

Two Ways to Critique Hattie: theoretically and empirically
There are two kinds of potential problems with the "recipe" that Hattie (along with Marzano and all the other proponents of "clear learning objectives") is selling.

One kind of problem is theoretical: it seems possible that this kind of learning is potentially dry, uninspiring, and limited--and perhaps especially boring to students who get the idea quickly;  also, it's not clear to me that, in the case of reading, the "learning goals" can be, to use Hattie's words, "challenging, specific and visible" (25).

A second class of problems, that I'll deal with in a post later this week, have to do with the empirical evidence Hattie is using: (a) the evidence for this recipe's efficacy seems to be quite thin in the case of reading; (b) the evidence for this recipe's efficacy seems to be mostly short-term, not long-term; (c) Hattie's evidence is mostly from controlled experiments, not from natural experiments; (d) Hattie and especially Marzano argue loudly that their evidence shows that poverty can be overcome by good schools and good teaching, but as far as I know this has never actually been done by an actual school in the real world--or at least not in the United States--despite the fact that thousands of schools across the country are trying to put this kind of "visible learning" into practice.

A theoretical critique: Is "Visible Learning" possible in English class--and if so, is it doomed to be dry and boring?
Today I wrote the following on the board in my "Honors" American Lit. class: "Learning Targets:  (1) I will be able to write a good question about a sophisticated text and find a specific passage through which I can explore the question; (2) I will be able to discuss coherently the relationship between Hawthorne and Transcendentalism."

The cluster of A students who sit near the front of the room noticed what I had written and hooted derisively.  One of them asked, "Is that a joke?"  Another said, "No teacher in my entire school career has ever written learning targets on the board."  Another one said it reminded him of a description of a typical day at a Fall River charter school that was posted a few days ago on Edushyster, an anti-corporate Ed reform blog.  This led to a spirited discussion for a minute or so, before I shut it down so that we could get to our learning targets.

One student said that she liked the idea of learning targets, even in English class, but the most common knee-jerk response from the kids was that putting up a "learning target" every day in English class was laughable and ridiculous. Again, we didn't spend much time on this, but I believe their reaction was based on two ideas that I've written about before: (1) that, on the micro level, the skills we work on in English class are very complex and interdependent, so that isolating one of them each day is either absurd (today we're learning about commas in Hemingway) or obvious (the one I put up), while on the macro level the skills are  (2) daily learning goals promote short-term thinking, while English-class skills are gained over the very long term--learned through repeated practice over years.

What are the learning goals in the case of upper-grade reading?
It is worth asking, then, what people who support "visible learning" would say should be our learning goals.  The obvious place to look for these learning goals is in the Common Core.

The Common Core has two reading strands for grades 6-12, one covering "Literature" and one covering "Informational Text."  I looked at the Literature strand.  The Common Core specifies 9 standards for literature in the ninth and tenth grade.  Three of them are below:

CCSS.ELA-Literacy.RL.9-10.2 Determine a theme or central idea of a text and analyze in detail its development over the course of the text, including how it emerges and is shaped and refined by specific details; provide an objective summary of the text.

CCSS.ELA-Literacy.RL.9-10.4 Determine the meaning of words and phrases as they are used in the text, including figurative and connotative meanings; analyze the cumulative impact of specific word choices on meaning and tone (e.g., how the language evokes a sense of time and place; how it sets a formal or informal tone).

These standards seem fairly unhelpful. They are essentially the same as standards from earlier grades (1).  That would be fine in itself; as I've written, English class is a matter of practicing the same skills over and over.  The problem is that these standards don't tell us anything we don't already know.  We know that every English student in the world will be required to "determine the meaning of words and phrases as they are used in the text." This is quite a low-level goal. 

In the end, I don't think putting a "learning target" up on the board is a particularly bad idea.  I don't think it's necessary, but in the hands of a decent teacher it won't necessarily hurt.  The danger with the practice is that it may lead to reductive, simplistic, boring teaching, in which kids are taught a lesson completely focusing on the simple structure of Hemingway's sentences.  On the other hand, the danger with the opposite approach, the let's-just-read--and-talk-about-it approach, is that it may lead to a class that is way too loose.  Either way is dangerous, but it seems unlikely that telling kids specifically what they are going to learn in every lesson is going to lead to more learning overall.  Learning in English class is, as the Common Core standards admit both implicitly and (sometimes) explicitly, cumulative and long-term.  To pretend that students are going to be noticeably better at "determining the meaning of words and phrases as they are used in the text" after a day-long lesson, or even after a month-long unit, is silly.  Determining the meaning of words and phrases is a life-long task--and it is one at which the learning curve becomes flatter and flatter.  I am probably better in some ways at this now than I was when I was twenty-five; but I'm not sure anyone would notice the improvement.  Does this mean that reading a new author (as when I discovered Edward St. Aubyn last year) is useless?  No, of course not.  But it would be silly to set targets for that learning ("I will be able to describe Patrick Melrose's family dysfunction...").

That may mean that more advanced students will be the more dismissive of "visible learning"; perhaps, but I think it's not necessarily great for any students, and I'll talk more about that in a post on the empirical critique of Hattie's work later this week.

********************************************************************************
Footnote:
(1)
The fourth grade standards include the following:


CCSS.ELA-Literacy.RL.4.2 Determine a theme of a story, drama, or poem from details in the text; summarize the text.

CCSS.ELA-Literacy.RL.4.4 Determine the meaning of words and phrases as they are used in a text, including those that allude to significant characters found in mythology (e.g., Herculean).

Do students need to "build reading perseverance"?

I was at a Metco conference all day today, and one of the workshops I attended was on the Common Core.  It was interesting in lots of ways that I'll have to mull over for a while, but one minor thing I was struck by was the way the presenter talked about independent reading.  The presenter, a high-school principal, told the story of a teacher in her school who twice a week puts high-interest YA books on every desk, and for the first 30 minutes of the period has the students just read.  When one of the students was asked why he thought they were spending 30 minutes twice a week in reading, the kid said, clearly repeating a phrase the teacher had instilled in him, "Well, we're going to have to take the SATs, so we have to build up our perseverance."

Later in the workshop, the presenter put up a slide of "tips" for ways ELA teachers can prepare their students for the Common Core and the tests that will stem from it.  One of her slides was titled "FOCUS ON READING", and it had two "tips". One was about reading in the content areas and the other was this:

"Students need to build reading perseverance"

I wasn't sure what I thought about this.  On the one hand, I was happy to see teachers encouraged to have their students do in-class reading of pleasurable texts--particularly in connection with the Common Core, which does not, as far as I can see, pay much explicit attention to reading volume or making sure that students are actually reading.  On the other hand, I was like: What?!  Perseverance?!  That is so depressing!  We "persevere" in things that are difficult, painful, discouraging.  Reading a good book is not supposed to be difficult, painful and discouraging; it's supposed to be entertaining, engrossing, FUN!

Maybe I'm just lucky, but my experience is that for well over 90% of my students (whether in the Metco program or not), they don't have much trouble sitting and reading silently.  This year, I had one girl who complained the most about how much she disliked reading; over the past couple of weeks I have had to tell her more than once, when we were having a discussion or doing a writing activity, to stop reading and put her book away.  Reading is fun--that's what we need to be helping our students to discover.

More on other stuff later; it was an interesting day.

Teachers, Instruction, Relationships

(A quick post--I'm thinking through these issues--the brain surgery analogy makes sense to me, though it may come out of left field for some other people... Also, I'm not sure why the font switches size or style--I wrote it all out at the same time, I think!)

Teachers are not brain surgeons--we are much more important...
Teachers are not brain surgeons.  I sometimes wish we were.  Surgeons get paid a lot, and surgeons get to feel like they're DOING something, get to feel like there is a clear problem and they are going to fix it.  Plus, their job is a lot easier than the jobs of regular doctors or of teachers, because they don't have to deal with difficult people.  The people they deal with are easy to deal with, because they are asleep.

While we teachers may wish we had anaesthetists to put our students to sleep before we worked on them, and while we may wish that what made the difference was the actions we ourselves did directly, rather than those we helped other people do, our lives are not so simple.

A lot of teacher discourse sounds like we are like surgeons, focusing on what the teacher does to, not with, the students, just as much of the discourse about reading education avoids the term "reading" and relies heavily on terms like "instruction",  "skills", "strategies", and so on. These terms--like the term "teaching" itself--make it sound like we are either conveying information in a gradgrindian way (filling little vessels with facts!) or else instructing children in a step-by-step process, as if we are ambulatory instruction manuals.  

But English teachers are not instruction manuals.  What's ironic is that just as teachers in other disciplines are getting away from "instruction" by, for instance, flipping the classroom, so that the teacher works instead as a coach, we English teachers, who arguably have less to "teach" than teachers in "the content areas", are constantly being told about "instruction."

Why would we want to be instruction manuals?  Instruction manuals don't actually work very well, because most people don't like to use them. Most instruction manuals, like the one in the John Ashbery poem, lead not to understanding, but to our dreaming of Guadalajara, or of some other place we have never been.  

Teachers are neither surgeons, operating directly on the brains of etherized students, nor ambulatory instruction manuals, mechanically dispensing information.  We are people dealing with other people, and the important things we do are to create conditions in which kids will want to read and write, and to help kids pay attention to their own reading and writing.  This may involve instruction, but often it does not.

Relationship before task
I once peeked at the "Course Expectations" a wise colleague of mine was handing out on the first day of the year.  My colleague told his students that when people were doing difficult tasks, studies had shown that they could do them better and longer if there were only another person in the room.  His goal that year, he had written to his students, was to be, for them, that "other person in the room."

I thought about my colleague's words when I was talking with my wife the other day.  My wife has been doing a fair amount of SAT tutoring recently, and she tells me that parents often talk as if what she will be doing with their children is sharing with them the "strategies" and "tricks" that she learned in her time as a high-achieving student at ivy league colleges and her stints as a college instructor at elite universities.  The truth is, she said, that when she and her students work on the reading sections of the SAT, what she mainly does is sit with the students while the students figure it out for themselves.  What I really do, she said, is to help them to see their own thought processes.  In order to help them do so, she needs to build a strong relationship with the student--so she always spends some time chatting with her students about what's important to them, what's going on in their lives.

My department chair is right to be fond of the dictum, "Relationship before task."  The task is the responsibility of the students; the relationship is our responsibility. 

Building Relationships (more wise colleagues!)

In today's department meeting, we were treated to a presentation by a couple of our colleagues who have been systematically investigating the ways in which they build relationships with their students.  Relationships with students, they said, are built through the many small moments, the micro-interactions, that we have each day with individual students.  They offered a set of helpful guidelines for handling these little moments, many of which may present themselves to us as difficult or annoying.  Many of these guidelines were familiar from parenting books I've found helpful in dealing with my own kids at home (acknowledge, validate, try non-verbal response, give benefit of the doubt, start with observation, not judgment, offer student choice to provide solution, etc.), but I realized that I hadn't given much conscious thought to how they applied to dealing with students, and it was helpful to have them made explicit.

As these guidelines may show, our wise colleagues encouraged us to detach ourselves from our annoyance, see our students with compassion, and use these sometimes-difficult moments to build stronger human relationships with our students.  To illustrate their wisdom, our colleagues told a few stories and showed us some video they had taken.

One wonderful story was this: when one of these colleagues was a student teacher, she went with her mentor teacher and their students on an overnight trip early in the year.  At a certain point late in the night, one student went missing.  The mentor teacher was annoyed and upset, and clearly, said my colleague, angry with the student who had slipped away.  When they finally found the student, my colleague expected her mentor teacher to light into the kid, or at least show that she was upset.  Instead, she told us, the master teacher opened her arms, gave the kid a huge hug, and said, "We were so worried about you!"  For my colleague, this was an eye-opening experience--and as she told the story, we were right there with her.  A good teacher doesn't let her students off the hook, but always shows that she cares about them.  If she doesn't, they won't learn.  According to my colleague, that kid, who was going through a difficult time, had a good relationship with her teachers for the rest of the year.

The video that our colleagues shared was also really interesting.  The clips we watched were only a minute or two each, but it was amazing how much was happening in those minutes.  I've never seen so clearly how a good teacher can have a dozen meaningful interactions, with a dozen different students, in the course of only a minute or two.  A look, a wave, a few words--each time this happened, a relationship was strengthened and a student was helped to, as my wife says, see his or her own thought process.  My colleagues were holding up a mirror--a compassionate mirror--to their students, and their students clearly felt supported, cared for, and encouraged.

Not brain surgeons, not instructors, but people!
These colleagues have also spent a lot of time working together on planning classroom activities (how to introduce books, how to facilitate discussions, structures for bringing out responses and helping students to organize their thinking, etc.), but very little of what they do is what I would call "instruction".  They are not brain surgeons, not instruction manuals, but people, helping other people.