On sources

There is an editorial by E.D. Hirsch, Jr. in the Times today about standardized testing. I think it caught my eye because of the baseball bit, but I thought this was interesting. The writer reports on a study from 1988:

Experimenters separated seventh- and eighth-grade students into two groups — strong and weak readers as measured by standard reading tests. The students in each group were subdivided according to their baseball knowledge. Then they were all given a reading test with passages about baseball. Low-level readers with high baseball knowledge significantly outperformed strong readers with little background knowledge.

I thought, Wow! That's really important! And the conclusion Hirsch draws is that "The experiment confirmed what language researchers have long maintained: the key to comprehension is familiarity with the relevant subject. For a student with a basic ability to decode print, a reading-comprehension test is not chiefly a test of formal techniques but a test of background knowledge." Which neatly supports some of my own prejudices.

So I thought I'd write about this on the blog and say, "See?" But there was a problem: there's no link to the study. There's no reference or footnote to find it. There's no actual way of telling exactly what that study shows, or if it even exists.

So I did what anyone would do: I googled. And I found something that might be the study referenced, but a) I can only get to the abstract; and b) there are some very pertinent differences.

Here's the abstract:

The purpose of this experiment was to determine the effect of expert knowledge and metacognitive knowledge on strategy acquisition using an ecologically valid task. Fourth- and fifth-grade boys were poor readers and baseball experts were trained in the use of a reading strategy (asking why questions), with instruction being embedded in either baseball or nonbaseball stories. The boys were tested for strategy use, recall, and accuracy of monitoring 1 to 3 days after training, then again after 2 to 3 weeks later. Boys trained using baseball stories demonstrated greater strategy use at both post-tests than boys trained using nonbaseball stories, indicating that knowledge base aided in the acquisition of a reading comprehension strategy. Boys with higher declarative metacognition scores demonstrated better strategy acquisition and increased literal recall. All types of recall were higher for baseball stories.

[emphases in bold mine]

So, similarities: an experiment on reading comprehension involving baseball. Differences: a different grade level, only poor readers, using baseball stories for instruction on strategy, rather than using the baseball story as the test story itself.

All of which casts doubt for me on the validity of this editorial.

Where is this study? What does it say? What was its methodology? And have there been follow-ups? One study over 20 years ago does not a scientific conclusion make.

I hate feeling that I have to be skeptical of everything I read in the papers.