Volume 1, Number 1

Moulthrop and Kaplan question population

In "Seeing through the Interface: Computers and the Future of Composition," Nancy Kaplan and Stuart Moulthrop question the test population used by Marcia Peoples Halio in her article "Student Writing: Can the Machine Maim the Message?":

<1> To yield sound data (even in descriptive work), a comparative study must first establish that the test populations have comparable levels of ability before they begin the test conditions. This requirement is especially important when subjects are self-selecting rather than randomly assigned to the treatments (see Slavin). Halio claims that "all students in the computer sections have roughly comparable levels of writing ability" (17), but her methods were inadequate to establish even a rough baseline. Since Halio's test groups were regular classes, their comparability depends on her university's placement system, which assigns students to a "medium writing ability range" according to verbal SAT scores and a writing sample. Halio gives no account of the range of scores or of the inter-rater reliability for readers of the writing sample, but even had she done so, the SAT scores and writing samples could not have established comparability because the middle group, comprising neither honors nor remedial students, was designed to include the large majority of the university's first-year students.

<2> In his guide to research methods in education, Slavin explains that "so much of the variance [in any study] . . . is explained by student ability or past achievement that treatment effects are almost always small in relation to student to student differences" (28-29). In controlled experiments that compare two educational technologies, either the experimental groups are assembled randomly or a single group is introduced to both technologies. This procedure is necessary if the study is to eliminate the influence of social or personal differences--like the contrast between the "childish" attitudes of the Macintosh users and the "businesslike" orientation of the IBM users that Halio posits. With no reliable ways to establish students' abilities at the beginning of the term, Halio could make no valid claims based on differences she saw later.(256)

Return to the critiques of Halio. Move on to Moulthrop and Kaplan's Works Cited.
Move on to the conclusion, or return to Not Maimed but Malted.
Return to Daniel's home page or the CWRL home page.
iamdan

CWRLTable of Contents