Powerful Writers Program Evaluation

School Year 2003-2004

Prepared by Dr. Gary Troia

During the 2003-2004 school year, four teachers on the 4th/5th grade team at Beacon Hill Elementary School elected to participate in this programmatic evaluation. Beacon Hill is in south Seattle, a diverse area of the city with respect to race/ethnicity and socioeconomic status. Approximately 63% of the students at Beacon Hill are receiving free or reduced-price meals. The school has more than 86% minority enrollment, mostly students classified as Asian/Pacific Islander (51%), followed by students of Hispanic descent (24%). The participating teachers were asked to collect student writing samples from 3 high-performing students and 3 low-performing students in their classes over the year such that these samples represented a variety of writing genres and topics. These samples were scored and analyzed to determine to what degree teachers' implementation of a strong writing workshop model, supported through Powerful Writers, would positively impact their students' writing performance.

Writing samples were collected from 31 students in the four participating classrooms. For comparative purposes, only samples collected from the majority of students at the same time during the year were included in the analyses presented below. Consequently, the writing samples used for analyses included personal narratives written in September and November and expository papers written in April. Other writing samples, such as poems and fictional narratives, were collected but not analyzed either because they were not scored or were not present in a majority of student portfolios.

The samples were scored by two independent raters who are affiliated with the Northwest Regional Educational Laboratory using a four-point scale ranging from 1 (far below standard) to 4 (exceeding standard) for each of six writing traits: ideas, organization, voice, word choice, sentence fluency, and conventions. An average of the scores assigned by both raters for each writing sample was used in all analyses. The scoring reliability estimates for each trait are provided below:

Ideas = .82 Organization = .59 Voice = .74

Word Choice = .59 Sentence Fluency = .74 Conventions = .77

It should be noted that all of the traits were significantly and positively correlated with all other traits, meaning that a high score on a given trait was likely to be accompanied by high scores on other traits while a low score on a given trait was likely to be accompanied by low scores on other traits. The strongest relationships existed between: (a) sentence fluency and conventions (.84); (b) ideas and organization (.75); and (c) word choice and voice (.74).

In the first analysis, trait score means were compared across time points to examine growth in writing (see Table 1). Statistically significant differences were observed for 5 out of 6 traits (sentence fluency scores grew only slightly over the year). From September to April, students displayed significant growth in ideas (approximately a one-third point increase on the 4-point scale), voice (about a two-fifths point increase), word choice (about a one-third point increase), and conventions (about a half point increase). The half point increase in writing conventions represented a change from below standard to meeting standard for many students; 8 months of writing instruction and support from Powerful Writers helped students improve their spelling, capitalization, and punctuation skills. With respect to organization, students' scores rose significantly (about a one-third point increase, many students moving from below standard to meeting standard) from September to November, but fell by April. However, scores for text organization for writing samples collected in April generally exceeded organization scores for samples collected in September and, on average, met standard. Additionally, a drop in scores for organization is not surprising because the organization for narrative texts (which substantially improved between September and November) is quite different from the organization for expository texts (which were only collected in April). Expository texts tend to be much more difficult for students to write because they lack sufficient exposure to this genre through reading and because exposition entails a variety of organizational frameworks (e.g., cause-effect, compare-contrast).

Because personal narratives were collected in both September and November, but expository papers were collected only in April, a separate analysis was conducted to compare performance across these two genres, giving them equal weight (see Table 2). The trait scores for narratives at each time point were averaged to obtain a grand mean on each trait for narrative writing, and these grand means were compared with the mean trait scores for exposition. Significant differences were found for ideas, voice, word choice, and conventions; in all cases, scores for these traits were higher in expository texts (ranging from one-fifth to one-third point higher) than narrative texts. Keeping in mind that expository texts are typically more challenging for students, the better scores on 4 of 6 traits for expository texts collected later in the school year probably represents the most convincing evidence that writing growth took place for many of the students in the four classes.

The proportion of 93 papers scored (3 papers per student; 31 students) meeting or exceeding standard on each trait is summarized below. Over 62% of the papers met or exceeded standard for all 6 traits. When examining student performance, almost 30% of the students met or exceeded standard on all 6 traits on all 3 pieces of writing. Most importantly for comparisons with existing data, nearly 84% of the students met or exceeded standard on all traits on at least one paper. This finding compares quite favorably with the 2003 Direct Writing Assessment results for third graders at Beacon Hill, on which only 39% of students met or exceeded standard using the same criteria (a score of 3 or 4 on every aspect of writing measured for a single paper). It also demonstrates superior outcomes in comparison with the 2003 WASL Writing results—only about 68% of fourth graders met or exceeded standard on the WASL using similar criteria. Taken together, these findings indicate that students in these four classrooms, receiving strong writing instruction with the support of Powerful Writers, are making substantial gains in their writing and, as a group, are poised to excel on important high-stakes writing assessments.

Met or Exceeded Standard Exceeded Standard

Ideas 94.5% 62.6%

Organization 72.6% 37.4%

Voice 89.1% 53.9%

Word Choice 94.6% 46.2%

Sentence Fluency 79.2% 39.6%

Conventions 73.7% 37.4%