America's Choice

America’s Choice Comprehensive School Reform Design First-Year Implementation Evaluation Summary by Thomas Corcoran Margaret Hoppe Theresa Luhm Jonathan Supovitz  ABOUT CPRE The Consortium for Policy Research in Education (CPRE) unites five of the nation’s top research institutions in an exciting venture to improve student learning through research on policy, finance, school reform, and school governance. The members of CPRE are the University of Pennsylvania, Harvard University, Stanford University, the University of Michigan, and the University of Wisconsin-Madison. CPRE is currently examining how alternative approaches to reformsuch as new accountability policies, teacher compensation, whole-school reform approaches, and efforts to contract out instructional servicesaddress issues of coherence, incentives, and capacity. To learn more about CPRE, visit our web site at www.upenn.edu/gse/cpre/ or call (215) 573- 0700, and then press 0 for assistance. ABOUT AMERICA’S CHOICE The National Center on Education and the Economy (NCEE) is the nation’s leader in standards- based education. NCEE’s founders created this not-for-profit organization in the conviction that virtually all young people in the United States can and must achieve at the same high standards reached by their counterparts in other nations. NCEE exists to develop the policies and tools and provide the professional development and technical assistance that schools, districts, and states need to implement comprehensive programs of standards-based education and training. NCEE has made a special commitment to meeting the needs of low-income and minority youth. The America’s Choice School Network, originally NCEE’s National Alliance for Restructuring Education (NARE), offers a comprehensive, research-based design for schools and districts committed to standards-based education. NARE was named as one of the proven design models in the federal Obey-Porter legislation that provides grant funding for the Comprehensive School Reform Demonstration (CSRD) program. To learn more about America’s Choice, please call (202) 783-3668. You can also find information on the world wide web at www.ncee.org/ac/intro.html.  America’s Choice Comprehensive School Reform Design First-Year Implementation Evaluation Summary Thomas Corcoran, University of Pennsylvania Margaret Hoppe, MRH Evaluations Theresa Luhm, University of Pennsylvania Jonathan Supovitz, University of Pennsylvania Consortium for Policy Research in Education Graduate School of Education University of Pennsylvania February 2000 Copyright 2000 by the Consortium for Policy Research in Education America’s Choice, Comprehensive School Reform Design: First-Year Implementation Evaluation Summary 1 I. INTRODUCTION In the fall of 1998, the National Center on Education and the Economy (NCEE) contracted with the Consortium for Policy Research in Education (CPRE) to conduct the evaluation of the America’s Choice School Design. This is a summary of CPRE’s first report of a three-year evaluation of the design. The evaluation of America’s Choice seeks to answer four basic questions: Are schools successfully implementing the America’s Choice program design? What environmental characteristics are facilitating or impeding implementation? How effective is America Choice’s implementation strategy? And what are the impacts of the program on teachers and students? As America’s Choice is still in the early stages of implementation, most evaluation efforts are directed toward the questions about the implementation of the program and the conditions surrounding its implementation. In subsequent years, CPRE increasingly will emphasize its evaluation of the impacts of the program on students. This report describes the first year of the implementation of America’s Choice. Following this introduction, section two provides a description of America’s Choice and the theory behind the America’s Choice school design. Section two concludes with a set of reasonable expectations for the progress of America’s Choice in its first year. Section three describes CPRE’s findings concerning the implementation of America’s Choice, including many of the specific design components. Section four analyzes the role of the school district in the implementation of America’s Choice. The report concludes with a summary of the findings of the first year’s evaluation. This report is primarily based upon six data sources. First, CPRE researchers conducted telephone interviews with 21 design coaches and literacy coordinators across multiple America’s Choice schools. Second, CPRE visited ten schools, observed model classrooms and classrooms in which core assignments were being taught, and interviewed over 60 school staff members. Third, CPRE developed, disseminated, collected, and analyzed the results of a survey of the population of teachers and Leadership Team members in all 41 Cohort I America’s Choice schools.1 In all, about 1,900 surveys were distributed, with a 63 percent return rate. Fourth, CPRE attended several of the NCEE professional development sessions to train America’s Choice school leaders. Fifth, CPRE interviewed virtually all key NCEE staff. Finally, CPRE staff reviewed much of the relevant America’s Choice documentation, including training manuals, curriculum materials, and other NCEE documents. II. DESCRIPTION OF AMERICA’S CHOICE The America’s Choice School Design is a K-12 comprehensive school reform model designed by the National Center on Education and the Economy. America’s Choice focuses on raising academic achievement by providing a rigorous standards-based curriculum and safety net for all students. The key features of the America’s Choice School Design include: 1 A list of the participating schools is provided in the Appendix.  2 America’s Choice, Comprehensive School Reform Design: First-Year Implementation Evaluation Summary • Five design tasksstandards and assessments, learning environments, community services and supports, high-performance management, and parent and public engagement. • The New Standards Performance Standards. • The New Standards Reference Exams. • Curriculum aligned to the standards. • A safety net for all students. • Class teachers who follow students for three years. • A planning and management system focused on results. • On-site continuous technical assistance. • A comprehensive professional development program. In most cases, the primary reason a school or district chose to implement America’s Choice was a history of low student achievement. Many schools are under district or state pressure to raise achievement levels. School and district administrators generally explored several whole school reform models before deciding to adopt Americas Choice because they believed it would help them accomplish their goal. Over 80 percent of a school faculty should indicate their commitment to the Americas Choice design and agree to implement the program over three years. Each school must assign personnel as a design coach who coordinates the reform effort and acts as a liaison to NCEE, as a literacy coordinator who introduces the literacy program to the school staff, and as a community outreach coordinator who ensures that students get needed support services. In addition, schools must provide tutoring and other assistance to students falling behind, participate in America’s Choice professional development sessions, and use the New Standards Reference Exams. Additionally schools need the support of their district administration and must reserve $65,000 to $95,000 a year for three years to contract with America’s Choice for training, materials, and on-site assistance. THE THEORY BEHIND AMERICA’S CHOICE America’s Choice does not offer schools a script or a paint-by-numbers approach to reformed instruction. Rather, the core of the design is a set of basic principles about the purpose of schooling and how schools should operate and a set of tools for building a program based on those principles. The essential principles are as follows: • High expectations for student performance should be explicitly expressed in a comprehensive set of performance standards that drive the work of teachers and students. These standards should be made concrete through examples of student work that serve as benchmarks. • A common core curriculum should be aligned with the standards and understood by everyone. America’s Choice organizes the life of a school around such a core curriculum.  America’s Choice, Comprehensive School Reform Design: First-Year Implementation Evaluation Summary 3 Ideally, the topics and the materials are known by the teachers, the curriculum structure is understood by parents, and there is a strong commitment by all to getting the students to where they need to be. • Performance standards specify what students should know and be able to do at certain junctures, but also provide concrete examples of student work that link the standards to student performance. • Assessments should be aligned with the core curriculum that provide incentives for teachers and students to do the work and that provide feedback for identifying and correcting deficiencies in the program or in the work of students and teachers. • An orientation to results focuses on how resources are used, gaps in the curriculum, and gaps in knowledge. • Extended time for literacy that includes a 2 1⁄2 hour block for students in grades K-3 and a 2 hour block for students in grades 4-5. Students who are performing below grade-level in grades 6-12 are provided with a 2 hour literacy block for extra support. • An understanding that time is a precious resource for both teachers and students and should be used purposefully. • A commitment to teacher professionalism seeks to enable teachers to function as full professionals by requiring a high-quality professional development program that is aligned with the standards and in which content and pedagogy are intimately connected. • A compassionate commitment to students. America’s Choice views students as change agents because they will both demonstrate and demand the rituals and routines that have been effective for them, and the model understands that some students will need special assistance or more time to succeed. • Providing access to high-quality support by requiring schools to have a full-time design coach and a full-time literacy coordinator who receive extensive training by NCEE. • The formation of a school Leadership Management Team to coordinate implementation, to ensure the necessary resources for implementation, and to align other school activities with implementation of the design. • Using the power of demonstration to persuade teachers to try new approaches to literacy teaching. Literacy coordinators make it real in their own classrooms, and when they can do it themselves, they can show it to others. The America’s Choice design begins the process of changing instruction with writing. This establishes the credibility of the design by demonstrating improvement in student work prior to challenging deeply held beliefs about reading instruction.  4 America’s Choice, Comprehensive School Reform Design: First-Year Implementation Evaluation Summary • America’s Choice recognizes that the pace of change will vary from school to school and the model does not have a rigid three-year implementation schedule. Schools need models that show the way and they need tools and exemplars to build the scaffolding that will support their new architecture until it is institutionalized and self-sustaining. In support of these principles, America’s Choice offers a set of tools and building blocks that allow school staffs to transform the standards into a curriculum, a schedule, and an organizational design that sustains the focus on the standards and the results. What are some of these tools and building blocks? All America’s Choice schools are expected to use the New Standards Performance Standards and the related Reference Exams. Other assessment instruments such as portfolios have been developed to provide richer information about students and make it easier to observe, share, and discuss student work. America’s Choice also provides core curriculum units that will eventually cover more than 100 instructional days. At present, America’s Choice offers teachers a clear and explicit vision of good practice for literacy teaching and learning aligned with the standards, and it will eventually offer a complementary vision in mathematics. Further, there are ideas about scheduling that focus large blocks of time on literacy and mathematics. There are also notions about staffing and organization that build stronger relationships between adults and students by keeping them together from grade to grade. Safety nets provide support and extra instructional time for students when needed. There are even special curricula to help students catch up. Tested approaches to involving parents in their children’s education are still another component of the design. The America’s Choice design seeks to develop a school’s internal capacity for continued improvement. The designers believe that, if the building blocks are put into place, used well and sustained, then a school will experience substantial and lasting performance gains. EXPECTATIONS FOR THE FIRST YEAR OF AMERICA'S CHOICE The 1998-99 school year was the first year of implementation in schools choosing the America’s Choice School Design. In some cases, schools did not begin implementation until mid-year. The focus of the first year of any reform initiative is on putting the new infrastructure in place and on helping teachers and administrators to learn new roles and new skills. The America’s Choice design assumes that it will take time to produce changes in teaching and learning. However, the evaluation team believes that it is reasonable in the first year to expect the following: • A clear vision of reform is emerging and is shaping decisions at all levels of the school. • Staff in each America’s Choice school understand the theory underlying the reform, understand what is expected of them, and some begin to behave in ways that realize the potential of the reform. • New structures required by the design, such as the Leadership Management Team, have been put in place and are functional in each school.  America’s Choice, Comprehensive School Reform Design: First-Year Implementation Evaluation Summary 5 • Teachers and school administrators are beginning to make changes consistent with the reforms; there are indications those reforms are reaching classrooms and affecting students. III. IMPLEMENTATION FINDINGS America’s Choice schools made substantial headway in their first year of implementation of the design. Our survey and interview analyses suggested that school staff were supportive of standards and quickly accepted the design requirements. They implemented school-wide features such as the 25 Book Campaign and Book of the Month, and literacy coordinators developed model classrooms. Although training time was limited, design coaches worked with staff so they understood the America’s Choice design, became familiar with New Standards Performance Standards, and administered the New Standards Reference Exams. Schools instituted Leadership Management Teams that worked on finding ways to extend learning time and establish safety net programs. In this section of the report, we discuss our findings about the implementation of America’s Choice in the 1998-99 school year. As an organizing structure, we first discuss findings about the design in general, and then we look in greater depth at the implementation of components of the design including standards, assessments, the literacy program, and the high performance management system. This serves only as a loose structure, however, since the components of the design are intimately interrelated. THE AMERICA’S CHOICE DESIGN In less than one year, most schools made some headway in moving practice closer to the America’s Choice design. Our survey and interview data identified the extent to which schools were using strategies required by the America’s Choice design: • 25 Book Campaign. In all but one school we visited, staff were implementing the 25 Book Campaign and the Book of the Month as a result of adopting America’s Choice. Sixty percent of the language arts teachers were using the 25 Book Campaign with their students. More model classroom teachers (83 percent) than language arts teachers (60 percent) were participating. Over half of the K-8 language arts teachers reported implementing the 25 Book Campaign; far more grades 3-5 language arts teachers (75 percent) were doing so than teachers of grade 9-12 (17 percent). • Posting Standards in the Classroom. Over two-thirds of the teachers reported posting standards in their classrooms; 75 percent of teachers reported that they displayed models of student work that met standards. During our visits in the spring of 1999, we observed standards posted in many classrooms, but more frequently, these were state content standards, not New Standards Performance Standards. Posted student work often represented all the students in the class, not just work representative of the standards.  6 America’s Choice, Comprehensive School Reform Design: First-Year Implementation Evaluation Summary • A Consecutive Instructional Block for English Language Arts. Less than 40 percent of the teachers reported having a consecutive block of time for reading and writing. Only a handful of schools that we visited had a consecutive block for English language arts, and these were elementary schools that had instituted this practice prior to implementing America’s Choice. We found one high school that had a double language arts block for students performing below state standards, but again this was instituted prior to the adoption of America’s Choice. We did find that schools were exploring how to implement a consecutive block in the 1999-2000 school year for some, if not all, grades. • A Two and One-half Hour Block for English Language Arts. Over 70 percent of the K- 5 teachers reported more than six hours a week allocated to literacy, as compared to 36 percent of language arts teachers of grades 6-8 and 15 percent of the grades 9-12 language arts teachers. Based upon teachers’ survey results, the average time spent per day on reading and writing in grades K-2 was 130 minutes, and it was 128 minutes in grades 3-5. Instruction time for reading and writing in grades 6-8 dropped to 101 minutes per day and to 87 minutes in grades 9-12. • Teaching Different Genres. More grades 3-5 language arts teachers reported teaching responses to literature (90 percent), reports (78 percent), narrative accounts (84 percent) and narrative procedures (61 percent). By contrast, more grades 9-12 language arts teachers taught reflective essays (64 percent) and persuasion (58 percent). Very few K-2 language arts teachers taught reflective essays, persuasion, or narrative procedures, which were not normally part of their curriculum. • Use of Leveled Texts. More grades 6-8 language arts teachers (48 percent) reported using leveled texts in their classrooms than K-2 language arts teachers (39 percent), grades 3-5 language arts teachers (40 percent), or grades 9-12 language arts teachers (28 percent). This pattern may be due to misinterpretation related to basal reading systems, which have stories written on grade level but are not leveled literature books. • Resources for Core Assignments (fiction and non-fiction books). About 60 percent of the K-8 language arts teachers reported using classroom libraries as central to instruction, as compared to less than half of the grades 9-12 language arts teachers. • Full-day Kindergarten. All but one of the schools we visited had a full-day kindergarten in operation prior to the adoption of America’s Choice. Teacher Agreement with Statements about Design Components 68 Standards are posted in my classroom 100 71 75 50 25 0 52 School library has enough books for 25 Book Campaign Before/after school programs in place % Agreement America’s Choice, Comprehensive School Reform Design: First-Year Implementation Evaluation Summary 7 • Teachers Staying with Students for Three Years. About 30 percent of the teachers currently stayed with students over multiple years. Some elementary teachers were staying with students for two years, but none for three years. • Safety Net Programs: Before and After School and Saturday. Over two-thirds of the teachers reported there were programs before or after school for students who needed more time; less than one-third reported Saturday programs or that the school schedule was built to provide additional time for students in need. Sixty-one percent of principals reported there were before or after school programs for students, and 52 percent reported assistance was available on Saturday. • Safety Net Programs: Summer School. All the districts we visited had summer programs, but about half were either enrichment programs or were programs only for students not reaching benchmarks. Half of the grades 6-12 language arts teachers reported that summer school was mandatory for those who finished the year below grade level in literacy or mathematics, but only about 30 percent of the K-5 language arts teachers reported this to be the case. • America’s Choice Approach to Discipline. In the schools we visited, model classroom teachers had established rituals and routines in their classrooms and felt this reduced discipline problems. About half of the schools reported that they were working on a discipline policy, and the others had postponed it until next year because all their efforts were focused on other America’s Choice components such as model classrooms, English language arts blocks, and the 25 Book Campaign. Despite many challenges, design coaches and literacy coordinators were rolling out the America’s Choice design within four months of training. Our interviews with design coaches and literacy coordinators four months after their first training suggested that implementation had progressed in every elementary school, although its extent and breadth varied. Design coaches had begun introducing staff to the America’s Choice design elements and New Standards Performance Standards, and literacy coordinators had conducted training on the Writer’s Workshop and begun setting up model classrooms. Most had a clear understanding of their roles and, with the exception of two schools, had moved the school staff into work on the design. Some design coaches and literacy coordinators were making presentations to parents through the PTA, and parents were requesting information about books and writing. In all of the schools we visited, Book of the Month had been instituted and there was a high level of receptivity from the students and the staff. Others were enthusiastically engaging teachers in the 25 Book Campaign, using rubrics to assess student work, and planning how to change the discipline policy. Some had formed Leadership Management Teams, often blending them with existing school-based management committees. The Leadership Management Teams were making progress in ensuring that America’s Choice was part of the dialogue on school change and improvement. In a few schools we visited, progress was slower; they had only implemented one or two training sessions and the Book of the Month. These were settings with many obstacles to implementation, but optimism was high that more tasks would be completed as the year unfolded.  8 America’s Choice, Comprehensive School Reform Design: First-Year Implementation Evaluation Summary The following factors influenced the implementation progress of the America’s Choice design: • Principal Support. Principals generally were supportive and their support grew after the national conference in January. Both design coaches and literacy coordinators reported that principals were supporting their efforts by finding time for staff training, procuring resources, and assisting in selecting teachers for model classrooms. • Multiple Jobs. Many of the design coaches we interviewed had not been released from other job responsibilities. Most were assistant principals and a few were Title I teachers or staff developers. Design coaches were performing their regular job responsibilities as they conducted America’s Choice training and found this extremely stressful. • Resource Shortages. Shortages of resources occurred at different levels. In some schools, the resources for buying books for running records or the 25 Book Campaign were not available. Districts often provided fiscal support, but it took time to obtain it and time to procure books. Some experienced difficulty in obtaining the furniture and materials needed to organize model classrooms. Several design coaches reported having difficulty obtaining sufficient copies of the America’s Choice materials. These problems delayed training of all staff in a timely fashion and implementing the required tasks of the design. • Teacher Resistance. Both design coaches and literacy coordinators reported experiencing some resistance from staff, but the extent varied by school, grade, and experience of the teacher. Some found that resistance came primarily from experienced teachers. • Limited Time for Professional Development. A consistent challenge was the limited time available for professional development. Most schools and districts had already set their training schedules for the year, and there was little or no available time in the calendar for staff training in America’s Choice. Union contracts prohibited requiring teachers to spend additional time in professional development. Some principals used substitutes or teacher assistants to free teachers for training during the regular school day. Other principals used grade-level or faculty meetings for training. Some offered after- school training and provided stipends to encourage attendance. • High Schools. The roll out of America’s Choice in high schools was not progressing at the same pace as in elementary schools. Many found it more difficult to translate components of the design to departmental settings. Many high schools also were facing new state tests, and their staffs were overwhelmed with changing curriculum to support students in the new assessments. • New Textbooks. Introduction of new textbooks in some districts also was interfering with staff receptivity to the America’s Choice design. For example, in one setting, new textbooks had been adopted for all the core subject areas, so staff were trying to learn how to use new materials at the same time they were learning how to implement standards. These were not major barriers to implementation, but they were often cited as issues that delayed implementing design activities or caused staff resistance.  America’s Choice, Comprehensive School Reform Design: First-Year Implementation Evaluation Summary 9 Overall, design coaches and literacy coordinators reported they were moving forward and trying to meet the expectations set in their training. All were concerned they were behind schedule, but most were pleased with the receptivity of staff and students. Teachers in the America’s Choice schools were supportive of the design. Almost 90 percent reported that America’s Choice had the potential to benefit students; about two-thirds reported that it was a unifying strategy for the different programs in the school and was consistent with other programs in the school. Those deeply engaged in America’s Choice were more positive than all language arts or other academic teachers. When we interviewed teachers in the spring of 1999, we also spoke to individuals who were less supportive of the America’s Choice design. Their criticisms fell into four categories: not liking whole school reform in general; not supporting how America’s Choice became the school’s approach to whole school reform; not understanding or liking specific aspects of America’s Choice; or not seeing the requisite resources to implement the America’s Choice requirements. The issues or concerns that teachers voiced, for the most part, demonstrated a lack of understanding and preparation for America’s Choice strategies, such as the scoring rubric, rather than flaws in the design. Many teachers were confused by the New Standards Reference rubrics. Some were concerned that the New Standards Reference rubric was not consistent with the state rubrics; others thought the New Standards rubric was too difficult. In addition, some teachers voiced concern about resources. Some teachers lacked confidence that their colleagues could teach the design correctly and felt the only way to learn it was through seeing a classroom in another school or a demonstration by an NCEE staff member. There was a considerable range in the degree to which the design had penetrated school cultures. Both survey data and site observations suggest that there is extensive variation in the extent to which teachers and administrators have confidence in the underlying theories about teaching and learning which form a foundation for the America’s Choice reform. For example, over one third of the teachers on the survey agreed that special education and English Language Learners should not be expected to meet the standards. Four out of ten teachers felt that students are not ready for problem solving until they have acquired the basics. These responses and others suggest that the belief systems underlying standards-based reform are not yet widely held in the schools. Principals were very positive about the America’s Choice design. In their survey responses, principals unanimously reported that they understood the purpose of the program (100 percent) and that the design was consistent with other programs in the school (94 percent). They also perceived the design as a unifying strategy for the school’s different programs (97 percent), and reported that the reform approach had the potential to benefit students (97 percent). The schools we visited were at different points in the America’s Choice implementation schedule. School staffs learned about the America’s Choice design in October of 1998. Overall, they made considerable progress in implementing the design elements in eight months. We analyzed the ten schools we visited against the design rubric suggested by National Center on Education and the Economy. These schools, not necessarily representative of all the schools adopting America’s Choice in 1998-99, were at different points in implementing the design  10 America’s Choice, Comprehensive School Reform Design: First-Year Implementation Evaluation Summary elements. One hundred percent of the schools were implementing the 25 Book Campaign, had or were planning modifications to their master schedule, had staff who attended the literacy institute and the design coach training, and had made modified allocations of their human and non-human resources to support America’s Choice. Seventy-five percent or more of the ten schools we visited had started the training in America’s Choice (including introducing staff to standards); had administered the New Standards Reference Exams: had established model classrooms; had started leveling or acquired leveled books; had planned or started safety net programs before or after school as well as during the summer; had full-day kindergarten; had started the Book of the Month; had established a Leadership Management Team; had staff who attended the NCEE National Conference; and had delivered professional development seminars for the staff. The ten schools had made less progress in changing the learning environment (which included displaying student work against standards), changing the discipline policy, using Core assignments, having the principal monitor classrooms for implementation of standards, and attending principals’ networks. STANDARDS Most staff in the America’s Choice schools understood and supported the New Standards Performance Standards. Almost 90 percent of the teachers responding to the survey reported support for the New Standards Performance Standards. They understood the purpose of the New Standards (87 percent), perceived standards to have the potential to benefit students (90 percent), and felt that standards were relevant to the subjects they teach (80 percent) as well as the grades they teach (80 percent). The teachers we interviewed had some understanding of the purpose of the America’s Choice design, but many lacked details about the design components. In most schools we visited, teachers had received copies of the New Standards Performance Standards and training in how to apply them in instruction, but they had difficulty in differentiating between the new standards published by the state and the New Standards Performance Standards. When asked on the survey about the influence of two key components of the America’s Choice designNew Standards Performance Standards and Core assignmentsover half the teachers reported each component was changing the way they taught language arts. For both New Standards Performance Standards and Core assignments, a greater percentage of grades 3-5 language arts teachers somewhat to strongly agreed, indicating that the America’s Choice components have influenced their teaching. A greater percentage of grades 6-8 language arts teachers strongly agreed about the influence of New Standards than teachers in other grade 100 75 50 25 0 87 Understand purpose 90 Have potential to benefit students 80 Relevant for grades taught 80 Relevant for subject(s) taught Teacher Agreement with Statements About the New Standards Performance Standards % Agreement America’s Choice, Comprehensive School Reform Design: First-Year Implementation Evaluation Summary 11 spans. A larger percentage of K-2 language arts teachers than those in other grade spans strongly agreed about the influence of Core assignments. Less than half of the teachers reported either discussing standards-based classroom instructional units or matching student work to standards with colleagues. When asked how frequently teachers who teach the same students analyzed performance data in light of standards, about one- third said they never met and another third said they met only once or twice a year. These data suggest there was little dialogue between teachers about student work, but it must be noted that most elementary teachers work in self-contained classrooms. Most teachers reported that their schools were focused on standards. Overall, teachers reported that their schools were focused on standards. More than 80 percent of the teachers responding to the survey said that the goals of the school were to get students to attain high standards. About two-thirds reported that their school had specific student performance targets, the staff had high academic standards for students, and the standards for student performance were sufficiently understood and specific to guide instruction Teachers varied in their capacity to use standards. Teachers differed considerably in their capacity to use standards for improving instruction (for example, analyzing student work, selecting curriculum materials, using instructional techniques, organizing a standards-based classroom, or giving students individualized attention). Teacher perceptions of their capacity were influenced by participation in America’s Choice training and by the grades they taught. More model classroom teachers and secondary teachers reported being well-prepared to do instructional activities related to standards than did those who had not received training or who taught the primary grades. ASSESSMENTS Most teachers reported high expectations for all students, but they used different criteria for judging students’ work. Eighty-nine percent of the teachers somewhat or strongly agreed that, given adequate time, students can master the expected knowledge and skills; 64 percent somewhat or strongly agreed that special education students in regular classes should be expected to meet standards. About two-thirds of teachers somewhat or strongly agreed that students with limited proficiency in English who are placed in regular classrooms should be expected to meet standards. Over two-thirds of teachers reported that all students were capable of learning the material that is taught.  12 America’s Choice, Comprehensive School Reform Design: First-Year Implementation Evaluation Summary 100 89 75 69 67 64 Teacher Agreement with the Following Statements 50 25 0 Given adequate time, students can master expected know ledge and skills My students are capable of learning material taught LEP students in regular classes should be expected to meet standards Special education students in regular classes should be expected to meet standards More than two-thirds of teachers (72 percent) somewhat or strongly agreed that all teachers should use the same standards in evaluating student work. There was no consensus among teachers that all student work should be evaluated the same. This reflects preferences for teacher autonomy in instructional matters, a viewpoint also seen in teacher responses to survey items about curriculum decisions. Over half the teachers somewhat or strongly agreed that they should be able to make independent decisions about what standards (57 percent) and what content (63 percent) they taught. To evaluate students, most teachers were still relying on traditional non-standards-based measures. Over 80 percent reported using the following approaches: teacher-determined criteria; comparing current with past performance; a student’s effort; class participation; daily attendance; and classroom conduct. About half (51 percent) of the teachers compared student work against external standards and less than half (44 percent) compared students against the rest of the class. The data suggest that many teachers were not yet using performance standards as a basis for evaluating student work. However, teachers believed there was consistency in their practice. About two-thirds somewhat or strongly agreed that they used the same standards in judging the quality of all students, engaged in dialogue with teachers in adjacent grades about the criteria for evaluating students, and believed teachers should all use the same standards in evaluating student work. These data suggest that the criteria used for evaluating students are deeply entrenched in the culture of schools, and strongly-held beliefs must be changed before the use of performance standards becomes commonplace. More model classroom teachers than other language arts or other academic teachers were using the design’s assessment strategies in almost all of their lessons. We asked teachers how frequently they used specific practices in evaluating students. We found that slightly more than half of the teachers reported having one-on-one conferences with students and that three-quarters of teachers collected, corrected, and returned assignments often or in almost all lessons. About 60 percent of teachers had students correct their own assignments in class. About one-third % Agreement America’s Choice, Comprehensive School Reform Design: First-Year Implementation Evaluation Summary 13 reported collecting, correcting, and keeping assignments often or in almost all lessons; the larger percentage of teachers who did so were model classroom teachers. THE LITERACY PROGRAM The following are samples from classrooms we visited where the Writers’ and Readers’ Workshops were being implemented. Students were enthusiastic and the rituals and routines suggested by the America’s Choice design were being implemented. • The second grade classroom is well decorated. There are 27 students in the class and all are either African American or Latino. A reading corner with a carpeted area and a rocking chair is located in the back of the room. A box of books labeled “leveled readers” sits on the bookcase. In the back of the room a bulletin board has a banner labeled “Authors’ Club” and beneath it were student writing samples. The writing standards are posted on the wall next to the bulletin board. The objective for the lesson is written on the chalkboard: “Students will be able to continue writer’s workshop activities by engaging in peer conferencing to get ideas, and engage in teacher/student conferencing to make sure the student is on the right track.” • In a first grade classroom, the teacher is explaining in Spanish and in English to 26 Latino students that they are going to read a story about George and what happens to him in the morning. She asks, “What happens when you get up in the morning? What do you have to do to get ready?” Different students explain what they doput on clothes, brush teeth, and take off pajamas. The teacher asks, “What is something that couldn’t happen?” The children struggle to find an answer and the teacher rephrases, “Would you take off your pajamas, put on your clothes, and then take a bath?” The class answers in unison, “No.” The class then listens to a tape about a boy named George and they follow along in their books. When the tape is finished, several students ask to hear the story again. After hearing the story a second time and reading it aloud with the tape, the teacher gives each student group a sentence strip. They decide what would happen first, second, third, and so forth, then place the strip on the board in front of the class and discuss what will happen next. • In a fifth grade classroom, students are working in groups of four. They listen to one another’s written essays and provide suggestions. The class is multiethnic and multilingual. While most of the discussion among students is in English, some students often rephrase in Spanish, Russian, and different Asian languages. One student voluntarily decides to assist the observer. She takes out her portfolio and explains that each student has his or her own portfolio with a cover of individual design. “After writing an essay you share it with your partner who then explains what you said in writing. You can ask questions but you cannot criticize. You can offer suggestions and then you rewrite,” states the student. Model classroom teachers liked the new approach to teaching literacy, but still felt there was much more to be learned. Teachers who implemented the Writer’s Workshop and became model classroom teachers were relatively pleased with the new strategies. Several reported that the America’s Choice approach provided “structure, consistency, uniformity, and direction.”  14 America’s Choice, Comprehensive School Reform Design: First-Year Implementation Evaluation Summary Model classroom teacher after model classroom teacher reported they were implementing standards, conferencing with students, modeling reading and writing, implementing mini- lessons, permitting students to have choice in their writing or reading, implementing the rituals and routines they had designed with their students, and implementing the 25 Book Campaign. All had reorganized their rooms, acquired more resources, and shared their new-found learning informally with other teachers. Most teachers reported the success students were experiencing as the ultimate value of the design. As teachers implemented the literacy strategies in the classroom, they often found some strategies worked better than others. Several teachers identified specific challenges: • Core assignments were seen as valuable, but many teachers were concerned that they lacked sufficient structure and could not easily be turned into lessons. • Students were conferencing with each other, but teachers were not confident that they were providing valuable help to their peers or staying on task. • Rituals and routines. The design suggests that children should sit in different places, but teachers reported it made classroom management a problem when 30 or more students were present. • 25 Book Campaign. Some children were participating in the 25 Book Campaign and some were not, although several incentives had been provided. Some teachers attributed the lack of interest to lack of parental support. • Standards were being used and had been rewritten them in children’s language, but several teachers expressed difficulty in adjusting fourth grade standards to other grade levels. • Writers Workshop. Finding the time for embedding all the writing opportunities in an already crowded curriculum posed challenges for many teachers. Some found it difficult to do without specific guidelines on how to actually teach certain writing strategies. • While some teachers were using the mini-lessons for reviewing specific concepts (such as a story has a beginning, middle, and end), others found it difficult to implement them every day. Teachers often reported that the 10-15 minute lessons always seemed to take much longer, and a mini-lesson became almost the length of a regular lesson. There was wide variation in the time it took to establish model classrooms. Survey results showed that 156 teachers across the 41 schools became model classroom teachers in the 1998-99 school year. The time it took to establish a model classroom ranged from two to more than ten weeks. About half the model classrooms were established in a month or less.  America’s Choice, Comprehensive School Reform Design: First-Year Implementation Evaluation Summary 15 HIGH PERFORMANCE MANAGEMENT Principals viewed Americas Choice as an asset to their schools. Most principals were supportive of America’s Choice, believing that it was a valuable asset to their schools and that it would help children meet standards. All the principals we visited whose schools had implemented the Book of the Month were extremely satisfied with that approach. They discussed how everyone in the building was reading the book and discussing the story with students. One principal reported that the Book of the Month had motivated her to read more. Another principal reported that she thought parents were supportive because now they asked more questions about the stories and because no one had complained. Still another principal discussed how valuable the activity was with students having special learning needs because they were reading the same story that all the other children in the school were reading. By and large, the Book of the Month was something principals felt they could really champion as a literacy activity and use to unite the entire school community. Teachers reported that principals were exhibiting instructional leadership, but infrequently visited their classrooms or discussed student performance. On the survey, over three-quarters of the teachers somewhat or strongly agreed that their principals understood how children learn, had high standards for student achievement, supported teachers in enforcing rules for student behavior, communicated clear standards, provided professional development, encouraged teachers to share their professional development learning, visited classrooms, and looked at student work. Yet, fewer than three-quarters of the teachers reported that their principals carefully monitored student academic progress. About two-thirds of the teachers reported that their principal visited their classroom only once or twice a year. Leadership Management Teams were being established and making changes in their school structures, which teachers reported as beneficial. All the schools we visited had established Leadership Management Teams, although frequently they were not yet functioning according to the America’s Choice design. We heard that the Leadership Management Teams were active in moving the school organization closer to the America’s Choice design. All Leadership Management Teams had been involved with the 25 Book Campaign or the Book of the Month and several had worked closely with their principals in redesigning the classroom schedules so an extended block of time for literacy could be implemented next year. Some Leadership Management Teams had developed or were currently developing strategic plans, although some schools considered their strategic plans to be synonymous with the school improvement plans. Some Leadership Management Teams were meeting monthly, others weekly, and some even more frequently because they were planning many changes that had to be completed and approved by staff by the end of the school year. On the survey, over two-thirds of the teachers reported that the Leadership Management Team had already contributed to the school in the first year and over 80 percent somewhat or strongly agreed that the Leadership Management Teams had the potential to contribute to the school in the future. Over two-thirds of the teachers reported that their Leadership Management Teams had effectively delegated responsibilities and that their meetings were well organized.  16 America’s Choice, Comprehensive School Reform Design: First-Year Implementation Evaluation Summary Teachers reported having sufficient influence to obtain the resources they needed in their classrooms. About two-thirds of teachers reported that they felt comfortable asking for needed classroom resources and determining the instructional materials used in the classroom. Principals reported that teachers had considerable influence over school policy. Eighty-seven percent of the principals reported that teachers had some to a great deal of influence over how student progress was measured and over half of the principals (57 percent) reported that teachers had a great deal of influence over setting standards for student behavior. Parental involvement was a valued, but underutilized, resource. Over two-thirds of the teachers somewhat or strongly agreed that parents and guardians were warmly welcomed in the school, that regular communication was encouraged, and that the school worked hard to build trust in relationships. However, slightly more than 40 percent reported that teachers think of parents as partners in educating children. And only 20 percent or fewer reported that parents understood the standards for what their children were learning, were involved in planning or carrying out before or after school programs, or had been informed about America’s Choice. The America’s Choice design requires each school to have a staff position responsible for involving parents, known as the community outreach coordinator. Many schools did not implement this position in the first year of reform. Most of the schools that we visited had tried to share some information about the design with parents through the PTA, school meetings, or school newsletters. IV. THE DISTRICT ROLE In general, school staffs selected their own whole school designs, but districts influenced these choices in many ways. Districts helped schools obtain funding or provided funding; they guided schools to models that fit the district’s philosophy, context, and need. Districts supported schools during the first year of implementing America’s Choice by paying for materials and training, instituting inter-district networks, changing school leadership, and using district staff to assist with implementation in a school. The districts played an influential role in the adoption and continuation of America’s Choice but, although there are plans to do so, NCEE had not yet provided formal training and support to school district personnel. In the five districts CPRE evaluators visited in the spring of 1999, we found some similarities and some differences in the roles district leadership played in determining which schools adopted whole school designs and which selected Americas Choice. We also found that districts were playing an important role in the implementation of America’s Choice. The districts encouraged the adoption of whole school reforms because of pressures to improve student achievement. In the districts we visited, schools had been encouraged to adopt whole school reforms either because their test scores were low relative to other schools in the district or because there was a district-wide effort to help children reach challenging new standards.  America’s Choice, Comprehensive School Reform Design: First-Year Implementation Evaluation Summary 17 District staff typically took the lead in identifying and securing resources to support schools in their adoption of America’s Choice. Fiscal support for whole school reform came from both federal and local sources. Districts often used local dollars to support whole school reform, and the schools supplemented this with Title I funds. Several schools applied for federal dollars through the Comprehensive School Reform Development (CSRD) program. As prescribed by federal statute, the faculty in each school had to vote for a reform model and for the CSRD application. In practice, however, it was staff at the district level, not staff in the individual schools, who identified the resources and made sure that interested or targeted schools received CSRD or some equivalent funding. In several schools we visited, the staff reported they either had no input or that they had been pressured to adopt America’s Choice. Both school and district administrators reported there were many factors that attracted them to Americas Choice, but district influence was the most frequently mentioned. Reasons given for choosing America’s Choice included the following: • The district staff liked the structural and organizational changes Americas Choice called forcreating a literacy team, increasing time for English and mathematics instruction, expanding academic safety nets for children in the regular school year as well as the summer. They believed that these changes would be beneficial for low-performing schools. Thus, the schools selected the model because it was aligned with their districts philosophy about reform. • The district was already moving toward the balanced literacy approach; the model was aligned with their philosophy for teaching literacy. • The superintendent wanted principals to select a model that could be used from kindergarten through grade 12 in all the schools in the district. • The staff in the schools were told by district leadership that America’s Choice would help keep them off the state’s takeover list because it was standards-based. • The schools were already involved with NCEE as part of the former National Alliance to Restructure Schools, and they wanted to build upon this partnership by adopting Americas Choice. Districts had to provide additional resources to schools so they could implement many of the Americas Choice design requirements in 1998-99. Districts served many roles in supporting the implementation of Americas Choice. They provided additional financial support to purchase reading books, furniture, tests, and other resources required by the Americas Choice design. They paid for substitutes to cover classes while staff received training or paid stipends to teachers who participated in after-school training. Districts paid travel costs in 1998- 99 for training school design coaches and literacy coordinators and for school personnel to attend the NCEE national conference. District and school staff reported that many of these extra costs associated with the America’s Choice design (such as materials, training, and testing) were identified after training. Because districts had agreed to support the America’s Choice design, the districts agreed to provide the additional resources to meet the design requirements.  18 America’s Choice, Comprehensive School Reform Design: First-Year Implementation Evaluation Summary Districts supported implementation of the design by assigning district personnel to be liaisons between NCEE and schools, using staff developers for training, hosting principals’ or literacy coordinators’ networks, and revising and expanding summer school programs. In each district we visited there was an individual assigned to be a liaison to NCEE and the America’s Choice schools. These individuals had direct contact with the superintendent and made sure that district leadership was regularly briefed on the progress schools were making and on the requisites for implementing the America’s Choice design. The liaisons mobilized resources within the district, monitored school implementation of America’s Choice, and leveraged district support as the need arose. Districts used their own staff developers and curriculum coordinators to support America’s Choice activities within schools. District personnel were sent to America’s Choice schools to assist with the 25 Book Campaign, training on standards, and conducting focus walks through the schools. Some districts transferred principals to establish better environments for whole school reform in general, and America’s Choice in particular. District staff also provided follow- up support to literacy coordinators and design coaches after NCEE training. One district hosted regular meetings with literacy coordinators so they could share strategies, de-brief after training, and plan implementation of model classrooms. Another district hosted design team meetings so principals could hear from design coaches and literacy coordinators about NCEE training, discuss implementation, and plan strategies. In addition, districts supported America’s Choice schools by expanding their safety net programs for students, particularly by offering summer programs. Districts were unable to ensure that each school had sufficient staff to free design coaches and literacy coordinators from other job responsibilities. The Americas Choice design calls for a full-time design coach and literacy coordinator in each school. We found some schools that had full-time design coaches or full-time literacy coordinators, but in many cases, one or both were part-time positions held by school staff who had other important responsibilities. Unions were supporting whole school reform but also were concerned about safeguarding their constituents’ interests. Union representatives were neither in favor of nor opposed to the America’s Choice design; all said they supported reforms that helped students. Part of the problem of providing full-time design coaches and literacy coordinators arose from contractual agreements and concerned such issues as the additional workload from the paperwork and non- teaching responsibilities of design coaches and literacy coordinators, the time for professional development beyond the time agreed upon in the contract, and the literacy coordinator’s supervisory responsibilities over teaching staff. Most state and district policies appeared to be adequately aligned with the Americas Choice design, but teachers perceived many districtwide practices as incongruent with the design. School staff in one district reported that new textbooks in all core subject areas took time to master, which detracted from their efforts to focus on America’s Choice. Most district staff reported that the New Standards Reference Examinations would help students prepare for the high-stakes state exams, but school personnel saw the Reference Exams as one more test that took time away from instruction or student preparation for the state test.  America’s Choice, Comprehensive School Reform Design: First-Year Implementation Evaluation Summary 19 District personnel reported that all other initiatives were aligned with the philosophy and strategies suggested by the America’s Choice design, but school personnel did not always agree. School staff in one district said that the new reading series required in all schools was not aligned with the balanced literacy approach. In another district we heard that the new mathematics textbook and take-home reading and mathematics exercises were not aligned with the America’s Choice approach. In still another district, we heard that paraprofessionals were key staff, but the America’s Choice design provides no role for paraprofessionals and no training to ensure they understand the new instructional strategies. Districts were already looking for evidence of effectiveness in schools that adopted America’s Choice. The schools made a three-year commitment to implement America’s Choice, but some school staff members were concerned that the reform strategy would not be around long enough to make a difference. They hoped that their districts would provide sufficient time and resources to allow it to work. Interviews with district staff suggested that evidence of effectiveness would be a critical variable in determining whether whole school reform in general and America’s Choice in particular would be continued. District representatives said that student achievement and other outcomes such as improved student attendance, fewer referrals to special education, reduced suspensions, and more parental involvement would be critical in determining the design’s effectiveness. About half the districts were conducting external evaluations to ascertain the impact of whole school reform. V. SUMMARY In section two of this report, we established a set of reasonable expectations for first year progress of America’s Choice schools. These included a clear understanding of the reform, the implementation of the structures of America’s Choice, and behaviors that realize the potential of the reform. Overall, schools participating in the America’s Choice comprehensive school design met these expectations and made tremendous progress in their first year of implementation. An overwhelming majority of school faculties were supportive of the concept of standards-based reform in general, and the America’s Choice design in particular. School staff exhibited a general understanding of the model, although there was variation in their understanding of some of the specific components of the design, like the role of the New Standards Reference Exam data. School Leadership Team members also understood the tasks and goals of the first year implementation schedules. There also was substantial evidence that teachers and administrators were beginning to make changes consistent with the reform. School staff had begun to implement many of the design requirements. For example, design coaches were leading training for classroom teachers, and literacy coordinators had begun setting up model classrooms. Although training time often was limited, design coaches worked with staff so they understood the America’s Choice design, became familiar with New Standards Performance Standards, and administered the New Standards Reference Exams. The fact that model classroom teachers consistently exhibited  20 America’s Choice, Comprehensive School Reform Design: First-Year Implementation Evaluation Summary beliefs and practices that were more consistent with the America’s Choice philosophy than other school faculty suggests that the design was having an influence on instruction. The reforms also were beginning to influence the organizational structure of America’s Choice schools. Schools instituted Leadership Management Teams that were working to find ways to extend learning time and to establish safety net programs. They also had implemented schoolwide features of the design, such as the 25 Book Campaign and Book of the Month. Barriers to implementation, including multiple job responsibilities, resource shortages, teacher resistance, and training time constraints resulted in unevenness in their progress. While school staffs were beginning to enact the structural components of America’s Choice, our results indicate that it will take more time for the belief systems underlying the reform to become deeply rooted. School staffs were divided in their attitudes toward the beliefs about teaching and learning, which underlie the America’s Choice reform. Indications of this division were found in teachers’ responses to questions about their expectations for children, how they assess their students, and how principals view their instructional roles. Finally, even though America’s Choice is a school-level reform, districts also were playing an important role in its implementation. Districts were supporting implementation in several ways including providing resources for schools, supporting school-based America’s Choice training, hosting principals’ or literacy coordinators’ networks, and revising and expanding summer school programs. As described in this report, the first cohort of America’s Choice schools made tremendous progress in their first year of implementation of the model. Facing challenges both within and outside their schools, they succeeded, albeit at different paces, in laying the foundation for standards-based reform. Their ongoing efforts will determine the long-term influence of America’s Choice on their students.  America’s Choice, Comprehensive School Reform Design: First-Year Implementation Evaluation Summary 21 APPENDIX SCHOOLS PARTICIPATING IN AMERICA’S CHOICE IN 1998-99 School Public School 169 Gage Park High School Harlan Community Academy McNair Academic Center Nicholson Specialty School Senn Metro High School Academy Public School 27 Public School 32 John J Pershing Intermed. School #220 Public School 179 Public School 183 Public School 42 Honaunau School Ewa Beach Elementary Highland Intermediate School Honowai Elementary School Nanakuli Elementary Waipahu Elementary School ABGS Middle School Early Childhood Center Franklin School Fulton School Hempstead High School Jackson Annex Jackson Main Ludlum School Marshall Pre K Prospect School Rhodes School Adiai Stevenson School #29 Chester E. Dewey School #14 Kodak Park Elementary #41 Lincoln Park Elementary #44 Martin Luther King Elementary #9 Roberto Clemente School #8 Harry Daniels School Ulysses Byas Elementary School Salmon River Central High School Salmon River Elementary St. Regis Mohawk Elementary Vashon High School School 16 District, State Brooklyn, NY Chicago, IL Chicago, IL Chicago, IL Chicago, IL Chicago, IL District 15, District 15, District 20, District 20, District 27, District 27, Hawaii, HI Hawaii, HI Hawaii, HI Hawaii, HI Hawaii, HI Hawaii, HI Hempstead, NY Hempstead, NY Hempstead, NY Hempstead, NY Hempstead, NY Hempstead, NY Hempstead, NY Hempstead, NY Hempstead, NY Hempstead, NY Hempstead, NY Rochester, NY Rochester, NY Rochester, NY Rochester, NY Rochester, NY Rochester, NY Roosevelt, NY Roosevelt, NY Salmon River, NY Salmon River, NY Salmon River, NY St. Louis, MO Yonkers, NY New York, NY New York, NY New York, NY New York, NY New York, NY New York, NY  NONDISCRIMINATION STATEMENT The University of Pennsylvania values diversity and seeks talented students, faculty and staff from diverse backgrounds. The University of Pennsylvania does not discriminate on the basis of race, sex, sexual orientation, religion, color, national or ethnic origin, age, disability, or status as a Vietnam Era Veteran or disabled veteran in the administration of educational policies, programs or activities; admissions policies, scholarships or loan awards; athletic, or University administered programs or employment. Questions or complaints regarding this policy should be directed to Executive Director, Office of Affirmative Action, 1133 Blockley Hall, Philadelphia, PA 19104-6021 or (215) 898-6993 (Voice) or (215) 898-7803 (TDD).