 |
 |
 |
 |
 |
|
National Accessible Reading Assessment Projects December 10, 2004
OverviewThe first meeting of the General Advisory Committee (GAC) of the National Accessible Reading Assessment Projects (NARAP) was held December 10, 2004, in Alexandria, Virginia. NARAP's purpose is to support research to make large-scale assessments of reading proficiency more accessible for students with disabilities that affect reading. NARAP currently comprises two projects. One project, conducted by the Educational Testing Service (ETS), focuses on students with learning disabilities and will use a component approach to measuring reading proficiency. The second project, the Partnership for Accessible Reading Assessment (PARA), seeks to develop an approach that is based on input from knowledgeable constituencies who use reasoned judgment to talk to each other about the meaning of the reading construct and how access can be assured in the assessment in light of the construct being assessed; PARA is a partnership of the National Center on Educational Outcomes (NCEO); the National Center for Research on Evaluation, Standards, and Student Testing (CRESST); and Westat, Inc. Dr. Linda Cook of ETS and Dr. Louis Danielson and Dr. David Malouf of the Office of Special Education Programs, U.S. Department of Education, welcomed the meeting participants. Project team members discussed the collaborative effort and presented overviews of the two NARAP projects. The meeting participants also learned about the work of the Definition Panel and the organization and role of the GAC, reviewed the GAC charter, and provided feedback to the project team. WelcomeDr. Linda Cook, Project Co-Director, Designing Accessible Reading Assessments Project, Educational Testing Service Dr. Cook introduced Dr. Louis Danielson and Dr. David Malouf of the Office of Special Education Programs (OSEP), who presented opening remarks. Dr. Danielson's division within OSEP manages a number of discrete funding projects authorized by the Individuals with Disabilities Education Act. Dr. Malouf is involved with assessment and accountability programs, and with the development and use of technology to benefit students with disabilities. Dr. Louis Danielson, Director, Research to Practice Division, Office of Special Education Programs, U.S. Department of Education Dr. Danielson said that since the early 1990s much progress has been made in the area of assessment, but that local-level challenges remain in implementing the mandates of the Individuals with Disabilities Education Act (IDEA) and the No Child Left Behind Act (NCLB). NCLB has increased the attention given to students with disabilities, and to the differential exclusion rates in reading assessment that exist among the states. For example, one state does not permit reading accommodations for students with disabilities, whereas other states allow students to have the assessment tests read orally if needed. In states that do not permit oral reading of the tests, students might be scored as non-participating or receive a zero score. Other issues include whether Braille is an allowable accommodation, including for high school exit exams, for students who are blind. NCLB has also raised issues related to how to assess grade-level reading for students with significant mental retardation. PARA and ETS both have the capacity to address these important, compelling issues. Dr. David Malouf,Research Analyst, Office of Special Education Programs, U.S. Department of Education Dr. Malouf, NARAP's Project Officer, reviewed the U.S. Department of Education's priority for research on accessible reading assessments. The priority is for "projects to conduct systematic programs of research and development to make large-scale assessments of reading proficiency more accessible for students who have disabilities that affect reading." Projects are to "address the assessment of students with a full range of disabilities that affect reading, including particularly blindness and other visual impairments, deafness and other hearing impairments, learning disabilities, and mental retardation." Two projects were funded in the last competition, although others may be funded in the future. The competition set forth four goals: Goal 1: Formulate a definition of the construct of "reading proficiency" that provides a basis for research and development for accessible large-scale tests of reading proficiency that provide (a) a valid measure of proficiency against academic standards, and (b) individual interpretive, descriptive, and diagnostic reports for the full range of students with disabilities that affect reading. Goal 2: Conduct a program of research on the assessment of reading proficiency to determine the effects of various factors of test development, design, and administration on accessibility, validity, and comparability for students with disabilities that affect reading. Goal 3: Develop research-based principles and guidelines for making large-scale assessments of reading proficiency more accessible for students who have disabilities that affect reading. Goal 4: Develop and field-test instruments and/or methods for assessing reading proficiency that are suitable for large-scale administration for school accountability purposes; that are accessible to students who have disabilities that affect reading; that maintain validity and comparability of scores; and that can provide (a) a valid measure of proficiency against academic standards, and (b) individual interpretive, descriptive, and diagnostic reports for the full range of students with disabilities that affect reading. Goal 1 is collaborative, Goal 2 is independent research, Goal 3 is a collaborative process to develop principles, and Goal 4 is collaborative. It is hoped that by the end of the five-year funding period, substantial progress will have been made toward making reading assessments accessible. The NARAP GAC will serve as an advisory committee to the two funded projects and any future projects. IntroductionsThe GAC members and other meeting participants introduced themselves, and the GAC members stated their interests, which include to:
Overview of the ProjectsDr. Cara Cahalan-Laitusis, Project Director, Designing Accessible Reading Assessments Project, Educational Testing Service Dr. Martha Thurlow,Co-Director, Partnership for Accessible Reading Assessment Dr. Cahalan-Laitusis and Dr. Thurlow presented an overview of NARAP and reiterated the NARAP projects' four goals (see above). They explained that the NARAP project teams seek to work together in several ways, including by adopting a single name for the collaborative entity (NARAP); hosting a shared Web site (www.narap.info); having shared committees, including the GAC and the Definition Panel; and collaborating on activities. An executive committee will ensure that collaboration continues to occur throughout the course of the project. The GAC's purposes are to review and advise on the general project plans and to provide linkages with significant stakeholder groups. The GAC will include representatives of the National Center for Education Statistics; the National Assessment Governing Board; organizations representing test developers, reading educators, and researchers; disability groups; and other appropriate projects and organizations. The GAC will meet at least annually, but will communicate more frequently. Dr. Cahalan-Laitusis and Dr. Thurlow asked the Committee for advice about how to maximize collaboration as the projects move forward. Suggestions included to send monthly e-mails to inform GAC members about activities, send brief e-mail reports that can be reconfigured for members' specific audiences and posted on members' Web sites, and cross-link members' Web sites. A suggestion was also made to use the GAC as a sounding board for philosophical questions as the projects develop. Dr. Cahalan-Laitusis described the ETS project, which focuses on learning disabilities and will take a "component approach" to assessing reading. This approach will isolate into subtests the individual components of reading (e.g., comprehension or word recognition) and will allow for accommodations (e.g., audio presentation) to be specific to subtests without impacting the validity of the entire test. In Year 1 (2005), the research agenda includes:
Operational test data will look at how applicable current assessments are with different student groups. Year 2 of the research agenda (2006) will focus on new item development. Activities will include:
Dr. Thurlow explained the Partnership for Accessible Reading Assessment (PARA) is a joint effort involving NCEO, CRESST (the research arm), and Westat, Inc. (the field-test arm). The Institute for Applied Psychometrics in Minnesota will assist with the development aspects of the project. The project will focus on all disabilities that affect reading, particularly as the project identifies the issues that exist and the possible solutions. The field-test phase (Years 4 and 5) will include both classroom-based testing (which will include all students in those classrooms) and over-sampling of students with learning disabilities, speech-language impairments, mental retardation, and deaf/hearing impairment. The PARA team intends to build on and recognize the expertise of others. It recognizes the importance of addressing the needs of students with diverse disabilities. The team is also committed to engaging external experts and obtaining input from scholars, key national stakeholders, expert advisers, and end users of the assessments. The project will be flexible and will evolve based on the GAC's and Definition Panel's work. PARA's research agenda, which will begin in Year 2, will address the following general issues and questions:
Plans for Year 1 include the development of a definition of reading proficiency and to hold meetings with advisory groups, including the GAC and Technical Advisory Committees. The definition will be consistent with NCLB requirements and will provide a valid measure of reading proficiency against academic standards. Projects will develop documentation, analyze the definition in relation to current state and national standards, analyze the definition in relation to different disability populations, obtain input from external groups (e.g., through focus groups), and refine the definition as needed on the basis of input from external sources. Work of the Definition PanelDr. Deborah Dillon, Co-Director, Partnership for Accessible Reading Assessment Dr. John Sabatini, Project Leader for Goal 1, Designing Accessible Reading Assessments Project, Educational Testing Service Dr. Dillon and Dr. Sabatini, co-leaders of the Definition Panel Work Group, discussed the role of the Work Group. The Work Group also includes Ann Clapper of NCEO, Laurie Cutting of ETS and the Kennedy Krieger Institute, and Lee Galda and David O'Brien of the University of Minnesota . The Work Group's roles are to establish the 14-member Definition Panel's membership, develop meeting agendas, outline tasks, and identify and design protocols and products that will emerge from the work of the Definition Panel. Tasks of the Definition Panel are to:
Definition Panel meetings will include a teleconference to get input on directions and to begin to identify research to review, document review meetings, and a face-to-face panel meeting(s) to gain panel members' consensus on the working definitions of key terms and the definition of "reading proficiency." Dr. Sabatini noted that the Definition Panel will focus on defining English-language reading proficiency for any student, and will look at how the definition can be used with students with disabilities. Dr. Malouf commented that states and national testing programs are struggling with these issues, and that the definition should be sound and compelling. Dr. Dillon said that the definition of reading proficiency will be strengthened as more input is gathered. Dr. Sabatini added that the Work Group wants to ensure that the definition extends the potential for research and will be continually improved and defined. It is hoped that the GAC members will provide input and support in developing the definition, and that they will eventually endorse/approve the definition. A participant noted that the endorsement process for large membership organizations can be lengthy and that large organizations need time and consistent information to move to the point of endorsement. A suggestion was made to distinguish consensus and endorsement by the GAC from endorsement by the GAC members' organizations. DiscussionThe meeting participants discussed the project collaboration and work of the GAC. The Committee members noted that:
Development of the NARAP GAC CharterDr. Ann Clapper, Development Coordinator, Partnership for Accessible Reading Assessment Dr. Clapper explained that the project team has drafted a charter that outlines the GAC's purposes, roles and responsibilities, and membership, as well as NARAP's function and role. Feedback about the draft charter was requested in the following areas. GAC MembershipThe meeting participants suggested that a limit be placed on the number of organizations represented on the GAC. They proposed that, in addition to the members listed in the draft charter, the GAC members include representatives of:
Dr. Clapper suggested that a profile be created for each GAC member. Members will be asked to complete profile questionnaires and will have the opportunity to review the profiles before dissemination. The questionnaire could ask members about electronic links to include on the NARAP Web site. A suggestion was made to post brief descriptions of the member organizations on the NARAP Web site. The profile questionnaire will also ask members which national conferences they attend so that GAC subcommittee meetings might be held at the conferences. The members also suggested that information be shared through an electronic bulletin board, e-mails with documents attached, a listserv, or a Web log. There were no comments about whether the charter should specify at what point new members can be added to the GAC. PurposesThe members had no comments about the GAC purposes listed in the draft charter. A participant suggested that GAC members disseminate NARAP products and information, and gather feedback from their own audiences/constituents (this is included in roles and responsibilities). Roles and Responsibilities of Committee MembersThe participants noted that:
Function and Role of NARAPDr. Clapper asked how the project staff can facilitate NARAP's work. The members suggested that the staff could:
TermDr. Clapper explained that the GAC will also advise on any future NARAP projects. GAC members are expected to serve for the five-year project time span. Members who are unable to serve for the full five years will be asked to help identify replacements to represent their organizations. The Executive Committee will discuss whether a substitute should be sent if a member is unable to attend a GAC meeting. A suggestion was made to identify a consistent, informed alternate from each organization. The alternate's qualifications should be submitted for review by the NARAP team, and information about the alternate will be included on the profile questionnaire. Other Issues/SuggestionsThe meeting participants also noted that:
ClosingDr. Cook said that the GAC members will receive frequent e-mail communication from the project team. The project team would like to take advantage of the GAC members' experience and the power of their organizations, and to obtain input from members and their organizations. Members may send comments, concerns, and questions to Mari Quenemoen (quen0006@umn.edu), who will respond to or forward requests to other appropriate NARAP staff. Dr. Cook thanked the participants for attending the meeting and providing their input and insight. Appendix A
|
Contact narap@umn.edu
with questions or assistance with this site.