|
Definition Panel Meeting
January 29, 2005
Minneapolis/St. Paul Airport Hilton
Minneapolis, MN
Overview
The initial meeting of the Definition Panel of the National Accessible Reading Assessment Projects (NARAP) was held January 29, 2005 in Minneapolis , MN . (See Appendix A for a list of participants.) NARAP is a collaboration of projects funded by the U.S. Department of Education, Office of Special Education Programs, whose overall purpose is to support research to make large-scale assessments of reading proficiency more accessible for students with disabilities that affect reading. One of the goals in the original priority for funding was to formulate a definition of the construct of “reading proficiency” as a basis for research and development for accessible large-scale tests of reading proficiency that provide (a) a valid measure of proficiency against academic standards, and (b) individual interpretive, descriptive, and diagnostic reports for the full range of students with disabilities that affect reading. A 15-member Definition Panel was established to guide the development of the definition of reading proficiency and the members were convened in order to begin work on this charge.
The meeting started with opening remarks from representatives from the Office of Special Education Programs followed by a presentation from NARAP project leaders on the Definition Panel’s role, the timeline for their work, and other issues specific to the context of the Panel’s work. NARAP’s Definition Panel leaders reviewed the agenda for the meeting and the rest of the day was spent with Panel members working in small and large group discussions designed to accomplish the meeting’s goals.
Top of page.
Introductions and Overview
Dr. Deborah Dillon, Co-Director, Partnership for Accessible Reading Assessment
Dr. Deborah Dillon of the Partnership for Accessible Reading Assessment (PARA) project welcomed the participants and introduced Dr. Louis Danielson and Dr. David Malouf of the Office of Special Education Programs (OSEP), who presented opening remarks.
Dr. Louis Danielson, Director, Research to Practice Division, Office of Special Education Programs, U.S. Department of Education
Dr. Danielson greeted the meeting participants on behalf of the Office of Special Education Programs at the U.S. Department of Education and relayed his office’s support for the work of NARAP. He reminded the group that many of the challenges associated with assessing students with disabilities stem from the lack of accessibility of current assessments and stressed the need for improved assessments that adequately address the constructs being measured while minimizing the need for accommodations. He noted the importance of starting from the beginning and thinking again about, and defining clearly, the construct to be measured. Dr. Danielson thanked the Panel members for their willingness to assist NARAP at this stage in the process and expressed his appreciation for their time and commitment to the project.
Dr. David Malouf,Research Analyst, Office of Special Education Programs, U.S. Department of Education
Dr. Malouf, NARAP’s Project Officer, reviewed the U.S. Department of Education’s priority for research on accessible reading assessments. The priority is for “projects to conduct systematic programs of research and development to make large-scale assessments of reading proficiency more accessible for students who have disabilities that affect reading.” Projects are to “address the assessment of students with a full range of disabilities that affect reading, including particularly blindness and other visual impairments, deafness and other hearing impairments, learning disabilities, and mental retardation.”
Dr. Malouf explained that two projects were funded in the last competition. One of the funded projects is the Designing Accessible Reading Assessments (DARA) project, housed at Educational Testing Service (ETS) and the other is the Partnership for Accessible Reading Assessment (PARA), a consortium of the National Center on Educational Outcomes (NCEO), the Center for Research on Evaluation, Standards, and Student Testing (CRESST), and Westat. Dr. Malouf also noted that other projects may be funded in the future.
Dr. Malouf highlighted the four goals set forth in the competition:
Goal 1: Formulate a definition of the construct of “reading proficiency” as a basis for research and development for accessible large-scale tests of reading proficiency that provide (a) a valid measure of proficiency against academic standards, and (b) individual interpretive, descriptive, and diagnostic reports for the full range of students with disabilities that affect reading.
Goal 2: Conduct a program of research on the assessment of reading proficiency to determine the effects of various factors of test development, design, and administration on accessibility, validity, and comparability for students with disabilities that affect reading.
Goal 3: Develop research-based principles and guidelines for making large-scale assessments of reading proficiency more accessible for students who have disabilities that affect reading.
Goal 4: Develop and field-test instruments and/or methods for assessing reading proficiency that are suitable for large-scale administration for school accountability purposes; that are accessible to students who have disabilities that affect reading; that maintain validity and comparability of scores; and that can provide (a) a valid measure of proficiency against academic standards, and (b) individual interpretive, descriptive, and diagnostic reports for the full range of students with disabilities that affect reading.
Dr. Malouf reviewed how the two currently funded projects under NARAP (DARA and PARA ) will carry out activities for each goal. Goal 1 is collaborative, Goal 2 is independent research, with each project carrying out separate lines of research, Goal 3 is a collaborative process to develop principles, and Goal 4 is also a collaborative venture between both projects.
Dr. Linda Cook, Co-Director, Designing Accessible Reading Assessments
Dr. Cook told the Definition Panel that the definition of reading proficiency that the members generated would provide the basis for the research and development activities for both the PARA and DARA projects. She highlighted the range of perspectives represented on the Definition Panel, including members with expertise in reading, research, blind/visual impairment, deaf/hearing impairment, learning disabilities, mental retardation, assessment, English language learners, English language learners with disabilities, cognitive processing, policy, and technology. Dr. Cook also reviewed the specific tasks of the Definition Panel:
- Review and analyze the collection of key documents
- Develop a definition of reading proficiency
- Provide guidance to the work of the focus group process and revise the definition of reading proficiency if necessary
- Communicate with others about the work of the Definition Panel
Dr. Martha Thurlow, Co- Director, Partnership for Accessible Reading Assessment
Dr. Thurlow explained that NARAP has two other joint advisory groups currently operating. A General Advisory Committee, made up of representatives from key organizations including test developers, reading educators, researchers, parents, and disability groups, and a State Advisory Board consisting of leaders from eight state departments of education. Future NARAP advisory groups will include two Technical Advisory Committees (one for each project) and a joint Principles Committee.
Dr. Thurlow reviewed the timeline for the Definition Panel’s work:
- January-Provide an initial draft of a definition of reading proficiency
- February-Disseminate the definition to the General Advisory Committee and the State Advisory Board
- March-June-Conduct focus groups
- July-Compile the results
- August-Finalize the definition
Dr. Thurlow explained that there were members of the Definition Panel who were participating in the meeting through Web conferencing, and that PARA and DARA staff members present, both on and off site, would be listening to the meeting but not participating.
Top of page.
Review of Agenda and Task
Dr. Deborah Dillon, Co-Director, Partnership for Accessible Reading Assessment
Dr. John Sabatini, Project Leader for Goal 1, Designing Accessible Reading Assessments
Dr. Dillon reviewed the agenda items for the day:
- Develop a robust definition of reading
- Discuss associated reading proficiencies implied by the definition for learners in grades 4-8
- Consider the definition of reading with respect to the full range of students
- Identify key concepts and terms
- Review and provide initial feedback on questions for focus groups
Dr. Sabatini reminded the group that its task was to formulate a definition of reading proficiency that can be used as a basis for research and development for accessible large-scale tests of reading proficiency. The definition must also be consistent with the requirements of the No Child Left Behind (NCLB) Act of 2001 and provide a valid measure of proficiency against academic standards and individual interpretive, descriptive, and diagnostic reports for a full range of students with disabilities that affect reading.
Dr. Dillon explained that the Panel’s first task was to develop a robust definition of reading based on the elements of the definitions provided in the materials from the following reports:
National Reading Panel Report—2000
“Reading comprehension is the construction of the meaning of a written text through a reciprocal interchange of ideas between the reader and the message in a particular text” (Harris & Hodges, 1995, definition #2, p. 39) (p. 4-5).
“In the cognitive research of the reading process, reading is purposeful and active (Pressley & Afflerbach, 1995). According to this view, a reader reads a text to understand what is read, to construct memory representations of what is understood, and to put this understanding to use. A reader can read a text to learn, to find out information, or to be entertained. The various purposes of understanding require that the reader use knowledge of the world, including language and print. This knowledge enables the reader to make meaning of the text, to form memory representations of those meanings, and to use them to communicate with others information about what was read” (p. 4-39).
Programme for International Student Assessment (PISA)—2000
“Reading literacy is understanding, using and reflecting on written texts, in order to achieve one’s goals, to develop one’s knowledge and potential and to participate in society.” (From The construct of reading literacy for PISA, Chapter 2, “Reading for Change,” p. 25).
Progress in International Reading Literacy Study Assessment (PIRLS)—2001
Reading is defined as “the ability to understand and use those written language forms required by society and/or valued by the individual. Young readers can construct meaning from a variety of texts. They read to learn, to participate in communities of readers, and for enjoyment.” (p. 33)
National Assessment of Educational Progress (NAEP)—2009
“Reading is an active and complex process that involves understanding written text; developing and interpreting meaning; and using meaning as appropriate to type of text, purpose, and situation.” (Summary of Reading Framework, NRC Presentation Handout, Dec. 2004, p. ii)
RAND Report—2002
“We define reading comprehension as the process of simultaneously extracting and constructing meaning through an interaction and involvement with written language. We use the words extracting and constructing to emphasize both the importance and the insufficiency of the text as a determinant of reading comprehension. Comprehension entails three elements:
- The reader who is doing the comprehending
- The text that is to be comprehended
- The activity in which comprehension is a part” (p. 11)
Top of page.
Discussions
Small Group Discussion-Session 1
Dr. Dillon divided the panel into three groups and asked each group to discuss key aspects of the definitions from the five reports and then draft a definition of reading based on the elements from those definitions. She also asked the members of each group to identify significant issues or concerns that they might have relative to the definition, such as what must be included or excluded in the definition and the implications of including or excluding particular aspects of the definition.
Members of each group and group leaders (*) were:
Group 1: Kamil*, McCardle (on phone), Rose, Simmons (on phone), Strickland, Wagner (on phone), O’Brien (note taker)
Group 2: Alvermann*, Dillon, Edyburn, Holbrook, Williams, Clapper (note taker)
Group 3: Afflerbach*, Browder, Deshler, Gersten, Wiemer, Byrn (note taker)
Large Group Discussion-Session 1
The panel members reconvened as a large group and Dr. Dillon asked that each group leader share his or her group’s draft definition.
Kamil reported that Group 1 decided to adopt/adapt the NAEP definition:
- Accessing, decoding and understanding written, electronic, and multimedia text
- Developing and interpreting meaning
- Using the meaning appropriate to the text, purpose, and situation
Alvermann (Group 2) reported that her group had arrived at the following definition:
Reading is fluent decoding, using vocabulary and comprehending. Reading involves multiple forms of text, the reading and the activity all interacting in the presence of mediating influences embedded in a dynamic sociocultural context. Group 2 also recommended that the graphic from the RAND Report be included as part of the definition, with a modification of one of the circles to represent mediating influences.
Afflerbach’s group (Group 3) defined reading as:
The interaction and strategic effort of the reader to construct and understand meaning(s) from text to achieve one’s purpose of personal fulfillment, academic pursuit, and work in order to participate in society.
Following the reports from the small groups, the entire Definition Panel discussed the three definitions. Dr. Dillion suggested that the group think about commonalities and differences across the definitions and try to coalesce the definitions into one that would work for the entire Panel.
The Panel did not reach consensus on a definition of reading by the end of the discussion period but its members noted that in general the definition should be streamlined, accompanied by substantial explanation, and that it must be able to be operationalized and assessed.
Small Group Discussion-Session 2
Following a break, the Definition Panel broke into small groups again and discussed reading proficiencies implied by the definition for learners in grades 4-8. Dr. Dillon asked the group to consider the following questions during their discussions:
- Given the definition of reading that we generated, discuss and draft descriptive statements or claims about the reading proficiencies of “typical” learners in grades
4-8. What would these students be able to do as readers?
- Identify significant issues/challenges that may require a reworking of the initial definition of reading.
- Identify significant issues and challenges for the PARA and DARA development teams—what innovations must occur with R&D?
Large Group Discussion-Session 2
Following lunch, Dr. Dillon suggested that the Panel share information generated in the last small group session and once again reflect on the definitions generated earlier by the small groups to see if one stood out, if the three could be combined in some way, or if one could be tweaked in order to come up with a draft definition. After a lengthy discussion, the following language was generated:
Reading is decoding and understanding written text. Understanding is determined by the purposes for reading, the context, the nature of the text, and the reader’s strategies and knowledge.
Panel members suggested that a sentence on decoding be added and it was determined that Wagner, McCardle, Simmons, and Holbrook would craft a decoding sentence. Wagner suggested that the decoding group send its ideas to him, and he would collect them for the group and share them with NARAP staff.
Dr. Dillon recommended that, in the interest of time, the discussion be brought to a close and that further work on the definition, and other definition related agenda items not addressed during the meeting, be carried out at a later date via email or conference call.
Top of page.
Focus Groups
Dr. Linda Cook, Co-Director, Designing Accessible Reading Assessments
Dr. Linda Cook explained that once the draft version of the definition was completed, a series of focus groups would be held to gather feedback on the definition of reading. She shared the current list of sites for focus groups and dates on which the focus groups will be held:
Learning Disabilities Association 3/3
Council for Exceptional Children 4/6
American Educational Research Association/
National Council on Measurement in Education 4/13
International Reading Association 5/1
Council of Chief State School Officers 6/19
Society for the Scientific Study of Reading 6/24
Dr. Cook asked Panel members to suggest additional groups that might be considered for focus group sites. Other organizations the group recommended for consideration included the International Dyslexia Association, the National Education Association, TASH, the Council for Children with Behavioral Disorders (CCBD) along with other appropriate divisions of the Council for Exceptional Children, Special Interest Groups in the American Educational Research Association, a group that is focused on physical disabilities, the National Association of State Directors of Special Education, and the National School Boards Association.
Dr. Cook indicated that the current list of participants in the focus groups included researchers, practitioners, teachers, and parents. She asked the Panel for suggestions about other constituencies who might be included in the focus group process and the Panel recommended that school psychologists, diagnosticians, professionals with disabilities, state policymakers, federal policymakers, and congressional staffers also be considered.
Dr. Cook distributed a draft copy of the questions that will be used during the focus groups and described how the focus group process would be managed. A brief discussion was held and Panel members suggested that NARAP consider convening homogeneous focus groups instead of heterogeneous focus groups because homogeneity would increase the amount of information generated.
Dr. Dillon thanked the Panel members for their participation and adjourned the meeting of the Definition Panel.
Top of page.
Appendix A: List of Participants
Presenters/Facilitators
Linda Cook
Co-Project Director and Project Leader for Goal 3
ETS - MS 07R, R129
Center for Validity Research
Rosedale Road
Princeton , NJ 08541
Phone: 609-734-5878
Fax: 609-734-1755
lcook@ets.org
Deborah R. Dillon
Co-Principal Investigator and Co-Director
Department of Curriculum and Instruction
University of Minnesota
125 Peik Hall, 159 Pillsbury Dr. SE
Minneapolis , MN 55455
Phone: 612-625-1362
Fax: 612-624-8277
dillon@umn.edu
Lou Danielson
Director, Research to Practice Division, Office of Special Education Programs
400 Maryland Avenue, SW , Room 4160
Washington , DC 20202-2600
Phone: 202-245-7250
Fax: 202-245-7617
louis.danielson@ed.gov
Dave Malouf
Office of Special Education Programs
U.S. Dept. of Education
400 Maryland Avenue, SW
Room 4078 PCP
Washington , DC 20202-2600
Phone: 202-245-7427
Fax: 202-245-7617
david.malouf@ed.gov
John P. Sabatini
Project Leader for Goal 1
ETS - MS 10R, R142
Center for Teaching & Learning
Rosedale Road
Princeton, NJ 08541
Phone: 609-734-5842
Fax: 609-734-1755
jsabatini@ets.org
Martha L. Thurlow
Co-Principal Investigator and Co-Director
NCEO
University of Minnesota
350 Elliott Hall, 75 East River Road
Minneapolis, MN 55455
Phone: 612-624-4826
Fax: 612-624-0879
thurl001@umn.edu
Definition Panel
Peter Afflerbach
Professor
University of Maryland, Department of Curriculum and Instruction
2311 Benjamin Building
College Park, MD 20742-1115
Phone: 301-405-3159
Fax: 301-314-9055
pa15@umail.umd.edu
Donna E. Alvermann
Professor
University of Georgia
309 Aderhold Hall
Athens, GA 30602-7125
Phone: 706-542-2718
Fax: 706-542-3817
dalverma@uga.edu
Diane Browder
Professor
University of North Carolina-Charlotte
Special Education & Child Development
9201 University City Boulevard
Charlotte, NC 28223-0001
Phone: 704-687-4012
Fax: 704-687-2916
dbrowder@email.uncc.edu
Donald D. Deshler
Director
University of Kansas Center for Research on Learning
1122 West Campus Road, Room 521
Lawrence, KS 66045-3101
Phone: 785-864-4780
Fax: 785-864-5728
ddeshler@ku.edu
Dave L. Edyburn
Associate Professor
University of Wisconsin - Milwaukee
Department of Exceptional Education
PO Box 413
Milwaukee, WI 53201-0413
Phone: 414-229-5251
Fax: 414-229-5500
edyburn@uwm.edu
Russell Gersten
Director
Instructional Research Group
2525 Cherry Avenue, Suite 300
Signal Hill, CA 90755-2013
Phone: 562-595-6300
Fax: 562-595-9788
rgersten@inresg.org
Cay Holbrook
Faculty of Education
University of British Columbia
2125 Main Mall
Vancouver, BC V6T1Z4
Phone: 604-822-2235
Fax: 604-822-3302
cay.holbrook@ubc.ca
Michael L. Kamil
Professor of Education
Stanford University
123 Cubberley
485 Lasuen Mall
Stanford, California 94305-3096
Phone: 650-725-5452
Fax: 650-725-7412
mkamil@stanford.edu
Peggy McCardle
Associate Chief, Child Development and Behavior Branch
National Institute for Child Health and Human Development (NICHD)
6100 Exec. Blvd., MSC 7510
Rockville, MD 20852-7510
Phone: 301-435-6863
Fax: 301-480-0230
pm43q@nih.gov
Susan Rose
Associate Professor
University of Minnesota, College of Education and Human Development
253 Burton Hall
178 Pillsbury Drive SE
Minneapolis, MN 55455
Phone: 612-624-6387
Fax: 612-624-8241
srose@umn.edu
Deborah C. Simmons
Professor
Texas A&M University, College of Education and Human Development
4225 TAMU
College Station, TX 77843-4225
Phone: 979-845-8050
Fax: 979-862-1256
dsimmons@coe.tamu.edu
Dorothy S. Strickland
Professor of Education
Rutgers University
10 Seminary Place
New Brunswick, NJ 08901-1183
Phone: 732-932-7496 ext 8353
Fax: 732-932-7552
strickla@rci.rutgers.edu
Richard K. Wagner
Professor and Associate Director, Florida Center for Reading Research
Florida State University
208 Longmire Building
P.O. Box 1270
Tallahassee, FL 32306-1270
Phone: 850-644-1033
Fax: 850-644-7739
rkwagner@psy.fsu.edu
Brenda Jane (B.J.) Wiemer
Special Services Department Chair
Kirk Day School
802 Rotherham Drive
Ballwin, MO 63011
Phone: 636-391-3795
bwiemer@earthlink.net
Joanna P. Williams
Professor of Psychology and Education
Columbia Teachers College
525 West 120th Street
New York, NY 10027-6696
Phone: 212-678-3832
Fax: 212-678-3837
jpw15@columbia.ed
Observers
Jamal Abedi
Director of Technical Projects
CRESST
Professor
University of California, Los Angeles
301 GSE & IS Building
Los Angeles, CA 90095-1522
Phone: 310-206-4346
Fax: 310-825-3883
jabedi@cse.ucla.edu
Katherine M. Byrn
Research Assistant
Department of Curriculum and Instruction, University of Minnesota
60 Peik Hall
159 Pillsbury Dr. SE
Minneapolis, MN 55455
Phone: 612-625-9472
byrn0039@umn.edu
Ann T. Clapper
Development Coordinator
NCEO
University of Minnesota
350 Elliott Hall, 75 East River Road
Minneapolis, MN 55455
Phone: 612-625-6032
Fax: 612-624-0879
clapp013@umn.edu
Laurie Cutting
Assistant Professor of Neurology, Assistant Professor of Education
Johns Hopkins University
Kennedy Krieger Institute
707 North Broadway, Suite 232
Baltimore MD 21205
Phone: 443-923-9250
Fax: 443-923-9255
cutting@kennedykrieger.org
Lee Galda
Reading Researcher
Department of Curriculum and Instruction
University of Minnesota
350 Peik Hall
159 Pillsbury Dr. SE
Minneapolis, MN 55455
Phone: 612-625-3310
galda001@umn.edu
Christopher J. Johnstone
Universal Design Analyst
NCEO
University of Minnesota
350 Elliott Hall, 75 East River Road
Minneapolis, MN 55455
Phone: 612-6261936
Fax: 612-624-0879
john4810@umn.edu
Sheryl Lazarus
Alignment Analyst
NCEO
University of Minnesota
350 Elliott Hall, 75 East River Road
Minneapolis, MN 55455
Phone: 612-625-6585
Fax: 612-624-0879
laza0019@umn.edu
Ross Moen
Assessment Coordinator
NCEO
University of Minnesota
350 Elliott Hall, 75 East River Road
Minneapolis, MN 55455
Phone: 612-624-6676
Fax: 612-624-0879
moenross@umn.edu
David G. O'Brien
Reading Researcher
Department of Curriculum and Instruction
University of Minnesota
125 Peik Hall, 159 Pillsbury Dr. SE
Minneapolis, MN 55455
Phone: 612-625-5337
dobrien@umn.edu
Mari Quenemoen
Logistics Coordinator
NCEO
University of Minnesota
350 Elliott Hall, 75 East River Road
Minneapolis, MN 55455
Phone: 612-625-7241
Fax: 612-624-0879
quen0006@umn.edu
Sandra J. Thompson
Research Coordinator
NCEO
University of Minnesota
350 Elliott Hall, 75 East River Road
Minneapolis, MN 55455
Phone: 612-625-5338
Fax: 612-624-0879
thomp178@umn.edu
Top of page.
|
with questions or assistance with this site.