 |
 |
 |
 |
 |
|
National Accessible Reading Assessment Projects December 7, 2007
OverviewThe fourth meeting of the General Advisory Committee (GAC) of the National Accessible Reading Assessment Projects (NARAP) was held December 7, 2007 in Washington, DC. The purpose of the meeting was to update GAC members of NARAPs progress since its second meeting on December 8, 2006 in Washington, DC. GAC members were updated on all three NARAP projects, including the Technology-Assisted Reading Assessment (TARA) project, the Partnership for Accessible Reading Assessment (PARA) project, and the Designing Accessible Reading Assessments (DARA) project. GAC members learned about and discussed the upcoming research for each project and the next steps for NARAP. Welcome and GreetingsMartha Thurlow Martha Thurlow welcomed and thanked all the GAC members for their attendance. She highlighted the day’s agenda and reminded the group of their charge: to review and advise on the general plan of the collaborating projects of NARAP and to provide liaison with the groups represented by the General Advisory Committee members. Martha also discussed the current activities of the NARAP projects. All newly completed papers and reports can be found in the meeting binder. Reports and past presentations can be found on the NARAP website. Overview of TARA Project and Results Elizabeth Stone Elizabeth Stone shared an overview of the TARA project, which focuses on improving state reading assessments for students with visual impairments. She also explained the psychometric work done to evaluate comparability of measurement characteristics for students without disabilities and students who are blind or visually impaired. The research results discussed at the meeting were those from differential item functioning (DIF), differential distractor functioning (DDF), and a review and comparison of the standard, large print, and braille test forms. All research was completed on the 4 th and 8 th grade English Language Arts components of a large-scale, state standards based test. Overall, the DIF and DDF results showed that the tests performed comparably for the students without disabilities and students who are blind or visually impaired. The group found that there were differences between the standard and large print forms and aspects of the large print and braille forms that should be considered as possibly impacting the performance of the students who are blind or visually impaired. Elizabeth Stone also discussed the Evidence Centered Design (ECD) test development process, putting the TARA test development effort into a structured framework. TARA project members are currently working on drafting Evidence Centered Design Models based on the research findings and will be developing pilot test items beginning Summer 2008. The process involves stating the purpose of the test, presenting the high level claim about what the test will measure, describing the population of test takers and the test structure, and outlining proficiency levels that will function as the outcomes. The research, as well as the survey and interviews of TVIs will feed this test development work. Martha Thurlow shared results and progress on the surveys and interviews of teachers of students with visual impairments. The surveys, which sought information on instruction as well as assessment, show that assistive technology seems to be a large part of instruction. However, because students use different technology for different purposes, traditional standardization of assessments may be a challenge. The teacher interviews have only been preliminarily analyzed, and further analysis is currently underway. Discussion with GAC members included concerns about the availability of appropriate technology for students to be required to take computer-based assessments. Discussion also focused on the sample sizes and population used for the psychometric work. Members of the GAC asked about procedures for defining grade-level texts, which has implications for the development of the test. Overview of the PARA Project Progress Ross Moen Ross Moen began the overview of the progress of the PARA project by sharing results from the Item Characteristics, Student Characteristics, and Segmented Text studies. The purpose of these studies is to seek universal solutions in order to minimize the need for accommodations on large-scale standardized assessments. For the Item Characteristics Study, results for students with disabilities on state assessments were analyzed from three states. Comprehensive results based on differential item functioning (DIF) analysis and differential distractor functioning (DDF) analysis indicated that it is important to look at results by disability category as well as overall. They also suggested that for at least one of the assessments studied,there did not appear to be bias against students with disabilities. The Student Characteristics study included teacher questionnaires and interviews to determine whether teachers are able to identify students who are likely to be less accurately measured by assessments. Ross Moen also discussed the Segmenting Study, which looked at whether segmenting text during an assessment would reduce the need for accommodations. Deborah Dillon shared progress on the Calibration and Motivation studies for the PARA project. The purpose of the Calibration Study is to calibrate the measurement tools that will be used in a large-scale reading assessment for students with disabilities. Results from the test will determine passages and items that will be used for the Motivation Study. Deborah Dillon passed out several sample passages that will be used in the Calibration Study. Once passages have been calibrated and the final passages are chosen, the Motivation Study will take place. The purpose of the Motivation Study is to determine whether improving the motivational characteristics of a large-scale reading assessment increases the accessibility for students with disabilities. AC members discussed the sample sizes for the Motivation Study, and expressed the desire to obtain results at the category of disability level. There were also questions about logistics of the test, including whether students will have a chance to practice choosing passages before the test and whether students will have to respond to every item on the assessment. Questions were asked about whether improving motivational characteristics would be an accommodation if the assessment is given to both students with and without disabilities and the motivational characteristics also help the students without disabilities. Overview of DARA Project and ResultsLinda Cook Linda Cook shared an overview of the DARA project, which is designed to look at whether students with reading-based learning disabilities receive differential performance gains from a read aloud accommodation when compared to students without disabilities. DARA is also investigating whether reading comprehension tests and test items take with and without read aloud accommodations measure similar constructs. The specific questions that the DARA studies addressed were:
The first question was addressed by carrying out a differential boost study using New Jersey Public school fourth and eighth grade students taking the Gates MacGinitie Reading Test. The second question was addressed by a series of factor analyses and differential item functioning analyses that were carried out using data from the differential boost study and also using fourth and eighth grade data collected at the administration of a large state standards-based assessment of English-language arts. The results of the differential boost study showed that students with reading based learning disabilities show a differential score gain when the Gates MacGinitie reading comprehension test is taken with a read aloud accommodation/modification. The results of the factor analysis and DIF studies indicated that the Gates MacGinitie reading assessment and the English-language arts assessment appear to measure the same single construct when taken with and without a read-aloud accommodation/modification. Next steps for DARA include completing the analysis from cognitive labs, as well as design and carry out simulations for a multi-stage reading assessment and field trial of prototype assessment. Principles and Guidelines WorkMartha Thurlow Martha Thurlow shared the progress on Goal 3 of the NARAP projects. The purpose of Goal 3 is to develop research-based principles and guidelines for making large-scale assessments of reading proficiency more accessible for students who have disabilities that affect reading. The three NARAP projects met in July 2007 to begin to draft the evidence-based principles. The principles, which have been further developed through bi-weekly conference calls, will be shared with a “Principles Committee” in order to review and comment on the proposed principles and guidelines that are gleaned from NARAP and other research. The principles and guidelines are intended to be finalized in Summer 2008. GAC members commented on the current principles and guidelines, and stated that there may need to be more consistency throughout the principles, making sure to acknowledge the population in focus and make clear that the document reflects that the principles are being designed for large-scale assessments. NARAP Next StepsLinda Cook Linda Cook thanked the GAC for their thoughtful participation in the discussions throughout the meeting. She stated that the presentations from the day’s meeting would be added to the website so that GAC members are able to share and talk about them with their colleagues. The members of the GAC requested regular updates on the progress of the projects, and notification when items are added to the website.
|
Contact narap@umn.edu
with questions or assistance with this site.