Letter+Offers+a+Warning+on++the+Unreliability+of+Value-Added+Evaluations

=The letter from assessment experts the N.Y. Regents ignored = =By Valerie Strauss = =//Published in the Washington Post//= =//5/22/11//=

=Here’s the letter that 10 assessment experts sent to the New York State Board of Regents earlier this month urging it not to approve a system that links student standard test scores to the evaluations of teachers and principals. = =The Regents ignored their evidence showing such “__value-added __” systems are unstable and prone to error — and __approved it anyway __. Starting in the fall, 40 percent of the evaluations of teachers and principals in New York public schools will based on student test scores. = =**May 15, 2011 ** = =**To The New York State Board of Regents: ** = =As researchers who have done extensive work in the area of testing and measurement, and the use of value-added methods of analysis, we write to express our concern about the decision pending before the Board of Regents to require the use of state test scores as 40% of the evaluation decision for teachers. = = = =As the enclosed report from the Economic Policy Institute describes, the research literature includes many cautions about the problems of basing teacher evaluations on student test scores. These include problems of attributing student gains to specific teachers; concerns about overemphasis on “teaching to the test” at the expense of other kinds of learning; and disincentives for teachers to serve high-need students, for example, those who do not yet speak English and those who have special education needs. = =Reviews of research on value-added methodologies for estimating teacher “effects” based on student test scores have concluded that these measures are too unstable and too vulnerable to many sources of error to be used as a major part of teacher evaluation. A report by the__ RAND Corporation __ concluded that: = =//The research base is currently insufficient to support the use of VAM for high-stakes decisions about individual teachers or schools. // = =<span style="background-color: transparent; color: #000000; font-family: serif; font-size: 130%; text-align: start; text-decoration: none; vertical-align: baseline;">The Board on Testing and Assessment of the National Research Council of the National Academy of Sciences stated, = =<span style="background-color: transparent; color: #000000; font-family: serif; font-size: 130%; text-align: start; text-decoration: none; vertical-align: baseline;">//<span style="background-color: transparent; color: #000000; font-family: serif; text-align: start; text-decoration: none; vertical-align: baseline;">…VAM estimates of teacher effectiveness … should not be used to make operational decisions because such estimates are far too unstable to be considered fair or reliable. // = =<span style="background-color: transparent; color: #000000; font-family: serif; font-size: 130%; text-align: start; text-decoration: none; vertical-align: baseline;">__<span style="background-color: transparent; color: #0000ee; font-family: serif; text-align: start; vertical-align: baseline;">Henry Braun __, then of the Educational Testing Service, concluded in his review of research: = =<span style="background-color: transparent; color: #000000; font-family: serif; font-size: 130%; text-align: start; text-decoration: none; vertical-align: baseline;">//<span style="background-color: transparent; color: #000000; font-family: serif; text-align: start; text-decoration: none; vertical-align: baseline;">VAM results should not serve as the sole or principal basis for making consequential decisions about teachers. There are many pitfalls to making causal attributions of teacher effectiveness on the basis of the kinds of data available from typical school districts. We still lack sufficient understanding of how seriously the different technical problems threaten the validity of such interpretations. // = =<span style="background-color: transparent; color: #000000; font-family: serif; font-size: 130%; text-align: start; text-decoration: none; vertical-align: baseline;">According to these studies, the problems with using value-added testing models to determine teacher effectiveness include: = =<span style="background-color: transparent; color: #000000; font-family: serif; font-size: 130%; text-align: start; text-decoration: none; vertical-align: baseline;">*Teachers’ ratings are affected by differences in the students who are assigned to them. Students are not randomly assigned to teachers – and statistical models cannot fully adjust for the fact that some teachers will have a disproportionate number of students who may be exceptionally difficult to teach (students with poor attendance, who are homeless, who have severe problems at home, etc.) and whose scores on traditional tests have unacceptably low validity (e.g. those who have special education needs or who are English language learners). All of these factors can create both misestimates of teachers’ effectiveness and disincentives for teachers to want to teach the neediest students, creating incentives for teachers to seek to teach those students those expected to make the most rapid gains and to avoid schools and classrooms serving struggling students. = =<span style="background-color: transparent; color: #000000; font-family: serif; font-size: 130%; text-align: start; text-decoration: none; vertical-align: baseline;">*Value-added models of teacher effectiveness do not produce stable ratings of teachers. Teachers look very different in their measured effectiveness when different statistical methods are used. In addition, researchers have found that teachers’ effectiveness ratings differ from class to class, from year to year, and even from test to test, even when these are within the same content area. Henry Braun notes that ratings are most unstable at the upper and lower ends of the scale, where many would like to use them to determine high or low levels of effectiveness. = =<span style="background-color: transparent; color: #000000; font-family: serif; font-size: 130%; text-align: start; text-decoration: none; vertical-align: baseline;">*It is impossible to fully separate out the influences of students’ other teachers, as well as school and home conditions, on their apparent learning. No single teacher accounts for all of a student’s learning. Prior teachers have lasting effects, for good or ill, on students’ later learning, and current teachers also interact to produce students’ knowledge and skills. Some students receive tutoring, as well as help from well-educated parents. A teacher who works in a well-resourced school with specialist supports serving students from stable, supportive families may appear to be more effective than one whose students don’t receive these supports. = =<span style="background-color: transparent; color: #000000; font-family: serif; font-size: 130%; text-align: start; text-decoration: none; vertical-align: baseline;">These problems are exacerbated further by the fact that the kind of grade-level tests and end-of-course tests used in New York are not designed to measure student growth. = =<span style="background-color: transparent; color: #000000; font-family: serif; font-size: 130%; text-align: start; text-decoration: none; vertical-align: baseline;">While value-added models based on student test scores are useful for looking at groups of teachers for research purposes – for example, to examine the results of professional development programs or to look at student progress at the school or district level, they are problematic as measures for making evaluation decisions for individual teachers. = =<span style="background-color: transparent; color: #000000; font-family: serif; font-size: 130%; text-align: start; text-decoration: none; vertical-align: baseline;">We urge you to reject proposals that would place significant emphasis on this untested strategy that could have serious negative consequences for teacher and for the most vulnerable students in the State’s schools. = =<span style="background-color: transparent; color: #000000; font-family: serif; font-size: 130%; text-align: start; text-decoration: none; vertical-align: baseline;">//<span style="background-color: transparent; color: #000000; font-family: serif; text-align: start; text-decoration: none; vertical-align: baseline;"> **<span style="background-color: transparent; color: #000000; font-family: serif; text-align: start; text-decoration: none; vertical-align: baseline;">Eva Baker **, Distinguished Professor, UCLA Graduate School of Education // = =<span style="background-color: transparent; color: #000000; font-family: serif; font-size: 130%; text-align: start; text-decoration: none; vertical-align: baseline;">//<span style="background-color: transparent; color: #000000; font-family: serif; text-align: start; text-decoration: none; vertical-align: baseline;">Director, National Center for Research on Evaluation, Standards and Student Testing (CRESST) // = =<span style="background-color: transparent; color: #000000; font-family: serif; font-size: 130%; text-align: start; text-decoration: none; vertical-align: baseline;">//<span style="background-color: transparent; color: #000000; font-family: serif; text-align: start; text-decoration: none; vertical-align: baseline;">President, World Educational Research Association, 2010-2012 // = =<span style="background-color: transparent; color: #000000; font-family: serif; font-size: 130%; text-align: start; text-decoration: none; vertical-align: baseline;">//<span style="background-color: transparent; color: #000000; font-family: serif; text-align: start; text-decoration: none; vertical-align: baseline;">Past President, American Educational Research Association // = =<span style="background-color: transparent; color: #000000; font-family: serif; font-size: 130%; text-align: start; text-decoration: none; vertical-align: baseline;">//<span style="background-color: transparent; color: #000000; font-family: serif; text-align: start; text-decoration: none; vertical-align: baseline;"> **<span style="background-color: transparent; color: #000000; font-family: serif; text-align: start; text-decoration: none; vertical-align: baseline;">Linda Darling-Hammond **, Charles E. Ducommun Professor of Education, Stanford University // = =<span style="background-color: transparent; color: #000000; font-family: serif; font-size: 130%; text-align: start; text-decoration: none; vertical-align: baseline;">//<span style="background-color: transparent; color: #000000; font-family: serif; text-align: start; text-decoration: none; vertical-align: baseline;">Past President, American Educational Research Association // = =<span style="background-color: transparent; color: #000000; font-family: serif; font-size: 130%; text-align: start; text-decoration: none; vertical-align: baseline;">//<span style="background-color: transparent; color: #000000; font-family: serif; text-align: start; text-decoration: none; vertical-align: baseline;">Executive Board Member, National Academy of Education // = =<span style="background-color: transparent; color: #000000; font-family: serif; font-size: 130%; text-align: start; text-decoration: none; vertical-align: baseline;">//<span style="background-color: transparent; color: #000000; font-family: serif; text-align: start; text-decoration: none; vertical-align: baseline;"> **<span style="background-color: transparent; color: #000000; font-family: serif; text-align: start; text-decoration: none; vertical-align: baseline;">Edward Haertel **, Vida Jacks Professor of Education, Stanford University // = =<span style="background-color: transparent; color: #000000; font-family: serif; font-size: 130%; text-align: start; text-decoration: none; vertical-align: baseline;">//<span style="background-color: transparent; color: #000000; font-family: serif; text-align: start; text-decoration: none; vertical-align: baseline;">Chair, Board on Testing and Assessment, National Research Council // = =<span style="background-color: transparent; color: #000000; font-family: serif; font-size: 130%; text-align: start; text-decoration: none; vertical-align: baseline;">//<span style="background-color: transparent; color: #000000; font-family: serif; text-align: start; text-decoration: none; vertical-align: baseline;">Vice-President, National Academy of Education // = =<span style="background-color: transparent; color: #000000; font-family: serif; font-size: 130%; text-align: start; text-decoration: none; vertical-align: baseline;">//<span style="background-color: transparent; color: #000000; font-family: serif; text-align: start; text-decoration: none; vertical-align: baseline;">Past President, National Council on Measurement in Education // = =<span style="background-color: transparent; color: #000000; font-family: serif; font-size: 130%; text-align: start; text-decoration: none; vertical-align: baseline;">//<span style="background-color: transparent; color: #000000; font-family: serif; text-align: start; text-decoration: none; vertical-align: baseline;"> **<span style="background-color: transparent; color: #000000; font-family: serif; text-align: start; text-decoration: none; vertical-align: baseline;">Helen F. Ladd **, Edgar Thompson Professor of Public Policy and Professor of Economics, Sanford School of Public Policy, Duke University // = =<span style="background-color: transparent; color: #000000; font-family: serif; font-size: 130%; text-align: start; text-decoration: none; vertical-align: baseline;">//<span style="background-color: transparent; color: #000000; font-family: serif; text-align: start; text-decoration: none; vertical-align: baseline;">President, Association of Public Policy and Management // = =<span style="background-color: transparent; color: #000000; font-family: serif; font-size: 130%; text-align: start; text-decoration: none; vertical-align: baseline;">//<span style="background-color: transparent; color: #000000; font-family: serif; text-align: start; text-decoration: none; vertical-align: baseline;"> **<span style="background-color: transparent; color: #000000; font-family: serif; text-align: start; text-decoration: none; vertical-align: baseline;">Henry M. Levin **, William Heard Kilpatrick Professor of Economics and Education, Teachers College, Columbia University // = =<span style="background-color: transparent; color: #000000; font-family: serif; font-size: 130%; text-align: start; text-decoration: none; vertical-align: baseline;">//<span style="background-color: transparent; color: #000000; font-family: serif; text-align: start; text-decoration: none; vertical-align: baseline;">Past President, Evaluation Research Society // = =<span style="background-color: transparent; color: #000000; font-family: serif; font-size: 130%; text-align: start; text-decoration: none; vertical-align: baseline;">//<span style="background-color: transparent; color: #000000; font-family: serif; text-align: start; text-decoration: none; vertical-align: baseline;">Past President, Comparative and International Education Society // = =<span style="background-color: transparent; color: #000000; font-family: serif; font-size: 130%; text-align: start; text-decoration: none; vertical-align: baseline;">//<span style="background-color: transparent; color: #000000; font-family: serif; text-align: start; text-decoration: none; vertical-align: baseline;"> **<span style="background-color: transparent; color: #000000; font-family: serif; text-align: start; text-decoration: none; vertical-align: baseline;">Robert E. Linn **, Professor Emeritus, University of Colorado at Boulder // = =<span style="background-color: transparent; color: #000000; font-family: serif; font-size: 130%; text-align: start; text-decoration: none; vertical-align: baseline;">//<span style="background-color: transparent; color: #000000; font-family: serif; text-align: start; text-decoration: none; vertical-align: baseline;">Past President, American Educational Research Association // = =<span style="background-color: transparent; color: #000000; font-family: serif; font-size: 130%; text-align: start; text-decoration: none; vertical-align: baseline;">//<span style="background-color: transparent; color: #000000; font-family: serif; text-align: start; text-decoration: none; vertical-align: baseline;">Past President, National Council on Measurement in Education // = =<span style="background-color: transparent; color: #000000; font-family: serif; font-size: 130%; text-align: start; text-decoration: none; vertical-align: baseline;">//<span style="background-color: transparent; color: #000000; font-family: serif; text-align: start; text-decoration: none; vertical-align: baseline;"> **<span style="background-color: transparent; color: #000000; font-family: serif; text-align: start; text-decoration: none; vertical-align: baseline;">Aaron Pallas **, Professor of Sociology and Education, Teachers College, Columbia University // = =<span style="background-color: transparent; color: #000000; font-family: serif; font-size: 130%; text-align: start; text-decoration: none; vertical-align: baseline;">//<span style="background-color: transparent; color: #000000; font-family: serif; text-align: start; text-decoration: none; vertical-align: baseline;">Fellow, American Educational Research Association // = =<span style="background-color: transparent; color: #000000; font-family: serif; font-size: 130%; text-align: start; text-decoration: none; vertical-align: baseline;">//<span style="background-color: transparent; color: #000000; font-family: serif; text-align: start; text-decoration: none; vertical-align: baseline;"> **<span style="background-color: transparent; color: #000000; font-family: serif; text-align: start; text-decoration: none; vertical-align: baseline;">Richard Shavelson **, Dean Emeritus and Margaret Jacks Professor Emeritus, Stanford University // = =<span style="background-color: transparent; color: #000000; font-family: serif; font-size: 130%; text-align: start; text-decoration: none; vertical-align: baseline;">//<span style="background-color: transparent; color: #000000; font-family: serif; text-align: start; text-decoration: none; vertical-align: baseline;">Past President, American Educational Research Association // = =<span style="background-color: transparent; color: #000000; font-family: serif; font-size: 130%; text-align: start; text-decoration: none; vertical-align: baseline;">**<span style="background-color: transparent; color: #000000; font-family: serif; text-align: start; text-decoration: none; vertical-align: baseline;"> //<span style="background-color: transparent; color: #000000; font-family: serif; text-align: start; text-decoration: none; vertical-align: baseline;">Lorrie A. Shepard, Dean & Distinguished Professor, University of Colorado at Boulder // ** = =<span style="background-color: transparent; color: #000000; font-family: serif; font-size: 130%; text-align: start; text-decoration: none; vertical-align: baseline;">//<span style="background-color: transparent; color: #000000; font-family: serif; text-align: start; text-decoration: none; vertical-align: baseline;">Past President, American Educational Research Association // = =<span style="background-color: transparent; color: #000000; font-family: serif; font-size: 130%; text-align: start; text-decoration: none; vertical-align: baseline;">//<span style="background-color: transparent; color: #000000; font-family: serif; text-align: start; text-decoration: none; vertical-align: baseline;">Past President National Academy of Education // = =<span style="background-color: transparent; color: #000000; font-family: serif; font-size: 130%; text-align: start; text-decoration: none; vertical-align: baseline;">//<span style="background-color: transparent; color: #000000; font-family: serif; text-align: start; text-decoration: none; vertical-align: baseline;">Past President National Council on Measurement in Education // = =<span style="background-color: transparent; color: #000000; font-family: serif; font-size: 130%; text-align: start; text-decoration: none; vertical-align: baseline;">**<span style="background-color: transparent; color: #000000; font-family: serif; text-align: start; text-decoration: none; vertical-align: baseline;"> //<span style="background-color: transparent; color: #000000; font-family: serif; text-align: start; text-decoration: none; vertical-align: baseline;">Lee S. Shulman, Charles E. Ducommun Professor Emeritus, Stanford University // ** = =<span style="background-color: transparent; color: #000000; font-family: serif; font-size: 130%; text-align: start; text-decoration: none; vertical-align: baseline;">//<span style="background-color: transparent; color: #000000; font-family: serif; text-align: start; text-decoration: none; vertical-align: baseline;">President Emeritus, Carnegie Foundation for the Advancement of Teaching // = =<span style="background-color: transparent; color: #000000; font-family: serif; font-size: 130%; text-align: start; text-decoration: none; vertical-align: baseline;">//<span style="background-color: transparent; color: #000000; font-family: serif; text-align: start; text-decoration: none; vertical-align: baseline;">Past President, American Educational Research Association // = = = = = = = = = = = =News= =home=