Comments on: Margins of Error http://www.quickanded.com/2008/08/margins-of-error.html The Quick and the Ed is an education blog published by Education Sector, an independent think tank in Washington D.C. The Quick and the Ed offers in-depth analysis on the latest in education policy and research. Sat, 20 Mar 2010 08:26:20 -0400 http://wordpress.org/?v=2.8.1 hourly 1 By: Ian http://www.quickanded.com/2008/08/margins-of-error.html/comment-page-1#comment-114 Ian Mon, 25 Aug 2008 04:35:00 +0000 http://www.quickanded.com/wordpress/?p=1216#comment-114 <i>If you think students need 80% correct to be proficient, set the cut score at 75% to account for error. But having done so, don't then proceed to tell parents that schools have met AMOs under NCLB when in fact they have not.</i><br/><br/>So are you saying that the schools have <i>already</i> done this, or that they <i>should</i> do this?<br/><br/>It's true that if you have a 100% sample of test scores you don't need a confidence interval. You have sampled your entire population. That said, no one is interested in the population of <i>test scores</i> - they are interested in what the students know. You have only <i>sampled</i> what it is that students know. Ignoring the systematic error associated with the instrument, we still have a random error associated with the measurement. Different versions of a test may be identical on average, but individuals will perform differently on different versions of a given test. Even with a given test, individuals will perform differently at different times. You might be hungry one day, sleepy another, and in peak mental condition the third.<br/><br/>One could, of course, fudge the standards up front, as you suggested, and build in a "margin of error". But why build junk like that into the system? Why not use something more reliable - like a margin or error? You know - something that you can actually calculate from the data... If you think students need 80% correct to be proficient, set the cut score at 75% to account for error. But having done so, don’t then proceed to tell parents that schools have met AMOs under NCLB when in fact they have not.

So are you saying that the schools have already done this, or that they should do this?

It’s true that if you have a 100% sample of test scores you don’t need a confidence interval. You have sampled your entire population. That said, no one is interested in the population of test scores – they are interested in what the students know. You have only sampled what it is that students know. Ignoring the systematic error associated with the instrument, we still have a random error associated with the measurement. Different versions of a test may be identical on average, but individuals will perform differently on different versions of a given test. Even with a given test, individuals will perform differently at different times. You might be hungry one day, sleepy another, and in peak mental condition the third.

One could, of course, fudge the standards up front, as you suggested, and build in a “margin of error”. But why build junk like that into the system? Why not use something more reliable – like a margin or error? You know – something that you can actually calculate from the data…

]]> By: Corey Bunje Bower http://www.quickanded.com/2008/08/margins-of-error.html/comment-page-1#comment-110 Corey Bunje Bower Fri, 22 Aug 2008 18:58:00 +0000 http://www.quickanded.com/wordpress/?p=1216#comment-110 Regardless of what they call it, it's still the right idea. Imagine that the cut-off for AYP is that 50% of students in a school are able to pass a test. If you test everybody in the school ten times, you might have 45% pass one time and 55% pass another. But the school is only tested once. So if 45% of students in the school pass, we can't really be sure that the school didn't live up to expectations -- it might have just been a testing day that was at the bottom of the distribution. Regardless of what they call it, it’s still the right idea. Imagine that the cut-off for AYP is that 50% of students in a school are able to pass a test. If you test everybody in the school ten times, you might have 45% pass one time and 55% pass another. But the school is only tested once. So if 45% of students in the school pass, we can’t really be sure that the school didn’t live up to expectations — it might have just been a testing day that was at the bottom of the distribution.

]]> By: Anonymous http://www.quickanded.com/2008/08/margins-of-error.html/comment-page-1#comment-109 Anonymous Fri, 22 Aug 2008 18:13:00 +0000 http://www.quickanded.com/wordpress/?p=1216#comment-109 I found this post amusing given your past attempts at statistical wizardry. Let's re-tell the story, applying the Carey 51% action rule.<br/><br/>You: That's terrible! Are you sure?<br/><br/>Dr: Well, there are few absolute certainties in medicine...But I'm pretty sure it's pneumonia.<br/><br/>You: How sure?<br/><br/>Dr: 51% sure. <br/><br/>What would you do? Hey, what matters is your kid is sick! If there's a 51% chance he needs to be treated for pneumonia, then we begin treatment--now! It sure beats the status quo.<br/><br/>Another lesson learned from Q&E. When it comes to statistics--kids, don't try this at home. I found this post amusing given your past attempts at statistical wizardry. Let's re-tell the story, applying the Carey 51% action rule.

You: That's terrible! Are you sure?

Dr: Well, there are few absolute certainties in medicine…But I'm pretty sure it's pneumonia.

You: How sure?

Dr: 51% sure.

What would you do? Hey, what matters is your kid is sick! If there's a 51% chance he needs to be treated for pneumonia, then we begin treatment–now! It sure beats the status quo.

Another lesson learned from Q&E. When it comes to statistics–kids, don't try this at home.

]]> By: Anonymous http://www.quickanded.com/2008/08/margins-of-error.html/comment-page-1#comment-108 Anonymous Fri, 22 Aug 2008 13:07:00 +0000 http://www.quickanded.com/wordpress/?p=1216#comment-108 As a statistician, I refer you to the very important Todd Snider song, "Statisticians Blues". It's friday afterall:<br/><br/>Live with band: http://www.youtube.com/watch?v=3d_VU2XUP-E&feature=related<br/><br/>Live by himself: http://www.youtube.com/watch?v=BMQdtyot38s&feature=related<br/><br/>Enjoy. As a statistician, I refer you to the very important Todd Snider song, "Statisticians Blues". It's friday afterall:

Live with band: http://www.youtube.com/watch?v.....re=related

Live by himself: http://www.youtube.com/watch?v.....re=related

Enjoy.

]]> By: Kevin Carey http://www.quickanded.com/2008/08/margins-of-error.html/comment-page-1#comment-105 Kevin Carey Thu, 21 Aug 2008 16:10:00 +0000 http://www.quickanded.com/wordpress/?p=1216#comment-105 When you ask states themselves why they use confidence intervals, they refer to sampling error, not testing error. <br/><br/>It's true that tests only assess a subset of what students need to know, and test results are subject to measurement error. But the proper way to account for measurement error is in setting cut scores on the test. No states requires students to get 100% correct to pass. So when policymakers decide what score score is good enough, that's the place to make allowances for the imprecision of the instrument. If you think students need 80% correct to be proficient, set the cut score at 75% to account for error. But having done so, don't then proceed to tell parents that schools have met AMOs under NCLB when in fact they have not. When you ask states themselves why they use confidence intervals, they refer to sampling error, not testing error.

It’s true that tests only assess a subset of what students need to know, and test results are subject to measurement error. But the proper way to account for measurement error is in setting cut scores on the test. No states requires students to get 100% correct to pass. So when policymakers decide what score score is good enough, that’s the place to make allowances for the imprecision of the instrument. If you think students need 80% correct to be proficient, set the cut score at 75% to account for error. But having done so, don’t then proceed to tell parents that schools have met AMOs under NCLB when in fact they have not.

]]> By: Corey Bunje Bower http://www.quickanded.com/2008/08/margins-of-error.html/comment-page-1#comment-104 Corey Bunje Bower Thu, 21 Aug 2008 15:33:00 +0000 http://www.quickanded.com/wordpress/?p=1216#comment-104 It wouldn't be 84% likely that Obama is the next president, it would be 84% likely that more people in the population currently plan on voting for him.<br/><br/>And the MOE around test results doesn't reflect the fact that not every kid was tested, it reflects the fact that that one test is only a sample of their ability -- if 100 different tests were given, the scores would differ each time -- and recognizes that a student's score would vary each time they took a test based on how they're feeling and what questions happen to be on the test. It wouldn’t be 84% likely that Obama is the next president, it would be 84% likely that more people in the population currently plan on voting for him.

And the MOE around test results doesn’t reflect the fact that not every kid was tested, it reflects the fact that that one test is only a sample of their ability — if 100 different tests were given, the scores would differ each time — and recognizes that a student’s score would vary each time they took a test based on how they’re feeling and what questions happen to be on the test.

]]>