Monday, August 25, 2008
The Schools We Need, Continued
A few additional thoughts on the post below about margins of error, etc., based on the comments in this and earlier posts. (An aside: My take on anonymous commenting is pretty much akin to my take on anonymous blogging: Why? It's a free country, nobody's going to yank you out of your house in the middle of the night because of what you write about No Child Left Behind. Identify yourself, we'll all be better for it. Particularly, I should add, if you're claiming some kind of expertise. If you're really a statistics professor, say so; I'm not giving handles like "StatsProf" any more credence than "Anonymous" or "SomeGuyYellingOnTheStreetCorner.")
First, some people have raised the issue of measurement error vis a vis the tests themselves. And it's true that (A) tests only cover a fraction of what we want to students to know, and (B) tests are an imperfect measure of that fraction. But the place to make allowances for that should be the process of setting cut scores. States don't require students to get 100 percent of the answers right to be deemed "proficient." (In fact, all states have a separate, higher "advanced" level which is also less than 100 percent.) Once states determine what less-than-100 score is proficient, it makes sense to build in some cushion to account for imperfections in the instrument, random variation, etc. This is particularly true if the test is being used for individual high-stakes purposes, i.e. graduation and grade retention policies. But having made those allowances, don't layer on new allowances with confidence intervals and tell the public that students are meeting NCLB performance thresholds when in fact they're not.
Second, there's certainly no unanimity in academia on the issue of confidence intervals. Some very qualified, well-trained people see it one way, some see it another.
Third, what's consistently lost in all of these discussions is the need to balance the risks of Type I and Type II errors. When implementing NCLB over the last six years, states have been almost exclusively interested in making sure that good schools aren't mistakenly identified as "in need of improvement" or "failing" under NCLB. And that's a legitimate thing to worry about. But they've all but ignored the risk of under-performing schools being mistakenly not identified as needing improvement. There's no way to completely eliminate both risks; states have to find a reasonable middle ground between them. Instead, many states have been obsessed with finding new ways to make sure there is absolutely no possible way for a good school to be falsely identified as underperforming, and in doing so have knowingly let schools and districts that are obviously not serving students well slide by for years and years. Of course, focusing only on minimizing the first kind of risk makes sense if you view NCLB as fundamentally illegitimate and malign, but if that's the case just say so and stop pretending your concerns are limited to proper use of statistics. -- |
Comments