A Developing Scandal in Oklahoma

In September a survey of Oklahoma high school students found only 2.8 percent were able to achieve a passing score on the U.S. Citizenship Test. The actual test has a 92.4 percent pass rate, so it would be quite a problem if Oklahoma high schoolers performed so poorly. But, thankfully, the results appear to have been fabricated.

The survey asked ten basic civics questions, along the lines of who the first president was (the survey claimed only 23 percent of Oklahoma high schoolers correctly identified George Washington) and what we call the first 10 Amendments to the Constitution. According to the survey, no students correctly answered more than eight questions, and only 11 percent knew how many years a senator’s term is, and only 10 percent knew how many justices serve on the Supreme Court.

What’s distressing for the education space is how quickly these results spread. State and national media covered the story, and the results were used in posts by education blogs and bloggers such as Flypaper, Common Core, Matthew Ladner (at Jay P. Greene’s blog), Robert Pondiscio (on the Core Knowledge site), and Diana Senechal (at joannejacobs.com). These are some of the education blogosphere’s biggest names, but none of them questioned the results as dubious. It’s rather remarkable, in retrospect, that so many people were willing to take these amazingly poor findings as solid evidence of the failings of American public schools.

Update: Matthew Ladner responds in the comments section to say he and the Oklahoma Council of Public Affairs (the think tank commissioning the survey) are investigating the survey’s validity. Fair enough. But even if they’re “valid” in the sense that no one deliberately made up the results and the surveyors actually did talk to 1,000 representative Oklahoma high schoolers, they’re still not “valid” in the sense that they would allow anyone to claim that “only 2.6 percent of Oklahoma students could pass the citizenship test.”

The survey was conducted over the phone in a low-stakes manner to high school students. Is it reasonable to expect that such a survey design is going to give good results? That high school students will answer a stranger’s call,  presumably on a land line, and try their best? Is it fair to compare this format to the results of immigration applicants, who can see the test questions (and answers!) in advance, can study, and have a vested interest in memorizing the answers and scoring well? I think not.