In September a survey of Oklahoma high school students found only 2.8 percent were able to achieve a passing score on the U.S. Citizenship Test. The actual test has a 92.4 percent pass rate, so it would be quite a problem if Oklahoma high schoolers performed so poorly. But, thankfully, the results appear to have been fabricated.
The survey asked ten basic civics questions, along the lines of who the first president was (the survey claimed only 23 percent of Oklahoma high schoolers correctly identified George Washington) and what we call the first 10 Amendments to the Constitution. According to the survey, no students correctly answered more than eight questions, and only 11 percent knew how many years a senator’s term is, and only 10 percent knew how many justices serve on the Supreme Court.
What’s distressing for the education space is how quickly these results spread. State and national media covered the story, and the results were used in posts by education blogs and bloggers such as Flypaper, Common Core, Matthew Ladner (at Jay P. Greene’s blog), Robert Pondiscio (on the Core Knowledge site), and Diana Senechal (at joannejacobs.com). These are some of the education blogosphere’s biggest names, but none of them questioned the results as dubious. It’s rather remarkable, in retrospect, that so many people were willing to take these amazingly poor findings as solid evidence of the failings of American public schools.
Update: Matthew Ladner responds in the comments section to say he and the Oklahoma Council of Public Affairs (the think tank commissioning the survey) are investigating the survey’s validity. Fair enough. But even if they’re “valid” in the sense that no one deliberately made up the results and the surveyors actually did talk to 1,000 representative Oklahoma high schoolers, they’re still not “valid” in the sense that they would allow anyone to claim that “only 2.6 percent of Oklahoma students could pass the citizenship test.”
The survey was conducted over the phone in a low-stakes manner to high school students. Is it reasonable to expect that such a survey design is going to give good results? That high school students will answer a stranger’s call, presumably on a land line, and try their best? Is it fair to compare this format to the results of immigration applicants, who can see the test questions (and answers!) in advance, can study, and have a vested interest in memorizing the answers and scoring well? I think not.
Lowering Student Loan Default Rates: What One Consortium of Historically Black Institutions Did to Succeed 
College and Career-Ready: Using Outcomes Data to Hold High Schools Accountable for Student Success
03/19/10 | Ben Miller is quoted on college graduation rates and the NCAA Tournament in the Detroit Free Press.
03/19/10 | Linda Perlstein links to Education Sector's "College- and Career-Ready" event.
03/17/10 | Ben Miller's column on graduation rates and NCAA Tournament eligibility is mentioned on The Wall Street Journal's The Daily Fix.
03/17/10 | Kevin Carey's critique of Diane Ravitch's latest book is quoted in Doublethink's Conventional Folly blog.
03/15/10 | Ben Miller pens an op-ed for US News & World Report on using graduation rates to determine NCAA Tournament eligibility.
In one multiple choice test several of my students thought “The League of Extraordinary Gentlemen” was the proper name for the Spanish Inquisition.
To be fair to my students, nobody expects the Spanish inquisition. http://www.youtube.com/watch?v=gldlyTjXk9A
As a former history teacher these results are not that dubious. First, fill in the blank questions with no word bank always had the lowest performance. Add a word bank and kids did much better. Multiple choice questions were the same way.
There is nothing wrong with questioning the results, but you better have proof to back up your claim that the results were fabricated. Having a hunch doesn’t count – that is as valid (or less valid) as my teaching experience.
Duplicate the survey yourself and try again.
[...] Matthew Ladner says he and the Oklahoma Council of Public Affairs, which commissioned the poll, are investigating the survey’s validity. It’s possible that students didn’t bother to try on a [...]
Claus, you’re in the right here. It looks like you beat me to this latest story too with this post from Monday: http://tinyurl.com/ygulafn
Chad-
I noted in the Oklahoma study that the first time pass rate for immigrants was over 90% and said that we ought not to expect a similar result for some of the very reasons you suggest.
Is there a similar investigation of the identical Arizona survey, which yielded similar results? (Was it carried out by the same firm?)
Not every blogger swallowed the results whole. I questioned the results of the Arizona survey in early July. I questioned the Oklahoma results in September.
Sorry for tooting my own horn here, but I felt like the lone voice in the wilderness back then. So when I saw the 538.com posting last night, I felt vindicated.
Things are certainly not nearly what they should be in civics and history education, but we’ve got to check our impulse to believe every rotten thing we hear about public schools and the students in them!
[...] students can answer basic civics questions is the day I go back to my fields a happy man. At The Quick and The Ed, Chad Alderman says “it’s rather remarkable, in retrospect, that so many people were [...]
Chad-
Both myself and OCPA are investigating the validity of the survey. Thus far we have been provided with the call logs, and are awaiting receipts for the marketing list purchases of students. If the allegations against the polling firm are valid, then both myself and the OCPA will have been victims of a fraud. If not, the polling firm will have been the victim. Once we have gathered the requested evidence, we will consider whether to conduct a replication poll with another firm.
Regarding your “why did anyone believe this” point consider the following evidence from a multiple choice civics exam given to college students:
http://www.americanciviclitera.....dings.html
Simply put, even college results are abysmal. Like the U.S. citizenship test itself, our survey was given in an open answer format rather than multiple choice, and to high school students rather than college students. One would expect students to perform significantly worse without the correct answer staring them in the face in a multiple choice format. The dynamics of a sit down test versus a telephone interview may also have played a role.
We are doing our best to investigate the situation, and I can only ask for your patience at this point.
Matthew Ladner