Data Boondoggles and the “New Stupid”

It’s pretty easy to find examples of multi-million dollar, troubled data projects. California’s $34 million data system is a mess. And in my home city of Washington, DC, we’ve tried and failed several times to build a state longitudinal database. One reason I wrote Putting Data Into Practice was to try to help both policymakers and educators avoid these mistakes. The paper chronicles New York City’s ARIS system, which also experienced a very troubled start, and offers important lessons for districts and states beginning similar projects: Districts and states must design technology tools in an iterative fashion, work closely with educators in the field, create prototypes of a number of tools, and test each with teachers, principals, and other users before taking them into full-scale development.*

Worse, though, are the many examples of superficial and faddish uses of data (h/t Core Knowledge). Rick Hess, in a smart 2008 Educational Leadership article, warns of the glib use of data as “the new stupid.”

Research across education and a variety of other fields shows that the effective use of data involves real change–it’s much more than just mounting a few brightly colored spreadsheets on the wall. Thomas Davenport, President’s Chair in Information Technology and Management at Babson College and author of Competing on Analytics, cautions that “companywide embrace of analytics impels changes in culture, processes, behavior, and skills for many employees.”  He also notes the time-consuming process of building both the human capabilities and work processes to support the effective use of data. This is also true in New York City schools:

Just as important, says Emily Weiss, chief of staff for the Division of Performance and Accountability, the district also had to “build the demand for data.” And to do that, they had to make it relevant.

There are two important prerequisites to building demand for data, Weiss says. First, educators must understand what the data is and what it means – they must become “data and assessment literate” — before they can understand how to use it for instruction. Second, the operational structure of the school must accommodate teacher collaboration based on data. That means teachers must be encouraged to share data and talk about what they think it means. “We need to move to conversations among educators,” says Weiss, “not just individual data analysis.”

That sort of collaboration — “shared accountability across multiple educators for the same students,” Weiss calls it — represents a significant change in the closed-door culture of many schools.

Using data effectively needs to be embedded into the practice of good teaching. You can’t just build a data system and expect that instruction will magically improve.

* The dynamic nature of this work is really important to emphasize. Providing feedback on the paper, one person who used to work on ARIS told me that “what actually happens is that you build to an initial set of priorities (features and audiences) and those expand over time so you plan on expanding and adjusting the system as new requirements become clear.”