Chapter 2 Analysis development in education

2.1 What does it look like in education agencies?

Now more than ever, the use of data is a critical component of decision-making within organizations. As computing costs continue their precipitous decline and the volume of raw data grows exponentially, leaders in education agencies are eager to leverage these trends to better inform their choices. The most significant barriers to a data-informed education agency are no longer financial or technological. Instead, personnel and culture tend to be the largest obstacles for organizations looking to improve their use of data.

For years, educators and policy makers have been encouraged to embrace “data-driven decision-making,” but few education organizations have personnel that focus specifically on data analysis. Instead, it has become yet another skill we’ve asked educators to add to their already-overflowing professional responsibilities.

Larger education organizations are fortunate enough to have dedicated data analysts, particularly state education agencies (SEA’s), but structural and cultural barriers often prevent them from operating as effectively as they could. Analysts are often program-specific staff working in isolation. Collaboration among data analysts in SEA’s is frequently an exception, not the rule.

Education agencies of different sizes and missions face different data problems. School systems collect and process a variety of information generated by their students and employees, then report the results to a wide range of audiences.Their data could be coming from millions of students and thousands of educators in a large state to a hundred students and a dozen teachers in a start-up charter school, but they both have the same goal in doing so: improving the outcomes of their students.

Non-profit organizations focusing on education policy face different challenges. Since they aren’t directly generating much data on student, school, or educator performance, they typically rely on publicly-available data sources. These organizations then use that information to support programs and policies that will help more students succeed.

Regardless of an education agency’s size or mission, they all have a similar job. They collect funding, then allocate it to programs and/or staff to ultimately improve education for kids. In order to inform their efforts, they collect data to guide their decision-making.

The key question these agencies must continually ask themselves: are we collecting the right data and analyzing it appropriately to inform our work? If the answer is a strong “yes” with significant supporting evidence, well done!

If not, this book can help.

2.2 Who are we talking about?

Education agency employees vary in their interactions with data. Some directly manage student information systems (pretty data-heavy work) while others mostly consume data from reports and briefings. Is there a clear point on that spectrum which clearly delineates someone as an “analyst?”

Probably not, but it’s helpful to set a working definition for our purposes:

An analyst in an education agency is anyone whose primary work is focused on the collection, transformation, and communication of data.

Under this definition, and analyst is less defined by the tools and techniques they use than the amount of time they spend working with data. Some analysts are spreadsheet warriors while others write code using R, Python, and/or other languages. Some analysts will only need the SUM() and AVERAGE() functions in Excel while others will apply sophisticated machine learning techniques.

At the end of the day, anyone who spends most of their time at work collecting, transforming, and sharing data is an analyst.

2.3 Why should we care about the culture of data analysis?

Data analysis is difficult work. Doing it well requires an understanding of the field you’re studying, statistical knowledge, some programming ability, and most importantly, the ability to translate your results to non-analysts. At rstudio::conf() in January 2017, Hilary Parker gave a presentation that pinpoints some of the most critical challenges faced by data analysts. In her talk, she notes that:

“Creating an analysis is a hard, error-ridden process that gets ‘yadda-yadda-ed’ away” because we’re reluctant to share our own workflows and/or limit another analyst’s “creativity.”

These informal norms that exist between analysts can not only limit their own professional growth, it inevitably leads to mistakes that go unnoticed because there’s only been one pair of eyes on a project. To address this problem, Parker recommends that we start to think about ourselves as “analysis developers” and borrow some concepts from programmers and engineers.

Errors will happen. Parker notes that we can learn from operations engineers, particularly Sidney Dekker’s book, “The Field Guide to Understanding Human Error.” The lesson of this book is to not blame a person for their errors and instead to look at the process they used. This helps to make the after-action review conversations less personal and more focused on solving the problem at hand.

Reducing error is a big reason to care about developing a healthy, collaborative culture among “analysis developers,” but there are many other benefits to improving how these folks work. It can help improve the quality and depth of analysis as they start to learn from one another. A more collaborative environment can also improve the speed of analysis as analysis development moves from an ad-hoc approach to one that follows a more standardized routine. This will also help to improve institutional knowledge of the analysis development process, making it easier to integrate new staff and maintain stability after a personnel transition.

Education agencies could develop more accurate, high-quality, and reproducible analyses, but it order to do so, they need to deliberately build a strong collaborative culture among their analysis developers.