‘Data-driven’ in Michigan

How many 10,000s are there in 200,000?

Russ Stolberg, eighth-grade science teacher at Olivet Middle School, leads students through an exercise on calculating atomic mass. Stolberg can find out quickly if students have grasped a concept by posing a question, offering several answers and asking students to “vote” for the correct answer using handheld clickers.

When math teacher Jennifer Ball included that question on a test earlier this year, she knew not every fourth-grader would choose the correct answer. (It’s 20). But she also knew that wrong answers had the potential to tell her as much as right ones.

So, as conventional and charter public school teachers across Michigan are increasingly doing, Ball loaded the results into a database and used a computer program to find out which wrong answers were selected most often and by whom. Using the same program, Ball also could compare her students’ answers with similar questions on past Michigan Educational Assessment Program tests, determine if her test reflects what the state says should be taught in fourth-grade math and, in some cases, compare her results with those of other math teachers.

Doing each of those analyses individually for dozens of students would overwhelm a teacher, but having access to a warehouse of data and the software to manage it makes the difference, according to Ball, a teacher at Olivet Middle School.

"It’s amazing to me to think I have that system available," she said. "It really brings forth patterns and trends."

When teachers get together to analyze those patterns, it can lead to insight and improvement at the student, classroom and building level, teachers and administrators told Michigan Education Report.

Like Olivet, other schools also are adopting "data-driven decision making," a process of digging into assessment results to find out exactly what students know and when they know it, and hopefully to shed light on which students haven’t grasped certain concepts and why.

"The deeper we dig into things, I think the more we are learning," teacher Kristina Priesman said of the math team at Olivet Middle School, which currently is focusing on improving instruction in fractions. "We can actually focus on the specific areas of fractions that students are struggling with and not just spend more time on fractions in general."

Ball and Priesman both participated in a pilot project to foster data-driven decision making throughout their intermediate school district and, eventually, statewide. Funded by a $1.5 million federal grant, the project so far has resulted in creation of a Web site that helps educators analyze Michigan Educational Assessment Program scores, followed up by intense professional development on how to translate their findings into improvement plans.

Educators who participated in the project have established professional learning communities in their own schools — teams of teachers who meet regularly to review data, set goals and even develop lesson plans together, trying out each other’s best practices, according to Rebecca Rocho, assistant superintendent of general services and legislation at the Calhoun Intermediate School District. The Calhoun ISD is leading the grant project in consortium with the Macomb and Shiawassee intermediate districts.

"It’s those conversations that this project is really about," Rocho said. "We’re trying to break down those isolated classrooms and get teachers talking."

Schools have always collected volumes of academic and demographic data about students, Rocho said, but haven’t necessarily used it to drive school improvement. Some information was stored in separate databases that didn’t "talk" to each other. Other data was accessible, but mainly to tech-savvy teachers who understood how to query it. And given the volume of data, it could be hard to tell which facts and figures were important.

"Data alone is often not meaningful," Rocho said. "As you look at data, you need to know what the questions you should be answering are."

Collaborating with the Center for Educational Performance and Information in the Office of the State Budget, and with local educators, the Calhoun ISD identified six key questions to put to MEAP data. Those questions are now the basis of Data for Student Success, a Web-based program that educators can use as a tool to analyze their MEAP scores in more detail than many have done in the past.

The program goes beyond the typical breakdown of students into proficiency levels, and adds reports on how many students are "near proficient" as well as proficient on specific concepts within each subject. For example, educators can quickly learn how well their students read narrative text or calculate length and area. The system can report on individual students or cohorts of students over time, and academic records can be compared to demographic information like the student’s age, ethnicity and attendance record.

Now in the second year of a three-year development and rollout plan, a final version of Data for Student Success is expected to be available to all teachers and administrators in 2008-2009.

While the Data for Student Success model focuses on MEAP scores, some schools also are building local data warehouses to track each teacher’s classroom assessments throughout the year. A number of districts now ask all educators who teach the same subject in the same grade to give the same quarterly or year-end assessments so the results can be compared. They also keep records on what’s been tried with struggling students in the past — tutoring, after-school study groups — and which approaches resulted in academic improvement.

Being able to analyze results as assessments are given, rather than once-a-year MEAP scores, is a big advantage, Rocho said.

"The really important data is what is going on with a student right now," she said.

Some schools also are turning to more frequent tests. Olivet third- through 10th-graders take the Northwest Evaluation Association’s Measure of Academic Progress test at least once a year. Students at National Heritage Academies, a network of 55 public charter schools in six states, including 35 in Michigan, take it three times a year. NWEA is a nonprofit association based in Oregon that offers educational testing services, data analysis and other academic resources.

At South Arbor Charter Academy, an NHA school in Ypsilanti, students take the computerized NWEA exam in language arts, reading and math. The results show not only where South Arbor students currently stand compared to their peers nationwide, but also predict how much academic growth that student should be able to achieve in a given year. The results are broken down into subcategories which relate to specific academic skills.

"We’re able to use that to drive instruction," Principal Tim DiLaura said. "Teachers can track data and watch students progress from test to test. We’re able to calculate the percent of growth a student has achieved."

Knowing the students’ strengths and weaknesses early in the school year — teachers receive a test report within 24 hours — helps them target instruction, he said.

"More and more schools are realizing that we need to manage achievement," DiLaura said. "We need to know where we’re starting at."

At Walters Elementary School in Marshall Public Schools, Principal Susan Townsend schedules regular "data checkpoint" meetings with each teacher to talk about how students did on recent assessments and to discuss improvement plans. She makes time for teachers to prepare data reports by taking over each classroom for several hours a month.

Teacher collaboration is the key to using data well, said Principal Mark Bensinger at Olivet Middle School. Every teacher at his school is now a member of a Collaborative Action Team that meets monthly to review data and set specific goals in a given subject.

The team process has brought teachers together, he and others said.

"The culture and climate of our school is not just a little better. The change is incredible," Bensinger said. "Implementation of professional learning communities has allowed us to develop a culture of mutual accountability and ownership for decisions."

"Nothing we have done is incredibly technically difficult," he added. "Once we overcame some of our traditional views of staff meetings … things started clicking."

Similarly, South Arbor Principal DiLaura said that once the data points out strengths and weaknesses, "That’s where we want teachers to bring their craft to the challenge."

But the ability to analyze data apparently comes with its own achievement gaps among Michigan schools.

In a presentation to the state board of education in November, Linda Hecker of the Michigan Department of Education said that administrators in some underperforming schools don’t understand achievement data.

"Auditors found that many schools don’t have any idea why they didn’t make AYP (adequate yearly progress), so they can’t focus as well on improving themselves," said Hecker. She added later, "They got their (MEAP) data, but sometimes they weren’t skilled in reading the data. Sometimes there were three or four subgroups in different areas and they didn’t understand how to put those pieces together."

The state hopes that intermediate districts will lead the way in serving as data warehouses and offering professional development on how to use data, as several already do, according to Bruce Umpstead, director of the Office of Educational Technology and Data Coordination at the Michigan Department of Education.

"We are primarily looking to our ISD partners to support local districts," Umpstead said, responding by e-mail to questions from Michigan Education Report. School districts aren’t required to use data-based decision making in setting school improvement goals, but "state and federal performance requirements are making the effective use of data imperative," he wrote.

No Child Left Behind legislation has led school districts nationwide not only to test students every year, but to develop data systems to track individual student performance over time, noted Margaret Ropp, director of CEPI and Michigan’s representative to the National Center for Education Statistics.

"We’ve always been getting data in," Ropp said, but now the emphasis is on "getting data back out to local districts securely and longitudinally."

Though Data for Student Success focuses on K-12 education, Ropp said supporters of data-driven decision making nationally see a need to link educational data from preschool through college. A "P-20" system, as it’s called, would point out such things as how many students from a given high school needed remedial coursework in college, or whether preschool attendance is related to reading scores.

Struggling students are not the only ones to benefit from such programs, noted Townsend, the principal at Walters Elementary. Data analysis also points out students who need more challenge.

"So often when we look at improvement, it’s for the lower end," she said. "These warehouses are wonderful. You have more time to get to the nitty-gritty."

That’s the challenge facing Russ Stolberg and other science teachers at Olivet Middle School, where 92 percent of eighth-grade students tested proficient in science in the fall of 2007.

Stolberg analyzes his students’ performance across the MEAP, looking for clues to strengths and weaknesses that might affect science scores. Learning that some students didn’t do well at reading informative text, he and his colleagues decided to add a timed reading — reading a scientific article and answering questions about it — to one of their own classroom assessments.

"We really have to fine tune our approaches," Stolberg said.

#####

Lorie Shane is the managing editor of the Michigan Education Report, the Mackinac Center’s quarterly education policy journal. Permission to reprint in whole or in part is hereby granted, provided that Michigan Education Report is properly cited.