Many education reformers today like to use student scores from standardized exams, such as the Michigan Educational Assessment Program (MEAP) test, to evaluate schools' progress in improving education. Tests certainly can be useful tools of measurement; however, there are several reasons why the MEAP is of questionable effectiveness when it comes to weighing how successfully school districts are meeting the needs of students.
Gary Wolfram, Ph.D., former member of the Michigan State Board of Education, is George Munson Professor of Political Economy at Hillsdale College.
The primary reason to be skeptical is that the whole construct of MEAP-driven reform is based on a flawed paradigm and a flawed system. Ninety percent of K-12 educational services in America are provided through government-funded schools. What we call the "public school system" is actually a political system that itself determines what is taught, how it is taught, and how well it is taught, without much reference to the needs and desires of the parents and kids who use the system. Unlike services that are delivered through a voluntary market process, education as a government monopoly yields no indicators to provide a direct measurement of how well schools are serving their customers.
Consider this analogy. If Farmer Jack's is doing a poor job of providing retail food services to its customers, we do not need to test Farmer Jack's employees or the produce they sell to find this out. Farmer Jack's will simply lose customers to Spartan stores or other supermarkets. In fact, even if Farmer Jack's is doing a good job, if it is not doing as good a job as Spartan stores, we will know this by observing where customers decide to get their food.
This process does not work for K-12 schools because the vast majority of them are owned and run by the government. Since there are large barriers to entering the K-12 school market-most conspicuously the fact that citizens have to pay for government schools whether they use them or not-customers have very limited choices, a fact that makes it difficult for them to switch to other providers if the product is not good. Since we cannot observe customer response, how can we tell if the $14 billion spent annually on Michigan's government-produced K-12 education is put to good use? The answer, so far, has been testing.
Using tests to measure school quality sounds reasonable enough-until we think about how the testing is done. The government produces the tests, so by definition the tests will be designed and scored through a political process. Now suppose for a moment the tests are completely reliable in testing the knowledge of students, something that is far from certain. There is no mechanism to ensure that what knowledge the bureaucrats design the test to measure will be what parents want their children to know. In fact, it is fair to say that close to zero percent of parents have ever seen one question on the MEAP test. So what the MEAP system actually tests is what the legislature or bureaucrats wish to have tested.
But suppose the MEAP does accurately test what the bureaucrats want children to know and that what the bureaucrats want children to know is exactly what parents want the children to know. The MEAP still wouldn't be a good measure of school quality. Take the example of charter school test scores. A report on charter schools from the Hudson Institute bases its criticism of the use of tests to compare charter schools with ordinary public schools on the fact that "the (MEAP) data reveal as much about where charter-school students are coming from as about how they're doing once enrolled." Since charter schools are so new, prior schooling experience of students has a strong effect on the early test scores of charter schools.
Using MEAP data to compare different school districts can also be misleading since the MEAP scores are "cut scores." That is, if my raw score is at a certain "cut" level, I will have passed the test satisfactorily. If my raw score is one point lower, I will not have passed the test satisfactorily.
Now imagine two school districts, A and B, each with 100 4th-grade students who took the 4th-grade MEAP reading test. School District A's students all have a raw score one point above the cut score. School District B's students all have raw scores one point below the cut score. While the raw score differential between the two districts would be two points out of perhaps 68 points, or about a 3-percent difference, we would find District A had 100 percent of its students with a satisfactory score, and School District B with 0 percent of its students with a satisfactory score. It is quite possible for the raw score differential between districts to be much different than the cut score differentials that are published by the Michigan Department of Education and used for comparison.
The MEAP test can provide us with rough information about what students know in certain subject areas, and this might be useful information. Studies that compare different state measures of school accountability usually rate Michigan's MEAP tests relatively highly. However, the MEAP is at best a flawed attempt to solve a problem that requires a more fundamental solution. We must change the delivery system of K-12 education to one where parents decide what is best for their children and where entrepreneurs can enter the market and provide quality educational services when the needs of parents are not being met. Only then can we truly provide an intelligent answer to the question, "Are schools improving?"