Writing tests that game the system

When tests are involved, one generally thinks that gaming the system means figuring out how to pass the test without having the quality that the test is designed to measure. This is an old and well-known strategy. Now, we have a second-level version of this effect. If one has control over the test, it isn't necessary to figure out how to beat it, just change it.

Who would set up a system in which the test taker is given the authority to re-write the test? Apparently Bush learned something from his early days in which all the tests he faced were re-written for his benefit.

According to a story in the New York Times

In Mississippi, 89 percent of fourth graders performed at or above proficiency on state reading tests, while only 18 percent of fourth graders demonstrated proficiency on the federal test. Oklahoma, North Carolina, Alabama, Georgia, Alaska, Texas and more than a dozen other states all showed students doing far better on their own reading and math tests than on the federal one.

How can that be? It's simple.

The No Child Left Behind law

requires states to participate in the National Assessment, [but] states are allowed to use their own tests in meeting the law's central mandate — that schools increase the percentage of students demonstrating proficiency each year. The law requires 100 percent of the nation's students to reach proficiency — as each state defines it — by 2014. … And because states that fail to raise scores over time face serious sanctions, there is little incentive to make the exams difficult, some educators say.

In other words, the law has the effect built into it. Like most other mechanical systems, it encourages those who must deal with the system to look at how the system works rather than what the system is intended to do. The system tells states to get your scores up or lose money. Since the system also allows states to write the tests that measure whether their scores are up, what else would one expect them to do except write tests that produce results that the system requires of them.

It's clear that this is a direct effect of the fact that the law is written in this way.

G. Gage Kingsbury, director of research at the Northwest Evaluation Association, a nonprofit group that administers tests in 1,500 districts nationwide, said states that set their proficiency standards before No Child Left Behind became law had tended to set them high.

"The idea back then was that we needed to be competitive with nations like Hong Kong and Singapore," he said. "But our research shows that since N.C.L.B. took effect, states have set lower standards."

I was glad to see that Inez Tenenbaum, state superindent of education in South Carolina, whom I supported in her failed race for the Senate last year, did not compromise.

South Carolina is a state that set world-class standards, Mr. Kingsbury said. … "Unfortunately it's put us at a great disadvantage," said Inez M. Tenenbaum. … "We thought other states would be high-minded too, but we were mistaken."

South Carolina's tough exams make it harder for schools there to show the annual testing gains demanded by the federal law.

This year less than half of the state's 1,109 schools met the federal law's benchmark for the percentage of students showing proficiency, a challenge that will get tougher each year. As a result, legislators are pushing to lower the state's proficiency standard, Ms. Tenenbaum said, an idea she opposes.

It's such an obvious and direct effect. If you can't meet the standard, but if you have control over the standard, change the standard.

Where do various factions line up on this issue?

Most corporate leaders favor national testing, said Susan Traiman, a director at the Business Roundtable, a group that represents corporate executives.

Opponents include liberal groups that dislike all standardized testing; the testing industry itself, which has found lucrative profits in writing new exams for all 50 states; and political conservatives who fiercely resist any intrusion on states' rights to control curricula and tests.

The positions of the liberal and conservative groups reflect political positions rather than positions based on this particular issue. That leaves the two business factions. Most businesses want strong testing because they want competent employees. The testing industry doesn't care about the consequences of the tests, they just want to maximize the amount of money they can make.

So here too, what's important is the effect, not the intent. In the case of the Business Roundtable, the effect for them is consistent with the intent, which allows them to take the more ethical position. But being businesses, what really matters is the bottom line — as the position of the testing industry illustrates.