Across nearly every metric — grade level, test subject and both the gender and race of students — the average scores of this spring’s standardized exams dropped statewide from last year.
The lower results were largely expected after the 2019 testing window was marked by large-scale computer glitches and interruptions that came as Utah worked with a new testing company. Despite that, the state announced Thursday, it will — for now — move forward with using the scores as usual.
The dips were minor enough, said Darin Nielsen, assistant superintendent of student learning, that the results from the RISE tests overall are still considered statistically valid. (The deviation from 2018 data varies but is generally no lower than 0.05.) That determination came after three separate analyses of the numbers, as well as a state audit.
Still, as a precaution, the Utah Board of Education will ask the Legislature for flexibility in using the results — including a potential reprieve from issuing school grades that are based on them.
“There’s certainly an uneasy feeling,” Nielsen said. “But there’s been extensive work to try to determine the impacts on student learning. And everything looks OK.”
Here’s what parents, students and teachers need to know about the decision.
What are the test results used for?
Students’ test scores are used for a handful of what the Utah Board of Education calls “accountability measures.” Perhaps the biggest among those is assigning annual school grades.
By law, each year the state board is required to analyze every school’s performance and give out a letter grade A through F. Those rankings are based largely on how well students do on standardized tests. There are also points for improvements in their scores.
Those grades are then used to determine, in part, which schools are failing and may need to be closed.
Rich Nye, the superintendent of Ogden City School District, told the state board during its monthly meeting Thursday that it didn’t seem fair to use the results for grades given the major disruptions. At least two of his schools had significant problems with testing.
“We know that the RISE data plays a big role, a critical role,” he said. “And we’re worried.”
School grades typically come out in early December, but that will likely be pushed back to January.
Many teachers are also evaluated each year based on how well their students do. An educator could be required to change lesson plans, or could be reassigned, reprimanded or fired for low scores. And several have expressed concerns over those consequences to The Salt Lake Tribune.
(Rick Egan | The Salt Lake Tribune) A student practices a year-end test with his class at Elk Run Elementary school on Wednesday, April 22, 2015.
The scores also are used to track student achievement. Those kids who score well, along with other accolades, can be bumped up a grade level. Those who don’t may be held back.
What would happen if they didn’t use the results?
Annual testing is required by federal law in grades three through eight (as well as at least once in high school). The exams focus on language arts, writing, science and math.
If the state didn’t have usable results it would likely be breaking the law and could lose federal education funding — including money for low-income Title I programs.
Utah would also have to apply for a waiver from the Every Student Succeeds Act that mandates at least 95% of students participate in the tests. Too many students opting out of the exam or scores that are invalid can cause complications and result in fines.
This year, participation in the tests in Utah went up to 94.75% — higher than it’s been for the past five years (in 2018 it was 91.95% and the state had to count those who didn’t take the tests as zero scores). But if the state invalidated results completely, Utah wouldn’t meet the required bar.
“At least the participation is moving in the right way,” said Scott Marion, executive director of the Center for Assessment, a third-party company that helped Utah review its data this year. “The more data that we have in the system, the more opportunity for validity.”
Having no scores would also require the Legislature to allow the Utah Board of Education to skip assigning school grades.
My school’s score is down. What does that mean?
Schools or districts that were particularly impacted by disruptions and glitches will not be allowed to appeal the use of their scores, but the board voted Thursday to work with legislators, overall, on other possible remedies and flexibility and maybe suspending school grades. And it will be adding a disclosure to the website for school grades noting that testing disruptions statewide may affect the letter assigned.
Marion said that his company, as well as the state, thoroughly studied the data “as closely as we could and from as many different angles as we could.” His conclusion that using the results is fair is based on the numbers as a whole and by averages. Individual schools and school districts, however, may see lower scores.
“Kids are pretty resilient,” he said. “This is what we’ve seen in state after state. It’s not good to have this and it’s a headache for schools, but the reality is, we didn’t see a large effect.”
(Rick Egan | The Salt Lake Tribune) Terry Shoemaker, executive director of the Utah School Superintendents Association, makes a comment during a House Government Operations Standing Committee meeting on Thursday, February 1, 2018.
Some challenged that portrayal. “I think it would be vastly inaccurate to say that this testing session didn’t impact our teachers and our students and our schools,” said Terry Shoemaker, executive director of the Utah School Superintendents Association. “The confidence level has been fairly negative. We don’t feel like this has been a fair session for our students.”
Why are there problems with the data?
In February 2018, the state signed a contract with a new company — Questar Assessment Inc. — to conduct its year-end tests. Despite concerns reported in other states, Utah Board of Education members approved a 10-year, $44 million deal as part of an effort to rebrand the annual exam, give it a new name, RISE, and encourage more students to take it.
But by the end of April this year, Utah had experienced five major testing interruptions. The outages here delayed more than 18,000 public school students in completing their assessments in April and May. For one day, no one was able to take a science exam. On at least four others, testing was stopped entirely for some school districts. The window to complete the assessments had to be expanded into June.
“There were some students who because of problems with the delivery system were not able to test at all,” Marion said. “At least 1,000 tests in one school were inadvertently reset by Questar.”
According to a state audit released last month, 3,546 exams have come back with missing scores.
That report raised serious questions about the validity of the results and suggested that had Utah not decided to switch from previous testing vendor, American Institutes for Research, it’s possible the glitches could have been avoided.
The review suggests the state should have better weighed and understood the costs of switching companies — when there was likely no pressing need to do so.
What do we expect to happen next year?
In response to criticism — and questions about why the state Board of Education signed a deal with a company that has a history of similar malfunctions, stolen data and cyberattacks in other states — members voted in June to cancel its contract with Questar.
According to the contract, the state can charge the company up to $50,000 each day there was a major disruption. That would amount to about $250,000.
Since then, the board also announced it would return to using AIR to conduct the exams. The contract will be for three years and $21 million. So next year, students will be using a familiar platform once again. And testing should return to normal.
