After five years of working with a standardized testing company, the state decided last year to switch to a different provider with the hope that it might help rebrand the annual exam, give it a new name and encourage more students to take it.
That didn’t pan out.
During its inaugural rollout this spring, the new vendor, Questar, had computer glitches that delayed thousands of students in taking their assessments. Teachers reported that the scores that showed on computer screens didn’t match the reports they later downloaded. And administrators have said they’re worried the results will be unreliable.
Now — with more doubts about the tests than before — Utah education leaders intend to go back to the previous provider.
On Tuesday, Darin Nielsen, the state's assistant superintendent of student learning, announced that the Utah Board of Education will return to using American Institutes for Research (AIR) to conduct the exams. The contract will be for three years and $21 million.
“We’re fortunate to have that as an option,” Nielsen said during a presentation to lawmakers on the Public Education Appropriations Subcommittee. “They’re the only vendor that’s available on this short timeframe that has the capability.”
The announcement comes less than two weeks after the state school board voted to terminate its $44 million, 10-year contract with Questar over the interruptions with the RISE test. And it was made, in part, as the company has responded to similar problems with testing in other states, too.
In some, the tests were marked by cyberattacks. One school district threw out its results because the software was so unreliable. In another, all of the students had to start over when the programming shut down and didn’t save their responses.
Many of those issues came up before Utah signed its contract with the company. Nielsen was asked to talk to lawmakers Tuesday to discuss why the state chose to move ahead anyway.
“The board made a decision based on what the vendor promised to do,” he responded.
Nielsen also detailed several more issues with Questar that were not previously made public — and happened in the lead-up to spring testing this year. After the state school board signed the contract with the company in February 2018, the company missed several deadlines to provide materials, he said.
On Oct. 1, it failed to deliver assessment tools. On Nov. 1, it didn’t have winter tests ready. On Nov. 27, it had issues developing the student rosters. On Jan. 9, it reported problems with putting IDs on files. On March 19 of this year, it failed to go live with spring testing as scheduled.
“Our level of concern started increasing,” Nielsen acknowledged.
By the end of April, Utah experienced five interruptions. The outages here delayed more than 18,000 public school students in completing their assessments this April and May. For one day, no one was able to take a science exam. On at least four others, testing was stopped entirely for some school districts. The window to complete the assessments had to be extended into June.
Because computers froze when students were in the middle of tests, no other student could log on to the same device. And Questar couldn’t fix the problem without rebooting the systems — something that took 24 hours. Each glitch, then, pushed back testing another day.
Some were not able to recover their work. Others were not able to finish. In a few schools, Nielsen said, even students who submitted their answers were later told there was no record of them taking the test.
That happened, too, one day after state officials met with the company’s president to express concerns.
“This whole situation is unfortunate,” responded state Sen. Lincoln Fillmore, R-South Jordan. “If people could go back and do it differently, I’m sure they would.”
Lawmakers criticized the state school board for not doing more research on Questar beforehand. Nielsen said the state’s mandated process stopped them from doing that. But several on the subcommittee responded that they didn’t understand or didn’t know about any provision that required the school board not to do a Google search for a business before signing a contract with it.
“I think the frustration with this is clearly it was a mistake to hire this company,” said Rep. Robert Spendlove, R-Sandy.
In the disclosure section of the Questar contract, where the bidding vendor was required to detail any accusations of poor performance, there were three separate incidents listed. The oldest dated to 2014. The issues involved Educational Testing Services (ETS) software — the same one used in Utah.
In Texas, Questar said, its parent company, ETS, paid $5 million for “the disruption of testing and reporting schedules.” In California, ETS lost more than $3 million for not providing the correct testing materials and not delivering scores in the set timeframe. With AP and SAT exams ETS offered through the College Board, it had “minor to moderate failures” in administering the tests in the schools where it had contracts.
Nielsen said the other bidders also reported issues. Lawmakers contended, though, that these seemed rather large to overlook — and have far-reaching consequences.
“We can’t change what’s happened in the past. But I’m concerned with what we’re doing now,” added Rep. Marie Poulson, D-Cottonwood Heights. “This is potentially devastating to schools.”
Annual testing is required by federal law in grades three through eight (as well as at least once in high school). Those scores are then used to determine which schools get funding, how they’re graded statewide and which ones are struggling enough to possibly merit closure.
The exams focus on language arts, writing, science and math and are used to assess how well students are improving year to year. Results can tell teachers, most important, who is falling below grade level. And a school gets an annual grade based on performance.
“It doesn’t seem fair to those schools to get a grade,” Poulson said.
Nielsen said the school board will receive the data on scores on July 15. A third party will examine them and report back by Aug. 1 on the validity of the numbers.
“I encourage people to wait and see,” he added, noting that “we won’t impose invalid results” on anyone.
Questar, which is based in Minnesota, previously told The Salt Lake Tribune after its contract was cancelled: "While we regret this decision, Questar Assessment Inc. is going to do everything possible to ensure a smooth transition.”
The board had said it would terminate the agreement “rather than risk continued interruptions." Meanwhile, it can still seek damages from Questar for the glitches. According to the contract, the state can charge the company up to $50,000 each day there was a major disruption. That would amount to about $250,000.
The state switched vendors last year after previously contracting with AIR to conduct what was then called the SAGE test.
SAGE had failed to gain traction in Utah since it was implemented in 2013. More and more parents each year opted their students out of the test. That is allowed under state law, though the school board has said it undermines the accuracy of using the exams for accountability rankings.
Most board members had hoped the new RISE tests would encourage more parents to have their kids take the assessment. Roughly a million tests are completed in Utah each year (with many students taking more than one based on subjects).
Now, the state will go back to using AIR, which had bid with Questar for the new contract and was declined.
In its proposal, AIR wrote of its experience: “Utah has tested 100% online with virtually no disruptions for multiple years.” It was late once in giving the state its scores and paid $200,000 for that mistake. But it also helped Utah lease its questions to others, which earned the state $15 million.
Rep. Dan Johnson, R-Logan, said returning to the proven company seems like a good idea. He said: “We need to develop confidence down the road.”
