Surveillance company Banjo could see its Utah business halted for as long as a year as the state auditor begins to review whether its platform is susceptible to bias or privacy concerns.
Civil liberties groups and media outlets began raising the alarm about the secretive company months ago, when Utah Attorney General Sean Reyes and staff were working to secure as many state government contracts for the company as possible. But those agencies only took action on privacy concerns after learning about CEO Damien Patton’s past association with KKK groups, which included involvement in a synagogue shooting.
Since then, all known Utah agencies with Banjo contracts, including the University of Utah, Intermountain Healthcare and local police departments, have suspended their contracts while awaiting a third-party audit. The Attorney General’s Office has since handed off that review to State Auditor John Dougall.
“The timeline is up to him. [Our] office would like it done as soon as possible, but it’s probably going to take months,” said Rich Piatt, speaking as a spokesman on the matter for Reyes. “I think they’ve got a lot to comb through.”
Banjo announced last week it would stop ingesting government information in Utah and indicated its support of an audit.
“All of Banjo’s government and private customers are on pause while the State of Utah conducts its audit," Banjo spokesman Linden Zakula said in a statement. "We believe that any company working with the government should be subject to oversight and [look] forward to the findings of this audit which we believe will demonstrate that Banjo is handing all data according to the contract, is not a surveillance company in any way, and that no bias is in our technology.”
The audit will have two goals, Piatt said.
First, “In light of allegations and revelations about Damien Patton, the audit will help ensure there’s not anti-Semitism or racism in the underlying algorithm,” Piatt said. Second, “It will verify and certify that privacy is protected in the way the company has told us they will protect it.”
The Attorney General’s Office was already planning on doing an audit of Banjo this spring, Piatt said, but the priority level dropped as concerns about the coronavirus pandemic surged. Revelations about Patton’s past renewed upped the urgency for a review.
State contracts with Banjo will remain on pause until the audit is complete, Piatt said.
Dougall, the state auditor, said that process could take months if not a year. His office is currently forming a panel that includes experts from the tech and civil liberties sectors.
“Once we get that put together, we’ll review the status of things and get input on the best way to proceed,” Dougall said. “My office has done audits in the past about data privacy, but algorithm bias is not something we’ve done before.”
The trick will be persuading Banjo to provide enough of its proprietary information to ensure the audit is robust, according to Suresh Venkatasubramanian, a professor at the University of Utah School of Computing with expertise in artificial intelligence and ethics.
“What you have to do, really, is try to break the system, use it to understand what the vulnerabilities are,” Venkatasubramanian said. “This is not going to happen very easily without the cooperation of Banjo.”
And one audit won’t be enough. The state will need to conduct regular, ongoing oversight of Banjo’s platform.
“Systems change over time,” Venkatasubramanian said. “In a year or two years, it might look like a very different system than the one you audited.”
Banjo’s contracts with state and local governments allowed it to ingest UDOT traffic camera feeds, 911 calls, emergency vehicle locations as well as private and public security cameras, all in the name of improving emergency response.
But privacy experts worried the data could be combined with other information to deanonymize personally identifiable information. Hackers aren’t the only concern — other government agencies could access the data to increase state surveillance of individuals without any scrutiny from the public.
Similar apprehensions caused a stir last summer when a Washington Post investigation revealed Utah had opened its driver license database to the Federal Bureau of Investigation and Immigration and Customs Enforcement. Those federal agencies then scanned Utahns’ photos with facial recognition software, including people who had never committed a crime.
“The issues with [Banjo’s] lack of transparency and the way these systems are being used is of concern regardless of who’s running the company,” Venkatasubramanian said, adding, “the point is not that they’re not doing it now, it’s that it’s possible to do it.”
12:49 p.m. May 5: This story was updated to include comment from Banjo.
