Ariel Bogle* says researchers have tricked a Tesla car without hacking its system, and it’s a sign of things to come in the race to fool artificial intelligence.
Photo: JP Valery
A sticker on the road. A pile of salt.
These are things that could cause a semi-automated car to drift.
Using white dots stuck on the tarmac, a recent study from Tencent Keen Security Lab in China pushed a Tesla Model S on to the wrong side of the road.
This is not computer hacking.
At least, not in the darkened-room-and-hoodies sense of the word.
As more of daily life moves not only online but into the orbit of intelligent machines, computer scientists and lawyers are debating a fuzzy line: when are you hacking a computer and when are you simply tricking it?
Tesla said it already fixed the key vulnerability raised by the report, adding that drivers can override autopilot at any point, and “should always be prepared to do so”.
Yet what makes the work so interesting is the researchers didn’t have to alter the car’s code.
They just used its own cameras and sensors, which look for lane markings, against it.
At Harvard University, Ariel Herbert-Voss studies adversarial machine learning — where an attacker uses external signals to force an AI system into making an incorrect prediction, like choosing the wrong lane.
Ms Herbert-Voss grew up hacking computers and doesn’t see much of a distinction, if any at all, between hacking a system and tricking it.
Hackers usually want to make money, she said, or to “cause some general chaos”.
“In most cases it just involves fooling a system somehow, and usually they want to take the path of least resistance.”
“So, if you can fool a car by just having a bunch of stickers on the road, I guarantee you hackers are going do that.”
But what are the police going to do about it?
The law of tricks
The story goes like this.
The US introduced anti-hacking laws after members of former President Ronald Reagan’s administration saw the film War Games, in which a computer almost starts World War III.
The Computer Fraud and Abuse Act, implemented in mid-1980s, made it a Federal crime to hack into a computer system.
But what about tricking an automated system, without bothering to hack it?
Ryan Calo, Co-Director of the University of Washington’s Tech Policy Lab, recently published a paper asking this question: “Is tricking a robot hacking?”
Unlike the traditional understanding of hacking — entering a system, stealing information or changing its code — this threat includes prompting an AI system to make what Mr Calo called “errors of consequence”.
“You’re not doing it by breaking into the system,” he said.
“You’re just understanding how the model works and then influencing it, affecting it, forcing it to do the wrong thing.”
While the results could be just as serious as traditional hacking, Mr Calo and his colleagues are concerned this doesn’t fit neatly within current US regulation.
Australian Federal law is a little more prepared for this grey area, according to Professor Kieran Tranter, who researches law and technology at the Queensland University of Technology.
Our criminal code prohibits not only getting into code and changing it, but also potentially affecting its inputs.
“So arguably, doing adversarial machine learning … or just doing things to confuse the robots could still be covered by the Australian laws,” he said.
While Australia’s criminal law may be broad enough to cover these scenarios, the bigger threat, Dr Tranter said, is the “known unknown”.
“Often the most interesting and malevolent uses of technology are the ones no-one’s ever thought of,” he said.
AI responsibility
It is the “known unknowns” that raise another sticky question: Who is responsible when a system can be fooled?
For now, companies can sometimes be penalised if they fail to secure their systems against malicious hacking.
In Mr Calo’s view, the same ought to apply to systems that are too trickable.
He also wants the laws to be clarified so that researchers and others are not criminalised for pushing the boundaries and testing whether such systems can be fooled.
This is an issue in Australia, where the Government has proposed outlawing the reidentification of anonymised government data.
In some cases, it might be possible to reconstruct the data used to train a machine learning system, by asking it the right questions.
This is potentially a serious invasion of privacy for the people whose data is involved.
We risk “emphasising comfort over understanding our vulnerabilities” if we make laws prohibiting such investigations, said Dr Vanessa Teague, a cryptography expert at the University of Melbourne.
The results might be embarrassing or cause community concern — but they might also be very important.
Ms Herbert-Voss said the Tesla study shows that intelligent systems must be built for robustness: against hackers, certainly, but also hazards as mundane as bad weather.
What if, instead of white stickers, a scattering of de-icing salt dragged a self-driving car into oncoming traffic?
Learning to make mistakes
Apart from manipulating an existing system, researchers are also looking at how AI could be trained or manipulated to make mistakes in the future.
Machine learning tools often rely on large datasets to teach themselves about the world — to distinguish a kerb from a driveway, or from lane markings, for example, they may need to be trained on millions of such images.
But this also provides a vulnerability.
Only remember Tay, Microsoft’s ill-fated chatbot, who was designed to learn by interacting with humans on Twitter. It wasn’t long before she was tweeting “feminism is cancer”.
“There are opportunities for you to inject malicious behaviour into the very training of the algorithm, which then later will perform the way that you, the attacker, wants,” Mr Calo said.
No matter how automated and sophisticated a system is, there will always be ways to exploit it.
“I think a lot of people fall into the trap where they think that if it’s a decision from a machine, it’s infallible,” Ms Herbert-Voss said.
“But humans are trickable and so are machines.”
* Ariel Bogle is the online technology reporter in the ABC RN science unit. She tweets at @arielbogle.
This article first appeared at www.abc.net.au.
