Devin Coldewey* says Google’s medical researchers were humbled when an AI screening tool fell well short of expectations in real-life testing.
Photo: Google
AI is frequently cited as a miracle worker in medicine, especially in screening processes, where machine learning models boast expert-level skills in detecting problems.
But like so many technologies, it’s one thing to succeed in the lab, quite another to do so in real life — as Google researchers learned in a humbling test at clinics in rural Thailand.
Google Health created a deep learning system that looks at images of the eye for evidence of diabetic retinopathy (DR), a leading cause of vision loss around the world.
But despite high theoretical accuracy, the tool proved impractical in real-world testing, frustrating both patients and nurses with inconsistent results and a general lack of harmony with on-the-ground practices.
It must be said at the outset that although the lessons learned here were hard, it’s a necessary and responsible step to perform this kind of testing, and it’s commendable that Google published these less than flattering results publicly.
It’s also clear from their documentation that the team has already taken the results to heart.
The research paper documents the deployment of a tool meant to augment the existing process by which patients at several clinics in Thailand are screened for DR.
Essentially nurses take images of diabetic patients’ eyes (a “fundus photo”), and send them in batches to ophthalmologists, who evaluate them and return results … usually four–five weeks later.
The Google system was intended to provide ophthalmologist-like expertise in seconds.
In internal tests, it identified degrees of DR with 90 per cent accuracy; the nurses could then make a preliminary recommendation for referral or further testing in a minute instead of a month.
Sounds great — in theory.
But that theory fell apart as soon as the study authors hit the ground.
As the study describes it: “We observed a high degree of variation in the eye-screening process across the 11 clinics in our study.”
“The processes of capturing and grading images were consistent across clinics, but nurses had a large degree of autonomy on how they organised the screening workflow, and different resources were available at each clinic.”
“The setting and locations where eye screenings took place were also highly varied across clinics.”
“Only two clinics had a dedicated screening room that could be darkened to ensure patients’ pupils were large enough to take a high-quality fundus photo.”
The variety of conditions and processes resulted in images sent to the server not being up to the algorithm’s high standards.
“The deep learning system has stringent guidelines regarding the images it will assess,” the study said.
“If an image has a bit of blur or a dark area, for instance, the system will reject it, even if it could make a strong prediction.”
“The system’s high standards for image quality is at odds with the consistency and quality of images that the nurses were routinely capturing under the constraints of the clinic, and this mismatch caused frustration and added work.”
Images with obvious DR but poor quality would be refused by the system, complicating and extending the process.
And that’s when they could get them uploaded to the system in the first place.
“On a strong internet connection, these results appear within a few seconds,” the study said.
“However, the clinics in our study often experienced slower and less reliable connections.”
“In one clinic, the internet went out for a period of two hours … reducing the number of patients screened from 200 to only 100.”
Even the best-case scenario had unforeseen consequences.
Patients were not prepared for an instant evaluation and setting up a follow-up appointment immediately after sending the image.
“As a result of the prospective study protocol design, and potentially needing to make on-the-spot plans to visit the referral hospital, we observed nurses at clinics 4 and 5 dissuading patients from participating in the prospective study, for fear that it would cause unnecessary hardship,” the study said.
As one of those nurses put it: “[Patients] are not concerned with accuracy, but how the experience will be — will it waste my time if I have to go to the hospital?”
“Some people aren’t ready to go so won’t join the research; 40–50 per cent don’t join because they think they have to go to the hospital.”
It’s not all bad news, of course.
The problem is not that AI has nothing to offer a crowded Thai clinic, but that the solution needs to be tailored to the problem and the place.
The instantaneous, easily understood automatic evaluation was enjoyed by patients and nurses alike when it worked well, sometimes helping make the case that this was a serious problem that had to be addressed soon.
And, of course, the primary benefit of reducing dependence on a severely limited resource (local ophthalmologists) is potentially transformative.
But the study authors seemed clear-eyed in their evaluation of this premature and partial application of their AI system.
As they put it: “When introducing new technologies, planners, policymakers, and technology designers did not account for the dynamic and emergent nature of issues arising in complex healthcare programs.”
“The authors argue that attending to people — their motivations, values, professional identities, and the current norms and routines that shape their work — is vital when planning deployments.”
The paper is well worth reading both as a primer in how AI tools are meant to work in clinical environments and what obstacles are faced — by the technology and those meant to adopt it.
* Devin Coldewey is a Seattle-based writer and photographer. His website is coldewey.cc.
This article first appeared at techcrunch.com
