Face Recognition for Pills

AI tools can abstract away manual tasks, improving the experience for the user and increasing productivity.

Thumb default avatar
Mar 05, 2019

In this paper we demonstrate an example of using AI in a healthcare application to abstract away an otherwise manual task.  Our technology shows how pill identification can be performed using a ubiquitous mobile-device camera, rather than requiring the user to manually enter appearance characteristics (e.g. shape, color, imprints).  This technology can link other pertinent information about the medication, such as dosage, interactions and contraindications.

The motivation for this work came out of conversations with pharmacists, EMTs, firefighters, home-caretakers, nurses, and physicians we learned about challenges in patient care and how technology can address these with improved outcomes in the quadruple aim.  We heard the frequent problem of rapid identification of medication and lack of technology tools to perform the task.  EMTs described medication identification as one of the single most important tasks that they perform daily.  Patients and their families frequently do not know what medications they are taking, yet this is life-critical information to treat the patient before transporting.  EMTs look for important classes of drugs, such as blood thinners, and drug interactions between pills.  At a time of alarming increase in prescription medication usage, the need to quickly and accurately identify the over 4,000 unique pill appearances is becoming increasingly frequent for patients and providers alike. These small gains in efficiency add up. If a worker saves 10 minutes a day in a five-day work week, this is over 40 hours per year.  The time of highly trained medical workers can be reallocated to patient care activities.

With better understanding of care teams and workflows, we set about building a computer-vision based tool to query pills from mobile devices, using data from a previous NIH pill identification competition.  There are multiple challenges involved in identifying medications based on the pill appearance.  First, you need to detect the target pills contained in the image. The main challenge comes from these sources’ variability given the unstructured camera angle, and varying pill orientation and lighting conditions. To overcome them, we use a type of deep learning known as a Convolutional Neural Network (CNN). The idea is to extract discriminative features that are invariant to the sources of variation mentioned above. An additional source of difficulty in this healthcare application is the predominate number of round white pills that are nearly only distinguishable by the imprint. Applying approaches based on learning similarities, which are successfully used in other applications such as face identification, much more difficult due to the more subtle difference. The previous published methods also use a CNN approach but optimize a similarity loss function. The existence of these very similar pills causes a mismatch with the intrinsic goals of similarity-based system. This context is what directed us to using a classification approach, which turned out to have higher performance, and that is presented in our paper.

Medium default avatar

Jessica Lundin

Engineering Manager, Microsoft Research


Thumb download
Yehuda Idan Ratinsky about 2 months ago

That's a nice startup idea that is quite easy to implement. However, is there a reason the title is "Face Recognition for Pills"? as opposed to "Image Recognition for Pills"? 

Face Recognition is normally used when referring to tracking down a person (whom has a face...) with a camera...  Image recognition is the general technology term... I thought maybe 'Face' for the surface area of the pill? but that wouldn't be so accurate because a pill is a 3D object, many of which have the unique identifying imprint only on one side of an oval shaped pill...