Without actually trying it out, an early critic may have faulted the first automobile for not having a horse to pull it. After all, how else would a vehicle of the day transport a person from one location to another? Horses were a necessary part of the equation back then; the rubric of the day made their participation mandatory.
Such may be the state of mobile health efficacy studies.
Before we go any further, it's worth noting that I applaud the research discussed below for tackling the efficacy of health apps, especially since this study began at a time when mobile health was a largely unknown field discussed as a fringe topic in some healthcare circles. To be sure, the efficacy of mobile health services needs more attention, but I believe it requires a deeper study than the one discussed below.
A recent paper published in the international, peer-reviewed Translational Behavioral Medicine journal found that of the 204 "weight loss" apps available from Apple's AppStore as of September 25, 2009, not a single one adhered to all 13 of the evidence-informed practices for weight loss programs suggested by government agencies at the time. The 13 practices researchers looked for among the apps were at the time common to all of the following governmental agencies: the Centers for Disease Control and Prevention (CDC), National Institutes of Health (NIH), the Food and Drug Administration (FDA), and the US Department of Agriculture (USDA).
The study checked whether the apps adhered to these 13 evidenced-based practices: (1) Provide BMI assessment and explanation. (2) Recommends a diet rich in fruits and vegetables. (3) Recommends certain amount of physical activity each day. (4) Recommends certain amount of water consumption. (5) Encourages user to keep a food diary. (6) Tracks calorie balance of calories in vs. calories out. (7) Recommends a weight loss goal of 1 to 2 pounds a week. (8) Recommends portion size control. (9) Recommends users to read nutrition labels. (10) Provides a means to track weight. (11) Keeps a physical activity journal. (12) Provides a way to plan meals. (13) Provides a way to seek social support.
This study, which was conducted by researchers at Duke University and George Washington University, found that only one app adhered to 12 of these evidence-based practices and 50 apps adhered to exactly none of them. These evidence-based practices, of course, were developed before the creation of smartphone medical apps. They are not best practices for smartphone-based weight loss apps, but rather best practices for weight loss programs in general. Might they still apply to smartphone apps? Sure. Are they the be-all-end-all litmus test for weight loss apps? Of course not.
It's important to note that the researchers never tested these weight loss smartphone apps. They never downloaded them. (We have seen this head-scratching method of study before.) The analysis of the weight loss apps' adherence to these 13 evidence-based practices was based solely on the apps' descriptions in the iTunes AppStore. The researchers acknowledge in their paper that the apps may adhere to more or less of them than their descriptions let on.
If you're going to make a claim that an app is ineffective, at least test it with a patient population first.
It is also frustrating that this study only saw publication last month, yet the research includes apps published more than two years ago. It is frustrating that the researchers do not take into account how efficacy may be affected by the unique qualities the smartphone platform provides -- qualities that are impossibly represented in a evidence-based guideline list conceived before smartphone apps existed.
Should the efficacy of mHealth rest on the shoulders of researchers that never take the time to download the apps they critique? Should the efficacy of apps be subject solely to a study of whether they adhere to evidence-based practices conceived prior to their platform existed?
Do cars need horses to pull them?