Using practical search strategies to identify health apps: A case study with smoking cessation
Using practical search strategies to identify health apps: A case study with smoking cessation
Patric Gibbons, MS4, Edwin D. Boudreaux, PhD, Brianna L. Haskins, MS University of Massachusetts Medical School
Journal MTM 7:1:16–23, 2018
Background: Mobile applications (apps) to improve health are becoming increasingly popular among patients. In order to keep pace with demand, clinicians, healthcare organizations, and insurers will need practical methods to efficiently evaluate apps that are of potential benefit to their patients. This case study aims to provide a practical framework for identifying and recommending health apps to patients.
Methods: This paper uses a modified version of a previously published practical method to evaluate and select health apps which target smoking cessation. The modified approach recommended the use of the following strategies: (1) Review the scientific literature, (2) Search app clearinghouse websites, (3) Social media query, (4) Search app stores, (5) Review app descriptions, ratings, and reviews, and (6) Pilot the app using the “Mobile Application Rating Scale” (MARS) to assess key app features. The apps returned from the first three steps were compared against the first 10 apps found through app store searches (steps 4 and 5).
Conclusions: Case study results showed overlap among four apps identified using each of the search and selection strategies. The MARS Application Rating Scale subsequently reduced the final list to three apps that had adequate ratings. Healthcare stakeholders currently lack efficient, practical strategies for evaluating the potential utility of a mobile application. The strategy to rapidly identify potential apps and assess their quality outlined herein is feasible and yielded three potential apps for smoking cessation by triangulating the results.
Introduction
The field of mobile health (mHealth) is flourishing and offers a platform for innovative and convenient methods to address patient health needs. Commercial businesses, health organizations, and app developers have been quick to take advantage of this burgeoning market with over 100,000 health and wellness applications (apps) currently available from the iTunes and Google Play stores.1 If current trends persist, the health app market is expected to be valued at $26 billion by 2017 with over 1.7 billion people having downloaded at least one health app.1,2 The healthcare industry has begun to recognize value in this trend of “do it yourself” healthcare as 86% of physicians now believe that mobile apps will be an important tool in managing the health of their patients over the next five years.3
Keeping pace with the exponential commercial development and dissemination of mobile apps has been a tremendous challenge for healthcare professionals and researchers. Only apps that can be classified as “medical devices,” or those intended to help diagnose, treat, or prevent disease, are under the purview of the FDA, and represent only a small portion of apps available.4 Otherwise, there are currently no regulatory bodies to oversee the development of health-related apps, such as those intended to track and monitor progress in changing health behaviors. Without such regulation, the onus lies with the clinician, healthcare organization, or insurer to offer app recommendations. However, the inexhaustible supply of apps has made it incredibly burdensome for clinicians to isolate clinically beneficial apps and to stay up to date with apps within their field.5
The goal of this paper is to provide direction for clinicians faced with such a dilemma, to explore a practical approach designed to guide healthcare providers in their recommendation of health apps which are effective and evidence based. This paper will focus on health apps which are not intended to diagnose, treat, or prevent disease, as those fall under the regulatory jurisdiction of the FDA.4 The approach to app evaluation and recommendation outlined below will combine a previously published series of app selection strategies proposed by Boudreaux et al. (2014) with an innovative Mobile Application Rating Scale. Together, these tools offer a framework for app selection and recommendation that is systematic, and offers a more objective, comprehensive, and reliable measure of app quality than either tool offers on its own. Following a description of methodology, the approach is demonstrated with an illustrative example using smoking cessation.
Methods
Practical Search and Selection Strategies
Six strategies adapted from Boudreaux et al. (2014) were utilized and include: (1) Review the scientific literature, (2) Search app clearinghouse websites, (3) Conduct a social media query, (4) Search app stores, (5) Review app descriptions, user ratings, and reviews, and (6) Pilot the apps.10,13 For each of the first three strategies, the investigators recorded up to 10 apps that were recommended from each strategy. For the fourth and fifth strategy, the investigators searched the app stores and chose up to 10 from each based on their listing in the app store, which generally aligns with the popularity of the app. This allows for comparison between the five strategies and yields a list of apps that can be piloted. Apps can be subsequently recommended or discouraged after piloting with the Mobile Application Rating Scale (MARS) tool in the sixth strategy, which is the aforementioned new addition to the original Boudreaux et al. (2014) framework.
Strategy 1: Review the Scientific Literature
A PubMed search was conducted using the term “smoking cessation app” targeting more comprehensive reviews such as meta-analyses.
Strategy 2: Search App Clearinghouse Websites
Boudreaux et al. (2014) provides a summary of available clearinghouses. To conduct an application clearinghouse search, three highly recommended clearinghouses were surveyed: The National Center for Telehealth and Technology, iMedicalApps, and the NHS Choices Health Apps Library.6,7,8 The clearinghouses were searched for “smoking cessation app” and “smoking” and a list was compiled of the most reviewed and recommended or certified apps from each of the three clearinghouses.
Strategy 3: Conduct a Social Media Query
The social media query was executed by searching for the term “smoking cessation app” on Twitter©. The top 150-200 tweets were analyzed for whether they were sent by a physician or healthcare organization or if a certain app appeared in more than two tweets, which helped serve as a gauge of popularity.
Strategy 4: Search App Stores
A search of the iTunes store for smoking cessation apps revealed the first 10 apps as of May 2016 (Table 1). To identify these apps, app stores were searched using the terms “quit smoking.” The first 10 relevant or “top” apps per search term, for each app store, were documented. Included apps offered education or assistance with smoking cessation, while apps for clinicians, advertisements, and apps that were pro-smoking were excluded.9
Table 1: Top 10 apps as of May 2016 in the iTunes store
Strategy 5: Review App Descriptions, User Ratings, and Reviews
The apps listed in Table 1 were examined to ensure the app received an approximate rating greater than or equal to 3 stars, and to ensure that no excessive “malfunctions” were reported in the reviews. User ratings can be reviewed for the presence of any “malfunctions” which for the purposes of this paper was defined as any occurrence in the app that hinders usability. For example, apps that frequently crash (close without warning) or freeze will often be reported in user reviews.
Synthesis
After compiling a list of 10 apps from each of the first three strategies, the list was compared against the top 10 apps resulting from the fourth and fifth strategies. This comparison effectively allows the user to determine if there is any overlap between what is most recommended by the scientific literature and other vetting sources, and what is most likely to be available to the patient population. The method described in this paper uses the iTunes store as an example; however, the process is easily replicated for other platforms such as Google Play (Android).
Strategy 6: Pilot the Apps
After identifying top apps using one or more of the strategies outlined, the next step is to pilot the apps. This can be done informally or using a structured process like the Mobile Application Rating System (MARS).10 One dedicated reviewer was used for the MARS by the authors. A second reviewer was used to check the inter-observer reliability using four random apps as a test. Apps are scored on scale from 0 to 5, with 5 being the best possible score. When deciding which apps to advise, MARS authors recommend a score of 4.0 or above. This ensures that the app is held to higher standards and will therefore have a greater chance to address the health needs of patients.
The proposed strategies and their recommended data collection approaches are described below and outlined in Figure 1.
Figure 1: Flow chart summarizing four strategies used in an algorithm to produce a list of potentially useful apps for patients. For each of the first three strategies above, the user will then compare the top apps found against the top 10 apps returned from searching an app store. This comparison allows the user to compare and contrast the most effective apps as viewed by the medical community with the most popular apps among the public. Comparisons should also be made among strategies (e.g. looking for overlap between the social media query and the app clearinghouse search).
Results: Case Study
Strategy 1: Review the Scientific Literature
Our search yielded two meta-analyses, which were subsequently used to generate a list of 10 of the most highly recommended apps (Figure 2).11,12
Figure 2: Comparison of each step in the algorithm with the top 10 apps downloaded from the iTunes store. (A) Two apps were found in both the iTunes store top 10 and the literature search. (B) One app was found in common among the app clearinghouses search and the top 10 iTunes apps. (C) No overlap was found with the social media query.
Strategy 2: Search App Clearinghouse Websites
After searching for smoking cessation apps within The National Center for Telehealth and Technology, iMedicalApps, and the NHS Choices Health Apps Library, a total of 9 apps were discovered, which are shown in Figure 2.
Strategy 3: Conduct a Social Media Query
A social media query yielded four potential apps for smoking cessation (Figure 2).
Strategy 4: Search App Stores
The top 10 iTunes apps generated from this strategy are shown below in Table 1.
Strategy 5: Review App Descriptions, User Ratings, and Reviews
None of the apps listed had any significant malfunctions reported and all had an approximate rating of 3 stars or greater.
Synthesis
As seen in Figures 2 and 3, the comparative list generated from the study design yielded a very small amount of overlap between strategies. From this approach, four apps were found to have overlap for the iPhone platform: SmartQuit, SmokeFree, LIVESTRONG My Quit Coach, and Quit Smoking Now with Max Kirsten.
Figure 3: Overlap of mobile applications (shown in yellow) among all categories of the search and selection strategy.
Strategy 6: Pilot the Apps
The MARS evaluations conducted on these four apps yielded scores shown in Table 2, with SmartQuit having the highest rating.
Table 2: Summary of MARS ratings for the four apps returned from steps 1-3 of the search and selection strategy.
Discussion
Boudreaux and colleagues (2014) outlined several practical search and selection strategies that attempt to balance the need to be practical with the need to find effective apps that have evidence or credible content with expert evaluation to support their use.13 This case study builds upon the work of Boudreaux et al. (2014) by adapting six of their original seven strategies into a modified framework that is practical for physicians and other vested parties to utilize, and tests the modified approach through a tobacco cessation case study.
Strategy 1: Review the Scientific Literature
A practical strategy is to rely on systematic reviews and meta-analyses, when available.13 Search terms should be targeted when reviewing databases such as PubMed and include the word “app.” Providers should be aware of several caveats as they review the published literature. (1) mHealth is constantly evolving and literature can become outdated rather quickly. To account for this, it is best to try to use the most recent literature, if available. (2) Some reviews do not list names of specific apps, so contacting the corresponding author is the best recommendation in that scenario. (3) The search itself, even if focused on meta-analyses and review articles, can be time-consuming; however, collaboration with colleagues and resources such as medical librarians can help reduce the amount of labor involved.
Strategy 2: Search App Clearinghouse Websites
Due to the paucity of scientific literature surrounding some apps, healthcare providers may find it difficult to find evidence for or against the use of a particular app. Fortunately, many app clearinghouses have taken the responsibility of orchestrating systematic reviews of health apps and providing recommendations regarding their use.13
Clearinghouses, although useful for reviewing brief articles about apps and quickly assessing whether they are found acceptable by a content rater, have important limitations worthy of mention. Some clearinghouses do not provide extensive detail regarding the process they use to review apps. Other clearinghouses have individual authors and editors making recommendations, which introduces the potential for bias or influence of subjectivity. Finally, some clearinghouses may be sponsored by commercial entities with potential conflicts of interest.13 Nonetheless, clearinghouses are useful for interested parties and have the potential to become more rigorous and standardized as the field of mHealth continues to expand. The clearinghouses used for this paper were chosen because of their independence from potential financial conflicts of interest and popularity among the medical community.
Strategy 3: Social Media Query
If there is scarce literature or app clearinghouse reviews for a particular condition or health behavior, it is advisable for consumers or physicians to see what colleagues have found useful in the field. Other social media websites may be useful but Twitter is the most practical due to its ease of use for finding trending items and its enormous popularity.
Strategy 4: Search App Stores:
Any strategy for searching app stores is complicated by the fact that there are multiple platforms (e.g. iTunes for the iPhone and Google Play for the Android), which rely on unpublished ranking algorithms. Search terms are therefore important and need to be focused and specific to avoid apps that are irrelevant to the condition of interest. A search term that combines the pathology or health behavior with the desired outcome is best. For example, “smoking cessation” or “alcohol abstinence” or “blood pressure management” are effective search terms.13 Inevitably, some apps will return from this search that are either unrelated to the topic or have no basis in the outcome of interest (e.g. games or advertisements) and they should be removed from consideration.
Strategy 5: Review App Descriptions, User Ratings, and Reviews
Ratings and reviews helped determine the quality of app functionality. User ratings can be helpful in sifting through the top apps found in Strategy 4, determining not only the popularity of the app based on the number of ratings, but also to briefly assess the usability and functionality. Apps with hundreds or thousands of positive ratings may reflect features that make them more interactive, used more frequently, or make them more effective. Apps with low ratings or descriptions that do not seem in line with the desired health outcome can be removed from the consideration lists. Generally, on a five-star system, it is advisable to avoid recommending apps with an average rating of less than three stars.13
Synthesis
Figures 2 and 3 yielded a comparative list from the strategies and shows the four apps that were found to have some overlap across them. The overlap between strategies allows the user to narrow his or her focus. In other words, overlap in the apps returned from strategies 1-4 indicate that those apps are likely to have some combination of evidential support and/or clinical utility among the healthcare community and should be considered for evaluation using the MARS tool.
Strategy 6: Pilot the Apps
As seen from Table 2, it would be advisable to recommend SmartQuit, Smoke Free, or LIVESTRONG due to their high MARS ratings. SmartQuit was given the highest MARS rating due to the fact that it had several randomized controlled trials to support its use, which accumulates more points for the app in the MARS rating system. Overall, the list of 4 apps that was found using strategies 1-5 was able to be narrowed down to a final list of 3 due to the low MARS score for Quit Smoking Now (Table 2).
During the testing of the MARS, it demonstrated excellent inter-observer reliability (ICC = 0.79), which allows the consumer to be confident in being the only reviewer for a particular app.10 With that said, the authors encourage multiple raters, if available, to ensure valuable and satisfactory results. If this strategy of multiple testers is used, the MARS developers recommend that evaluators meet to clarify any terms they find puzzling and to pilot and review ratings together until an appropriate level of inter-rater reliability is achieved. The MARS also offers three videos on YouTube that serve as training modules, which can be useful for the team.
Integration of the MARS tool into the sixth strategy outlined by Boudreaux et al. (2014) aims to provide a more objective measure of a mobile application’s overall quality.10 Furthermore, the MARS yields a quantitative value that can be useful in comparing potential applications and thereby strengthens the confidence of the evaluator in selecting apps for potential use with patients. The two reviewers in this paper showed a high level of inter-observer reliability with an average percent difference in ratings of 3.22%. This gives further credence to the MARS as a reliable and objective measure of an app’s quality and to its utility in the search and selection process described by the authors.
Limitations
The two main limitations of the strategy presented in this paper are time and cost. Providers already pressed for time may find it cumbersome to prowl through literature and app clearinghouses; however, to date there are no other algorithms as rigorous or efficient to determine whether an app is suited to help patients achieve a clinical outcome or behavior change. Cost is another major limitation and should be considered when discussing applications with patients. A subscription to the full, upgraded version of SmartQuit, which was found to be one of the more highly recommended apps in our search, costs $49.99. If patients are unable to afford this particular app, the provider or interested party should recommend the next highest rated app returned from the search strategy (e.g. “Smoke Free” in this case study).
Another limitation is the lack of RCTs that have been conducted on mobile applications outside of SMS services. For smoking cessation, although some of the meta-analyses rate a significant number of apps, the adherence indices of these apps to established national guidelines and clinical practices are incredibly low.14,15 Therefore, it is easy to gather a list of apps from meta-analyses; however, most of these apps are low-quality and will most likely not be of benefit to the patients. The MARS tool helps to adjust for this and can filter apps that are unlikely to be of benefit but still made it through the initial stages of the search and selection process.
Conclusion
Using smoking cessation as a case study, the strategies outlined effectively provided three potential apps for physicians to help motivate their patients to quit smoking. Due to market growth exceeding the pace of scientific scrutiny and a lack of regulatory bodies to examine mHealth applications that fall outside the jurisdiction of the FDA, concerns remain about the validity, efficacy, and safety of many apps. Moreover, physicians have few if any tools to help guide them through the process of validating an application. This paper outlines a more robust methodology for healthcare professionals to conveniently examine the scope of mobile healthcare technology and help their patients make more informed decisions as to whether mHealth applications are a suitable option for their individual health needs.
References
1. Krisch, Joshua. “Questioning the Value of Health Apps.” The New York Times 16 Mar. 2015, well ed. Web. 25 June 2015.<http://well.blogs.nytimes.com/2015/03/16/healthapps-provide-pictures-if-not-proof-of-health/?_r=0>.
2. research2guidance, Global mobile health market report 2013–2017, 2013. <http://www.research2guidance.com/the-market-for-mhealth-app-services-will-reach-26-billion-by-2017/>.
3. “Http://www.pwc.com/en_US/us/health-industries/top-health-industry-issues/assets/pwc-hri-top-healthcare-issues-2015.pdf.”Health Research Institute: PWC (2014). Web. 25 June 2015. <http://www.pwc.com/en_US/us/health-industries/top-health-industry-issues/assets/pwc-hri-top-healthcare-issues-2015.pdf>.
4. US Department of Health and Human Services, Food and Drug Administration, Mobile Medical Applications: Guidance for Industry and Food and Drug Administration Staff, http://www.fda.gov/downloads/MedicalDevices/DeviceRegulationandGuidance/GuidanceDocuments/UCM263366.pdf September 25, 2013.
5. Powell AC, Landman AB, Bates DW. In search of a few good apps. JAMA Published online March 24, 2014.
6. National Center for Telehealth and Technology. US Department of Defense, 2015. Web. 8 July 2015. <http://t2health.dcoe.mil/>.
7. iMedicalApps. Ed. Iltifat Husain and Satish Misra. Medpage Today, 2015. Web. 8 July 2015. <http://www.imedicalapps.com/>.
8. NHS Choices Health Apps Library. National Health Servce, 2015. Web. 8 July 2015. <http://apps.nhs.uk/>.
9. A Haskins BL, Lesperance D, Gibbons P, Boudreaux ED. A systematic review of smartphone applications for smoking cessation. Translational Behavioral Medicine. 2017;7(2):292–9. 
10. Hides L, et al. 2014, Mobile Application Rating Scale (MARS): A new tool for assessing the quality of health mobile applications, Young and Well Cooperative Research Centre, Melbourne.
11. Abroms LC, Lee Westmaas J, Bontemps-Jones J, Ramani R, Mellerson J. A content analysis of popular smartphone apps for smoking cessation. Am J Prev Med. 2013 Dec;45(6):732–6.
12. Whittaker R, McRobbie H, Bullen C, Borland R, Rodgers A, Gu Y. Mobile phone-based interventions for smoking cessation. Cochrane Database of Systematic Reviews 2012, Issue 11. Art. No.: CD006611. DOI: 10.1002/14651858.CD006611.pub3.
13. Boudreaux ED, Waring ME, Hayes RB, Sadasivam RS, Mullen S, Pagoto S. Evaluating and selecting mobile health apps: strategies for healthcare providers and healthcare organizations. Transl Behav Med. 2014 Dec;4(4):363–71.
14. Hoeppner BB, Hoeppner SS, Seaboyer L, Schick MR, Wu GW, Bergman BG, Kelly JF. How Smart are Smartphone Apps for Smoking Cessation? A Content Analysis. Nicotine Tob Res. 2015 Jun 4.
15. Abroms LC, Lee WJ, Bontemps-Jones J, Ramani R, Mellerson J. A content analysis of popular smartphone apps for smoking cessation. Am J Prev Med 2013 Dec;45(6):732–6.
