Evaluation of an App: STAB-IT™ Staphylococcus aureus Bacteremia Is Terrible

---

Debra A. Goff, PharmD, FCCP¹, Kim Hawksworth, RPh¹, Julie E. Mangino, MD, FSHEA^2,3

¹Department of Pharmacy, The Ohio State University Wexner Medical Center, OH, USA; ²Division of Infectious Diseases in the Department of Internal Medicine, The Ohio State University, OH, USA; ³Department of Clinical Epidemiology, The Ohio State University Wexner Medical Center, OH, USA

Corresponding Author: debbie.goff@osumc.edu

Journal MTM 2:3:15–20, 2013

10.7309/jmtm.2.3.4

---

---

Introduction

Technology is changing the practice of medicine. In the “The Creative Destruction of Medicine: How the Digital Revolution Will Create Better Health Care”, Eric Topol, author and physician, describes how smartphones have radically changed our lives; yet many digital medical innovations (i.e.) mobile applications (apps) lie unused, because much of the medical community’s resistance to change.¹ Apps allow healthcare providers to access information at the point-of-care to enhance decision making. Recent reviews of infectious diseases apps have identified limitations to each app.², ³ Research addressing how physician’s utilize apps in medicine is rapidly growing, however, there is a paucity of data to show that physicians use of apps improve patient outcomes. The Ohio State University Wexner Medical Center (OSUWMC) is a 1,234 bed academic teaching hospital; our purpose is to develop an app for the iPhone/ iPad to educate our health care providers on the optimal management of Staphylococcus aureus bacteremia. This focus was chosen for our first app based on preliminary work assessing our outcomes of patients with Staph aureus bacteremia. Secondary goals include assessing app use and patient outcomes pre and post implementation of this app.

Methods

The OSUWMC Antimicrobial Stewardship Program (ASP) expanded in 2005 to include a multidisciplinary team of infectious diseases (ID) physicians, ID pharmacists, microbiologists, infection preventionists, epidemiologist and data manager. The ASP pharmacists use iPads to provide point-of care education to healthcare providers and patients. The OSUWMC information technology (IT) department provides technical and developmental support for iPhones, and iPads. This study was approved by the Office of Responsible Research Practices Institutional Review Board.

A web-based educational app called STAB-IT™ (S. aureus bacteremia is terrible) was developed to optimize the management of S. aureus bacteremia at OSUWMC after a pilot study identified opportunities for improving care.⁴ STAB-IT™ provides five point-of-care evidence based information categories regarding OSUWMC specific epidemiology, microbiology, diagnosis, management, antimicrobial options in addition to two other categories, patient education and case studies with an online quiz to provide immediate feedback. (Figure 1). Test questions developed by senior attending infectious disease physicians and reviewed by second year internal medicine residents, were designed to assess the optimal management of a patient with S. aureus bacteremia at OSUWMC. Examples of select OSUWMC data in STAB IT™ not found in other online resources (Up To Date^®), apps, or textbooks, included:

1. Epidemiology: prevalence of S. aureus at OSUWMC and our percent methicillin resistant S. aureus (MRSA), OSUWMC hand hygiene compliance program
2. Microbiology: distribution of vancomycin minimum inhibitory concentration (MIC), antibiogram data, description and link to OSUWMC publication⁵ on rapid polymerase chain reaction (PCR) Xpert^® MRSA/SA blood culture test, summary of vancomycin intermediate S. aureus and daptomycin resistant S. aureus isolates at OSUWMC
3. Diagnosis and Management: OSUWMC evidence based definitions of uncomplicated versus complicated S. aureus bacteremia, criteria for repeat blood cultures and cardiac imaging with a link to Infectious Disease Society of America (IDSA) clinical practice guideline for the Treatment of Methicillin-Resistant S. aureus Infections in Adults and Children⁶
4. Antibiotics: formulary guidelines with restrictions and prior ASP authorizations needed.
5. The patient education section includes medical illustrations providing visual diagrams on antimicrobial mechanism of action, endocarditis, drive-line infections, out-patient antimicrobial therapy, and proper hand hygiene.

Figure 1: Screen shots from STAB-IT™

STAB-IT™ was implemented May 31, 2011 and could be accessed on the hospital’s secure wired and wireless networks by an array of web enabled devices and desktop computers. During June-July 2011 the authors provided STAB-IT™ education to infectious diseases (ID) attending physicians and fellows and 76 clinical pharmacists who round with physician teams. After the implementation of STAB IT™, ASP pharmacists continued to refer physicians managing patients with S. aureus bacteremia to STAB-IT™ if an ID consult was not obtained. The authors provided a continuing medical education (CME) approved lecture to educate and demonstrate how to navigate STAB IT™ to internal medicine house-staff during the introductory lecture series required for all new physicians in August 2011, after a baseline pre-test assessment of knowledge on S. aureus bacteremia. Residents navigated the STAB-IT™ app while information was presented from the five categories within the app with emphasis on the information unique to patient care at OSUWMC. Learning outcome was tested by the pre and then post STAB-IT™ test. The internal medicine house-staff filled out evaluation forms to provide feedback to the authors.

To assess the effect of the availability of the STAB-IT™ app to improve the management of patients with S. aureus bacteremia, STAB-IT™ use was prospectively monitored and patient management and outcome was retrospectively collected. The utilization of STAB-IT™ was tracked using Google web analytics. Data collected included the number of visits to the STAB-IT™ web site, pages viewed per visit, and average visit duration. To assess the outcomes, patients eligible for assessment included all hospitalized adult patients admitted from September 1, 2010 to May 31, 2011 (i.e. pre-STAB-IT™) and Sept 1, 2011 to May 31, 2012 (i.e. post STAB-IT™) with S. aureus bacteremia. Patients were identified through records from the microbiology laboratory and Information Warehouse (IW), a comprehensive data bank specific to OSUWMC. Data collected included age, positive S. aureus blood cultures, first vancomycin trough level, negative blood cultures obtained prior to hospital discharge, ID consultation, time to ID consultation, length of stay (LOS), mortality, and hospital re-admissions within 30 and 90 days.

Statistical analysis for categorical variables was described using Fischer’s exact test; normally distributed continuous variables were determined using Student t-test and non-normally distributed continuous variables were determined by Wilcoxon rank-sum.

Results

Utilization of STAB-IT™ and physician feedback:

Google analytics identified 353 visits to STAB-IT™ from November 2011-May 2012. There were 217 (61.5%) new visitors and 136 (38.5%) returning visitors. The mean utilization was 50 visitors / month. Three different pages of the app were viewed/per visit as the mean of viewed pages. The mean visit duration was three minutes and eleven seconds. Internal Medicine house-staff evaluations of STAB-IT™ following the demonstration/lecture showed 95% had favorable responses; 5% preferred STAB-IT™ to be an app retrievable from the Apple App store, so it would work without the need to be on the OSUWMC secure network.

Learning Outcomes

Thirty-seven internal medicine house-staff attended the CME STAB-IT™ interactive lecture and took the pre and post STAB-IT™ case study quizzes. After demonstrating how to navigate STAB-IT™ there was a significant improvement from pre to post test scores (50.2% vs. 86.6% p=0.0001) see Table 1.

Table 1: Results of Pre and Post STAB-IT^™ physician test scores

Patient Outcomes

S. aureus bacteremia was identified in 222 patients (MSSA=97, MRSA=124, both=1) in the pre-STAB-IT™ group and 191 (MSSA=78, MRSA=113) patients in the post STAB-IT™ group. The median age was 58 years in both the pre and post group. Patients in both the pre and post groups had initial empiric antibiotics selected by a physician. The ASP pharmacist adjusted antibiotics when necessary after reviewing the rapid Xpert^® MRSA/SA blood culture test result. No statistically significant differences were seen in patient outcomes; pre/post LOS (14 days vs. 14 days, p=0.40), mortality (13% vs. 13%, p=1.0), hospital re-admissions within 30 or 90 days (8% vs. 8%, p=1.0 and 18% vs 17%, p=0.70), median days to ID consultation (3 days vs. 3 days, p=0.08) and median first vancomycin level (15.9mg/L vs 14.8mg/L, p=0.36). The only positive trends observed in the post STAB-IT™ group were documented negative blood cultures prior to hospital discharge (90% vs 93% p=0.31) and ID consultations (64% vs. 67%, p=0.54), although neither were statistically significant.

Discussion

New technologies and use of medical apps by both healthcare providers and patients continue to grow. However, there is a paucity of data showing evidence of use and effect of apps on patient outcomes. To our knowledge, this is the first study to document an in-hospital evidenced based app initiative which is then tied to a measure of impact on patient outcomes for management of a common disease state, i.e. S. aureus bacteremia. Our data shows that after demonstrating how to navigate STAB-IT™ the internal medicine physicians overall knowledge of managing patients with S. aureus bacteremia significantly improved based on CME pre and post testing results. STAB IT™ was viewed more than once a day each month to help manage an average of 23 patients per month with S. aureus bacteremia. Returning visitors represented almost 40% of STAB-IT™ users suggesting users found the app informative. We did not show a statistically significant improvement in the measured patient outcomes; however, the number of documented negative blood cultures prior to hospital discharge and ID consultations requested increased.

Franko et al.⁷ studied the prevalence of app use among providers at the Accreditation Council for Graduate Medical Education (ACGME) accredited training programs. Over half of the respondents reported using apps. The most requested app types were reference materials (55%), treatment algorithms (46%), and general knowledge (43%). The results demonstrated a trend toward increasing app use that was inversely correlated with level of training. Health care providers at academic medical centers may be more likely to be younger practitioners and “early adopters” of new technology. The millennial learner is more reliant on technology and embraces new learning tools to gain knowledge⁸.

Despite the increasing popularity of medical apps there are concerns about the lack of actual evidence that app use improves health outcomes.⁹ One study done in 1,017 US hospitals examined the impact of UpToDate^®, a widely used web-based computerized clinical management system, on LOS, mortality and quality performance.¹⁰ They found a small, but statistically significant association between use of UpToDate^® and reduced LOS (5.6 days vs 5.7 days; P < 0.001), risk-adjusted mortality rate (−0.1% to −0.6%; P < 0.05, and better quality performance metrics, suggesting that a computerized tool could improve care. In contrast to their review that examined six common medical and surgical conditions, our study focuses on a single condition. Our lack of significant differences in clinical outcomes is undoubtedly related to the relatively short (9 months) time period and small numbers (37) of physicians in the pre/post groups.

A recent study from the University of Pittsburgh Medical Center measured the diagnostic accuracy for melanoma detection from four consumer dermatology apps.¹¹ Three of the 4 apps incorrectly classified 30% or more of melanomas. However, if these apps improved melanoma self-detection they could potentially decrease mortality through earlier detection and could be a welcome addition. The authors stress physicians must be aware of apps without regulatory oversight and the potential harm to patients if incorrectly diagnosed. STAB-IT™ is not a consumer app, however, oversight of the content for accuracy and current relevant information was the highest priority for the authors/developers.

Our study has several limitations. This was a single center study with a small number of physicians undergoing pre and post testing in our large academic medical center. The web-based version of UpToDate^® was also available during both the pre and post STAB-IT™ app periods. If UpToDate^® was not available, it is possible that providers would have accessed the STAB-IT™ app more frequently to assist in the management of S. aureus bacteremia. UpToDate^® use is not tracked by the IT department therefore we could not determine if this influenced our results. However, the authors stressed during the educational CME that STAB IT™ app provided OSUWMC specific data not found in UpToDate^® and other commonly used resources. Secondly, the demonstration and education of the STAB-IT™ app was only to internal medicine house-staff, who served as their own controls. Scheduling conflicts did not allow us to educate the surgical house-staff in August 2011; however, the three surgical intensive care unit clinical pharmacists provide STAB-IT™ education to surgeons one on one continuously. Patient outcomes did not differ if patients managed by the surgical services (pre=21, post=18) were removed from the outcome analysis (data not shown).Thirdly, the number of daily visits to STAB-IT™ may have been inadequate to show a significant difference in patient outcomes. Since this is the first disease-based app created at OSUWMC, we do not have usage data on any other apps to make a comparison. The average visit duration of 3 minutes and 11 seconds suggests the provider spent a reasonable amount of time reading the information. Google analytics was not activated until November 2011, five months after the educational session to IM house-staff, due to an internal IT issue, so we are unaware if the visits were higher immediately after the introduction of the STAB-IT^™ demonstration.

Finally, our internal web-based STAB-IT™ app provides information for “best practices” that are specific to OSUWMC and therefore not all of the information could be applied to other hospitals. In the STAB-IT™ microbiology section there is discussion on interpretation of results from the Xpert^® MRSA/SA blood culture test. Bauer et al. have shown this rapid test coupled with ID pharmacist interventions improves the outcome of patients with S. aureus bacteremia.⁵ The Xpert^® MRSA/SA testing is, however, not readily available in all hospitals; therefore if STAB-IT™ was submitted to the Apple’s app store, this testing would not be applicable as a “routine” microbiology test at all institutions. In addition, financial limitations did not cover the expense of IT personnel to format and provide updates to STAB-IT™ for the Apple app store.

The concept of hospital specific apps was recently studied in a survey of physicians and medical students from the United Kingdom.¹² The survey showed 74.8% would favor a disease management app specific to their hospital. Physicians surveyed after using STAB IT™, the first hospital specific disease management app at OSUWMC, also showed a high (95%) favorability rating.

Conclusion

Physician knowledge of managing patients with S. aureus bacteremia significantly improved after learning how to navigate a web-based app STAB-IT™. In lieu of finding a statistically significant impact on patient outcomes, larger studies over a longer time period are needed to further define the relationship between app use and patient outcomes for this common clinical condition.

Funding

All authors have completed the Unified Competing Interest form at www.icmje.org/coi_disclosure.pdf (available on request from the corresponding author) and declare: all authors had partial financial support from The Ohio State University College of Medicine for the submitted work; no financial relationships with any organizations that might have an interest in the submitted work in the previous 3 years; no other relationships or activities that could appear to have influenced the submitted work.

Partial Financial Support: Provided to the authors from The College of Medicine, The Ohio State University

Acknowledgements

Anthony S. Baker CMI. for creating medical illustrations for STAB IT™, Cheryl Brilmyer M.A. for technical support of STAB IT™ and Jessica West MSPH. for querying IW.

References

1. Topol E. The Creative Destruction of Medicine: How the Digital Revolution Will Create Better Health Care. New York, New York Basic Books 2012.

2. Goff DA. iPhones iPads and Medical Applications for Stewardship. Pharmacotherapy 2012;32(7):657–61.

3. Moodley A., Mangino J.E., Goff D.A. Review of Infectious Deseases Applications for iPhone/iPad and Android: From Pocket to Patient. Clin Inf Dis 2013. doi: .

4. Lustberg M., Goff DA., Mangino JE. The Effect of Infectious Diseases Consultation on Management, Outcomes, and Cost of Methicillin Resistant Staphylococcus aureus Bacteremia. [abstract] Society of Healthcare Epidemiology of America San Diego CA. 2009;200.

5. Bauer K, West J., Balada-Llsat J., Pancholi P., Stevenson K. Goff DA. An Antimicrobial Stewardship Program’s Impact with Rapid Polymerase Chain Reaction Methicillin-Resistant Staphylococcus aureus/S. aureus Blood Culture Test in Patients with S. aureus Bacteremia. Clin Infect Dis 2010;51(9):1074–80.

6. Liu C., Bayer A., Cosgrove SE., et al. Clinical Practice Guidelines by the Infectious Diseases Society of America for the Treatment of Methicillin-Resistant Staphylococcus Aureus Infections in Adults and Children. Clin Infect Dis 2011;1–38.

7. Franko O., Tirrell TF. Smartphone app use among medical providers in ACGME training programs. J Med Sysy 2012;36:3135–9.

8. Bahner D.P., Adkins E., Patel N., Donley C., Nagel R., Kman N. How we use social media to supplement a novel curriculum in medical education. Med Teacher 2012;34:439–44.

9. Boyce N. The Lancet Technology: Maps, apps and evidence? Lancet 2012;379:2231.

10. Isaac T., Zheng J., Jha A. Use of UpToDate and outcomes in US hospitals. J Hosp Med 2011;7:85–90.

11. Wolf JA., Moreau J., Akilov O., et al. Diagnostic Inaccuracy of Smartphone Applications for Melanoma Detection. JAMA Dermatol 2013; Jan 16, 1–4. [Epub ahead of print]

12. Payne KF., Wharrad H., Watts K. Smartphone and medical related App use among medical students and junior doctors in the United Kingdom (UK): a regional survey. BMC Med Inform Decis Mak 2012;12:121.