Precision of App-Based Model for End-Stage Liver Disease Score Calculators

Background: The prioritisation of patients with end-stage liver disease for liver transplantation requires a quantification of clinical disease severity. The Model for End-Stage Liver Disease (MELD) score is used to prognosticate survival for these patients and is therefore useful to prioritise for transplantation. The MELD score utilises a complex equation, which is now available for calculation using a range of smartphone applications (‘apps’). There is however no published data on the precision of these app-based calculators in calculating a MELD score.

Methods: In a cohort of 46 adults patients awaiting liver transplantation, the precision of 14 free and pay-for-use Apple iPhone app-based MELD score calculators in calculating the MELD score was compared with the actual MELD equation using kappa statistics.

Results: Kappa statistics demonstrated agreement of 0.70 to 0.97 (mean of 0.78; 95% CI, 0.6 – 0.95) between the app-based calculators and the MELD equation.

Conclusion: This study showed substantial but not perfect precision of  app-based MELD score calculators compared to the actual MELD equation. This is an important finding in assessing the validity of app-based MELD score calculators and further studies evaluating the growing number and availability of app-based medical calculators are required.

Introduction

End-stage liver disease is associated with a high morbidity and mortality¹. For selected patients with end-stage liver disease, liver transplantation is a suitable management option. With advances in surgical technique and post-transplant care, the survival rates after liver transplant continue to improve². Unfortunately, the scarcity of donor livers in Australia limits the availability of transplantation for patients with end-stage liver disease.

The allocation of livers to potential recipients requires a balance between clinical urgency and the likelihood of improved survival post-transplantation. A robust prognostic model is therefore necessary to assess disease severity and quantify the risk of mortality without transplantation. The Model for End-Stage Liver Disease (MELD) score is a prospectively validated index used in end-stage liver disease for this purpose^{3, 4}. It has been adopted globally by many transplant programmes, including The Transplantation Society of Australia and New Zealand (TSANZ)⁵. The MELD score is determined from three laboratory values: the serum bilirubin, the serum creatinine and the international normalised ratio for prothrombin time (INR). These values form part of a logarithmic equation to produce a numerical score (see Figure 1). The resultant score estimates three-month survival, with a higher score reflective of a lower survival rate. The MELD score is therefore used in the prioritisation patients on the liver transplant waiting list and is calculated periodically as donor livers become available or with change in the clinical state of potential recipients.

Figure 1. MELD score equation³. Note that the maximum serum creatinine for use in the MELD score equation is 4 mg/dL.

Information technology plays an integral role in the modern clinical environment. It is specifically used at transplant centres to coordinate pre-and post-transplant care, digitise radiology and maintain a database of patient records. With the increasingly widespread use of smartphones, such as the Apple iPhone (Apple Corporation, Cupertino, USA), in the medical workplace, clinicians can gain timely and portable access to reference materials, clinical tools and even patients’ results^6-8. There is an emerging volume of smartphone applications (‘apps’) that offer medical calculators that can be used at the bedside, however there is a paucity of studies assessing the accuracy of app-based medical calculators. Specifically, there is no published literature evaluating apps with MELD score calculators and their adherence to the actual MELD score equation. Our study aimed to assess the precision of the app-based MELD score calculators available for the iPhone by comparing the app calculated MELD scores with the actual, formula derived scores of a cohort of adult patients awaiting liver transplantation.

Methods

Subjects

A cohort of patients awaiting liver transplantation was selected in May 2012 from the transplant database of a tertiary hospital offering the sole statewide service for liver transplantation. All patients over the age of 18 were included in this study (n = 46). The most recent laboratory values for creatinine (mg/dL), bilirubin (mg/dL) and INR as of the 17^th May 2012  were obtained. Clinical details for each patient were recorded and included the patient’s sex, age and aetiology of liver disease.

Applications

App-based MELD score calculators (n=14) were obtained from the Apple iTunes Australia App Store (Version 10.6.1). Search terms for the apps were ‘MELD score calculator’, ‘MELD score’, ‘liver transplantation’, ‘liver disease’, ‘medical calculator’, ‘med calc’ and ‘medical application’. Both free and pay-for-use apps were included. Non-English apps were excluded from the study. Apps were downloaded and installed on an iPhone 4.

MELD Scores

A MELD score was calculated for each patient using the equation shown in Figure 1. This calculation was performed using Microsoft Excel 2008 for Mac (Version 12.3.4). MELD scores were subsequently calculated for each patient using each installed app on the iPhone.

Statistics

Statistical analyses were performed using a statistical package (Stata, Version 11, StataCorp LP, College Station, USA). A kappa statistic was used to evaluate the magnitude of agreement between the MELD scores of the cohort as calculated by the equation in Figure 1 (‘gold standard’) and the MELD scores as calculated by each app. A kappa statistic quantifies the agreement of two observations,  adjusted for agreement by chance alone. Kappa is standardised on a scale from -1 to 1. A kappa of 1 indicates perfect agreement, a kappa of 0 indicates agreement expected by chance alone and a kappa of -1 indicates agreement occurring even less often than chance⁹. Therefore in this study, a higher kappa statistic reflected a higher degree of agreement between the scores derived from the actual MELD equation and the scores obtained from a particular app, thereby conferring a higher degree of precision to the app. It has been proposed that the strength of agreement quantified by a kappa statistic can be graded as < 0 = poor; 0.01 – 0.2 = slight, 0.21 – 0.4 = fair; 0.41-0.6 = moderate; 0.61 – 0.8 = substantial and 0.81 – 0.99 = almost perfect⁹.

Results

Patient Characteristics

There were 46 adult patients on the liver transplant waitlist database at the time of data collection (see Table 1). The patients had a mean age (± SD) of 53 ± 9.90 years and were comprised primarily of males (76%). The most common aetiology amongst the patients was hepatitis C (39%), followed by hepatocellular carcinoma (30%) and hepatitis B (15%).

Table 1. Patient Characteristics. † polycystic liver disease, haemangioendothelioma, cryptogenic, familial amyloidotic polyneuropathy

Apps

14 app-based MELD score calculators were downloaded and installed from iTunes (see Table 2). Five were published by individuals. The apps ranged in price from $0 to $5.49, with a mean price of $2.93. Only one app’s function was solely to calculate a MELD score (MELD Calculator); the other apps had an ability to  calculate a variety of medical formulas, scores or classifications across a range of medical specialties. 11 of the 14 apps listed the MELD score equation utilised in the app.

Table 2. App-based MELD calculator

MELD Scores

The mean MELD score (± SD) using the equation in Figure 1 was 13.31 ± 7.69 (see Table 3).

Table 3. MELD characteristics and score.

The range of MELD scores obtained using the MELD apps was 13.32 to 15.39 (see Table 4).

Kappa statistics for the apps ranged from 0.70 to 0.97, with a mean of 0.78 (95% CI, 0.6 – 0.95). Three apps, Medical Observer, MediMath Medical Calculator and Mediquations Medical Calculator, achieved a kappa of 0.97 and the app, MediSolve, achieved a kappa of 0.94. The remaining apps had a  kappa score close to 0.70. The kappa statistics can be represented in Bland-Altman plots. Figure 2 demonstrates a Bland-Altman plot for the Medical Observer app. The close agreement between the app-based calculator and the MELD equation is represented by the narrow spread of points, yielding a high kappa statistic (0.97).

Figure 2.Bland-Altman plot of MELD equation and Medical Observer app MELD scores.

Figure 3 demonstrates a Bland-Altman plot for the GI Calculator app. The wider distribution of points reflects reduced agreement between the app-based calculator and the MELD equation. The corresponding kappa statistic (0.70) is therefore lower than the app represented in Figure 2.

Figure 3.Bland-Altman plot of MELD equation and GI Calculator app MELD scores.

Table 4. App-based MELD calculator scores and kappa statistics.

Discussion