Does Machine Learning Improve Prediction of VA Primary Care Reliance?

Objectives: The Veterans Affairs (VA) Health Care System is among the largest integrated health systems in the United States. Many VA enrollees are dual users of Medicare, and little research has examined methods to most accurately predict which veterans will be mostly reliant on VA services in the future. This study examined whether machine learning methods can better predict future reliance on VA primary care compared with traditional statistical methods.

Study Design: Observational study of 83,143 VA patients dually enrolled in fee-for-service Medicare using VA and Medicare administrative databases and the 2012 Survey of Healthcare Experiences of Patients.

Methods: The primary outcome was a dichotomous measure denoting whether patients obtained more than 50% of all primary care visits (VA + Medicare) from VA. We compared the performance of 6 candidate models—logistic regression, elastic net regression, decision trees, random forest, gradient boosting machine, and neural network—in predicting 2013 reliance as a function of 61 patient characteristics observed in 2012. We measured performance using the cross-validated area under the receiver operating characteristic (AUROC) metric.

Results: Overall, 72.9% and 74.5% of veterans were mostly VA reliant in 2012 and 2013, respectively. All models had similar average AUROCs, ranging from 0.873 to 0.892. The best-performing model used gradient boosting machine, which exhibited modestly higher AUROC and similar variance compared with standard logistic regression.

Conclusions: The modest gains in performance from the best-performing model, gradient boosting machine, are unlikely to outweigh inherent drawbacks, including computational complexity and limited interpretability compared with traditional logistic regression.

Am J Manag Care. 2020;26(1):In Press

Takeaway Points

Reliance on the Veterans Affairs Health Care System for outpatient services is a key metric for informing resource allocation and current operational priorities, including efforts to compete for veteran demand amid increasing healthcare options.

• The application of each of 6 candidate models for predicting future reliance exhibited very good performance based on previous thresholds.
• Each of the 5 machine learning models considered did not outperform, or modestly outperformed, a traditional regression model.
• Modest gains in performance of machine learning models are unlikely to outweigh practical limitations including computational complexity and difficulty in interpreting model composition.

The Veterans Affairs (VA) Health Care System is among the largest integrated healthcare systems in the United States. The VA provides comprehensive health services to eligible veterans, and most of the 8.3 million VA enrollees in 2017 were exempt from co-payments or other cost sharing.¹ VA enrollees are unique because, in addition to access to VA health services, more than 80% are dually enrolled in at least 1 other source of private or public insurance. Although most users of VA are exempt from cost sharing, many veterans 65 years or older supplement VA care with Medicare services. Extensive research has examined factors associated with greater dual use of VA and Medicare outpatient services. Dual VA-Medicare enrollees who were more reliant on VA outpatient care were younger and unmarried, lived closer to VA facilities, and were exempt from VA co-payments.^2-4

Previous research examining determinants of VA utilization and dual use has largely been descriptive and has not explored approaches to generate accurate predictions. More recent research outside VA has applied newer machine learning algorithms, demonstrating promise in producing predictions of future health outcomes, utilization, and costs with greater accuracy compared with more traditional approaches.⁵ Notably, several studies have examined prediction of costs within the context of improving risk adjustment, finding that supervised learning methods outperform such traditional methods as parametric regression.^6-8 Machine learning models are being increasingly applied within VA but have not been routinely used for the purpose of predicting reliance on VA health services. These models often have practical challenges, including greater computational complexity and lack of interpretability relative to traditional statistical models. Thus, it is important to understand trade-offs between potential improvements in model performance and these practical challenges.

Improving the ability to correctly predict patients’ expected utilization and costs is important for at least 2 reasons. First, more accurate predictions can help determine the adequacy of budget and staffing resources needed to provide care for a patient population. This is particularly important for the VA, which is funded annually through a budget appropriation from Congress. This appropriation is determined, in part, from the Enrollee Health Care Projection Model, which includes veterans’ expected reliance on the VA for care as a key input.^9,10 At the individual clinic level, predictions of expected VA primary care reliance are helpful in determining staffing distributions, including ancillary services such as social work and integrated behavioral health that are needed by more frequent users of VA.¹¹

More accurate predictions of VA reliance can also help better identify veterans expected to use outpatient services outside the VA. This is significant in light of VA’s current emphasis on competing for veterans as customers given increasing healthcare options available to VA enrollees.¹² In addition, accurate reliance predictions may inform care coordination efforts within VA’s patient-centered medical home.¹³ Receipt of fragmented care across health systems is often associated with poor clinical outcomes and increased hospitalizations.^14-17 As current policies continue to support veterans’ access to care outside the VA, accurate predictions of reliance will help identify patients who need further support to mediate the risks of care fragmentation. To address these policy-planning priorities, we examined whether supervised machine learning methods could improve the ability to predict future reliance on VA primary care among VA-Medicare dual enrollees.

METHODS

Data Sources

The primary data source was the VA Corporate Data Warehouse (CDW), which houses information on utilization of VA health services, demographic data, and International Classification of Diseases, Ninth Revision diagnosis codes.¹⁸ We linked CDW data with Medicare enrollment and claims data to ascertain enrollment in fee-for-service Medicare and utilization. We supplemented VA and Medicare administrative data with survey responses capturing patient experiences with VA outpatient care, as measured using the 2012 VA Survey of Healthcare Experiences of Patients (SHEP) Outpatient Module, which is adapted from the Consumer Assessment of Healthcare Providers and Systems.¹⁹ SHEP is managed by the VA Office of Reporting, Analytics, Performance, Improvement and Deployment, which routinely administers the survey to a random sample of veterans who obtained outpatient care at VA outpatient facilities. Finally, we obtained measures of local area health resources in veterans’ residence counties, as reported in the Area Health Resource File.²⁰

Study Sample

Using data from the 2012 outpatient SHEP, we identified 222,072 respondents enrolled in VA, of whom 90,819 were also continuously enrolled in fee-for-service Medicare in fiscal year (FY) 2012 and FY 2013. Consistent with prior studies,^2-4 we examined veterans dually enrolled in VA and fee-for-service Medicare because we were unable to capture utilization from other non-VA sources. We then excluded 7182 veterans with no primary care utilization in FY 2012 or FY 2013 whose reliance, by definition, was undefined. After excluding 494 veterans with missing covariates, the final study sample consisted of 83,143 dual VA–fee-for-service Medicare enrollees.