Machine learning charts disease symptoms from patient-physician conversations

Human scribes labeled the more than 2,500 transcripts with 185 symptoms, assigning each symptom mention a relevance to the ROS as it related to a patient’s experience. Input to the machine learning model was a sliding window of five conversation turns, or snippets, and output was each symptom mentioned, its relevance to the patient, and whether the patient experienced that symptom.

In the test set of 2,091, the team reported 5,970 symptom mentions, 79.3 percent of which were relevant to the ROS and 74.2 percent of which were experienced by patients. Across the full test set, the model achieved a sensitivity of 67.7 percent and a positive predictive value of a predicted symptom of 80.6 percent.

Broken down further, the sensitivity of the model was 67.8 percent for unclear symptoms, the authors said, and 92.2 percent for clearly mentioned symptoms. A symptom was considered “clearly mentioned” if two randomly selected scribes both independently assessed the likelihood of including any given symptom in the ROS as “extremely likely.”