Abstract

Objective Entrustable professional activities (EPAs) are core assessment tools for competency based education. EPA assessments provide a comment section to enrich the context of the assessment that is crucial for learner growth. Previous studies developed the Quality of Assessment for Learning score (QuAL), a validated rating tool for assessing the quality of narrative comments in competency-based medical education. We applied QuAL scoring to a large assessment database to assess the quality of comments. Design Assessment comments from a nationwide digital medical education platform were evaluated by QuAL with scores ranging from 0--5 with 0 indicating very low quality, 3 an average comment and 5 a high quality comment. This is based on three components (1) sufficient evidence about performance scored from 0--3 (2) offering suggestion for improvement, scored as 0 or 1 and (3) connection of suggestion to a described behavior, scored as 0 or 1. We leveraged previous work developing a publicly available machine learning model to score comments with QuAL. Python was used to automate the extraction and processing of comment data through the machine learning model. Participants Faculty from surgical training programs across the United States submitted EPA assessment comments for surgical residents and fellows. Results Of 12,841 EPA assessments, 7,211 (56%) contained a comment. Assessments without comment were excluded from analysis. The mean overall comment score was 2.91 representing average quality. The average evidence was 2.4/3 ($±$ 0.86). Only 29% of EPAs provided a suggestion with 26.5% connecting that suggestion to a described behavior. Conclusions The comment component of EPA assessments offers essential context to facilitate learner progress and faculty guidance of learners. Yet comments are inconsistently provided, and when they are provided, 3/4s lack suggestions for improvement and a connection to the behavior described. Future directions include faculty development initiatives regarding high quality comment utility and EPA platform driven prompts of essential comment components.

Back to List