Boston University School of Medicine Outcomes Assessment Methodology
Boston University School of Medicine (BUSM) employs a number of techniques to assess the outcomes of our educational activities. The type of technique we use will depend on the educational format of a continuing medical education (CME) activity and the level of outcome that we are aiming to assess.
Boston University School of Medicine espouses the model described by Donald Moore, Jr., PhD, Joseph Green, PhD, and Harry Gallis, MD[i],[ii] in developing our outcomes-based evaluations. We work closely with our partners to determine which techniques will yield the best results.
The Moore, Green, and Gallis model describes 7 outcome levels as follows:
- Level 1 Participation
- Level 2 Satisfaction
- Level 3A Learning: Declarative Knowledge (Knows)
- Level 3B Learning: Procedural Knowledge (Knows How)
- Level 4 Learning: Competence (Shows How)
- Level 5 Performance (Does)
- Level 6 Patient Health
- Level 7 Community Health
All of our activities are assessed for Levels 1, 2 and 3 by our registration data (Level 1) and our standard activity evaluation which asks participants to rate their level of satisfaction with the activity (Level 2) and the degree to which they believe the learning objectives were met (Level 3A).
Measurement of objectives achieved
Activity participants are tested based on the behavioral learning objectives established for a CME activity. For example, one of the objectives might be – “At the conclusion of this activity, participants will be able to list three of the currently approved statin drugs.” As part of the evaluation form, participants would be asked to actually list three of the currently approved statin drugs. In the absence of a pre-test, there is no guarantee that the learning occurred as a result of the activity. Yet, this process demonstrates whether or not the objectives were achieved – important information for the faculty and CME staff. This method can be employed with live meetings, enduring and Internet-based CME activities.
Boston University School of Medicine currently uses the following types of outcomes assessments to measure Level 3B, Level 4, and Level 5 outcomes, ie. procedural learning, competence, and performance based changes.
Pre- and post-tests
Activity participants complete multiple choice questions concerning activity content before and immediately after a CME activity. This method measures learning that occurred as a result of the activity. The benefit of this type of measurement is that the participants, the faculty and the CME staff have immediate feedback regarding what learning has occurred (Level 3B measurement). This method may not necessarily predict retention of the learning or change in performance. Pre- and post-tests can be used in conjunction with live meetings, printed enduring materials and Internet-based CME activities.
Commitment to Change
Participants of live and enduring material activities are asked to write one to three changes that they plan to make a change as a result of our activities (Level 4 measurement). Jocelyn Lockyer and her associates have found that a commitment to change (CTC) predicts actual change in practice[iii]. According to Lockyer, et. al., “Three quarters of CTCs were fully or partially implemented” in her study (p. 76). A summary of these reveals the immediate impact of the CME activity, providing useful needs assessment data for planning future activities.
Post activity surveys (“fax-back” surveys)
Post activity surveys go further in measuring change by venturing into performance based change – the Level 5 outcome. Participants are asked, at the conclusion of a CME activity, to list three changes that they intend to make as a result of the activity.
Within one to three months of the CME activity, the BUSM staff will fax or email our CME activity participants and ask them if they have fully implemented, partially implemented or were unable to implement the changes they intended to make.
The limitation of this data is that it is self-reported. However, in the absence of actual observation of a physician’s performance in practice, this information serves as a surrogate marker that, according to Lockyer’s research, is indicative of actual change.
Case based assessment
In a comparison of chart audits, standardized patients (where actors take on the role of patients and physicians are evaluated on their interactions with the “patients”), and case vignettes, case vignettes were found to be as effective as the other two methods in determining outcomes[iv]. Aimed at measuring Level 3B and Level 4 outcomes, we have asked physicians in live meetings to answer key multiple choice questions in response to a case presentation. The cases and questions are presented before and after the CME activity to measure learning. Case vignettes can also be administered to a control group, ie. a group of physicians who share a professional profile with the activity participants but who did not participate in the activity.
Performance Improvement Initiatives – Live
Following the AMA’s guidelines on awarding AMA PRA Category 1 Credit™ for performance improvement, Boston University School of Medicine CME staff will work with physicians to undertake customized practice-based performance improvement initiatives, facilitating the measurement of change through chart reviews. Participants audit a set of charts using a data collection survey created by BUSM faculty. The data from the charts are reviewed by BUSM faculty and a live educational intervention is developed based gaps in care revealed through the chart reviews. As a part of the educational intervention, the participants are asked to submit Performance Improvement Plans based on the results of the chart reviews. Members of BUSM will review and approve proposed plans, offering suggestions whenever necessary. Three months after the participants implement their Performance Improvement Plans, they are asked to audit a different set of charts to see if the intervention has had a positive effect on their practices and in turn on the health of their patients. 20 AMA PRA Category 1 Credits™ are awarded to participants for completing the initiative. Performance improvement activities will lead to Level 5 outcomes assessment data. Level 6 data can be attained through longitudinal studies, if resources allow.
Performance Improvement Initiatives – Virtual
Through a unique design, Boston University School of Medicine develops Virtual Communities of Practice working with physicians and allied health practitioners from varied locations to implement practice improvement initiatives. Activities are supported through a custom designed website, email and teleconferences.
Boston University School of Medicine will continue to seek new ways to obtain outcomes assessment data for our CME activities. In keeping with the charge set forth by Moore, Green, and Gallis, we will strive, in particular, to incorporate formative assessment processes into our activities that provide participants with opportunities for practice and feedback.
[i] Barnes, B, Davis, D, Fox, R. Continuing Professional Development of Physicians: From Research to Practice. Chicago. AMA Press, 2003: 249 – 274.
[ii] Moore, D, Green, J., Gallis, H. Achieving Desired Results and Improved Outcomes: Integrating Planning and Assessment Throughout Learning Activities. Journal of Continuing Education in the Health Professions. Winter 2009; 29: 1 – 15.
[iii] Lockyer, J, Fidler, H, Hogan, D, Pereles, L, Wright, B, Lebus, C, Gerritsen, C. Assessing Outcomes Through Congruence of Course Objectives in Reflective Work. JCHEP 2005; 25: 76-86.
[iv] Peabody, J, Luck, J, Glassman, P, Dresselhaus, T, Lee, M. Comparison of Vignettes, Standardized Patients, and Chart Abstraction: A Prospective Validation Study of 3 Methods for Measuring Quality. JAMA 2000; 283: 1715 – 1722.