Use of Portfolios to Assess Student Outcomes
National Board of Medical Examiners
The public in general and professional organizations in particular increasingly demand evidence of competence in medical practice and a physician's ability to meet the demands of today's society (IOM, 2001; 2003). Medical education has responded with a focus on educational outcomes (Whitcomb, 2004), case-based, authentic, curriculum (Friedman, 2001; Kincade, 2005) and experiences that support the development of physicians within a complex health care system (ACGME, 2005; AAMC Report V, 2001). The emerging definition of professional competence is difficult to evaluate using traditional assessment. The portfolio is an attempt to overcome the current limitations of assessment by integrating professional outcomes and placing them within an authentic learning context. Challenges in portfolio assessment include insufficient inter-rater reliabilities, questions of generalizability, a substantial faculty and learner time commitment, and balancing a prescriptive, standardized approach with individualization (Friedman, et. al. 2001; Case, 1994; Des Marchais et al 1995; Challis 1999: LeMahieu, et.al 1993; Herman et al, 1995).
The University of Missouri has developed a set of key competencies for our graduates (MU2020 key characteristics) that are consistent with national and international discussions of professional competence. To our knowledge few medical schools have successfully engaged faculty in developing an approach for assessment of professional competencies. This research proposal draws on the prior work at MU to address two global questions:
- How does the development of a set of descriptive anchors for each of the key characteristics influence the validity, reliability, reproducibility and trustworthiness of portfolio assessment?
- How do student contributions to the portfolio influence faculty assessment of portfolios?
Descriptive anchors of exemplary performance for each of the professional outcomes will be derived from the literature, clinical faculty, medical students and patients. These anchors will be used to develop a portfolio assessment tool. A two step judgmental review process will establish the content validity of the descriptive anchors. Inter and intra rater reproducibility will be established by using the assessment tool to evaluate the portfolios of third year medical students. Predictive validity will be determined by correlation of portfolio assessment with traditional measures of student success. The influence of student contributions to the portfolio assessment will be evaluated by determining the differences between individual faculty ratings of a student's portfolios rated with only required documentation and rated a second time with student's contributions included. An external advisory board will provide guidance to the research team and will review the appropriateness of the intermediate research projects.
The outcome of this project will be a portfolio assessment tool to evaluate student outcomes. It will be a useful addition in the assessment of learners and promote an enhanced understanding of professional competence.
For more information, contact:
Kim Hoffman, PhD
(Return to Current Research Projects)