Clinical competency committees: a guidebook for programs.

Clinical Competency Committees

A Guidebook for Programs

Edition

Kathryn Andolsek

Duke University

Jamie Padmore

Medstar-Georgetown

Karen E. Hauer

University of California at San Francisco

Andem Ekpenyong

Rush University Hospital

Laura Edgar

ACGME

Eric Holmboe

ACGME

This information is current as of January 2020

Overview

The Clinical Competency Committee (CCC) is a structure that has emerged as an

essential component of the evaluation process in graduate medical education

(GME). While some specialties and programs have utilized CCCs for years, this

structure is still relatively new to many others. Likewise, with the emergence of the

CCC as a requirement for accreditation (ACGME Common Program

Requirements), even seasoned programs and committees are facing questions

regarding the CCC’s structure, function, and process. Furthermore, scholarship on

group decision making, assessment, and CCCs themselves continues to provide

evidence to inform practices. The purpose of this manual, now in its third edition, is

to provide designated institutional officials (DIOs), program directors, faculty

members, CCC members, coordinators, residents, and fellows with information and

practical advice regarding the structure, implementation, function, and utility of a

well-functioning CCC. The materials were prepared for both individual learning and

application in a group setting. Ideally, institutions and programs will be able to use

these materials to have meaningful conversations with all stakeholders; enhance

CCC development, function, and outcomes; and improve transparency for residents

and fellows on the nature of assessment in competency-based education. This third

edition also contains updated material and new tools for programs to use to

continually improve their CCC processes, and a section on institutional

opportunities with CCCs has been added. Each major section has an

accompanying brief “summary” available as a separate, individual document.

This manual provides information related to the following topics:

1. Purposes of a CCC

2. CCC Structure and Membership

3. Preparing for CCC meetings

4. Running the CCC meeting, including effective group process

5. Post-meeting feedback, documentation. and follow-up

6. Legal issues

7. Other Uses for the CCC

8. Individualized Learning Plans

9. Institutional Oversight of CCCs

10. Current Research

Several appendices contain tools for institutions, programs, and CCCs. A robust

body of research to support the various aspects of CCCs, including assessment,

feedback, documentation, group dynamics, and outcomes, is now available. An

annotated bibliography is updated approximately every six months and is available

on the Milestones section of the ACGME website.

The ACGME welcomes feedback, and hopes this guidebook provides institutions,

programs, and faculty members with valuable information and tools to enhance

GME.

Table of Contents

Sections

Page

1. Introduction

2. Purpose of a CCC

3. CCC Structure and Membership

4. Preparing for CCC meeting

5. Running the CCC meeting

6. Post-meeting activities

7. Legal issues

8. Other Uses for the CCC

9. Individualized Learning Plans

10. Institutional Oversight of CCCs

11. Current Research

Appendices

Page

A. The High Performing Residency Assessment System

B. CCC Quiz

C. Case Studies

D. Designing the CCC

E. Examples of Assessment Methods for the ACGME Core

Competencies

F. Overview of Assessment Methods Aligned with Miller’s Pyramid

G. PPV (Predictive Probability Value) Tables

H. ADAPT Model of Feedback

I. R2C2 Evidence-Informed Facilitated Feedback

J. Institutional Checklist for CCCs

Introduction

The Clinical Competency Committee (CCC) became a central tool in graduate

medical education (GME) as the ACGME transitioned to its current model of

accreditation (previously dubbed the “Next Accreditation System”) as an important

element of competency-based education. ACGME International (ACGME-I)

programs use CCCs as well. The requirements for CCCs have evolved with the

most recently revised ACGME Common Program Requirements, effective July 1,

2019. All four sets of Common Program Requirements (Residency; Fellowship;

One-Year Fellowship; and Post-Doctoral Education Programs) have the same

requirements. The ACGME has categorized all requirements for CCCs as “Core”

requirements, meaning they are required for every graduate medical education

program. The requirements are discussed in Part 2 of this guidebook.

The objectives of this guidebook are to help institutions and programs:

1. Recognize the role and purpose of the CCC for individual programs in the

ACGME’s current accreditation model.

2. Design, create, implement, and continuously improve the program’s CCC.

3. Run an effective CCC meeting.

4. Provide feedback to residents or fellows allowing for improved constructed

individual learning plans.

5. Anticipate questions regarding “process” and considerations of academic

law.

6. Align the program’s own CCC processes with the best evidence from the

medical education literature.

7. Use the CCC to continuously improve the program’s curriculum,

assessment system, faculty development, and clinical

training/experiences/quality.

The guidebook also aims to help programs understand other opportunities for using

Milestones data and for designated institutional officials (DIOs) to recognize

opportunities to support CCCs at an institutional level.

This guidebook is intended to be a practical resource and a professional

development tool for institutional and program leadership, coordinator(s), and

faculty members. Residents and fellows may also benefit from this guidebook,

although a separate guidebook has been developed specifically for them:

Milestones Guidebook for Residents and Fellows. Institutional and program leaders

are encouraged to share these materials with their program faculty members and

leaders, and to use the exercises as part of faculty and coordinator professional

development. These materials can be reviewed individually or as part of a meeting.

The guidebook also provides suggestions for faculty development.

The CCC contributes to an effective resident/fellow assessment system as outlined

in Figure 1. In this figure, the CCC serves the critically important function of

synthesizing multiple quantitative and qualitative assessments regarding individual

resident/fellow performance. This figure highlights several important points:

1. The CCC’s deliberative process will depend on the quality of the assessment

program that should include a combination of assessment methods and a

number of different assessors. Ideally, the individuals who sit on the CCC must

understand the basics of good assessment and the assessment tools being

used by the program. Faculty members should have opportunities to enhance

their assessment skills and understand how their assessments fit into the

program’s overall assessment strategy.

2. Residents and fellows are adult learners and must be active agents in this

system; guided self-directed assessment behaviors by an individual resident or

fellow should be expected and strongly cultivated. Programs are urged to

encourage all of their residents and fellows to review the Milestones Guidebook

for Residents and Fellows (available at

http://www.acgme.org/Portals/0/PDFs/Milestones/MilestonesGuidebookforResid

entsFellows.pdf)

3. The program director of a residency, fellowship, or post-graduate educational

program is the ultimate arbiter of whether a resident or fellow is ready to

practice without supervision. The accountability of the program director and

the program cannot be over-emphasized: professional self-regulation depends

heavily on the informed judgment of education programs, as manifest by the

final summative evaluation of competence and entrustment made by the

program director.

Figure 1: Structure of a High Performing Resident/Fellow Assessment System

Residents = both residents and fellows

FB = Feedback loops

D = Assessment data and information

The model is more fully described in Appendix A

In this model the Data Synthesis Committee IS the CCC.

Holmboe ES, Yamazaki K, Edgar L, et al. Reflections on the first 2 years of

milestone implementation. J Grad Med Educ. 2015;7(3):506-511.

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4597976/

. 2020.

Part 1: Purpose of a CCC

The CCC serves several purposes for multiple stakeholders: the program itself,

program directors, faculty members, program coordinators, residents and fellows,

the institution, and the ACGME (Table 1). The ultimate purpose is to demonstrate

accountability as medical educators to the public: that graduates will provide high

quality, safe care to patients while in training, and be well prepared to do so once in

practice.

Table 1: Purposes of a CCC

Purpose of CCC

Program

● Develop shared mental model of what resident/fellow performance should

“look like” and how it should be measured and assessed

● Ensure the right combination of assessment tools to effectively determine

performance across the Competencies and specialty-specific Milestones

● Increase quality, standardize expectations, and reduce variability in

performance assessment

● Contribute to aggregate data that will allow programs to learn from each

other by comparing residents’ and fellows’ judgments against national data

● Improve individual residents’/fellows’ progress along a developmental

trajectory

● Identify early those residents/fellows who are challenged and not making

expected progress so that individualized learning plans can be designed

● Identify advanced residents/fellows to offer them innovative educational

opportunities to further enhance their development

● Identify weaknesses/gaps in the program as a first step in program

improvement

● Model “real time” faculty development

Program

Director

● Fulfill public accountability by ensuring that residents/fellows who

successfully complete a program can practice without supervision

● Engage faculty members, and others when appropriate, to make informed

decisions regarding performance

● Enhance credibility of judgments about resident/fellow performance

● Identify opportunities for faculty development around supervision and

assessment, both formative and summative

● Facilitate the program director’s role as “advocate” for the resident/fellow

● Improve feedback for residents and fellows

Table 1 (continued)

Faculty

Members

● Facilitate faculty members’ development of a shared mental model of what

is expected within each of the Competencies and specialty-specific

Milestones

● Improve documentation by simplifying and creating “more actionable” and

efficient assessment tools for the direct observation of residents/fellows in

the clinical learning environment

● Fulfill the professionalism inherent in the faculty member’s role by

contributing high quality teaching and assessment as part of the program

● Contribute accurate, rich descriptive assessment information to the CCC

Program

Coordinators

● Optimize resident/fellow data management systems

● Synthesize assessment data

● Improve methods to share data with the CCC

● Collaborate with program directors to ensure residents and fellows receive

feedback and follow-up, and that Milestones assessments are reported to

the ACGME

● Help improve CCC process by observing the meeting dynamics and

providing feedback

Residents/

Fellows

● Improve the quality, amount, and timing of feedback; normalize

constructive feedback

● Offer insights and perspectives of a group of faculty members

● Enhance self-directed learning

● Compare performance against established Competency benchmarks

(rather than only against peers in the same program)

● Allow earlier identification of sub-optimal performance that can inform

individualized learning plans and improve individualized interventions

● Improve “stretch goals” for residents/fellows to help high performing

residents/fellows achieve even greater competence

● Provide transparency regarding performance expectations

Institutions

● Ensure residents/fellows are making expected progress and those who are

not are provided an opportunity for early intervention

● Provide foundational expectations for faculty members as assessors of

performance through direct observation

● Ensure CCCs adhere to pertinent institutional policies

● Share best practices from within the institution, nationally and

internationally

● Identify opportunities to enhance resources necessary to optimize CCC

functioning at an institutional level

ACGME

● Enhance progress toward competency-based education with outcomes

data

● Establish national benchmarks for the trajectory of resident/fellow skills

acquisition that can be used for each specialty

● Provide better measures for public accountability

● Enable continuous quality improvement of GME programs

● Document the effectiveness of the nation’s GME to prepare graduates to

meet the needs of the public

A program’s creation of a CCC is, in itself, a “developmental” process. Next, this

guidebook will briefly review of the current ACGME requirements for a CCC,

effective July 1, 2019. Programs may identify gaps and potential enhancements

through their CCCs by comparing what they have in place to meet the

requirements. For programs either beginning to institute a CCC, or looking to

enhance an existing CCC, the next few pages offer a practical roadmap.

Part 2: CCC Structure and Membership

Designing and Creating a CCC

To design, create, and operate a CCC, it is useful to start with the requirements.

The ACGME requirements for a CCC are found in the Program Requirements (both

in the Common Program Requirements for Residencies, Fellowships, One-Year

Fellowships, and Post-Doctoral Education Programs, and also in the specialty- and

subspecialty-specific Program Requirements). CCC requirements are generally the

same for all programs and are considered “core” requirements—or essential and

required for all GME programs. The CCC is mentioned in Program Requirement

III.A.4.b) for those specialties that allow exceptions to resident eligibility

requirements. CCCs are also mentioned in the Background and Intent related to

Program Requirement IV.A.3. regarding the Educational Program. Most of the

requirements and description of CCCs are found in Section V.A. of the Common

Program Requirements, which addresses Resident/Fellow/Post-Doctoral Education

Fellow Evaluation.

The ACGME International Foundational Program Requirements are similar. “The

program director must appoint the CCC [V.A.1.]; (It) should be composed of

members of the program faculty [V.A.2.a)]; have a written description of its

responsibilities, including its responsibility to the Sponsoring Institution and to the

program director [V.A.2.b)]; participate actively in reviewing all resident evaluations

by all evaluators [V.A.2.c).(1)]; and, making recommendations to the program

director for resident progress, including promotion, remediation, and dismissal.

[V.A.2.c).(2)]” [

www.acgme-

i.org/Portals/0/FoundInternational03312016.pdf?ver=2016-04-25-084913-527]

These are minimum requirements; once the program CCC fulfills the Common

Program Requirements, and any specialty-specific and board requirements, they

are free to innovate!

Step 1: Review Section V.A. of the relevant specialty-specific Program

Requirements.

While there are no specific requirements for the CCC in the Institutional

Requirements at present, there are at least two institutional requirements that

should be considered. The Sponsoring Institution is responsible for programs’

development of “promotion criteria” and criteria for renewal of a resident’s/fellow’s

appointment (Institutional Requirement IV.C.), and those conditions for

reappointment and promotion to a subsequent PGY level must be in the contract or

letter of appointment. (Institutional Requirement IV.B.2.d)).

Many CCCs may de

facto “act” as promotion committees and apply their judgement of resident/fellow

performance to recommend resident/fellow renewal and promotion to the next

program year. The Common Program Requirements specify that “at least

annually... a summative evaluation of each resident that includes their readiness to

progress to the next year of the program, if applicable” must be conducted.

[Common Program Requirement VI.A.1.e)] Although not stated explicitly, it is likely

that an effective CCC will have collaborated with the program director to identify the

promotion criteria, or at the very least, align Milestones performance with them. The

CCC should inform the program director of its review so that the program director

can truly exercise the responsibility, authority, and accountability for promotion of

residents. [Common Program Requirement II.A.4.]

It is important to recognize that the Milestones do not represent the totality of any

discipline, but rather form a foundational core. They are intended to be used as a

formative framework to guide curricula, assessment, and CCC deliberations in

programs. The Milestones will also ultimately guide and inform CCC deliberations

that lead to a summative judgment to allow the program director to decide upon an

individual resident’s/fellow’s readiness for entrustment decisions, promotion, and

graduation. However, the Milestones should not be used as the sole criteria for

these important decisions. Programs should read the companion Milestones

Guidebook that provides specific recommendations and guidance on how best to

use the Milestones in residency and fellowship programs

(http://www.acgme.org/Portals/0/MilestonesGuidebook.pdf

Programs should periodically review their policies, with input from the DIO and

institutional Legal and Human Resources team members, to address:

● Needed clarifications or adjustments in the criteria for promotion, program

completion, remediation, and/or non-renewal.

● Needed changes in the “agreement of appointment” necessary to reflect

Milestones reporting to the ACGME.

● Necessary changes in the grievance policy, ensuring program policies are

always aligned with the institutional policy.

Changes may not be necessary. However, the ongoing development of the CCC

provides an excellent opportunity to review current performance standards,

promotion/program completion criteria, and assessment processes, and align the

Milestones and the work of the CCC with them. The DIO, Office of GME, Legal,

and HR resources may provide useful guidance.

Step 2: Assess How Well You Know the CCC Requirements

Appendix B provides a multiple choice “quiz” on the current ACGME requirements

for a CCC; Appendix C includes a series of case studies.

Faculty development opportunities may include having the CCC members, the

members of the core faculty, and the program and/or institutional leadership take

the quiz, discuss the case studies, or use one or more readings as an

“educational” journal club. These resources may also be used with the program’s

residents/fellows to help them better understand the role of the CCC in the

program’s assessment process.

The ACGME’s CCC requirements are listed in Table 2. These are the same

across all programs, and the ACGME Review Committees are not able to further

specify requirements in these areas without approval from the ACGME Committee

on Requirements. There are minor differences in the wording but not the

underlying intent of the Common Program Requirements, for each type of GME

program: residency; fellowship; one-year fellowships; and post-doctoral education

programs. For example, requirement A.1.c).(2) in both the Common Program

Requirements for fellowship programs and for one-year fellowship programs, state

“that the program must provide [objective performance evaluation] to the CCC for

its synthesis of progressive fellow performance and improvement toward

unsupervised practice.”

Requirement A.1.c).(2) in the Common Program Requirements for post-doctoral

education programs state that, “the program must provide (objective performance

evaluation) to the CCC for its synthesis of progressive post-doctoral fellow

performance and improvement…”

Appendix D provides a template that may help programs design and/or evaluate

the CCC, by “walking through” its various components. Filling in the blanks can

generate a draft document that will provide a written description of the

responsibilities of the CCC.

The ACGME no longer requires a written description of the CCC. However,

programs may still benefit from a written description, which may serve as an

important communication tool for residents/fellows and faculty members. It may

also provide a concise description of the expected roles for faculty members on

the CCC, that they could use to support their own promotion, and/or to help the

program director negotiate for needed resources from the Division, Department, or

Institution.

Creating, developing, and improving a CCC to optimize its function does require

considerable time and effort. The long-term effectiveness of a CCC can be

facilitated by institutional support from the DIO for shared resources across

programs within an institution, and for appreciating that there will be a learning

curve for new programs. Ultimately, the CCC process will help programs do what

they have always been responsible for doing, but with greater structure, clearer

purpose, and more standardization across programs nationally.

Table 2 lists the CCC functions for a residency as described in the Common

Program Requirements (Residency), and for programs with ACGME Osteopathic

Recognition.

Table 2. Common Program Requirements for a CCC - Residency Version

(effective July 1, 2019)

Description of Requirements in Specialty/Subspecialty Programs

Common

Program

Requirement(s)

or Background

and Intent

section

(The program must) provide (objective performance evaluation based on

the Competencies and the specialty-specific Milestones, and must use

multiple evaluators, e.g., faculty members, peers, patients, self, and other

professional staff members) “to the CCC for its synthesis of

progressive resident performance and improvement toward

unsupervised practice.”

V.A.1.c).(1)-

V.A.1.c).(2)

The program director/designee with “input from the CCC” must meet

with/review with each resident/fellow the semi-annual performance

evaluation, including specialty-specific Milestones progress, assist in

developing individualized learning plans, and develop plans for those

failing to progress.

V.A.1.d)

The program director must provide a final evaluation for each resident

upon program completion; specialty -specific milestones, and when

applicable specialty-specific Case Logs must be used as tools to ensure

residents are able to engage in autonomous practice; the final evaluation

must “consider recommendations from the CCC.”

V.A.2.a).(2).(c).

The program director must appoint CCC members.

V.A.3.

There must be a minimum of three program faculty members, at least one

of whom is a core faculty member.

V.A.3.a)

Additional members must be faculty members from the same or other

programs, or other health professionals who have extensive contact and

experience with the program’s residents/fellows.

V.A.3.a).(1)

The CCC must review all resident/fellow evaluations at least semi-

annually.

V.A.3.b).(1)

The CCC must determine each resident/fellow’s progress on the specialty-

specific Milestones.

V.A.3..b).(2)

The CCC must meet prior to residents’/fellows’ semi-annual evaluations

and advise the program director regarding progress.

V.A.3.b).(3)

The (‘delineation of resident responsibilities in patient care, progressive

responsibility for patient management and graded supervision’) are

generally described by PGY level and specifically by Milestones progress

as delineated by CCCs.

Background and

Intent for IV.A.3

If the Review Committee allows program applicants to be accepted

through an “exception” process, these residents/fellows “must have an

evaluation of their performance by the CCC within 12 weeks of

matriculation.”

III.A.4.b)

FOR PROGRAMS WITH OSTEOPATHIC RECOGNITION

Osteopathic

Recognition

Requirements

The Director of Osteopathic Education or a designee should be a member

of the program’s CCC.

V.A.1.a)

The program’s CCC or a sub-committee of the CCC must review the

progress of all designated osteopathic residents in the program as it relates

to [osteopathic principles and practice] OPP.

V.A.1.b)

[The CCC or a subcommittee of the CCC must] include at least two

osteopathic faculty members, which may include the Director of

Osteopathic Education.

V.A.1.c).(1)

The CCC must review all designated osteopathic residents’ evaluations

semi-annually as these relate to the Osteopathic Recognition Milestones.

V.A.1.c).(2)

The CCC must prepare and ensure the reporting of Osteopathic

Recognition Milestones evaluations for each designated osteopathic

resident semi-annually to the ACGME.

V.A.1.c).(3)

The CCC must advise the program director and Director of Osteopathic

Education regarding resident progress, including promotion, remediation,

and dismissal from a designated osteopathic position.

V.A.1.c).(4)

General Principles for CCCs

The size of the residency or fellowship will affect how the program director

constructs the CCC process. For the purposes of this guidebook, “small programs”

have fewer than 15 total learners; “medium programs” have 15 to 75 learners; and

“large programs” have more than 75 learners.

One committee or more:

● Large programs may need to have several CCCs. There is a great deal of

variety in how programs construct sub-CCCs. Some programs create sub-

CCCs based on PGY, having separate sub-CCCs responsible for each PGY

cohort; others have a sub-CCC follow a cohort longitudinally from entering

residency through graduation; and still others have separate sub-CCCs for

large curricular themes or components within the program (e.g., a CCC that

will review resident/fellow scholarship, procedural competence/Case Logs,

or quality improvement activities). Others simply divide the residents into

more manageable numbers. Some CCCs have each member responsible

for a subset of the Competencies.

● If sub-CCCs are used, it is essential that they still have robust

membership and review processes to ensure all residents and fellows

are thoroughly reviewed, discussed, and provided with an opportunity

to receive high quality feedback. There also needs to be a mechanism

to integrate information from sub-CCCs and ensure each sub-CCC

has a shared mental model with the overall program and is using the

same standards and procedures.

● For medium-sized or small programs, a single CCC may be able to oversee

all residents/fellows. Program directors will use their discretion to determine

whether one CCC is sufficient based on the curricular design of the program

and local resources.

Committee membership:

● Appointment. The program director must appoint the CCC, which at a

minimum must include three faculty members, at least one of which is a

core faculty member in the program. Three is the smallest number of

individuals required for a truly good discussion. Some program directors will

want to use “term limits” and have a subset of CCC members rotate off each

year to refresh the CCC periodically and retain experience.

● Size. The literature suggests that a group size of five to seven individuals is

generally considered ideal, and no more than eight to 10 individuals in a

group is recommended for optimal committee functioning. CCC members

should regularly teach and observe residents/fellows.

● Diversity. The literature suggests that diverse groups make better decisions

than homogenous groups. To the extent possible, program directors should

try to balance CCCs in terms of academic rank, gender, race/ethnicity,

program role, and professional focus. (Hauer, 2016)

● Additional Members. The program director may appoint additional CCC

members from the same or other programs, or other health professionals

who have extensive contact and experience with the program’s residents

(e.g., nurses, physician assistants, nurse practitioners, social workers, etc.).

● Osteopathic Representation. If the program has Osteopathic Recognition

and has osteopathic-focused residents/fellows, there must be two

osteopathic-focused faculty members on the CCC, and the Director of

Osteopathic Education (or a designee) must also be on the committee.

● Chief Residents. Chief residents who have completed a core residency

program may serve on the CCC. Chiefs who are residents in the same

ACGME-accredited program (the chief title distinguishing their final year of

training) cannot serve on the CCC. It is important to make sure any chief

selected is comfortable with this role. A chief who completed the program

within the last year or two years may be too personally close to the

residents to be candid in this evaluation activity.

● Role of Advisors/Mentors. Program directors may want to consider whether

there is an inherent conflict of interest if a faculty member is an advocate for

a resident/fellow (as the resident’s advisor or mentor) and a “judge” of

performance (as a CCC member). On the other hand, advisors and mentors

may benefit from being observers to the CCC and contributing information

to the discussion and hearing the deliberation. This may better help them

convey the impressions of the CCC when they provide feedback to their

resident/fellow advisees.

Other considerations:

● The “right size” of a committee is large enough to reflect the diversity of

perspectives and small enough to be manageable. CCC members must be

able to attend meetings, actively participate, and engage in faculty

development about their CCC role.

● CCC members must be committed and able to attend all or nearly all

meetings; erratic attendance will not allow the continuity critical to assessing

resident/fellow performance over time. Each member must be willing to

make honest decisions, even when it is challenging.

● With regards to term limits and duration of service, consider whether

appointments should be “in perpetuity,” or for a defined time period. In

perpetuity appointments should be coupled with regular addition of new

members for fresh perspectives; if enacting term limits, consider staggering

appointments so that not everyone on the CCC turns over at once.

● Some programs have found value in having a “public member” to represent

a societal view, similar to the practices of many organizations’ boards,

including the ACGME’s. This is not an ACGME requirement, but anecdotally

some programs have described benefits of adding a non-faculty member,

such as a social worker, patient safety officer, or member of a

hospital/health system/school patient advisory board.

● Small programs may be challenged in identifying three CCC members if

they have a limited number of faculty members. Many fellowships will likely

be in this position. Three program faculty members, one of whom must be a

core faculty member (denoted as such on the program’s Faculty Roster),

are essential. In addition to program faculty members, consider inviting

faculty members from the core residency program, other related disciplines,

or settings in which the resident(s)/fellow(s) have substantial exposure

and/or provide substantial consultation. Many small programs are also tied

to specific clinical settings; consider inviting faculty members from such

settings who have ongoing contact with the resident(s)/fellow(s) to sit on the

CCC (e.g., a nurse leader from a dialysis unit for a nephrology fellowship, a

nurse anesthetist for a surgery fellowship, a patient safety officer, or a

discharge planner from a specific clinical unit).

● Medium-sized programs may also encounter some of the same challenges

in finding faculty CCC members as small programs.

CCC Chair:

Some boards or Review Committees may place restrictions on who can chair a

CCC. The American Board of Anesthesiology (2019), for example, does not allow

the program director to chair the CCC.

Other boards are silent on this issue.

Programs should reflect on who would be the right chair for the CCC: the program

director? the associate program director? another faculty member? a rotating

responsibility among members? voted on by CCC members? Program directors

should select the individual who will best solicit broad input regarding

resident/fellow performance and ensure all voices are heard.

CCC chairs should work with the CCC members to ensure a safe environment in

which all can freely share their judgements and concerns. The chair can mitigate

“hierarchy” within the group by having the most junior member(s) speak first. The

chair should ensure all residents are discussed, not just those perceived as having

problems or concerns. Table 3 identifies additional guidelines for the optimal CCC

chair.

Table 3: Guidelines for Committee Chairs, adapted from French et al. (2014)

Chairs should:

● Be the Milestones “expert” for the committee or designate another

committee member who will serve in this role.

● Encourage a confidential positive working environment and open

communication from all members.

● Ensure members know their roles, as well as the latest versions of the

Milestones and the CCC process.

● Engage members in developing a shared mental model for the

Milestones and the assessment tools.

● Use best practices in effective group processes; for instance, employ a

structured format to gain information from each committee member;

obtain input using the same order of members, get perspectives of the

most junior member first (See Part 4, Running the CCC Meeting).

● Keep meetings on task and move toward the common goal.

● Make certain the coordinator or designated member maintains

documentation and meeting minutes.

● Understand the typical assessment methods used by the program, as

well as their limitations.

● Develop a plan for the professional development of CCC members

(perhaps a dedicated period of time at the beginning or end of each

meeting, or an assigned article to read before the meeting).

● Anticipate biases on the part of both oneself and committee members,

and intentionally cultivate greater insight on biases and strategies to

mitigate them.

Program director role:

The Common Program Requirements do not proscribe a specific CCC role for the

program director. The program director can be a chair (except in anesthesia

programs), a CCC member, or an observer, or not attend CCC meetings at all. The

Background and Intent for this section of the Common Program Requirements

describes the rationale for this degree of flexibility. If present, the program director

should not detract from the participation of other team members by prematurely

inserting a personal perspective on a given resident’s/fellow’s performance. In the

same way, the program director should not determine the Milestones ratings of

each resident/fellow and then bring these to the CCC for ratification. The CCC

should be able to perform its assessment of resident/fellow competence freely,

judged against the Milestones, to convey to the program director.

Program directors who attend CCC meetings should defer to the chair, to make

sure other CCC members’ voices are encouraged (e.g., asking other members to

discuss residents/fellows and reach consensus decisions before adding their own

comments). Some program directors find it extremely useful to have another faculty

member chair the CCC, so they can function better as the resident/fellow advocate

and mentor and avoid the residents/fellows viewing the CCC’s judgments as “only”

those of the program director. On the other hand, the program director indeed has

the final responsibility for reporting and determining the Milestones ratings for each

resident/fellow and should also ensure the residents/fellows are aware of how their

performance on the Milestones has been reported to the ACGME.

Coordinator role:

Program coordinators are essential in the CCC process through their involvement

with many, if not all, aspects of the program, and their knowledge of the

residents/fellows. Program coordinators frequently distribute and collect results

from assessment tools. They may also participate in multisource feedback by using

assessment instruments to share valuable and often unique perceptions of an

individual resident’s/fellow’s abilities in interpersonal and communication skills,

teamwork, and professionalism.

Program coordinators may attend CCC meetings in an administrative role at the

discretion of the program director. They can assist in the collection, preparation,

organization, and distribution of assessment data; take minutes; and capture key

aspects of the discussion. They can observe group process using some of the tools

and frameworks provided below and provide feedback to the CCC as part of a

continuous quality improvement (CQI) process. Following a CCC meeting, the

program coordinator can facilitate the communication of results to the program

director (if not in attendance); schedule meetings with individual residents/fellows

and the program director or designated faculty member to review decisions,

including Milestones status; and electronically submit Milestones information on

each resident/fellow to the ACGME. The coordinator can also capture information in

the CCC “debriefs” that may lead to improvements in the CCC process at the next

meeting. However, the program coordinator cannot be a CCC member, or make

judgments in or after the meeting regarding resident/fellow performance.

Coordinators should provide assessment and feedback through the program’s

assessment system, such as by participating in multisource assessment

instruments.

Members of the CCC:

Each member of the CCC will have various tasks to complete prior to, during, and

after each meeting. Table 4 summarizes these.

Table 4: Role/responsibility of each CCC member, modified from French et al.

Guidelines for Committee Members:

● Understand the purpose and responsibilities of the CCC

● Know role on the committee

● Recognize sources of likely biases and take steps to mitigate their impact

● Work with other members to develop a shared mental model of the

Milestones

● Follow through with assigned tasks (such as pre-review and synthesis of

resident/fellow performance data)

● Participate in ongoing professional development (the Milestones, best

practices in assessment, effective group process, understanding and

identifying bias)

● Facilitate a collegial, respectful atmosphere within the committee

● Use best practices to support a robust group process

● Ensure own honest “voice” is heard along with those of colleagues

● Maintain confidentiality

● Help orient new members

● Contribute to ongoing improvement of the CCC processes

Meetings:

Logistics of meetings should include location, frequency, and length. CCCs may

wish to meet more frequently than the minimum requirement of twice yearly. There

is no one way to accomplish their task. A study of 116 emergency medicine

program directors found that slightly over half met quarterly, and a third monthly.

(Doty, 2016) Approximately 40 percent of the CCCs reviewed the entire program at

a single sitting, and a third reviewed an entire class of residents at a meeting, such

as all PGY-1s.

References

1. ACGME. ACGME Common Program Requirements (Fellowship).

https://www.acgme.org/Portals/0/PFAssets/ProgramRequirements/CPRFello

wship2019.pdf. 2020.

2. ACGME. ACGME Common Program Requirement (One-Year Fellowship)

https://www.acgme.org/Portals/0/PFAssets/ProgramRequirements/CPROneY

earFellowship2019.pdf. 2020.

3. ACGME. Osteopathic Recognition Requirements.

https://www.acgme.org/Portals/0/PFAssets/ProgramRequirements/801Osteo

pathicRecognition2018.pdf?ver=2018-02-20-154513-650. 2020.

4. ACGME. ACGME Common Program Requirements (Post-Doctoral Education

Program).

https://www.acgme.org/Portals/0/PFAssets/ProgramRequirements/CPRPost-

Doctoral2018.pdf. 2020.

5. ACGME. ACGME Common Program Requirements (Residency).

https://www.acgme.org/Portals/0/PFAssets/ProgramRequirements/CPRResid

ency2019.pdf. 2020.

6. ACGME International. ACGME International Foundational Program

Requirements for Graduate Medical Education.

https://www.acgme-

i.org/Portals/0/FoundInternational03312016.pdf?ver=2016-04-25-084913-

527. 2020.

7. ACGME. ACGME Institutional Requirements.

https://www.acgme.org/Portals/0/PFAssets/InstitutionalRequirements/000Insti

tutionalRequirements2018.pdf?ver=2018-02-19-132236-600. 2020.

8. American Board of Anesthesiology. 2019 Policy Book. Page 14.

http://www.theaba.org/PDFs/BOI/2019-Policy-Book

. 2020.

9. Doty CI, Roppolo LP, Asher S, et al. How do emergency residency programs

structure their Clinical Competency Committees? A survey. Acad Emerg Med.

2015;22(11):1351-1354.

https://onlinelibrary.wiley.com/doi/full/10.1111/acem.12804

. 2020.

10. French JC, Dannefer EF, Colbert CY. A systematic approach toward building

a fully operational Clinical Competency Committee. J Surg Educ.

2014;71(6):e22-e27.

https://www.sciencedirect.com/science/article/abs/pii/S193172041400107X?v

ia%3Dihub. 2020.

11. Hauer KE, Cate OT, Boscardin CK, et al. Ensuring resident competence: A

narrative review of the literature on group decision making to inform the work

of Clinical Competency Committees. J Grad Med Educ. 2016;8(2):156-164.

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4857505/

. 2020.

Part 3: Preparing for CCC Meetings

Developing a Shared Mental Model

Perhaps the most important aspect of preparing for a CCC meeting is to make sure

the members develop a shared mental model of what resident/fellow performance

looks like, and understand their roles and responsibilities on the committee, as well

as how the CCC operates to judge resident/fellow performance. Developing a

shared mental model of the Competencies and the Milestones is essential. This will

usually necessitate a “meeting before the meeting,” or allocating sufficient time at

the beginning of a CCC meeting for this discussion before a new CCC gets started

on its first reviews. CCCs should also engage in ongoing dialogue to enrich and

deepen their understanding and mental model over time. Having a written

description of the CCC process (though no longer required by the ACGME) and

providing faculty development for committee members, can facilitate this. Some

programs find it useful to discuss a relevant article at a CCC meeting as part of

faculty development. See the references and annotated bibliography for some

suggestions.

Faculty members should reach a common understanding on the meaning of the

narratives of each milestone in the context of the specialty. This will almost always

require group conversation. It may be worthwhile to have each faculty member

perform self-assessment, using the specialty-specific Milestones, as a faculty

development exercise. Faculty members should be trained to compare each

resident’s/fellow’s performance to the Milestones as a whole, not just to the

performance of other or ‘typical’ residents/fellows in the program.

Note that as Milestones 2.0 is available for individual specialties, a Supplemental

Guide is also available. The Supplemental Guide includes the intent of the

subcompetency along with examples for each level, assessment methods, and

resources. The Supplemental Guide can be used to develop a shared mental

model and determine examples for each program. The CCC can individualize the

Supplemental Guide and use it as it considers resident development over time.

Members may also benefit from individually assessing recent program graduates

using the new Milestones, and then discussing as a committee to determine a

group consensus as another potential faculty development exercise.

Inventory Where Milestones are Represented in the Program

Competency-based medical education entails defining outcomes of education and

training, which then guide development of milestones to chart progress through the

course of a residency and fellowship program. Each program must determine which

of its assessment tools and activities address each milestone. CCCs should

inventory (or review an inventory conducted by others) where each milestone is

currently taught and assessed in the program to create a map that guides design

of curricular experiences. Teaching may occur on a specific rotation, or in the

context of a program activity, such as “leading morbidity and mortality rounds.”

Appendix E lists assessment methods for each of the Competency domains with

some representative examples.

The inventory should help to identify gaps in both curriculum and assessment. The

CCC can identify how to best address these gaps, perhaps by delegating the

review to a designated faculty member.

Shift from Assessment of Learning to Assessment for Learning

The foundation of competency-based education requires robust assessment with

regular honest feedback to each learner. This requires a large amount of data. The

CCC is in the perfect position to analyze how assessments drive learning, inform

the judgement of resident progress, and improve the overall quality of the program.

The assessment information and data that inform CCC deliberations necessitate a

comprehensive and intentional overall program assessment strategy. It should

follow several key principles:

● The Milestones were never meant to be used as a standalone assessment tool,

especially for short rotations (e.g., two to 12 weeks). Some programs continue

to use the entire Milestones Set for end-of-rotation evaluations. This typically

works poorly despite the fact this may seem a logical expedient, and even

helpful to faculty members, to better acquaint them with the Milestones and the

skills, attitudes, and behaviors they need to assess. However, there are several

major issues. First is the concept of cognitive load – the more you ask faculty

members to judge in shorter periods of time, the more difficult it is to truly

assess all the Competencies. Faculty members may feel pressed to assess

residents on milestones they did not directly observe, leading to range

restriction (i.e., using a very limited range of the Milestone levels), “straight

lining” (i.e., residents rated exactly the same on all Milestones), and halo

effects (i.e., strength in one area, such as Medical Knowledge, “spills over” into

ratings of other areas, especially if they were poorly assessed).

● Programs may consider a “retreat” to take each milestone and map out where it

is taught and assessed, as well as how it is assessed in the program. This will

highlight any gaps and opportunities for improvement. Frequently this can be

done collaboratively, either with other programs in the same state or region in

the same specialty, or with other programs of different specialties within the

same institution.

● The assessment program will need to include multiple forms of assessment

with multiple sampling using multiple assessors. No single assessment method

or tool is sufficient to judge something as varied and complex as clinical

competence. While end-of-rotation evaluations have some value, an

overreliance on global, end-of-rotation evaluations should be avoided.

● The combination of assessments will depend to some extent on the specific

needs of the specialty and the local context. Consult the Milestones Guidebook

for more information.

● At a minimum, core methods of assessments should include direct observation

of a specific component (e.g., care of individual patients, procedures, hand-

offs), multi-source feedback, multiple choice test/in-service examination,

longitudinal evaluations (e.g., rotational evaluation forms), audit of clinical

performance, and simulation where appropriate. The specific assessment tools

used will depend on the specialty and local context. The key point to remember

is that the true assessment “instrument” is not the tool or form itself, but rather

the individuals using it. The tool or form simply guides the individual performing

the assessment. CCCs should be cautious not to place an overreliance on

global, end-of-rotation evaluations, which too often fail to provide meaningful

comments and are limited by their scales and items.

● Faculty members and others involved in assessing residents/fellows will need

training in the use of and interpretation of data from the selected assessment

tools.

Some opportunities for assessments include the methods included in Appendices E

and F. Please note the lists are not comprehensive; consult the Milestones

Guidebook and the recent overview by Lockyer et al. (2017).

Preparing for Specific CCC Meetings

Another key pre-meeting activity is preparing the assessment data for review. It is

important to plan how all assessment information, including information that occurs

at the meeting, and from information gained through hallway conversations or other

informal sources, will be collected and summarized. Many resident management

systems (RMS) have tools available to aggregate evaluations, such as spider

graphs (aka radar plots), visual plots, and dashboards. These have been shown

both to make CCC discussions more efficient and to help in giving feedback to the

residents following the CCC meeting. Some learning management systems have

the ability to perform basic statistics on assessment data and may display visually

with dashboard tools such as spider graphs. While this is helpful, a word of caution:

simple means (i.e., averages) of aggregated assessments can be misleading,

especially if ranges and confidence intervals are not provided. In these cases, an

important outlier assessment might be missed and not properly reviewed and

discussed. Also remember the cardinal GIGO (“garbage in, garbage out”) rule: if the

quality of the assessments being used to produce aggregate data, such as

averages, is poor, then not even fancy statistics can make the assessment

information better.

It is also important for CCCs to examine the assessment data longitudinally. This

can be especially helpful once residents and fellows have acquired several cycles

of Milestones judgments, typically starting in their second year. The 2019

Milestones National Report includes predictive probability value (PPV) tables for

most GME programs that can assist CCCs in identifying residents with suboptimal

developmental trajectories within a subcompetency.

Larger CCCs may assign members a subset of the residents/fellows for whom to

review the assessment information in advance and prepare a preliminary review.

An individual member may be responsible for reviewing all measures of the

assigned residents’/fellows’ performance and preparing a synopsis that is brought

to the meeting and discussed with the full CCC. Some programs have individual

members complete Milestones assessments on each resident or fellow and have

the coordinator aggregate the information in advance of the meeting.

Suggested practices:

1. Synthesize performance information (done by the coordinator or assigned

CCC member) in advance of meeting.

2. Share written performance information about individual resident/fellow

performance during the CCC meeting (e.g., in a handout, a projection in the

room).

3. Train CCC members on how to interpret aggregated, synthesized performance

information about individual residents/fellows. This means that CCC members

must understand the nature and quality of the synthesized assessment data.

4. Maintain the confidentiality of the information. Failure to do so will undermine

trust in the Milestones and the CCC process.

5. Increase the use of direct observation, video logs, outcomes from actual

clinical performance, patient experience, and team member data as data

sources.

6. Practice good group process.

7. Review the specialty’s PPV tables in the 2019 Milestones National Report

8. Identify what’s not being assessed that may be critical to authentically judge

Milestone performance. CCCs have been shown to neglect resident/fellow

quality improvement, patient surveys, and actual clinical performance data.

Prior to the implementation of the Milestones, Hauer’s 2015 study of 34 program

directors at five institutions discovered that most CCCs relied on global, end-of-

rotation evaluations rather than using programmatic assessment with multiple tools

and data points, focused on problem residents more than they spent time

discussing the typical residents, and lacked faculty development or training of CCC

members. A small, single-institution study found that faculty members’ evaluations

received substantial weight in CCC deliberations in a large internal medicine

residency that used sub-CCCs, but the sub-CCCs weighed comments next in

importance. (Epkenyong, 2017) A study of 14 pediatric CCCs found that only two

considered participation in quality improvement projects (Schumacher, 2018).

Finally, a study by Watson, et al. (2017) found patient surveys identifying 13 of 19

factors the CCC used as important to evaluation; performance data was not

included in the 19 factors. Each of these studies speaks to the need to collect

robust data and provide deliberate, ongoing faculty development for those who

serve on the CCC, especially in the Milestones era.

Dashboards

A strategy for efficient data synthesis and display facilitates the work of the CCC.

Data synthesis remains challenging for many programs that may share too much,

too little, or poorly organized data with CCC members. A dashboard offers a

platform for high-level data display, combined with drill-down options for more detail

on quantitative and qualitative measures of learner performance. This information,

combined with display of metrics indicating expected levels of performance,

enables evidence-informed feedback discussions between residents/fellows and

their faculty advisors or coaches to inform robust learning planning.

Some programs document their CCC deliberations through their resident

management system (RMS). The RMS can create a Milestones evaluation

composite, and often graphical plots, which can be shared electronically with a

resident/fellow and stored with all the other resident/fellow evaluations. (Friedman,

2016; Johna, 2015)

Key Point: Whatever method is used to “pre-digest” and organize the data for

review, programs should ensure processes and/or standard protocols are in place

to ensure a systematic, consistent approach to the pre-review and the meeting

preparation process. Programs should not simply use statistical means (i.e.,

averages) or a single type of data to make CCC determinations. Narrative data

collected from assessment tools represents important additional information for the

CCC. As noted above, the Milestones do not represent the totality of the discipline,

and informed human judgment is still a critical component of the CCC process.

Much important and useful assessment information is attained through effective

group discussion at the CCC meeting.

References

1. Ekpenyong A, Baker E, Harris I, et al. How do clinical competency committees

use different sources of data to assess residents' performance on the internal

medicine milestones? A mixed methods pilot study. Med Teach.

2017;39(10):1074-1083.

https://www.tandfonline.com/doi/abs/10.1080/0142159X.2017.1353070?journal

Code=imte20. 2020.

2. Friedman KA, Raimo J, Spielmann K, Chaudhry S. Resident dashboards:

helping your Clinical Competency Committee visualize trainees' key

performance indicators. Med Educ Online. 2016;21:29838.

https://www.tandfonline.com/doi/full/10.3402/meo.v21.29838

. 2020.

3. Hauer KE, Chesluk B, Iobst W, et al. Reviewing residents' competence: a

qualitative study of the role of Clinical Competency Committees in performance

assessment. Acad Med. 2015;90(8):1084-1092.

https://journals.lww.com/academicmedicine/fulltext/2015/08000/Reviewing_Resi

dents__Competence__A_Qualitative.25.aspx. 2020.

4. Johna S, Woodward B. Navigating the Next Accreditation System: A dashboard

for the Milestones. Perm J. 2015;19(4):61-63.

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4625996/

. 2020.

5. Lockyer J, Carraccio C, Chan MK, et al. Core principles of assessment in

competency-based medical education. Med Teach. 2017;39(6):609-616.

https://www.tandfonline.com/doi/abs/10.1080/0142159X.2017.1315082?journal

Code=imte20. 2020.

6. Schumacher DJ, Michelson C, Poynter S, et al. Thresholds and interpretations:

How Clinical Competency Committees identify pediatric residents with

performance concerns. Med Teach. 2018;40(1):70-79.

https://www.tandfonline.com/doi/abs/10.1080/0142159X.2017.1394576?journal

Code=imte20. 2020.

7. Watson RS, Borgert AJ, O Heron CT, et al. A multicenter prospective

comparison of the Accreditation Council for Graduate Medical Education

Milestones: Clinical Competency Committee vs. resident self-assessment. J

Surg Educ. 2017;74(6):e8-e14.

https://www.sciencedirect.com/science/article/abs/pii/S1931720417302490?via

%3Dihub. 2020.

Part 4: Running the CCC Meeting, Including Effective Group Process

The CCC meeting can serve multiple purposes and the chair should ensure that all

members have a shared understanding about the role of the CCC and its goals. In

addition to rating residents’/fellows’ performance on the Milestones, other important

tasks can be accomplished, such as faculty development for the members, and

monitoring the quality of the assessment system. This section focuses on the many

processes and procedures that occur during CCC meetings. CCCs are charged

with rendering judgements about resident/fellow progress on the Milestones. It is

important to consider the following tips to bring the committee closer to realizing

this goal.

1) At the beginning of (or prior to) the meeting:

a. The CCC must have a shared understanding of its role in the assessment

system. Hauer et al. found that program directors had two different

perceptions of the role of the CCC: that of “problem identification” (i.e.,

focused on identifying the struggling residents/fellows) and “developmental”

(i.e., focused on helping all residents/fellows on their trajectory towards

achieving the Milestones). (2016) Although CCC chairs should contemplate

this prior to the meeting, during the meeting they should take opportunities

to move the group toward a developmental approach to benefit all learners

in the program.

b. Members should avoid coming to the meeting with a decision already

predetermined. It is inappropriate to use the CCC to simply confirm a

“verdict” about a resident or fellow from one member’s opinion or a set of

data. This may seem tempting; however, it significantly undermines the

benefit of having a group discussion, and pre-determined verdicts can be

inaccurate.

c. Plan for the group to discuss and agree on some ground rules or

“touchstones” for how the group will work together. Touchstones are simply

principles of engagement the group agrees to observe and to which

members hold each other accountable. For example, one touchstone might

be “all member opinions will be considered respectfully.”

2) During the meeting:

a. There should be a consistent and structured process for presenting each

resident/fellow during the meeting to ensure all members have an

opportunity to voice their opinions and any information that any given

member has about an individual resident/fellow is shared by the group.

CCC chairs should develop a structure or format to use during discussion

of each resident/fellow to ensure that key elements are not missed and so

that any relevant information that CCC members may have about a

resident/fellow that is not captured in the formal assessments can be

introduced. Some CCCs choose to assign residents/fellows to specific

members and ask the latter to present a summary about each of their

residents’/fellows’ performance. Donato and colleagues’ description of their

internal medicine residency CCC is one in which the resident is presented

in a “debate-like” format. (2016) Mentors present their resident(s) to the

CCC describing their accomplishments, and a second reviewer presents

challenges. The committee then discusses the presented information and

the mentor provides feedback to the resident.

b. Discussions of each resident/fellow may be enhanced by using charts,

pictures, spreadsheets, or other visual aids during the meeting. As noted

earlier, the PPV tables from the Milestones National Report can also help

guide the discussion. See Appendix G for an example of a PPV table.

c. At all times the chair should use guidelines for effective group process. It is

important to be mindful of issues of hierarchy that can negatively affect the

group dynamic. This is particularly an issue when a more senior faculty

member serves as CCC chair. One clear measure of the effectiveness of

the CCC is the willingness of all members to speak up. Consider using the

following techniques to minimize the effects of hierarchy:

i. Always start with the most “junior” person or the person most at risk in

the hierarchical chain.

ii. Chairs should, as a general rule, state their opinion last. In addition,

program directors, if present, should avoid stating their opinion early

on, if at all, depending on their role with the CCC. If present, the

program director’s role may be best as an observer, to “listen” to the

conversation and provide clarifying information if necessary, but not to

voice opinions, at least not until later in the discussion.

d. Effective group process is critical to the success of a CCC. The underlying

premise is that under the right circumstances, groups make better

decisions than individuals. The following are some examples of this

phenomenon, both within medical education and beyond:

i. Schwind et al. – deficiencies in surgical residents were uncovered via

group discussion, not during individual faculty member review. (2004)

ii. Hemmer et al. – professional lapses in medical students were only

identified as a result of formal group discussion. (2000)

iii. Thomas et al. – group discussions prior to the completion of the

evaluations of internal medicine residents in continuity clinic resulted

in higher reliability. (2011)

iv. The Wisdom of Crowds – author James Surowiecki, New Yorker

columnist, uses examples from a range of fields to demonstrate that

under the right circumstances (e.g., having diverse opinions, avoiding

groupthink) groups make better decisions than individuals. (2005)

e. CCC chairs and program directors need to be aware of the importance of

having a diversity of opinions expressed to enrich the group decision-

making process. The perspective each member brings to the discussions

(based on clinical expertise, research, medical education expertise, etc.)

is important. Research shows that minority opinions, even when “wrong,”

can lead to better decisions. In addition, CCC chairs need to have an

understanding of the factors that enable groups to function effectively

and

the potential biases that CCCs can encounter.

f. CCCs should have a shared mental model as to which assessment data

they need for their decision-making process. CCC chairs should take note

when this is not the case and should continue to work on building or

refining their “Milestones map” (see section on “Other uses of the CCC”).

In addition, committee members will likely bring information about many

residents and fellows not captured on completed assessment tools and

forms. The CCC provides a forum to hear this previously unshared

information. This information is critical to making a robust overall

assessment of each resident’s or fellow’s progress. However, if a program

finds that most of the useful information comes from CCC discussion and

is not written down on any assessment forms, it should consider revising

its assessment tools or processes and/or faculty development to solicit

better written/recorded information. Members may also need to spend

time discussing the “value” of different types of assessment data. They

may struggle with how much to value their own first-hand knowledge of a

resident’s or fellow’s performance versus information provided by

colleagues on formal assessments.

g. CCCs should have a shared understanding of how decisions will be made,

including how to deal with inadequate assessment data or lack of data. A

few studies have attempted to outline how CCCs make decisions.

Through observations of CCC meetings, Pack and colleagues describe

the process of making sense of assessment data that are difficult to

understand, and how the discussion of how to use the data enriches the

decision-making process. (Pack, 2019) Chahine and colleagues

performed a review of the literature on group process, and developed a

theoretical framework consisting of three “orientations” or approaches to

decision making by CCCs, namely “schema” (use of rules, guidelines),

“constructivist” (members work together to develop meaning and

understanding), and “social influence.” (2017) Also at play in this

framework are “moderators,” such as time, leadership style, etc., which

have an impact on the process.

h. Strategies to organize the conversation flow to ensure a systematic

approach with minimal bias can be inferred from the literature on clinical

reasoning. For example, Lambe (2016) and Croskerry (2003) found that

using cognitive forcing strategies allows for structured approaches to what

is discussed and how. Another recommendation from a review on

strategies to optimize clinical reasoning decision making is guided

reflection, in which decision making is slowed to avoid quick assumptions

and to lead individuals to consider information more deliberately. (Lambe)

The framing effect describes how individuals are swayed by whether a

scenario or option is portrayed positively or negatively. Bringing

awareness to the risk of the framing effect and overconfidence influencing

decisions is important for CCCs. (Saposnik, 2016)

i. How the decisions are made by the group is also important. The best

approach is for CCC members to choose the best description of the

residents’ or fellows’ abilities using the Milestones narratives, not the

numeric levels. Some studies have suggested that rating based on

narrative tends to be more discriminating. Too often when individuals start

by choosing a number rating, they will be more likely to try and justify that

rating. Encourage all members to focus on the narrative Milestones

descriptors.

j. The chair can determine the frame of reference that the CCC members

are using when rating the residents/fellows. The Milestones framework is

criterion-based, referring to specific expectations for all residents/fellows

to meet. However, sometimes faculty members may find themselves

comparing a given resident’s/fellow’s performance to themselves (self) or

to other residents/fellows (peer). If a resident/fellow has not rotated

through an experience over the past six months, and that hinders the CCC

in making a determination on one of the milestones, the CCC should

maintain the Milestone judgment from the previous reporting period.

k. The committee members will need to determine how best to spend their

time, e.g., time spent on struggling learners can often consume the

majority of the meeting, leaving little opportunity for discussing how to

assist residents/fellows with satisfactory or an even higher performance

level to create stretch goals. Large programs may address this in several

ways. To avoid having to discuss too many residents/fellows in one

meeting, some programs develop “subcommittees,” e.g., one for each

post-graduate year. These subcommittees may meet prior to the large

CCC meeting to discuss their assigned residents/fellows. In these

instances, subcommittees review each resident’s/fellow’s data and

discuss this in their meetings. During this process, they identify concerns

to bring to the full CCC. Other CCCs set up more than one meeting per

six-month cycle, e.g., once per month or every other month. They may

intentionally devote some of their meetings to be “formative” (i.e., focused

on ensuring they have all the necessary data and discussing performance,

but not assigning Milestones ratings), while other meetings are

“summative,” during which the committee actually rates the

residents/fellows on the Milestones. All residents/fellows, from the lower

performing residents and fellows to the “superstars,” need individualized

educational learning plans. Competence is the “floor,” and not the “ceiling”

of performance, and higher performing residents/fellows can be

challenged to develop further. Longer discussions tend to produce better

decisions and will likely produce better feedback.

l. Time pressure or trying to cover too many residents/fellows in one

meeting can produce lower quality decisions.

m. There should be a clear process for allowing the CCC to forward their

concerns about a given resident’s/fellow’s performance, their suggestions

for remediation, and their expectations for follow-up to either another

committee or the program director. There should also be clarity about the

expected outcomes of the meeting, which include not only the Milestones

ratings generated twice per year, but also feedback from the CCC to the

residents/fellows. The group will need to come to a consensus about the

type of feedback generated and develop a process for delivering it. The

CCC may also provide feedback to other stakeholders, such as the

program and core faculty members.

n. CCC chairs may choose to include some time during the meeting for

faculty development, such as regarding developing shared mental models

(as above), rater training, the pros and cons of various assessment

methods, building a “Milestones map,” etc.

3) Post-meeting:

a. The discussion about each resident/fellow should be captured and

documented (see Part 6: Legal Issues and Considerations). The

discussion and judgments of the CCC are legitimate and important

assessment information and should become part of each

resident’s/fellow’s record. This information should also serve as the

template for the feedback session with each resident/fellow. See section 5

for details regarding providing feedback to residents/fellows.

b. Transparency is an important principle in the ACGME’s accreditation

model. Accurately documenting and sharing the key components and

judgments with residents and fellows is a critical aspect of this principle.

c. Taking time at the end of each meeting to debrief how the meeting went

can improve processes at future meetings. The chair can generate a

discussion among the group by simply asking what went well, what could

be improved, and how members would like to see things move forward.

The coordinator, serving as an observer, can also provide useful feedback

for the whole group if appropriately guided and empowered to do so.

Thus, assess if the CCC is meeting its goals and determine how to

improve the next meeting.

d. In conjunction with assessment of residents/fellows, CCCs will

increasingly assess the performance of program. In assessing

resident/fellow performance against the Milestones, it will become clear

what is missing from the program’s assessment “toolkit,” and if there are

curricular gaps and redundancies. CCC deliberations can generate

behaviorally-specific feedback that will be useful to learners. But CCCs

will also identify feedback useful for faculty members. Some faculty

members will be recognized as role models for the timeliness, quality, and

quantity of their evaluations. The CCC can help these individuals to be

recognized, perhaps as part of promotion and tenure considerations, or

through incentives. Others may be tapped to coach fellow faculty

members whose evaluations could be improved.

e. The CCC should provide a synopsis of its findings to the Program

Evaluation Committee for its use in improving the assessment system

within the program.

f. The CCC, therefore, has an important role in the continuous educational

quality improvement of faculty members and the program, in addition to its

role in assessing residents/fellows.

It is recommended that the CCC revisit its purpose, shared mental model, and

procedures annually. Ongoing faculty development for CCC members to help

prevent the development of groupthink or drifting from the original aims and

procedures is critical.

As listed above, there are many factors to consider when planning or conducting a

CCC meeting. Should CCCs choose to maintain a written “policies and procedures”

document, it should be updated at least annually. CCCs should include the above-

mentioned processes in this document. Doing so not only fosters a quality

improvement approach to the workings of this committee but allow for greater

transparency of the CCC’s work to stakeholders.

Anticipating, Recognizing, and Mitigating Bias

Ensuring a fair and equitable assessment system constitutes a fundamental

obligation of the CCC to ensure that learners are afforded maximal opportunities to

learn and thrive in the program.

Especially as the diversity of learners continues to increase, CCC members require

awareness and training regarding bias in evaluations of learner performance.

Multiple studies and experts describe concerns about the risks of bias influencing

the evaluations of learners from students to postgraduate residents/fellows based

on gender and race/ethnicity. Emerging studies suggest that bias affects both

numerical and qualitative evaluations of learner performance. Quantitative ratings

of student and resident/fellow performance have been shown to be systematically

lower for women than men, (Dayal, 2017; Klein, 2019) and lower for

residents/fellows from backgrounds underrepresented in medicine. (Teherani,2018;

Backhus, 2019; Boatright, 2017) Narrative resident/fellow performance data also

reinforces stereotypes through use of different words to describe the performance

of different groups based on gender or race/ethnicity. (Rojek, 2019; Mueller, 2017;

Gerull, 2019; Salles, 2019; Ross, 2016; Isaac, 2011)

To address this important risk of bias influencing resident/fellow performance

ratings during CCC discussions, programs should do the following:

• CCC membership should include diverse members in terms of gender and

race/ethnicity. Diverse groups outperform homogeneous groups in terms

of the quality of their work and decision making. (Hong, 2004)

• All CCC members should participate in training on diversity, equity,

inclusion, and bias. Training can entail deepening one’s understanding of

unconscious bias and racism that permeates health care and medical

education. CCC members should appreciate how bias based on learners’

race/ethnicity or gender can impact both quantitative and qualitative

ratings of learner performance. Learners who are not white or who are

women receive lower numerical ratings and are less likely to be selected

for the Alpha Omega Alpha Honor Medical Society. (Boatright, 2017;

Mueller, 2017; Teherani, 2018) Women also receive less favorable ratings

on some milestones than men in some specialties (Dayal, 2017; Klein,

2019; Santen, 2019). Another study from Hamstra (2019) showed that in

pediatrics and some family medicine milestones, women scored higher.

Review of narrative comments about learner performance shows how

certain words may be systematically used more to describe individuals

based on the groups to which they belong. (Mueller, 2017; Rojek, 2019)

Through ongoing discussion and reflection, CCC members can share and

address their own perspectives and biases in order to recognize and

mitigate unconscious biases. (Morgan, 2018)

• The CCC should examine the program’s own data for any systematic

group differences in performance that signal bias in the evaluation data. In

their roles using CCC data to continuously improve the program, CCC

members should be vigilant for signs indicating how issues of race and

racism may be influencing residents’ learning experiences. (Karani, 2017)

• The CCC should discuss and reflect upon their performance ratings to

identify any areas in which bias may be influencing ratings and discuss

improvements to their processes. This reflection process can be

structured by reviewing the CCC’s data, including ratings for learners

based on gender and race/ethnicity. This review can uncover systematic

differences, as have been observed in some Milestones ratings showing

higher assessment of men than women in certain milestones traditionally

thought of as more ‘male’ characteristics. (Santen, 2019; Dayal, 2017)

References:

1. Backhus LM, Lui NS, Cooke DT, Bush EL, Enumah Z, Higgins R.

Unconscious bias: Addressing the hidden impact on surgical education.

Thorac Surg Clin. 2019;29(3):259-267.

https://www.thoracic.theclinics.com/article/S1547-4127(19)30015-5/fulltext

2020.

2. Boatright D, Ross D, O'Connor P, Moore E, Nunez-Smith M. Racial

disparities in medical student members in the Alpha Omega Alpha Honor

Society. JAMA Intern Med. 2017;177(5):659-665.

https://jamanetwork.com/journals/jamainternalmedicine/fullarticle/2607210

2020.

3. Chahine S, Cristancho S, Padgett J, Lingard L. How do small groups make

decisions? A theoretical framework to inform the implementation and study

of Clinical Competency Committees. Perspect Med Educ. 2017;6(3):192-

198. https://link.springer.com/article/10.1007%2Fs40037-017-0357-x

. 2020.

4. Croskerry P. Cognitive forcing strategies in clinical decision making. Ann

Emerg Med. 2003;41(1):110-20.

https://linkinghub.elsevier.com/retrieve/pii/S0196064402849459

. 2020.

5. Dayal A, O’Connor DM, Qadri U, Arora VM. Comparison of male vs. female

resident Milestone evaluations by faculty during emergency medicine

residency training. JAMA Intern Med. 2017;177(5):651-657.

https://jamanetwork.com/journals/jamainternalmedicine/fullarticle/2607209

2020.

6. Donato AA, Alweis R, Wenderoth S. Design of a Clinical Competency

Committee to maximize formative feedback. J Community Hosp Intern Med

Perspect. 2016;6(6):33533.

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5161790/

. 2020.

7. Friedman KA, Raimo J, Spielman K, Chaudry S. Resident dashboards:

helping your Clinical Competency Committee visualize trainees’ key

performance indicators. Med Educ Online. 2016;21:29838.

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4818357/. 2020.

8. Ekpenyong A, Baker EA, Harris I, et al. How do Clinical Competency

Committees use different sources of data to assess resident performance

on the internal medicine milestones? A mixed methods pilot study. Medical

Teacher. 2017;39(10):1074-1083.

https://www.tandfonline.com/doi/abs/10.1080/0142159X.2017.1353070?jour

nalCode=imte20. 2020.

9. Gerull KM, Loe M, Seiler K, McAllister J, Salles A. Assessing gender bias in

qualitative evaluations of surgical residents. Am J Surg. 2019;217(2):306-

313. https://linkinghub.elsevier.com/retrieve/pii/S0002-9610(18)30631-7

2020.

10. Hauer KE, Chesluk B, Iobst W, et al. Reviewing residents’ competence: a

qualitative study of the role of Clinical Competency Committees in

performance assessment. Acad Med. 2015;90(8):1084-1092.

https://journals.lww.com/academicmedicine/fulltext/2015/08000/Reviewing_

Residents__Competence__A_Qualitative.25.aspx. 2020.

11. Hemmer PA, Hawkins R, Jackson JL, Pangaro LN. Assessing how well

three evaluation methods detect deficiencies in medical students’

professionalism in two settings of an internal medicine clerkship. Acad Med.

2000;75(2):167-173.

https://journals.lww.com/academicmedicine/Fulltext/2000/02000/Assessing_

How_Well_Three_Evaluation_Methods_Detect.16.aspx. 2020.

12. Hong L, Page SE. Groups of diverse problem solvers can outperform

groups of high-ability problem solvers. PNAS. 2004;101(46):16385-16389.

https://www.pnas.org/content/101/46/16385.long

. 2020.

13. Isaac C, Chertoff J, Lee B, Carnes M. Do students’ and authors’ genders

affect evaluations? A linguistic analysis of medical student performance

evaluations. Acad Med. 2011;86(1):59–66.

https://journals.lww.com/academicmedicine/Fulltext/2011/01000/Do_Studen

ts__and_Authors__Genders_Affect.22.aspx. 2020.

14. Karani R, Varpio L, May W, et al. Commentary: Racism and bias in health

professions education: How educators, faculty developers, and researchers

can make a difference. Acad Med. 2017;92(11S Association of American

Medical Colleges Learn Serve Lead: Proceedings of the 56th Annual

Research in Medical Education Sessions):S1-S6.

https://journals.lww.com/academicmedicine/fulltext/2017/11001/Commentar

y__Racism_and_Bias_in_Health_Professions.2.aspx. 2020.

15. Klein R, Julian A, Snyder ED, et al. Gender bias in resident assessment in

graduate medical education: Review of the literature. J Gen Intern Med.

2019;34(5):712-719.

https://link.springer.com/article/10.1007%2Fs11606-

019-04884-0. 2020.

16. Lambe KA, O'Reilly G, Kelly BD, Curristan S. Dual-process cognitive

interventions to enhance diagnostic reasoning: a systematic review. BMJ

Qual Saf. 2016;25(10):808-20.

https://qualitysafety.bmj.com/content/25/10/808.long

. 2020.

17. Morgan AU, Chaiyachati KH, Weissman GE, Liao JM. Eliminating gender-

based bias in academic medicine: More than naming the "Elephant in the

Room". J Gen Intern Med. 2018;33(6):966-968.

https://link.springer.com/article/10.1007%2Fs11606-018-4411-0

. 2020.

18. Mueller AS, Jenkins TM, Osborne M, et al. Gender differences in attending

physicians' feedback to residents: A qualitative analysis. J Grad Med Educ.

2017;9(5):577-585.

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5646913/

. 2020.

19. Hamstra, SJ, Yamazaki, K, Holmboe, E. Gender group differences in

Milestone ratings: Exploring differences in ratings by individuals and Clinical

Competency Committees. Journal of Graduate Medical Education.

2019;11(6):722-731.

https://www.jgme.org/doi/full/10.4300/JGME-D-19-

00734.1?mobileUi=0. 2020.

20. Pack R, Lingard L, Watling C, Chahine S, Cristancho S. Some assembly

required: tracing the interpretative work of Clinical Competency Committees.

Med Educ. 2019;53:723-734.

https://onlinelibrary.wiley.com/doi/full/10.1111/medu.13884

. 2020.

21. Rojek AE, Khanna R, Yim JWL, et al. Differences in narrative language in

evaluations of medical students by gender and under-represented minority

status. J Gen Intern Med. 2019;34(5):684-691.

https://link.springer.com/article/10.1007%2Fs11606-019-04889-9

. 2020.

22. Ross FJ, Metro DG, Beaman ST, et al. A first look at the Accreditation

Council for Graduate Medical Education Anesthesiology Milestones:

Implementation of self-evaluation in a large residency program. J Clin

Anesth. 2016;32:17-24.

https://www.sciencedirect.com/science/article/abs/pii/S0952818016000477?

via%3Dihub. 2020.

23. Salles A, Awad M, Goldin L, et al. Estimating implicit and explicit gender

bias among health care professionals and surgeons. JAMA Netw Open.

2019;2(7):e196545.

https://jamanetwork.com/journals/jamanetworkopen/fullarticle/2737309

2020.

24. Santen SA, Yamazaki K, Holmboe ES, Yarris LM, Hamstra SJ. Comparison

of male and female resident Milestone assessments during emergency

medicine residency training: A national study. Acad Med. 2020;95(2):263-

268. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7004441/

. 2020.

25. Saposnik G, Redelmeier D, Ruff CC, Tobler PN. Cognitive biases

associated with medical decisions: a systematic review. BMC Med Inform

Decis Mak. 2016;16(1):138.

https://bmcmedinformdecismak.biomedcentral.com/articles/10.1186/s12911

-016-0377-1. 2020.

26. Schwind CJ, Williams RG, Boehler ML, Dunnington GL. Do individual

attendings’ post-rotation performance ratings detect residents’ clinical

performance deficiencies? Acad Med. 2004;79(5):453-457.

https://journals.lww.com/academicmedicine/Fulltext/2004/05000/Do_Individ

ual_Attendings__Post_rotation.16.aspx. 2020.

27. Surowiecki J. The Wisdom of Crowds. New York, NY: Anchor Books; 2005.

28. Teherani A, Hauer KE, Fernandez A, King TE Jr, Lucey C. How small

differences in assessed clinical performance amplify to large differences in

grades and awards: A cascade with serious consequences for students

underrepresented in medicine. Acad Med. 2018;93(9):1286-1292.

https://journals.lww.com/academicmedicine/fulltext/2018/09000/How_Small

_Differences_in_Assessed_Clinical.16.aspx. 2020.

29. Thomas MR, Beckman TJ, MauckKF, Cha SS, Thomas KG. Group

assessments of resident physicians improve reliability and decrease halo

error. J Gen Intern Med. 2011;26:759-764.

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3138588/

. 2020.

Part 5: Post-Meeting Activities: Feedback, Documentation, and Follow-Up

Feedback to the resident or fellow is an essential activity of the Milestones

assessment system. Research has clearly shown that feedback is one of the most

effective educational tools faculty members and programs have to help residents

and fellows learn and improve.

The Milestones should be used to help residents

and fellows develop action plans and adjustments to their learning activities and

curriculum. Feedback sessions should be conducted in person.

Program directors will have different processes within their programs based on

program size. Some program directors may provide feedback to residents/fellows

themselves. Others will delegate this responsibility to one or more of the CCC

members or a separate set of advisors. Research is clear that interpreting and

understanding multi-source performance data, as represented by the Milestones,

should be facilitated and guided by a trusted advisor working with a resident/fellow

over time. This relationship has been termed an ‘educational alliance’ that

strengthens the residents’/fellows’ ability to engage with the feedback provider in

reflecting on their own performance and incorporating feedback into a plan for

improvement. (Telio, 2015, Ramani, 2019) Feedback is most effective when

structured as an ongoing dialogue rather than a one-time transmission of

information. This process builds learners’ skills as Master Adaptive Learners

equipped to engage in lifelong learning. (Cutrer, 2017)

The faculty members providing feedback should be trained to serve in a coaching

role. Training entails building skills in discussing feedback, guiding reflection, and

creating learning plans. (Armson, 2019) A coach uses strategies to help residents

build on their strengths and address areas for improvement as part of their learning

experience. (Palamara, 2018) Similarly, residents should receive training about how

to maximize the benefit of a coaching relationship, including how to receive and use

feedback even when it may feel uncomfortable. (Deiorio, 2017)

There are many different models that can be used to provide feedback, like ADAPT

- Ask-Discuss-Ask-Plan Together (Appendix H), and R2C2 (Appendix I). (Fainstad,

2018; Sargeant, 2018) Regardless of the particular model used by a program, the

basic features of high-quality feedback include:

1. Timeliness: The results of CCC deliberations and Milestones determinations

should be shared with the individual resident or fellow soon after the meeting

has occurred.

2. Specificity: The Milestones help to facilitate this criterion by providing

descriptive narratives. However, as noted above, the Milestones do not

represent the totality of a discipline, and many other important points of

feedback will likely arise in a CCC meeting that should also be captured and

shared with the individual resident or fellow. Generalities (often called “minimal”

feedback), such as “you’re doing great,” or, “should read more,” are not helpful

in promoting professional development, especially in the context of Milestones

data.

3. Balance reinforcing (“positive”) and corrective or constructive

(“negative”) feedback: It is important to include both in specific terms. An

imbalance between too much reinforcing or conversely corrective feedback can

undermine the effectiveness. The popular feedback sandwich (positive-

negative-positive) is not actually very effective and not routinely recommended.

Models for giving feedback are provided in Appendix H (ADAPT model) and

Appendix I (R2C2).

4. Learner reaction and reflection: It is very important to allow the individual

resident or fellow to react to and reflect on the feedback and Milestones data.

The two models provided below are excellent ways to facilitate this process.

Reaction and reflection help garner resident/fellow buy-in and development of

individualized learning plans (ILPs). Residents should be strongly encouraged,

in partnership with a faculty advisor and coach, to create their own ILP every six

months.

5. ILPs: Creating and executing an ILP after Milestones review is critical to

professional development and is often neglected in feedback. As Boud and

Molloy (2013) argue, feedback hasn’t occurred until the learner has actually

attempted an action or change with the information. Feedback is more than just

information giving and dissemination. (Lockyer, 2017)

6. Feedback should start with where the resident/fellow was at the last feedback

meeting and a review of the action plans created then.

CCCs should also provide feedback to the Program and the Program Evaluation

Committee as to which Milestones have been easier to assess based upon the

assessments presented. This feedback is critical for the program to improve its

curriculum (where the content of the milestone is taught) and where and how it is

assessed.

References:

1. Armson H, Lockyer JM, Zetkulic M, Könings KD, Sargeant J. Identifying

coaching skills to improve feedback use in postgraduate medical education.

Med Educ. 2019;53(5):477-493.

https://onlinelibrary.wiley.com/doi/full/10.1111/medu.13818. 2020.

2. Boud D, Molloy E. Feedback in Higher and Professional Education. 1st ed.

Sydney: Routledge; 2013.

3. Cutrer WB, Miller B, Pusic MV, et al. Fostering the development of master

adaptive learners: A conceptual model to guide skill acquisition in medical

education. Acad Med. 2017;92(1):70-75.

https://journals.lww.com/academicmedicine/Fulltext/2017/01000/Fostering_the_

Development_of_Master_Adaptive.24.aspx. 2020.

4. Deiorio N, Hammoud M. Coaching in Medical Education: A faculty handbook.

https://www.ama-assn.org/system/files/2019-09/coaching-medical-education-

faculty-handbook.pdf. 2020.

5. Fainstad T, McClintock AA, Van der Ridder MJ, Johnston SS, Patton KK.

Feedback can be less stressful: Medical trainee perceptions of using the

prepare to ADAPT (Ask-Discuss-Ask-Plan Together) Framework. Cureus.

2018;10(12):e3718. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6428363/.

2020.

6. Lockyer J, Carraccio C, Chan MK, et al. Core principles of assessment in

competency-based medical education. Med Teach. 2017;39(6):609-616.

https://www.tandfonline.com/doi/abs/10.1080/0142159X.2017.1315082?journal

Code=imte20. 2020.

7. Palamara K, Kauffman C, Chang Y, et al. Professional development coaching

for residents: Results of a 3-year positive psychology coaching intervention. J

Gen Intern Med. 2018;33(11):1842-1844.

https://link.springer.com/article/10.1007%2Fs11606-018-4589-1

. 2020.

8. Ramani S, Könings KD, Ginsburg S, van der Vleuten CPM. Twelve tips to

promote a feedback culture with a growth mind-set: Swinging the feedback

pendulum from recipes to relationships. Med Teach. 2019;41(6):625-631.

https://www.tandfonline.com/doi/abs/10.1080/0142159X.2018.1432850?journal

Code=imte20. 2020.

9. Sargeant J, Lockyer JM, Mann K, et al. The R2C2 Model in residency

education: How does it foster coaching and promote feedback use? Acad Med.

2018;93(7):1055-1063.

https://journals.lww.com/academicmedicine/fulltext/2018/07000/The_R2C2_Mo

del_in_Residency_Education__How_Does_It.30.aspx. 2020.

10. Telio S, Ajjawi R, Regehr G. The "educational alliance" as a framework for

reconceptualizing feedback in medical education. Acad Med. 2015;90(5):609-

614.

https://journals.lww.com/academicmedicine/fulltext/2015/05000/The__Educatio

nal_Alliance__as_a_Framework_for.21.aspx. 2020.

Part 6: Legal Issues

The CCC can be an extremely beneficial structure to support legal constructs

required for academic decision-making. There are two Supreme Court decisions

that provide the context and framework for academic due process, including the

concept of a Clinical Competency Committee (See Key Legal Cases Supporting

Professional Judgment in GME).

Academic due process consists of three components:

1. notice (of deficiencies); and,

2. opportunity to cure; and,

3. a careful and deliberate decision-making process.

“Notice” and “Opportunity to Cure” are the heart of academics. Continuously,

residents and fellows are receiving feedback about how they might approach their

roles as clinicians, diagnosticians, and educators, in a way that is enhanced. This

notice may come in the form of routine feedback at the bedside, during rounds, in a

meeting, or in a structured assessment or evaluation process. Notice, or feedback,

can be formal or informal; verbal or written. An opportunity to cure simply

recognizes time following that feedback for the learner to demonstrate improvement

and change. An opportunity to cure may be a short period of time (even

immediate), or a more prolonged period (weeks or months), depending on the issue

at hand.

The CCC can serve as all, or part, of the ‘reasonable decision-making process.’ In

both Missouri v. Horowitz (“Horowitz”) and Michigan v. Ewing (“Ewing”), the faculty

evaluation committee was identified as being a core component of the reasonable

decision-making process; specifically, the cases reference “a regularly called

meeting of the faculty for the purpose of discussing student (resident/fellow)

performance.” While not specifically referred to at the time of the decision as the

“CCC,” this structure of a faculty committee is the legal construct supporting the

importance of what is now referred to as a CCC in today’s evaluation systems in

medicine. The Ewing case further supported the idea that a faculty decision-making

committee providing academic performance decisions that are conscientious and

made with careful deliberation (i.e., not arbitrary or capricious) constitutes

reasonable decision-making. When making academic decisions regarding

resident/fellow performance, promotion, or dismissal, the CCC provides a structure

recognized by the highest court in academic cases.

Reasonable decision-making processes should include all available information on

the learner’s performance, including “notices” and “opportunities to cure.” This

includes both verbal and written feedback, structured and unstructured

observations of performance, solicited and non-solicited feedback from others, and

any other information helpful to a careful decision-making process.

Documentation

When defending a legal case, contemporaneous documentation of events, actions,

or conversations are very helpful for confirming whether something actually

happened. Although there is no law requiring evaluations or performance feedback

to be in writing, the ACGME requires written rotational evaluations and semi-annual

evaluations of performance. Of course, it is natural within an academic clinical

setting that a faculty member provides a resident/fellow with routine verbal

feedback. Although it is not recorded, this verbal feedback also constitutes notice

and opportunity to cure (Horowitz).

While it is always helpful to have written performance documentation, lack thereof

should not deter evaluators from doing the right thing and using this information as

part of the overall evaluation process. One critical role of the CCC is to elicit

feedback from faculty members regarding performance in a variety of settings and

situations, and for the faculty to discuss performance based on individual

experiences and opinions. In many situations, this discussion at the CCC may be

the first time that issues emerge and indicate a pattern of performance or behavior.

This discussion is the heart of the CCC and should not be discounted just because

there is not a rotational evaluation or other assessment tool or form to support the

discussion. Research shows that the discussion among the faculty members in the

CCC often provides more accurate and robust information regarding learner

performance than the written evaluation alone, which may not represent a complete

view of actual performance. In fact, Schwind et al. demonstrates that verbal

discussion of actual performance is more accurate than what is provided in the

written evaluations. (2004)

These discussions are not only valuable to the formation of individual performance

evaluations, but also to demonstrate a “fair and reasonable decision-making

process” by the program.

The documentation of the CCC meeting itself can be one of the most valuable

documents to an institution when defending a resident/fellow dismissal or adverse

action. The ACGME does not have a requirement as to how the CCC meeting

should be documented, however, many programs will find it worthwhile to retain

minutes of CCC meetings. These minutes may be:

1. A written document reflecting the discussion of each resident’s/fellow’s

performance.

2. A concise summary of each resident’s/fellow’s performance and any action

or follow-up items.

3. Confidential (i.e., not shared with anyone other than the resident/fellow,

CCC, and program leadership).

4. Archived in accordance with the institution’s document retention policy in

consultation with legal counsel.

Some institutions may prefer #1 to be brief and use the Milestones reported to the

ACGME as #2.

Decision Process

The ACGME requires the CCC to make recommendations on resident/fellow

performance to the program director for review and action; thus, the program

director, and not the CCC, is the final decision maker. Still, in most situations, the

feedback and consensus of the CCC is critical in informing the program director of

the faculty’s expert opinion regarding progress and promotion.

In general, discussions of the CCC should lead to a “consensus” decision. That is,

after presentation of all data, and engagement of the members in a discussion of

their experience with, and opinion regarding the progress of a resident/fellow, the

Milestones assessment will be reached by consensus.

Nevertheless, the CCC may find a situation in which strongly held differing opinions

that are not modified through discussion fail to result in consensus. The Chair must

recognize and be prepared for these circumstances and appreciate their value. In

fact, these differences of opinion are crucial to authentic assessment, and another

opportunity to discuss the opinions provided. Govaerts et al. (2019) remind us that

such tensions are inherent in competency-based education (assessment of learning

versus assessment for learning; formative versus summative assessment) and

suggest the integration of Polarity Thinking

as one mechanism.

The CCC members should discuss the possibility of disagreement at the outset and

should consider describing how they will proceed. Although a written description of

the CCC is not required, it may be useful for the CCC to describe its processes to

clarify them for one another, and to help communicate to the residents/fellows and

other faculty members. The ACGME provides no specific guidance in this setting.

The committee should establish its own policy and apply it consistently, taking into

account input from the designated institutional official (DIO) and Legal office. While

it is recognized that decisions regarding actions such as remediation, probation,

and promotion can be difficult and programs may resort to voting, the ACGME and

the authors of this Guidebook strongly discourage voting as a decisional approach.

As the Milestones are designed to guide a developmental judgment, CCCs should

not vote on individual sub-competencies and milestones. If programs do choose to

use voting, it is very important to be clear about what exactly the vote means from

the outset, and to ensure that the process of voting does not provide a false sense

of power to the committee. For example, is a vote being taken to determine if

performance is not at an expected competence level, or is the vote to recommend a

disciplinary action, remediation, or dismissal? If a vote is held and there is a narrow

result, (e.g., four to three), the program director’s expert decision on a

resident/fellow may seem less clear cut if the resident/fellow subsequently appeals.

Regardless of whether a vote is taken, the CCC must remember that the decision

of the committee is advisory to the program director, and the program director has

the responsibility to be the final decision maker. With these mechanisms in place

and followed, fundamental fairness to both residents/fellows and committee

members is provided, and challenges to process consistency and fairness are

prospectively addressed.

Peer-Review Privilege

Peer-review statutes fall under state law, and thus vary from state to state. In

general, peer-review privilege has some common tenets that generally do not apply

to CCCs and resident/fellow performance evaluation.

Generally speaking, peer-review privilege:

● protects discussion of clinical performance for the purpose of internal quality

assessment, not evaluation and decisions communicated to external parties;

and,

● applies to in-person meetings where the information is maintained internally,

not communicated outside of the peer-review process (e.g., to clinical

advisors, other departments, or external agencies).

Each institution should review its peer-review statute with its legal counsel to

determine if it should be applied to the CCC. Likewise, given the ever-changing

legal environment and number of cases being heard regarding resident

performance, an institution’s legal counsel should regularly review new case law

and decisions in the state for updated rulings and orders issued by courts.

Notwithstanding a program’s natural tendency to want to maintain strict

confidentiality, if conducted in accordance with these guidelines, the discussions

and recommendations of the CCC are generally helpful when defending a

program’s decision to dismiss a resident/fellow (see Horowitz and Ewing). Carefully

prepared CCC minutes can provide one of the strongest legal defenses to support

dismissal actions by demonstrating the three core tenants of academic due

process: notice of deficiencies; opportunity to cure; and a reasonable decision-

making process.

The ACGME has been very clear that the Milestones are not intended for non-

residency (or fellowship) use. They are not used by the American Board of Medical

Specialties member certifying boards or by the certifying boards of the American

Osteopathic Association for specialty board certification. “The ACGME does not

have evidence that individual Milestones data

can be validly used in any other

context beyond provision of individual resident and fellow feedback, especially for

any higher stakes decisions,” such as licensure. The ACGME assumes that most

state medical licensing boards (including osteopathic medical boards) heed the

ACGME declaration that the Milestones are not designed for any non-GME

program use. In April 2018, the ACGME released a position paper, “Use of

Individual Milestones Data by External Entities for High Stakes Decisions - A

Function for Which they Are not Designed or Intended,” which describes the

potential adverse consequences if the Milestones are used by external entities.

Appeals and Due Process

The members of the faculty must be encouraged to provide candid and robust

evaluations that are reflective of actual performance. Evaluations are based on

each faculty member’s observations, judgments, and expectations. A faculty

member should complete evaluations in an honest and good-faith effort to provide

feedback to the resident/fellow with the goal of identifying both strengths and

deficiencies, in order to help the resident/fellow to improve academic performance.

Programs should be aware that allowing residents/fellows to appeal performance

evaluations (rotational evaluations, semi-annual evaluations, etc.) can send a

message to the residents/fellows that faculty member or program director feedback

is negotiable. It can also suggest to faculty members and program directors that

their feedback, usually critical, can be subject to scrutiny and overturned if a

resident/fellow complains. Programs should discuss with legal counsel the impact

of allowing residents/fellows to appeal performance evaluations or academic

evaluation decisions. Most institutions do not allow due process for routine

feedback, including assessment and evaluations, and the ACGME does not require

it for these purposes.

The ACGME does encourage programs using progressive disciplinary processes

(probation) to allow these actions, as well as termination or non-promotion,

resulting from CCC decisions to be eligible for appeal to ensure the department and

institution follow the policies in place regarding the decision-making process.

Key Legal Cases Supporting Professional Judgment in GME

University of Missouri v. Horowitz (1978)

Board of Curators of Univ. of Mo. v. Horowitz, 435 U.S. 78, 98 S. Ct. 948, 55 L. Ed.

2d 124 (1978).

Case Summary: Ms. Horowitz excelled in her first two years of medical school, but

received criticism from the faculty as she began her clinical rotations. She was

provided feedback in her rotational evaluations regarding her attendance, slovenly

appearance, hygiene, and bedside manner. Despite feedback, Ms. Horowitz’s

behavior did not improve. The school’s faculty evaluation committee ultimately

recommended her dismissal from medical school. Ms. Horowitz appealed the

decision to the Dean. The Dean allowed Ms. Horowitz the opportunity to be

evaluated by seven independent physicians. At the conclusion

of the rotations, the faculty provided feedback to the Dean of varied opinion.

Based on the feedback of the independent faculty evaluators, the Dean upheld the

dismissal decision. This case and the issue of academic due process were

ultimately argued in front of the Supreme Court. The Court supported the

University’s decision based on the following:

● Ms. Horowitz was provided notice of her deficiencies through private verbal

feedback and her rotational evaluations.

● Ms. Horowitz was provided an opportunity to cure her deficiencies.

● The decision was made carefully and deliberately. The regularly called

meeting of the faculty, called for the purpose of evaluating academic

performance, was noted as being a reasonable decision-making process

consisting of faculty members, expected to evaluate student performance.

● The Court decision noted that under this particular set of circumstances the

rotation with the seven physicians was much more process than was due.

University of Michigan vs. Ewing, (1985)

Regents of Univ. of Mich. v. Ewing, 474 U.S. 214, 106 S. Ct. 507, 88 L. Ed. 2d 523

(1985).

Case Summary: Mr. Ewing was enrolled in the six-year BS/MD program. After four

years, he was eligible to write the NBME Step 1 exam. Mr. Ewing failed the exam

and was subsequently dismissed from medical school. He sued, citing at least 11

other students who failed the exam and were allowed to stay enrolled in school and

retake the test; some were allowed to retake the exam three and four times. The

decision to dismiss Mr. Ewing was made by the faculty committee charged with

reviewing academic performance. This committee reviewed Mr. Ewing’s entire

academic record and determined that based on his overall performance (including

several incompletes, required repeats of courses, and the lowest score ever

recorded on the NBME exam at this school), he did not have the ability or aptitude

required of a physician and had no chance of succeeding. The Court sided with the

school noting:

1. “The narrow avenue for judicial review of the substance of academic

decisions precludes any conclusion that such decision was a substantial

departure from accepted academic norms as to demonstrate the faculty did

not exercise professional judgment.”

2. The decision was “conscientious and made with careful deliberation,” citing

the regularly called faculty meeting structure, and the Promotion and

Review Board.

3. The faculty rightly reviewed Mr. Ewing’s entire academic record, not just a

single test, rotation, or incident, to provide context to the decision.

References

1. ACGME. Use of Individual Milestones Data by External Entities for High

Stakes Decisions - A Function for Which they Are not Designed or Intended.

https://www.acgme.org/Portals/0/PDFs/Milestones/UseofIndividualMilestones

DatabyExternalEntitiesforHighStakesDecisions.pdf?ver=2018-04-12-110745-

440. 2020.

2. Govaerts MJB, van der Vleuten CPM, Holmboe ES. Managing tensions in

assessment: moving beyond either-or assessment. Med Educ.

2019;53(1):64-75.

https://onlinelibrary.wiley.com/doi/full/10.1111/medu.13656

. 2020.

3. Schwind CJ, Williams RG, Boehler ML, Dunnington GL. Do individual

attendings’ post-rotation performance ratings detect residents’ clinical

performance deficiencies? Acad Med. 2004;79(5):453-457.

https://journals.lww.com/academicmedicine/Fulltext/2004/05000/Do_Individu

al_Attendings__Post_rotation.16.aspx. 2020.

Part 7: Other Uses for the CCC

Although the primary role of the CCC is that of documenting residents’/fellows’

performance on the Milestones, by virtue of their review and synthesis of a vast

array of assessment data from different learning experiences, CCCs are uniquely

qualified to render judgements not only on resident/fellow performance, but on the

curriculum and the quality of the assessment system. CCCs can play many

important roles in the assessment system, including:

1. Assessing transferring residents/fellows

2. Contributing to the Annual Program Evaluation

4. Faculty development

5. Quality improvement of the assessment system

6. Assisting the institutional Graduate Medical Education Committee (GMEC)

7. Continuous educational quality improvement

8. Simplifying a program’s individual and collective assessment tools

In this section, we describe an expansion of the role of the CCC outlined in the

Common Program Requirements in order for programs and institutions to make the

most of these committees’ important work.

Transfers/Admittance from Non-ACGME-Accredited Programs

When residents/fellows transfer into an ACGME-accredited program with previous

experience from a non-ACGME-accredited program, an assessment on the

Milestones is required within 12 weeks of matriculation. Programs should also

review the resident’s/fellow’s Milestones results from the previous program when

applicable. The CCC may be used to assess the competence of those

residents/fellows who are either transferring into a program from a period of prior

education or applying to a fellowship with core residency education experience from

a non-ACGME-accredited program.

Annual Program Evaluation

Through the process of reviewing vast amounts of resident/fellow assessment data,

CCCs should take the time to develop “Milestone maps” (i.e., a spreadsheet to

track where each milestone is taught and assessed). CCCs can use these maps to

determine the extent of the curricular content for each milestone, including the

teaching and assessment methods and the learning experiences where the

material is currently taught and/or assessed, or perhaps where it should be

taught/assessed. This process can illuminate any potential gaps or redundancies in

the curriculum. Then, CCCs can make recommendations for the development of

new rotations or learning experiences that may help address curricular concerns.

Such information can be formally submitted to the program director at least once

per year to be included on the agenda for the Program Evaluation Committee

(PEC) as it performs the Annual Program Evaluation. The PEC is expected to

review multiple data points during this review, including information about the

curriculum and aggregate Milestones data.

Faculty Development

CCCs review an inordinate amount of faculty ratings and narratives about

residents’/fellows’ performance and must make important decisions based on this

data. Thus, they can provide important insights on the usefulness of such data and

offer feedback to be used for the purpose of faculty development. Program

directors are expected to review the performance of their faculty members at least

annually, including providing faculty members with feedback on their evaluations.

The CCC can have a significant role in this key faculty development mission.

Faculty development is needed at three levels: 1) the program director; 2) the

engaged core and other faculty members who join the CCC; and, 3) the faculty

members “in the trenches” who may not be as fully involved in educational

programming or administration, but who have essential roles in actively teaching

and assessing residents/fellows. Each group will have different needs. Program

directors and CCC members will need a deeper understanding of the Milestones,

assessment, group process, and program evaluation. Faculty members need to

understand what key elements of assessment information they need to contribute

to the larger picture of each resident/learner.

Faculty professional development is a required program component. The ACGME

recognizes that although “evaluation is a core faculty competency… most [faculty

members] will need additional training in [the] evaluation process,” to include

evaluation process training (how to interpret aggregated evaluation data),

understanding how many assessments are needed for each Milestone, assurance

of data quality, and application of QI methods to the evaluation processes. The

CCC provides an opportunity for faculty development for other program faculty

members as well: to understand the CCC process and how its evaluations of

residents/fellows fit into the overall assessment of resident/fellow performance

using the Milestones.

Quality Improvement

In aggregate, CCCs can review not only assessment data generated by the core

faculty members, but also their own Milestones ratings. Using aggregate reports

from their residency management system, CCCs can review trends in the ratings

for specific milestones and initiate conversations to assist in explaining such trends.

For example, if most PGY-2s are scoring low on the practice-based learning and

improvement milestones, there can be multiple reasons for this finding. This may be

due to lack of data, inadequate data, inadequate performance, or perhaps

inadequate sampling of performance. In some situations, there may be no

assessment data at all for certain milestones. This finding may also signal gaps in

the curriculum, a lack of assessment, inadequate documentation, and presentation

of assessment data to the CCC or some combination thereof. Mechanisms should

be put in place to recognize these issues and attempts should be made to address

them prior to the next review cycle.

Assist the GMEC

The CCC can assist the GMEC in the oversight of the effectiveness of programs’

curricula by providing the aggregate Milestones data for each program and trends

in the data, and by making recommendations for programs’ Annual Program

Evaluations. Per the Institutional Requirements, the GMEC’s responsibilities must

include oversight of:

“the quality of educational experiences in each ACGME-accredited

program that lead to measurable achievement of educational outcomes as

identified in the ACGME Common and specialty-/subspecialty-specific

Program Requirements;”

(Outcome)

“the ACGME-accredited program(s)’ annual program evaluations and self-

studies;”

(Core)

Continuous Educational Quality Improvement

For the individual resident/fellow, the CCC offers insights and perspectives from a

group of faculty members, and comparison of an individual’s performance to a

national standard, the Milestones. For the entire program, the CCC serves as an

early warning system if a resident/fellow fails to progress, and therefore identifies

an opportunity for remediation. For the members of the faculty, CCCs can be an

opportunity to balance out the “hawks” and “doves,” and to develop a more

standardized, consistent, explicit approach to expectations of resident/fellow

performance. More importantly, through longitudinal dialogue and repeated

sessions, faculty members can develop a better shared mental model of

competence and reduce the variability in assessment judgments.

Simplify a Program’s Individual and Collective Assessment Tools

The CCC can identify which assessments are most useful, and where there are

gaps. A program may be able to eliminate administrative burden. It may not be

feasible or even necessary for faculty members to complete multi-page evaluation

forms, for example. As stated earlier, the true assessment instrument is not the tool

or form, it is the faculty member(s) or others using it. CCCs can help to identify

barriers and impediments to effective faculty members’ evaluations and create

faculty development or other intervention opportunities.

References:

1. ACGME. ACGME Common Program Requirements (Residency).

https://www.acgme.org/Portals/0/PFAssets/ProgramRequirements/CPRResiden

cy2019.pdf. 2020.

Part 8: Individualized Learning Plans and Final Evaluations: The Other Roles

of the CCC

The CCC has two other critically important roles that are core requirements for the

program.

1) The program director/designee, with input from the CCC, must: “assist

residents in developing individualized learning plans to capitalize on their

strengths and identify areas for growth” (Program Requirement V.A.1.d).(2)

and,

2) The ACGME-required final evaluation for each resident, completed by the

program director upon each resident’s program completion, must “consider

recommendations from the Clinical Competency Committee”

V.A.2.a).(2).(c)

Individualized Learning Plans (ILPs)

Learning contracts are “without question the single most potent tool I have come

across in my more than half-century of experience with adult education.”

(Knowles,1990)

Adult learning theory is premised on the construct that adults learn best when they

are actively engaged in the learning process and self-direct their own learning goals

and activities. (Knowles, Holton and Swanson, 2005). The ACGME Common

Program Requirements now reference ILPs [Common Program Requirements

V.A.1.d.(1)-(3)]: “The program director (or designee) must… assist residents in

developing individualized learning plans to capitalize strengths and identify areas of

growth.”

At the point of graduation, program directors must certify that each resident has

achieved competence as an independent, self-directed, and lifelong learner. The

Core Competency of Practice-Based Learning and Improvement is a fundamental

component of this certification. Self-directed learning is a process by which

individuals identify and/or acknowledge their own learning needs, find resources to

meet those needs, and subsequently evaluate their own achievements; it is integral

to maintaining professional competence. An ILP is also an important tool for the

struggling resident or fellow.

Although the ACGME expects all residents/fellows to have individual learning plans,

often CCCs and program directors only think about using them in for those learners

who are performing below expectations. Residents/fellows struggling to perform at

acceptable standards are often a source of frustration for faculty members and for

CCCs. When the CCC and/or the faculty acknowledge that a learner is not meeting

academic standards, there is sometimes hesitation regarding a path forward. Often,

this hesitation stems from learners who are perceived as lacking insight to their own

deficiencies. In addition to being unable to self-reflect on performance, these

learners also tend to disregard faculty members’ feedback (often perceived as

biased or misdirected), claim that the faculty members have not adequately taught

them or provided them with helpful feedback, and cannot acknowledge their own

personal role and responsibility in the learning process. But the inability to

accurately “self -assess” and have appropriate insight is almost universal among

those performing poorly; program directors and faculty members should expect it

and not be surprised by it. Indeed, the very lack of insight that makes

residents/fellows incapable of recognizing their own poor performance, frequently

makes it nearly impossible for them to succeed with a remediation plan.

An ILP should be formulated by the learner for the learner, and should include

personal learning objectives, as well as resources and strategies to achieve them.

While the learner should be able to create an initial ILP, the ILP content should be

guided by a facilitator (faculty member, advisor, coach, or program director). The

draft ILP created by the resident/fellow can provide enormous insight to the

program director and/or CCC; the information contained in the ILP is one way to

determine if the learner has the ability to self-reflect based on feedback received,

and the insight required to be successful in remediating. Those unable to “own”

their deficiency(ies) and construct or at least contribute in a major way to a plan to

address it, are unlikely to be successful.

Creating an ILP should actively engage learners to take ownership of their own

learning. (Li & Burke, 2010) ILPs allow the learner to focus on priority areas, re-

evaluate learning needs, and have regular discussions about achieving learning

goals.

Components of an ILP (Li & Burke, 2010):

1. Reflection on goals and self-assessment of strengths and weaknesses

2. Generation of specific learning goals and/or objectives

3. Specific plans or strategies to achieve each goal focused on what the

learner will do to improve

4. Mutual agreement on how the assessment of progress on each goal will be

determined

5. Eventual revision of goals or creation of new goals based on performance

6. Expected timeline

ILPs ARE:

• Formulated by the individual (resident/fellow) – made by the learner, for the

learner

• Guided by a facilitator (faculty member, advisor, coach, or program

director)

• An exercise in self-assessment and self-reflection

• Iterative

• An ACGME core requirement

• An indicator of insight and ability to become an independent lifelong learner

ILPs are NOT:

• Set in stone – they can and should be revisited by both the learner and the

facilitator

• A portfolio

• Evaluations

• The sole or major responsibility of the program director (or faculty) or the

program

CCCs do not “create” ILPs. This is the work of the residents/fellows, co-produced

with the program director (or designee), an advisor, or coach.

The Final Evaluation

Upon completion of a program, the program director must provide a final evaluation

for each resident/fellow (Common Program Requirement V.A.2.). In the past, this

was often referred to as the “Final Summative Evaluation.” While this evaluation

has been a perfunctory document and process for residents/fellows successfully

completing a residency or fellowship and progressing to the next stage of their

career, the final evaluation for residents/fellows who depart the program prior to

completion, typically for performance reasons, are even more important and can be

difficult to write.

The ACGME requires that the “specialty-specific Milestones, and when applicable

the specialty-specific Case Logs, must be used as tools to ensure residents are

able to engage in autonomous practice upon completion of the program” [Program

Requirement V.A.2.a).(1)]. The complete requirements for the final evaluation

(Common Program Requirement V.A.2.) are listed in Table 5 below.

When a resident/fellow is dismissed from the program, or resigns early due to

performance concerns, the final evaluation becomes the document of record

regarding the resident’s/fellow’s achievement to date in each of the Core

Competencies. It also describes areas in which the resident/fellow is either deficient

or has not attained a level of performance consistent with PGY level expectations.

Additionally, the final evaluation can contain important contextual information,

including the dates of enrollment in the program, any relevant prior

education/training information, description of any behavioral issues or concerns,

and a summary paragraph describing the program director’s overall impression of

the individual resident/fellow, their achievements, any ongoing concerns, or areas

for future focus. For residents and fellows in procedural specialties, the final

evaluation may also include a summary of procedural cases performed at the

institution.

Table 5: Common Program Requirements for the Final Evaluation (previously

the Final Summative Evaluation)

V.A.2.a)

be provided by the program director for each resident upon completion

of the program

V.A.2.a).(1)

specialty-specific Milestones, and when applicable the specialty-specific

Case Logs, must be used as tools to ensure residents are able to

engage in autonomous practice upon completion of the program

V.A.2.a).(2).(a)

become part of the resident’s permanent record maintained by the

institution, and must be accessible for review by the resident in

accordance with institutional policy

V.A.2.a).(2).(b)

verify that the resident has demonstrated the knowledge, skills, and

behaviors necessary to enter autonomous practice

V.A.2.a).(2).(c)

consider recommendations from the Clinical Competency Committee

V.A.2.a).(2).(d)

be shared with the resident upon completion of the program

The final evaluation should be a comprehensive, stand-alone document that

encompasses the scope of a resident’s/fellow’s performance while enrolled in the

program. The final evaluation should not be confused with a letter of

recommendation, which is typically prepared to promote or support an individual

with a positive bias. Instead, the final evaluation should be written with candor, in a

way that is fair and balanced with regard to actual performance.

The final evaluation can be an important tool for a program receiving an off-cycle

resident/fellow. If properly written, the program receiving the resident/fellow should

be able to use the final evaluation from the prior program to ascertain the

resident’s/fellow’s current performance level for each Core Competency,

understand the resident’s/fellow’s strengths and weaknesses, understand the

context in which the resident/fellow departed the prior program, and provide

continuity of education, supervision, and feedback.

The final evaluation is also valuable for well performing residents as transitioning

from residency to fellowship or into their first post-GME professional role.

The final evaluation SHOULD be:

• Sufficiently comprehensive to stand alone in the resident or fellow’s

permanent record as an historic document of enrollment, achievement, and

areas of concern

• Honest – fair and balanced

• Competency-based, including knowledge, skills, and behaviors

• Signed and dated by the program director

• Maintained in the permanent academic record of the program and/or

institution

• Provided to the resident/fellow when finalized and signed

• Provided to others upon request (as appropriate, and when indicated, with

the approval of the resident/fellow)

The final evaluation SHOULD NOT:

• Misrepresent actual performance in any way

• Serve as a letter of recommendation

• Be negotiated by the resident/fellow or anyone else (the content of the final

evaluation must be the program director’s honest view of performance at

the conclusion of the resident’s/fellow’s time in the program

Some programs ask departing residents/fellows to sign the final evaluation retained

for their permanent file.

The Milestones were not designed to be used for this purpose, and the authors of

this guidebook strongly recommend that programs NOT substitute the final

Milestones Report submitted to the ACGME for this final evaluation document.

The authors recommend contacting the Sponsoring Institution’s DIO to learn if the

program should also collaborate with the institutional Legal and/or Human

Resources entities for guidance.

Conclusion

Although CCCs do not create ILPs or final evaluations, they have critical input into

these important, required program processes. As such, they benefit from

understanding how their judgement of resident/fellow progress will be used by the

program director beyond Milestones ratings alone.

References:

1. ACGME. ACGME Common Program Requirements.

https://www.acgme.org/Portals/0/PFAssets/ProgramRequirements/CPRResiden

cy2019.pdf. 2020.

2. Knowles M. The Adult Learner: A Neglected Species. Houston, TX: Gulf Pub

Co; 1990.

3. Knowles MS, Holton EF III, Swanson RA. The Adult Learners: The Definitive

Class in Adult Education and Human Resources Development. 6th ed.

Burlington, MA: Elsevier; 2005.

4. Li ST, Burke AE. Individualized learning plans: basics and beyond. Academic

Pediatrics. 2010;10(5):289-292.

https://linkinghub.elsevier.com/retrieve/pii/S1876-2859(10)00196-8

. 2020.

Part 9: Institutional Oversight of CCCs

[Acknowledgement to Y. Wimberly, MD, MSc, FAAP, Associate Dean of Clinical

Affairs and Designated institutional Official, Morehouse School of Medicine]

Although there are no CCC-specific requirements in the ACGME Institutional

Requirements, DIOs and their team, as well as the GMEC, have a critically

important oversight role in ensuring each program’s CCC processes adhere to the

Common Program Requirements. They may be a source of resources, including for

faculty development, and provide ways to share lessons learned among the

institution’s programs.

In addition, there is at least one institutional requirement that may impact CCCs.

The Sponsoring Institution is responsible for programs’ development of “promotion

criteria” and criteria for renewal of a resident’s/fellow’s appointment (Institutional

Requirement IV.C.), and those conditions for reappointment and promotion to a

subsequent PGY level must be in the contract or letter of appointment (Institutional

Requirement IV.B.2.d)). Many CCCs may de facto “act” as promotion committees

and apply their judgement of resident/fellow performance to recommend

resident/fellow renewal and promotion to the next program year. The Common

Program Requirements require “at least annually...a summative evaluation of each

resident that includes their readiness to progress to the next year of the program, if

applicable.” [Common Program Requirement V.A.1.e)] Although not stated

explicitly, it is likely that an effective CCC will have collaborated with the program

director to identify the promotion criteria or at the very least, align Milestones

performance with them. The CCC should inform the program director of its review

so that the program director can truly exercise the responsibility, authority, and

accountability for promotion of residents/fellows. [Program Requirement II.A.4.]

For residents not making sufficient progress, other institutional requirements will

likely become important, such as due process and grievance policies. Ideally the

Sponsoring Institution and its programs are closely aligned and the DIO and the

program directors are effective collaborators.

The GMEC may wish to monitor CCCs through their oversight of each program’s

Annual Program Evaluation and Self-Studies [Institutional Requirement I.B.4.a).(4)].

The GMEC may wish to make some aspect of the CCC’s assessments one of the

performance indicators used and reported as part of the institution’s Annual

Institutional Review (AIR).

One institution’s experience with this is described using change management

strategies and realizing effective engagement. (Dagnone, 2019)

Appendix H provides a checklist of important CCC elements for DIOs and GMECs

to use in their monitoring. It can be modified to reflect program-/institutional-specific

practices. It outlines potential expectations for the DIO and GMEC, the program

director, the CCC Chair, CCC members, program faculty members and evaluators,

and residents/fellows. It can be used to review the current status of CCCs within an

institution and to identify potential areas for improvement, enhanced resources, and

strategies to disseminate best practices.

References:

1. ACGME. ACGME Institutional Requirements.

https://www.acgme.org/Portals/0/PFAssets/InstitutionalRequirements/000Institut

ionalRequirements2018.pdf?ver=2018-02-19-132236-600. 2020.

2. Dagnone D, Stockley D, Flynn L. Delivering on the promise of competency

based medical education - an institutional approach. Can Med Educ J.

2019;10(1):e28-e38. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6445322/

2020.

Part 10: Current Research

The ACGME Department of Research, Milestone Development, and Evaluation

maintains a bibliography of research related to the Milestones and Clinical

Competency Committees. The bibliography is updated approximately every six

months and can be found at

https://www.acgme.org/What-We-

Do/Accreditation/Milestones/Research.

Appendix A: The High Performing Residency Assessment System

At the program level, residents/fellows are assessed routinely through a

combination of many assessment tools. These include: direct observations; global

evaluation; audits and review of clinical performance data; multisource feedback

from team members, including peers, nurses, patients, and family; simulation; in-

service training examinations (ITE); self-assessment; and others. Increasingly, the

Milestones and entrustable professional activities (EPAs) are used as a guiding

framework and “blueprint” for expected performance. Assessment tools are

selected intentionally to allow routine, frequent, formative feedback to the

resident/fellow to affirm areas of successful performance and to highlight those

aspects that need to be improved. The CCC is the committee that synthesizes

data—quantitative from in-service exams and clinical performance audits, and

qualitative from observers and co-workers. Using the Milestones, the committee

forms a consensus decision, or a judgment, regarding each resident’s/fellow’s

performance. The CCC provides those conclusions to the program director, who

makes the final determination on residents’/fellows’ Milestone “level” at least twice

yearly. These are provided to the applicable ACGME Review Committee and, in

some cases, the pertinent specialty boards. The ACGME’s unit of analysis is the

program, and the Review Committees use aggregate Milestones information

comparing a program with all accredited programs in the given specialty.

The comparison against these benchmarks serves as one source of input into the

ACGME’s determination of program quality and accreditation decisions. The unit of

analysis is the “individual” for certification and credentialing entities. Collectively, all

residents/fellows, faculty members/program directors/programs, the ACGME, and

certification and credentialing entities are accountable to the public for honest

assessments of residents’/fellows’ performance and truthful verification of their

readiness to progress to independent practice. Data (D) is essential for the entire

system to engage in continuous quality improvement, especially to create

meaningful feedback (FB) loops within the program and back to programs from the

ACGME. Programs and residents and fellows can currently download their

Milestones report after each reporting period.

Appendix B: CCC Quiz

1. Requirements for a CCC are found in:

A. The ACGME Common Program Requirements

B. The ACGME Institutional Requirements

C. The CLER Pathways to Excellence document

D. Both A and B

E. None of the above

2. Which of the following requirements of CCCs is an ACGME core requirement?

A. Include faculty members from other programs and non-physician

members of the health care team

B. Review all residents/fellows at least semi-annually

C. Have a written description of the CCC’s responsibilities

D. Allow residents/fellows to exercise a grievance process if they disagree

with the Milestone determination of the CCC

3. The minimum number of CCC members is:

A. 1

B. 2

C. 3

D. 4

E. As many as necessary so that all divisions/subspecialties must be

represented

F. None of the above; there are no specific requirements on the numbers

needed

4. Who of the following should ALWAYS chair the CCC?

A. Program director

B. Associate program director

C. Department Chair

D. DIO

E. Head, GMEC

F. Most senior faculty member on the committee

G. None of the above

5. The CCC must include:

A. Patients

B. Nurses

C. Peer-selected residents or fellows

D. Members of the program faculty

E. Program director

F. All of the above

G. None of the above

6. How many residents/fellows must participate on the CCC?

A. 0

B. 1

C. At least one peer-selected resident or fellow

D. At least one from every year of the program

E. At least one chief resident

7. CCC members:

A. Determine each resident/fellow’s progress on achievement of the

specialty-specific Milestones

B. Only consider residents/fellows who need remediation

C. Only review residents/fellows in their final year of the program

D. Review the decisions the program director has already made regarding

each resident/fellow and provide advice

E. Vote on each resident’s/fellow’s performance

8. The CCC must:

A. Submit Milestones summaries to the ACGME

B. Meet with all residents/fellows to discuss their individual progress on the

Milestones

C. Design and implement any remediation plan necessary (and mentor the

resident/fellow throughout)

D. Review all resident/fellow evaluations at least semi-annually

E. Share Milestones evaluations with the specialty board and state

licensing board

9. According to the ACGME, the minutes of the CCC must be:

A. Fully transcribed

B. Retained as a summary of all residents/fellows

C. Retained only as a summary of the sub-optimally performing

residents/fellows

D. Provided to the ACGME

E. None of the above

10. According to the ACGME, all residents/fellows must be able to exercise a

grievance/due process (“appeal”) if they disagree with the CCC regarding the

Milestones determination it plans to report to the ACGME.

A. True

B. False

C. It depends

11. Who makes the final decision on a resident’s/fellow’s Milestones level?

A. The CCC

B. The resident’s/fellow’s advisor

C. The resident/fellow

D. The ACGME

E. The program director

12. In order to serve on a CCC, a chief resident must:

A. Have completed the core program

B. Be in the last year of the core program

C. None of the above; a chief resident cannot be on a CCC

13. Program coordinators:

A. May serve as voting members of CCCs

B. Can manage submission of Milestones data for the ACGME

C. Should not attend the CCC meeting

D. Should participate as voting members of the CCC

E. None of the above

14. Which of the following is true about CCCs?

A. The best size of a CCC is 12-15 members

B. At least one peer-selected resident should attend

C. Faculty members and/or health professionals with “different”

voices/options are encouraged to participate

D. The most senior persons should express their opinion first

E. None of the above

15. The most reliable assessment of performance is:

A. Multiple choice (written) examination

B. Global end-of-rotation evaluation

C. Multi-rater evaluation (multisource feedback)

D. Procedural log

E. Oral examination

F. Observation of actual performance

16. The literature suggests the idea size of a CCC is:

A. 3 to 5

B. 5 to 7

C. 7 to 9

D. 9 to 11

E. None of the above

17. Which of the following statements regarding Milestones assessments is true?

A. Programs should give faculty members the entire Set of Milestones for

them to use as part of their end-of-rotation evaluations

B. Faculty members should be encouraged to make inferences on the

performance of residents ONLY based upon the performance they have

directly observed

C. Faculty members should generally use the Milestones level that

corresponds to a resident’s year in training (i.e., Level 1 for a PGY-1

resident)

D. Information gained from informal “hallway” conversations can be useful

E. CCCs should use the average calculated by their resident management

system to determine the Milestones level

18. Groupthink is a phenomenon that occurs when the desire for group consensus

overrides people’s common-sense desire to present alternatives, criticize a

position, or express an unpopular opinion. Which of the following is a risk for

groupthink?

A. The CCC feels pressure to make a consensus decision with inadequate

time (decisional stress)

B. Low level of group cohesion

C. Lack of a strong dominating leader

D. The CCC cultivates an environment that encourages dissent

19. A CCC member says, “this is a strong resident, and I think a 2.5 Milestone

rating is appropriate,” and provides two supporting vignettes. This is mostly

likely an example of which type of cognitive bias that is common in groups?

A. Authority bias

B. Anchoring bias

C. Framing bias

D. Confirmation bias

20. Using what’s known from the literature to encourage good group processes, the

CCC should:

A. Encourage the most senior person to discuss a resident/fellow first

B. Have the CCC chair state opinions first

C. Avoid a structured format and use open forum for discussion

D. Use only the synthesis of a resident’s/fellow’s performance rather than

the underlying data used to make that synthesis

E. Ask one member to offer an opposing or different view to help represent

all possible perspectives

21. Feedback to residents/fellows following the CCC meeting is best accomplished

through an email providing them with a written report of their Milestones

performance.

A. True

B. False

22. Individualized Learning Plans are required by the ACGME:

A. Only for residents/fellows failing to progress

B. All residents/fellows

C. Only residents/fellows in the first year of the program

D. Only residents/fellows in the final year of the program

23. An applicant is accepted through an “exceptional candidate” exception and

matriculates into the program. A performance evaluation by the CCC must take

place:

A. Within 2 weeks

B. Within 8 weeks

C. Within 12 weeks

D. Within 20 weeks

24. Which of the following is/are other possible roles of the CCC?

A. Contributing information for use in Annual Program Evaluation

B. Assessing the competence of residents/fellows transferring from non-

ACGME-accredited programs to ACGME-accredited programs

C. Faculty development for core faculty members

D. All of the above

E. None of the above

Modified from an earlier quiz presented by Andolsek, KM and Nagler, A at the

2013 ACGME Annual Educational Conference

Appendix B: Quiz Answers

1. A

2. B

3. C

4. G

5. D

6. A

7. A

8. D

9. E

10. B

11. E

12. A

13. B

14. C

15. F

16. B

17. D

18. A

19. B

20. E

21. B

22. B

23. C

24. D

Appendix C: Case Studies

Mini Case Studies/FAQs/Common Dilemmas/Challenging Situations/Promising

Practices

1. Program director, “Dr. C,” is an accomplished clinician and well-regarded

educator. Dr. C recruits several faculty members to the newly constituted CCC,

but decides to chair the committee to ensure everything occurs correctly and

meets ACGME expectations.

Program directors and programs should think carefully about the role of the

program director in the CCC. The American Board of Anesthesiology precludes the

program director from serving as chair. The other boards and the ACGME

are silent on this issue. Even if there are no rules, it is worthwhile to think through

the role of the program director on the committee. The intent of the CCC is to

ensure all faculty members feel comfortable discussing each resident’s/fellow’s

performance. If the program director is the chair, how comfortable and motivated

are the faculty members expressing their own opinions, versus deferring to the

program director who may “know” many more details about the residents/fellows.

Do the faculty members essentially rubber-stamp the program director’s view? Or

can they provide independent and important judgments necessary to create a valid

consensus, maximizing the strengths of the process, which depend on several,

independent, thoughtful faculty members weighing in?

As with any group process, the program should think strategically about how to

create an atmosphere in the CCC in which all participants feel they can and should

speak candidly and that their opinions will be valued. This committee should be one

of the most important committees in a department and should be known as a place

where faculty members can speak freely and honestly regarding learner

performance in a setting that is supportive, confidential, and structured. Think

intentionally about ways to reduce a hierarchy, perhaps having more junior faculty

members speak first. A faculty chair other than the program director may help

facilitate this process.

In situations where the program director needs to chair the committee, consider

having the program director speak last, after all committee members have provided

meaningful input based on their own observations and experiences. The program

director can be a participant or an observer or not present at all, although many

programs will find it beneficial for the program director to be present to at least

observe and hear the conversations regarding resident/fellow performance.

2. A residency program has 90 residents in a three-year program. The CCC has

its first meeting and can’t imagine faculty members having enough time to

meaningfully review all 90 residents in a practical manner.

There are several options for CCC structure, and since a specific structure is not

dictated by the ACGME, this is an area for programs to be flexible and innovative.

● Some CCCs accomplish this by meeting more frequently—perhaps

three separate meetings at which 30 residents each are considered.

● Large programs may have separate CCCs for each PGY cohort (i.e.,

one for the first-years, one for the PGY-2s, and one for the PGY-3s).

Programs using this model may have the individual CCCs follow their

cohort across all years of the program or develop expertise in the

particular curriculum year.

● Some programs may organize their CCCs around specific activities

(e.g., one CCC to assess the QI activities, one for the research

activities, one for ambulatory versus inpatient activities, etc.).

● Some CCCs have organized similarly to an Institutional Review Board,

where one or two members will review a resident’s/fellow’s performance

in detail prior to the meeting and present their assessments and

recommendations to the committee at the meeting, soliciting feedback

from the group.

Programs will gain efficiency by having the CCC think through its expectations of

performance and identify what program assessments best speak to these. When

gaps in assessment tools are identified, it can help the program address them.

CCC members will benefit from faculty development on the Milestones, and on how

best to assess resident/fellow performance. Whatever methods are chosen, the

program coordinator plays a critical role in organizing and providing the right

information to the CCC and its members.

3. A program wants to “democratize” the CCC to reflect resident input by inviting

its chief resident to attend.

Some chiefs are still considered residents, while other chiefs are considered faculty

members. The ACGME precludes a resident (whether or not a chief) from being on

the committee. The rationale is that residents are colleagues of their fellow

residents, and it can be challenging to have them in a situation in which they

engage in high-stakes performance evaluation of these colleagues. The ACGME

allows a chief who has completed a core residency and is eligible for board

certification in his/her specialty to be a CCC member.

Though technically possible to have a faculty-level chief resident as part of the

CCC, the same concern may lead the program to not include such a resident—

they are often just a year away from being a resident themselves and know the

residents very well, and it may be too challenging to engage in the required tasks of

the CCC. On the other hand, input from all residents on their peers is desirable and

may be an important source of data for CCCs, particularly in resident

Professionalism and Communication and Interpersonal Skills milestones. The

program can accomplish this by having regular resident peer feedback as part of its

multi-source/multi-rater evaluation process. Likewise, residents can have a forum to

discuss peer performance and/or send concerns or accolades to the CCC for

review and inclusion in the faculty process.

4. The CCC wants to thoroughly document its process and keep extensive

minutes.

At a minimum, the program director will record the CCC consensus and report

resident/fellow performance on the Milestones to the ACGME. How much of the

discussion that informs the Milestones decision is up to the individual program.

Specific, behavioral feedback that would help a resident/fellow improve can be

conveyed as with any program evaluation. This information can be shared with the

resident/fellow as part of his/her twice-yearly evaluation meeting with the program

director, an assigned CCC member, or his/her advisor. The assessment data used

by the CCC to develop its consensus should already be available to

the resident/fellow for review. A written document reflecting the discussion of each

resident’s/fellow’s performance should be:

1. A concise summary of each resident’s/fellow’s performance and any action

or follow-up items

2. Confidential

3. Archived for several years*

*The program should consult with its Human Resources and Legal experts to

understand what should be retained, where it should be archived, and for how long.

5. The CCC and the program director disagree on the Milestone performance of a

particular resident/fellow.

The ACGME Common Program Requirements expect the CCC to provide input, but

the program director to make the final decision on resident/fellow performance

against the specialty-specific Milestones.

6. The CCC wants its faculty members to be more comfortable and candid in their

deliberations, and decides not to share its decision on resident/fellow

performance on the Milestones with the residents themselves.

Residents/fellows should be informed and aware of the Milestones performance

summary the program director is submitting to the ACGME. Currently, the ACGME

does not require programs to have the resident/fellow sign a copy of what is

submitted, but it is considered a best practice. It is required that a copy is

kept in the resident’s/fellow’s performance file. It is expected that programs will

use this as an opportunity to provide feedback to residents/ fellows on their

performance, and to discuss what is needed to get them to the next level. It should

be noted that the ACGME provides individual Milestones data to the

residents/fellows via the Accreditation Data System (ADS).

7. A resident doesn’t agree with the CCC and asks it to change its assessment.

The program director should work with the CCC to clarify and communicate the

program process on options if residents/fellows disagree with the CCC or the

program director’s subsequent assessment of milestone performance. Program

policies and procedures should differentiate the situations in which a resident/fellow

can exercise due process and grievance procedures for an adverse decision. Some

programs would consider the CCC consensus, as a judgement but one devoid of

adverse consequences and would not provide an opportunity for a resident to

“grieve” it. On the other hand, a resident could exercise due process if there were

an adverse program decision (suspension, non-renewal, non-promotion; or

dismissal) based upon the CCC’s Milestones evaluation. Programs should work

closely with the DIO and GMEC to ensure program policies are consistent with

institutional policies. HR and Legal contacts may also be useful.

8. The CCC has formed and at its first meeting is deliberating upon the residents’

performance. The Chair is uncertain if they should “call for a vote.”

The authors of this guidebook recommend CCCs not vote. Rather, they

recommend CCCs sufficiently discuss each resident/fellow so they can arrive at a

decision with which each of the members can agree, a true consensus. Calling for a

vote may lead to a situation in which the CCC may appear “divided” and set up a

situation in which its recommendation may be considered uncertain, leaving the

program more vulnerable to a future challenge.

Appendix D: Designing the CCC

Completing this table will provide programs with a draft of the required written

description of the CCC, which they can refine and use to educate residents/fellows

and faculty members.

Element

Describe the CCC on this element

Committee Membership

● Appointed by program director

● Minimum of three faculty members

● at least one core faculty member

● Size—“enough,” but all individuals are committed

and able to get to meetings

● Who on the faculty is best able to take on this

role? (i.e., sufficient resident/fellow contact; need

for subspecialty representation)

● Other members? (at the prerogative of and

appointed by program director)

● Physician faculty members from same or other

program(s)

● Health professions with extensive contact and

experience with the program’s residents/fellows in

patient care and other health care settings

● Chief residents who have completed core

program

● Term limits? (five years? the duration of the

residency/fellowship?)

● Staggered appointments? (may be useful to plan

overlap among those joining the committee and

leaving it)

Chair

● Are there requirements/restrictions imposed from

the specialty board or Review Committee

regarding who can chair (or not; e.g.,

anesthesiology program director cannot chair per

American Board of Anesthesiology)?

If no external requirements/restrictions:

● Consider pros and cons of who is best positioned

for this role (goal is to ensure all voices are

heard—if program director chairs, will everyone

simply defer to the program director)

● Program director?

● Associate program director?

● Another faculty member?

● Rotating among members?

Element

Describe the CCC on this element

Role/Responsibility of each member

● Where is this information

summarized/documented, and how is it conveyed

to CCC members?

● Confidentiality

● Attempts to mitigate bias

● Meeting attendance

● Term length

● Participation in required professional development

around this role

● Necessary preparation in advance of meeting (is

each member assigned a subset of

residents/fellows to review in advance?)

● Who conveys results to program director (if the

program director is not in attendance at a

meeting)?

Who is responsible for any remediation plan (a

member of CCC, or is this referred to another

individual or group within residency/fellowship?)

Role of the Program Director

● Chair (or not)

● A member

● An observer (perhaps he/she only attends but

refrains from providing input)

● Not present

● Provides feedback from CCC to the

residents/fellows (or not)

Role of Residents/Fellows

● Residents/Fellows are not permitted to be

members of the CCC

● In some programs “chief residents” are faculty

members, and not considered trainees; in this

case it may be appropriate to include them

● Residents/fellows are commonly asked to provide

multi-rater feedback on their peers; this

information is typically used by the CCC as one

assessment of resident/fellow performance on the

Competencies of Interpersonal and

Communication Skills and Professionalism

Element

Describe the CCC on this element

(Potential) Role of the Coordinator

Pre-meeting

● Schedule meeting and location

● Notify attendees

● Aggregate data sources (electronically or on

paper)

● Provide information to members before the

meeting so they can engage in any pre-work

● Summarize data, preparing “scorecards” or

“snapshots”

At the meeting

● Provide any information needed by committee

members

● Take minutes

● Document any necessary information to

resident/fellow record

● Record recommendations on each resident/fellow

by milestone

Post-meeting

● Communicate results to program director (if not

present)

● Schedule meetings with residents/fellows and

program director and/or designated faculty

member(s) to review CCC decisions, including

Milestone status

● With program director, submit Milestone

information on each resident/fellow to the ACGME

Shared Mental Model

● How do CCC members develop a shared mental

model of performance?

● What faculty development needs do they have?

● Reaching a common agreement of Milestones

narrative meanings

● Determining how many assessments (and of what

type) are needed for any given milestone

● Determining how to aggregate/interpret data

● Applying QI principles to the evaluation process

● How is this provided? Documented?

● Who is responsible for providing?

● How is any lack of consensus among members

managed?

Consider asking CCC members to self-assess

their performance with specialty-specific

Milestones.

Element

Describe the CCC on this element

Meetings

● When?

● Where?

● How frequently? at least twice yearly for most

specialties; could be more frequently, e.g.,

monthly, quarterly

● How long are meetings?

● What is necessary prep to be completed ahead of

meetings, and who contributes to it? What is

deliverable and who is responsible?

How the work of the CCC will be distributed?

● Some CCCs may be responsible for all the

residents/fellows

● Others may be responsible for a subset of the

residents/fellows, (e.g., all PGY-1s, or the

research component of all of the fellows)

● In a large program, there may be CCCs that each

review a specific subset of the residents/fellows

(e.g., three sub-committees of the CCCs each

review 1/3 of the residents/fellows)

Consensus versus Voting

● Preferable to have CCC reach consensus and not

vote

● How are disagreements among CCC members

managed? Documented?

● Program director is the final decision maker

● Guidance from institutional Human

Resources/Legal on how this is

managed/reflected

Integrating assessments from faculty members

external to the program

● If a faculty member not from the program makes

an assessment on resident/fellow performance

with which the CCC disagrees, it is expected that

CCC will take data from evaluations and apply

them to the Milestones to judge the progress of

residents/fellows

● The CCC will have the advantage of knowing how

each of the specialists evaluated the

residents/fellows and can apply that knowledge as

it marks residents’/fellows’ progress on the

Milestones

Element

Describe the CCC on this element

Minutes

● What information is captured at the meeting

electronically versus in writing? How is it retained?

● Are there institutional policies that address how

this information is retained (i.e., where? in what

format/for what duration?)?

Measures of Assessment/Tools used by the CCC

● Existing resident assessment data

● What are these?

● How many different types of tools (e.g., multi-rater

feedback, in-service training exam, chart audit of

clinical performance)

● How are these assessments documented?

● How are these assessment shared with

residents/fellows?

● Are there challenges (e.g., faculty members not

completing assessments; milestones for which no

assessment is currently done)? Can the CCC

work with the program to solve these issues?

Measures of Assessment/Tools used by the CCC

(continued)

● Faculty observations

● How are these organized (global end-of-rotation

evaluation, checklist from a procedure, simulation,

standardized patient, miniCEX)?

● How are these documented?

● Used in provision of feedback to

residents/fellows?

● Data from Milestone assessments

● Are these observations captured in such a way

that they provide useful input in Milestone

assessments

Inventory of the Milestones

● Where is each taught in the curriculum?

● How/where/by whom/ is each assessed?

● What are the gaps in teaching and assessment

and what are the plans for addressing them?

Element

Describe the CCC on this element

Are there expectations the program has of

residents/fellows that aren’t captured in the

current specialty Milestone(s)?

● How are these communicated to

residents/fellows? To faculty members?

● How are these assessed and documented?

If a resident/fellow is performing sub-optimally:

● Is the CCC (or a member of the CCC) responsible

for an individualized learning plan/ remediation

plan? Another member/group of faculty members?

What are the options for individualized learning/

remediation?

● Intensify mentoring

● Additional readings/structured reading plan

● Skill lab/simulation experiences

● Added rotations

● Repeat rotations/activities

● Extend education

● Counseling to consider another

specialty/profession

Transparency of the CCC process

● How do you describe the CCC process to your

residents/fellows and faculty members (e.g.,

program manual, web page)?

● Is the description of the CCC process up to date

and reflective of actual process?

If a resident/fellow disagrees with a CCC

assessment:

● Review with Human Resources and Legal the

desirability of a grievance process in this

instance (not required by the ACGME)

● Courts (in general) support faculty decisions:

“Made at routine meeting for the purpose of

evaluation”

“Shared understanding of performance”

“Reasonable process”

Residents given notice (of deficiency) and

“opportunity to cure” (ameliorates)

Conscientious decision making

Take into account the entire performance

record

Element

Describe the CCC on this element

How do the Milestones fit into promotion criteria?

ACGME Institutional Requirement IV.C.1.:

“The Sponsoring Institution must have a

policy” that requires each of its programs to

determine the criteria for promotion and/or

renewal of appointment…”

How do the Milestones fit into the program’s criteria

for promotion and/or renewal of a resident’s/fellow’s

appointment? Based upon program review:

● Do you need to make any adjustments in your

criteria for promotion and/or non-renewal?

● Do you need to change your agreement of

appointment to reflect Milestone reporting to the

ACGME?

● Do you wish to modify your grievance policy?

You may find that you do not need to make any

changes at all, but this is an excellent opportunity

to review your current processes and ensure they

align.

Element

Describe the CCC on this element

Using the CCC in continuous educational quality

improvement

● Following the CCC meeting, it may be useful to

debrief

● What types of assessments were particularly

helpful to the CCC in making decisions on

resident/fellow performance?

● Who among the faculty members generated the

most useful assessments (e.g., from explicit,

behaviorally specific narrative comments)

● Do the residents/fellows consistently demonstrate

challenges in their performance on a small subset

of the Milestones? (If so, this may be either a

curricular issue or the lack of an effective

assessment tool)

● What did the program learn from the CCC

experience to help improve the overall educational

and assessment process? (e.g., simplifying the

assessment system; applying examples from the

most useful assessment formats to those that

were least useful)

● What can the program learn from its best

assessors? How can they

acknowledge/reward/use these faculty members

as role models? How can these faculty members’

practices be transferred to other faculty

members?

● Based on this debrief, identify at least one way to

improve assessment in the program

● Specify who will do what, and what exact timeline

to implement the change

● Follow up on results of the improvement at the

next CCC meeting

● Did all faculty members feel able to honestly

represent their views on each resident/fellow?

What impeded/facilitated this ability, and can

enhancements be identified?

Consider making the CCC and its related processes

part of the Annual Program Evaluation

Modified from an earlier table presented by Andolsek, KM and Nagler, A at the

2013 ACGME Annual Educational Conference.

APPENDIX E: Examples of Assessment Methods for the ACGME Core

Competencies

Competency

Method

Example

Patient Care

Direct observation

Mini-Clinical Evaluation Exercise;

various apps.

Simulation

Partial task trainers for procedures;

virtual reality

Standardized patient

Objective standardized clinical exams

(OSCEs)

Clinical performance

review

Medical record audits using quality and

safety measures

Procedure log with

assessment of competency

Surgical Case Logs with/without

entrustment scales, potentially with

learner reflection

Faculty evaluations of

observed performance

Evaluation forms using developmental,

supervision, or entrustment scales

Video-captured

performance

Surgical or other procedure; patient

encounter

Virtual reality

Simulation of procedure/ encounter

Multi-source feedback/360-

degree

Feedback from patient experience, team

members, resident/fellow peers

Medical Knowledge

In-training Examination

(ITE)

Most specialties now have an ITE

provided either by their certification

board or a specialty society

Work-based assessments

of medical knowledge

SNAPPS framework; mini-clinical

evaluation exercise (MiniCEX);

Assessment of Reasoning Tool (ART)

Oral-guided chart review

Chart-stimulated recall

Interpersonal and Communication Skills

Multi-source feedback

(MSF)/multi-rater/360-

degree

Some tools available; most home

grown; Teamwork Effectiveness

Assessment Module (TEAM) multi-

source feedback instrument

Patient experience surveys

Consumer Assessment of Healthcare

Providers and Systems (CAHPS) suite

of survey tools

www.ahrq.gov/cahps/index.html

Practice-based Learning and Improvement

Self-assessment

Milestones self-assessment followed by

a compare/contrast review of CCC

Milestones ratings with a mentor or

advisor

Evaluation of

resident/fellow teaching

skills

Evaluation forms

Evidence-based practice

(EBP)

Clinical question logs; EBP

prescriptions; EBP assessment of

journal articles

Professionalism

Contribution to institution’s

error reporting process

Spontaneous error reporting; root cause

analysis

MSF/multi-rater/360-

degree

Some tools available; most home grown

Patient survey

CAHPS suite of survey tools

Systems-based Practice

Quality improvement (QI)

project

Can judge the quality of a QI project

using several tools; can measure the

impact of QI project through clinical

performance measures

Contribution to institution’s

error reporting process

Spontaneous error reporting; root cause

analysis

Appendix F: Overview of Assessment Methods Aligned with Miller’s Pyramid

(adapted from Lockyer, et al.)

Stage

Corresponding Methods to Assess Performance

Does

Medical record (chart) review

CCC

Direct observation

Efficiency data

End-of-rotation evaluations

Multisource feedback

Patient outcome data

Portfolio

Case Log with assessment

Project review

Shows How

Objective structured clinical exam

Oral case presentation

Simulated case

Skills station

Virtual/computerized patient management

Knows How

Chart-stimulated recall

Development of ILP

Mock oral boards examination

Oral questioning targeting patient management

Written test

Knows

Test (oral or written) targeting fact recall

Reference:

1. Lockyer J, Carraccio C, Chan MK, et al. Core principles of assessment in

competency-based medical education. Med Teach. 2017;39(6):609-616.

https://www.tandfonline.com/doi/abs/10.1080/0142159X.2017.1315082?jour

nalCode=imte20. 2020.

Appendix G. PPV (Predictive Probability Value) Tables

PPV tables provide the probability (in percentage terms) that a resident/fellow at

or below a certain Milestone rating (Level) would not achieve Level 4 by time of

graduation. In the example shown below, all PPVs for the Patient Care

Subcompetency (#03) in the Family Medicine Milestones that could be calculated

as of June 2019 are included in the table.

For example, a resident receiving a rating of 2.5 or lower at Milestone review

occasion four, at the end of the PGY-2 has a 54.7% probability (based on national

data) of not achieving Level 4 by the end of the three-year family medicine

residency in this sub-competency.

Example PPV Matrix for the Patient Care Subcompetency (#03) in Family

Medicine: Partners with the patient, family, and community to improve health

through disease prevention and health promotion.

The table shown in this example provides a matrix of all PPVs by Milestone rating

threshold and Milestone review occasions for a single sub-competency for a

single specialty. The tables that follow the box plots in this report for each

specialty provide the PPV matrices for the core specialties and sub-competencies

for which data was available as of June 2019.

PPVs are provided to help identify residents within a program who may be

struggling to match normative national data during each six-month block of the

educational program. This can then be used to support decisions for remediation

or ILPs.

Appendix H: ADAPT Model of Feedback

Reference:

1. ADAPT Model of Feedback. University of Washington. Accessible at:

https://depts.washington.edu/lgateway/elearning/feedback/story_html5.html

Appendix I: R2C2 Evidence-Informed Facilitated Feedback

(tri-fold and learning change plan forms may be found at

https://medicine.dal.ca/departments/core-units/cpd/faculty-

development/R2C2.html)

Reference:

1. Dalhousie University. R2C2 Feedback and Coaching Resources.

https://medicine.dal.ca/departments/core-units/cpd/faculty-

development/R2C2.html. 2020.

Appendix J: Institutional Checklists for CCCs

(links to institutional oversight of CCCs)

Adapted from Y. Wimberly, MD, Morehouse School of Medicine

✔ Recommended Practices

Comments

for DIO and GMEC

Program incorporates CCC information into APE and identifies

improvements

Regular review of overall resident/fellow milestone performance

and adverse events related to resident/fellow promotion,

program completion, withdrawal, dismissal

Institutional faculty development for CCCs

Program director/core faculty members, and CCC members

participated in program/institutional faculty development for

CCCs, Milestones, and assessment

Quality of CCC documentation across programs (any

expectation this is standardized across programs)

CCCs have “right” membership (> 3 faculty members, including

at least one core faculty member)

Program CCC processes consistent with institutional policies

Sharing of CCC experiences among the institution’s programs

Resident/fellow Milestone performance and/or program CCC

experiences/performance incorporated into Annual Institutional

Review

Program satisfaction with institutional tool(s) such as a resident

management system, used to aggregate performance data for

CCCs

For Program Directors

Need for program “grievance policy” for CCC decisions?

Program director’s CCC role described and acceptable to

specialty

Program faculty development for CCCs

Documented feedback to residents/fellows on their Milestones

performance

Identified enhancements in assessment based on CCC

experiences

CCC Improvements identified as part of Annual Program

Evaluation

Program director has final responsibility for resident/fellow

evaluation/promotion decisions

✔ Recommended Practices

Comments

For CCC Chair

CCC conclusions/recommendations are communicated to

program director

Process for how lack of consensus is managed within the CCC

Best practices in group meetings utilized

Review resident evaluations at least semi-annually

For CCC Members

Participate in faculty development (annually?)

Attend specified percent of CCC meetings

Provide requested pre-review of residents and/or meeting

“preparation” prior to meeting

Reach common agreement of Milestone narratives and

understand how assessed

Respect confidentiality

Work to recognize and mitigate biases

For Faculty/Evaluators

Provide timely, honest, high-quality assessments using

appropriate methods to allow CCCs to make informed decisions

regarding resident performance measured against the

Milestones

For Residents/Fellows

Complete Milestones self-assessment before each CCC

meeting?

Compare Milestones self-assessment with program

determination of Milestones assessment following meeting

Co-create an ILP for continued growth with program director or

designee