Evaluation of Residents
Resident evaluations may be done in a variety of ways. This brochure describes the most commonly used methods: supervisory reports, patient care observation, record review, chart stimulated patient presentations, oral examinations, videotape review, written examinations, review of patient logs, patient and staff satisfaction surveys or other means. There is increasing interest in graduate medical education for greater reliability and validity in the evaluation of resident competence.

Performance in Clinical Services,
Treatments and Didactic Seminars
Evaluations of residents by faculty are recorded on forms provided to them by the Training Program. Essentially, they ask for assessment of whether the resident in question met the goals and objectives of a clinical service, particular treatment method or seminar and whether each competency has been achieved. Every clinical service and didactic seminar has specific goals and objectives, given to the residents at the beginning of the year. They will be attached to each evaluation form. The goals and objectives of the core competencies are contained in an evaluation manual, which is distributed to all residents and faculty annually.

As noted above, each resident should be assessed midway through each clinical service and at the conclusion of each seminar. It is a vital role of faculty to give ongoing feedback to the resident about his or her increasing knowledge, skills and attitudes in all areas of training. Sensitive, open dialogue about a resident’s professional development is a critical part of this training program.

There are two grades for assessment in each category:
S (satisfactory) and N (needs improvement). After each, there is space for written comments. The evaluation form requires the name of the faculty member and resident, clinical service and/or treatment modality or seminar, date of evaluation and method(s) used for assessment. Seminars also require evaluation of attendance, preparation and participation.

If a faculty member feels a resident needs improvement, it is incumbent on that faculty member to specify:

• the exact nature of the weakness
  
  
• a plan for remediation by that evaluating faculty member
  
  
• a timetable specified for the specific deficiency to be improved

If there is a weakness, it should be identified early enough in a rotation so that the faculty member has ample opportunity to work with the resident to improve performance.

Residents are also required on each evaluation form to perform a self-assessment about his or her performance in every clinical rotation and seminar. The self-assessment, in narrative form, is intended to help the resident learn to monitor personal strengths and weaknesses. This is an important part of the practice-based learning and improvement competency. Self-assessment should also be part of the ongoing dialogue between faculty and residents, so residents can get feedback on their own perception of their personal evaluation of knowledge, skills and attitudes.

Methods of Evaluation
Evaluation of clinical competency is an essential function of the teaching faculty. Each faculty member is required to be familiar with the six core competencies and must strive in all educational efforts to assess how the resident is progressing in each area. The following are descriptions of the most common methods faculty members may use in the assessment of resident knowledge, skills and attitudes. Each faculty member must indicate on evaluation forms which methods were used to formulate evaluations. Faculty members are encouraged to increase their repertoire of assessments over time. The Program will provide guidance in helping educators learn new methods of evaluation in the annual faculty evaluation retreats.

• Supervisory Reports
  Supervisors may use personal notes about a resident’s performance. Such notes may be based on personal observation of the resident with patients, ongoing case discussions and review of medical records or written patient evaluations. The notes may be put together into a formal report to the Training Director, but, if used, should always be summarized in a resident evaluation form.
  
  
• Patient Care Observation
  The Training Program encourages faculty to observe residents caring for patients. This should be done in all clinical settings. In some services this is easier than in others. For example, there are many opportunities for direct observation in inpatient, partial hospital and consultation services to pediatrics, courts and schools. Faculty should take note of resident care in both formal interviews, e.g., diagnostic evaluations, individual and family meetings, and informal interactions, such as on the milieu.
  
  In outpatient care, each clinical supervisor is expected to observe resident interviews with as many patients as possible, both in diagnostic evaluation and in a psychotherapy session. While some faculty may be comfortable sitting in the same room with residents and their patients during a treatment session, others may choose to use another method such as watching the resident through the one-way mirror or having the resident videotape an interview or session with a patient or family.
  
  Following an observed interview or treatment session observed, the supervisor should discuss communication and interviewing techniques, clinical reasoning, case formulation and differential diagnosis, treatment planning and therapeutic skills used for that particular case.
  
  
  
• Record Review
  This method involves a faculty member reviewing a resident’s written medical record of a patient. It is useful for evaluating skill in documenting care, clinical reasoning, data gathering and synthesis, treatment planning, use of ancillary testing, use of hospital and community resources, communication with other professionals and use of best practice standards in clinical care.
  
  
• Chart Stimulated Patient Presentations
  This method may be done in supervision, in clinical examinations or at the close of a rotation. The resident brings in one or more charts of patients he or she has treated and uses chart documentation to describe and discuss patient care issues, including data gathering, clinical reasoning, methods of clinical care, prevention and educational methods, patient outcome, use of resources and use of systems of care in disposition planning. Formal discussions may also help evaluate the resident’s knowledge base and method of self-monitoring for practice-based learning and improvement.
  
  
• Case Reports
  Residents may be asked to present a case report in individual supervision, on rounds in hospital-based care or in clinical conferences. They may be written or oral. The case report is intended to allow a resident to present a comprehensive history and clinical evaluation of a patient. The evaluation should also include a thorough differential diagnosis, formulation, treatment plan and prognosis. In some cases, a case report may serve to highlight a particular diagnosis, clinical problem or treatment. In these situations, residents may supplement the case report with an article or brief literature review. The case report may be useful in assessing data gathering and synthesis, knowledge-base, clinical reasoning, methods of clinical pare, use of ancillary tests and systems of care and prevention methods.
  
  
• Checklist Evaluation of Live Performance
  (Formal Oral Examination)
  This method of evaluation is typically known as the “Mock Board Examination.” It is used with the resident evaluating a patient for a period of time, followed by a complete discussion of the case, including a presentation of clinical findings, formulation, differential diagnosis, treatment planning and prognosis. The examination is called a “Checklist” evaluation, because specific interview techniques and elements of an ideal case presentation are listed for the examiner to check. This is an effective method of resident competency assessment and should be used by as many supervisors as possible during training. In addition to rating the interview and presentation, the examination may be used to assess all six competencies, as well as medical knowledge relevant to the clinical case examined.
  
  
• Written Examinations
  The Training Program administers the PRITE and CHILD PRITE examinations annually. Other written examinations may include essay examinations in clinical rotations or seminars.
  
  
• Written Reports
  Supervisors should routinely review written clinical evaluations prepared by residents. Some of these should include a review of formal diagnostic evaluations, while others may include a review of emergency assessments or consultation reports. Some of these may have specific functions. For example residents should prepare a forensic evaluation and a school report and have them assessed by specialists on their clinical rotations. Other specialized evaluations, such as those used for adoption and custody, sexual or physical abuse or disability determination should be reviewed by supervisors. Reports are helpful in evaluating a resident’s data collection, use of medical knowledge, clinical reasoning, communication of findings appropriate to the reader, understanding of systems and professionalism.
  
  
• Patient Logs
  The resident patient logs should be reviewed by service chiefs to ensure an adequate volume and variety of patients on each clinical service.
  
  
• Portfolios
  A portfolio is a collection of products prepared by the resident that provides evidence of learning and achievement related to a learning plan. The resident can include video or audio recordings, self-reports of experiences or other documents that demonstrate such competencies as therapeutic effectiveness, ethical integrity, professionalism, self-directed learning and skill development, lectures given and continuing education experiences and written documents, such as review or research papers or case formulations. Patient logs may be included in portfolios.
  
  
• Other Methods That Involve Outside Evaluations
  Professionalism, patient care, interpersonal skills and communication, practice-based learning and improvement and systems-based care can all be assessed by additional measures such as:
  
  
  • Patient satisfaction surveys
    These may be based on samples of satisfaction with crisis intervention, consultations, psychotherapy, psychopharmacology, inpatient or partial hospital care. They must be done confidentially.
    
    
  • Evaluations by unit staff
    These may be done in inpatient, partial hospital or outpatient rotations. They should be done confidentially.
    
    
  • Evaluations from professionals in systems of care
    These may include medical professionals, allied health professionals or colleagues in pediatric, school, court or community settings.
    
    
  • Evaluations by participants in seminars or larger meetings, where residents give presentations
    
    
  • Peer review and faculty review of specific projects
    These may include graduation papers, academic presentations, participation in research projects, edited videotapes or Training Program enhancements.
    

   

Faculty Evaluation Retreat
Each year the Program will host a faculty retreat to review and discuss the methods of resident evaluation. This meeting will serve to enhance the reliability and validity of evaluations and to discuss new means of effective resident evaluation. It will also be used for faculty evaluation of the Training Program.

Faculty Resident Review Meetings
Residents are also evaluated by the faculty, as a whole, in two review meetings each year. During those meetings, service chiefs, attendings and supervisors discuss each resident with reference to progress in each of the core competencies. The Training Director takes detailed notes about the performance of each resident and places them in the resident’s permanent file. These meetings also serve to allow faculty members to review the Training Program.

Formal Written Examination
All child and adolescent residents are required to take the Child Psychiatry Residents in Training Examination (CHILD PRITE) each year. General psychiatry residents and child and adolescent residents in their PGY 4 year at MGH/McLean take the Psychiatry Residents in Training Examination (PRITE) each year. Child and adolescent residents (PGY 5 and beyond) are encouraged to take this examination as well. Both are standardized examinations produced by the American College of Psychiatrists (ACP), and have national norms for resident comparison.

Observed Clinical Examination
Each year, the Program conducts a formal clinical skills examination for residents. Following the format of the psychiatry Oral Board examinations, the child and adolescent psychiatry resident will interview an adolescent for 30 minutes, while observed by two faculty members. The resident will then take 30 minutes to present the case, including history, clinical findings, discussion of differential diagnosis, DSM IV diagnosis, treatment recommendations and prognosis. Faculty members will then provide a 30-minute critique of the interview and presentation. A standard checklist of clinical techniques of interviewing, case presentation, differential diagnosis and treatment planning is used for the evaluation. Resident performance is recorded by the faculty members and sent to the Training Director to be kept in the resident’s permanent record. The current oral examination is coordinated with all Harvard Child and Adolescent Psychiatry Training Programs. Faculty from one of the other two Harvard programs examines residents. In the future, we hope to provide a clinical examination exactly in the manner performed by the American Board of Psychiatry and Neurology, adding components including videotaped interviews of a preschool child and a vignette of a latency child. Consultation questions will be asked of the residents in the course of their discussions.

Biannual Resident Review Meetings
The Training Director will collect all faculty evaluations in an ongoing manner and keep a file for each resident, with the other evaluative measures noted above. He will fill out a training summary every six months, based on all the assessments collected at that time. Each resident will meet formally with the training director at least twice yearly to discuss progress towards the attainment of all the goals and objectives of the Training Program. During those meetings, the resident will also discuss his or her evaluations of faculty and the Training Program.

Longitudinal Resident File
The Training Director keeps a longitudinal file that contains all of the resident’s evaluations, PRITE exams, Oral examination checklists, patient care logs and any other material relevant to the assessment of the resident, e.g., unsolicited letters of commendation, patient or staff evaluations, presentations given at local and national meetings, publications and awards, among other documents. This will be part of the resident’s permanent record that also includes all application and preliminary interview material, records from adult residency and any additional documentation about the resident’s performance past and present. It will also include a checklist of seminars and clinical service rotations that are required as part of the residency program and indicate if they were successfully completed. The file will document any evidence of unethical behavior, unprofessional behavior or clinical incompetence. Where there is evidence, it will be comprehensively recorded, along with the responses of the resident. If disciplinary or remediation actions were taken, they will be documented with a clear description of the outcome. The record will include a final letter from the Training Director verifying whether the resident has successfully completed the program and demonstrated sufficient professional ability to practice competently, ethically and independently, based on the program’s defined core competencies.