Robert Karp, MD
Avrahom Gurwitz, MPH
Department of Pediatrics
SUNY-Downstate Medical Center
Brooklyn, NY
Darwin Deen
Department of Family Practice
Albert Einstein College of Medicine
Pretest for Part V Section 1
Q1. Pick best answer. An:"instrument" in the field of evaluation is:
A. A well played saxophone used as a background in a teaching video.
B. A observation tool that has been validated using several observers of the same activity getting same or almost same results
C. Both A and B
D. Neither A nor B
Q2. True or False: It is necessary to put structures in place before beginning an educational program.
Objectives
On completion of this module, residents will:
1. Be able to establish an evaluation protocol for a teaching program.
2. Develop an active learning exercise teaching residents how learn from role-play.
Introduction
Evaluation is an essential component in the implementation of any program or intervention. It is a dynamic process offering a way to insure that an intervention or program completes objectives or a learner achieves competence. Evaluation begins at the inception of an educational program and continues through its completion. It provides assurance that a program is headed in the right direction and allows for appraisal at crucial steps along this path. Without an evaluation protocol in place, one can easily imagine the implementation of a program (intervention), which is ineffective (has not completed its objectives), and results in wasted time, effort, and money. Effective evaluation also allows for recognition of phenomena not accounted for originally that may then be addressed. Without a system of continuous evaluation, the causes for intervention or teaching failure may never be known. Most importantly, a potentially valuable intervention may be abandoned when, if instituted with proper evaluation, it could have a substantial impact on medical practice or an individual's capabilities.
A. The components of evaluation
The process of evaluation is typically broken down into 3 parts. These represent evaluative processes used before ("Structural"), during ("Process/Formative"), and after ("Outcome") implementation of an intervention.
1. Evaluation of Structure
- Has a need to accomplish the task at hand been established?
- Are the structures necessary to achieve success in place?
2. Evaluation of Process - also called "formative" evaluation
- Are what is necessary to be successful being done?
- Is the learner being informed about the quality of his or her performance so that it can be improved? This is called "feed-back."
3. Evaluation of Outcome
- Have the correct outcome measures been chosen?
- Are the appropriate instruments being used to make that assessment? An appropriate "instrument" is an assessment tool with known validity and reliability. It is inexpensive and easy to use.
- Is the instrument being used properly?
B. Clinical Competency
Clinical Competency represents achieving the knowledge, skills and attitudes necessary to achieve a defined task. The Table shown below provides a matrix for what is needed in the three areas of evaluation (Structure, Process/Formative, and Outcome) and to assess the three domains of clinical competence (Knowledge, Skills, and Attitudes).
TEACHING CAPTION: It is necessary to ask "have structures necessary for success have been put in place?" and "are there ways to provide feed-back?" If not, it is unlikely that the learner will learn or the project at hand will complete its objectives.
Note: Dear Reader, my late colleague Richard Kravath showed this format to me. I cannot ask to find out from where. Please inform me if credit is due to another source.
C. A role-play exercise to reinforce concepts presented above and a bridge to Part 2.
You are at an afternoon case conference. There are 9 residents and medical students in attendance. Give a set of ground rules for a role-play including always speaking in the first person when playing the role and be sure to make the role play "safe." No assumptions should be made about the role-play, nor does the role-play continue beyond the time of the exercise.
1. Break the participants into groups of 3. There are three roles: 2 are parents and one is a pediatrician. Each plays his or her role without regard to own gender.
2. Put parental "expectations" on slips of papers in envelope. Try these or create your own: 1) a mother that doesn't want to breast feed, 2) parents who do not want to give their breast feeding infant vitamin drops, and 3) parents of an overweight, over fed baby whose parents want to continue supplementing breast feeds with additional bottle feeds.. [Source material for each of these issues can be found in the Teacher's Guide's Part I Section 2 (Pediatric Nutrition Notes, infant feeding), Part III Sections 1 to 3, promoting healthy weight gain, and Part IV Section 1-B Breast Feeding.]
3. Each group will randomly choose their role-play from the envelope
4. The groups of 3 will meet for 15 minutes. The parents will tell the pediatrician their expectations. The parents should maintain their expectations until an argument with sufficient evidence is presented to them. The information can't just be accurate, it also has to be presented in an effective way that shows respect for parental beliefs. This approach allows parents to give up their viewpoint and adopt another. Alternatively, the parents can hold to their beliefs or be equivocal, as happens in real life.
5. Each group will make a brief presentation of what they learned from this.
6. Always close with asking each player how they felt about the role and giving them a release from the role.
One cannot expect parents to change opinions and actions based on a single interaction. Instruct all groups to avoid confrontational language or hostile expressions. Rather, the parents should show genuine interest to what the pediatrician is saying, and the pediatrician should use "active listening" techniques. Parents will have a chance to say whether the arguments were convincing and the presentation empathetic. The role-play must be "safe" at all times for everyone involved.
The following might occur in a successful planning session:
Many parents feel that the principles of good nutrition we promote are incorrect: Why breast-feed, give a Vitamin D supplement, or not overfeed? Perhaps they have adopted a feeding system based on what they learned without actually ever having been taught. It is also possible that the parent might back up their practice with information gained from science as it was understood in the past. Rapid weight gain, for example, was always considered advantageous; bottle-feeding was considered an advance for women seeking a career; sunlight can produce Vitamin D.
The role of the pediatrician will be to listen to anecdotal stories with respect all the while providing a non-judgmental counter view in which basic principles of breast-feeding are reinforced. The best outcome for this role-play is for the participants to feel comfortable in the dialogue no matter how it turns out.
Residents review: Have everyone list out the possible ways used to evaluate:
1. Structure
2. Process/Formative/feed-back
3. Outcome
Some examples of what residents might present:
Structure: Is there evidence available that shows breast fed infants do better than bottle-fed ones? An example would be a paper showing that breast-feeding infant reduces the incidence of disease and promotes better development. Are you sure that they need vitamin supplementation? Why? Again, evidence is provided. Should parents who refuse to breast feed need to be convinced to change their minds? Why or why not? Should they be forced? Why or why not? What compromises would you make? How do you aid the partial or full bottle-feeding mother?
Process/Formative/feed-back: Have you observed how residents interact with parents? Are they effective? Have you shown them a better way? How does one interact with a resistant parent? What does one say and what does one not say? How do you instruct parents to find out more about breast-feeding? Can you help them assess the quality of material that appears on the web? [See module PII S1 on the 4 E's of interviewing: Empathy, Engagement, Enrollment, and Education.]
Outcome: Did the "parents" change their mind? Is that really an appropriate outcome? How much is "enough" for the results to be satisfactory? Are they really convinced? Have you successfully analyzed obstacles for success? Can you analyze partial success? Is a true or false post-test an adequate assessment of resident competence? Was the "resident" an effective communicator? Did he or she have an appreciation for the culture-centered concerns of the "parents?"
Annotated Answers
A1. The answer is B. We all appreciate the background music, but evaluation requires a tool, known as an "instrument", that is valid and reliable. Reliability is when one observers measures several phenomenon and gets same or almost the same results. Validity is when multiple observers of the same phenomenon get the same or almost the same outcome.
A2. The answer is True. Here is yet another give-away question to make an obvious point. It may be obvious that the elutriation process should be in place before starting to teach, or do just about anything, but the unfortunate reality is that structures are usually not put in place.
