Boston University Medical Campus
Nuclear Accident Preparedness Program

Chapter 7

CHAPTER 7

EVALUATION METHODS
Introduction

You are a patient in a hospital in need of having an open wound sutured. Which of the three nurses would you select to suture the wound? The only information you have on which to base your decision is the method by which each nurse has been evaluated and judged to be competent in suturing wounds.

Nurse A received a perfect score on an essay describing how to suture wounds.

Nurse B sutured “wounds” correctly on 4 out of 5 trials using mannequins in a lab setting.

Nurse C sutured wounds correctly on 5 out of 7 patients in a health center under supervision.

Nurse A, B or C? ____
Most people select Nurse C. Why?
The task of suturing wounds involves a number of skills including:

Nurse A clearly had mastered the important knowledge components of suturing wounds (as demonstrated by performance on the essay) but this method of evaluation cannot predict how well Nurse A might apply that knowledge in an actual setting.

All on-the-job conditions including an apprehensive patient, the pressures or other responsibilities, the need to set up the equipment, etc., were missing. Furthermore, the desired performance could not be observed based solely on an essay test. Performance such as putting the patient at ease, cleaning the injection site, actual suturing, etc. The only skill that can be evaluated based on Nurse A’s performance is writing or communicating.

The method by which Nurse B was evaluated is more desirable. Testing conditions more closely match on the job conditions including the actual equipment and a person-like model with which to work. And much of the desired performance could be observed including cleaning the site, selecting the instruments etc.

What is lacking are conditions of working with a living, frightened and suffering human-being as well as all the normal distractions of a clinical setting including responsibilities for other patients.

Clearly the testing conditions for Nurse C most closely match on-the-job conditions. The only difference is that she could turn to her supervisor if she requires guidance or assistance.

In general, in selecting a method of evaluating students for a specific job-related task, the goals of the evaluator should be to design testing conditions that match AS CLOSELY AS POSSIBLE on-the-job working conditions, and to specify desired student performance that matches AS CLOSELY AS POSSIBLE performance on the job.

And whereas “paper and pencil” tests might be appropriate for assessing knowledge components, they are often inadequate for assessing the degree to which students can apply that knowledge in an on-the-job setting.

Think of a SKILL as being the application of knowledge, then students who master skills can be assumed to have also mastered the job-related skills.

Therefore, the course designer must plan evaluation activities to assess:

1. The various knowledge components of a given task (especially to help students who cannot apply assumed knowledge at some point in the course.)

2. Skills needed for doing the task.

Chapter Purpose:

The purpose of this chapter is to describe a variety of methods that are appropriate for assessing skills and/or knowledge relevant to the tasks of the JOB DESCRIPTION.

What follows is a brief discussion of several testing methods.

A. Practical Examinations

Practical tests allow the instructor to observe and assess students as they perform critical tasks. A practical test is a test of performance. It test the actual on-the-job skills (thinking, doing, communicating) or comes as close as possible to testing these skills.

One simple way of testing performance is by constructing a checklist which outlines the critical steps in the procedure.

Directions for constructing practical examinations follow:

1. Identify Critical Steps. First decide what the critical (important) steps are in performing the task…

a. List the steps.
b. Cross out unnecessary steps, and
c. Order steps as they would be performed on the job.

2. List the steps one by one.

Each step should be written so that it can be checked “yes” or “no”. That is, each step should consist of a single idea. Statement a and b below can easily be checked yes or no. But you might have difficulty deciding between yes and no for statement c.

a. Uses cleansing agent. ___Yes ___No
b. Washes instruments thoroughly. ___Yes ___No
c. Using cleansing agent, washes instruments thoroughly. ___Yes ___No

3. Describe behaviors using strong action verbs.
In writing statements, do not use imprecise phrases like “answers quickly” or listens carefully”. Try to state exactly what you mean in terms of behavior. For example:
He/she computes the correct dosage of medicine.
He/she is able to describe the procedures for rehydration of infants.

4. Develop clear instructions.

Develop specific instructions to the person taking the test. If others are administering the test, ask them to read the instructions and the test. If any statements are unclear, explain the step and revise it accordingly.

Example:
An example of a practical exam follows:

Traditional birth attendants are being trained to use modern, aseptic techniques when they are helping mothers give birth.
Instructions to the student: “Here is a group of instruments that have been used at a recent childbirth. Sterilize the instruments.”
Checklist for Sterilizing Instruments:

a. Use cleansing agent ___Yes ___No
b. Washes instruments ___Yes ___No
c. Rinses sterilizer ___Yes ___No
d. Places instruments in sterilizer ___Yes ___No
e. Adds water to cover ___Yes ___No
f. Brings water to boil ___Yes ___No
g. Boils water for 15 minutes ___Yes ___No

Instructions to the teacher: “Observe the trainee and mark yes or no for each step. If the trainee fails to perform one step of the procedure or performs incorrectly, the trainee fails the exam.”

B. Observational Reports

This consists of documented observations of student performance over a period of time. Students are asked to perform tasks in settings close to an on-the-job environment. The observer often uses a checklist or a rating scale to assess the performance such as in practical examinations discussed above.

Example: 1. Throughout the duration of several cases, a health assistant is observed at different stages of patient management.

2. Once a day the head nurse observes each student nurse’s performance for five minutes and records ratings about that performance.

3. In a dental clinic, the teacher is required to evaluate each successive stage of a dental preparation.

C. Oral Examinations

Students respond to questions or to problem situations. Students may be asked to formulate and justify an approach to a problem. They may also be asked to assume a certain role and to respond from that viewpoint.

Example:
Given background information regarding:
a. The socio-economic make-up of a village.
b. Available resources in that village.
c. Health problems in that village.

Students assume the role a public health officer and discuss an appropriate plan to deal with the health problem.

Given an assignment to conduct sputum tests as appropriate in a village, students will describe:

a. Equipment and supplies needed.
b. The process of completing sputum tests.
c. How they will approach people to be tested.
d. How they will record information and follow-up.

D. Project Assignment

Students are given tasks or projects to complete in a given time period. Performance is evaluated on the basis of the product produced.

The project produced might include:

ú A research paper
ú A laboratory experiment or demonstration
ú A community survey
ú Tissue dissections
ú Dental preparations

E. Essay Examinations

Similar to oral examinations, except the student responds to questions or problems in writing.

Examples:
1. Given a letter from a community group requesting advice on how to design community health services, the student drafts a reply.
2. Given a copy of the clinical record of a child, the student writes instructions for follow-up care by the parents at home.

F. Objective Examinations

The student responds to questions by choosing the best answer(s) from those given (multiple choice, true-false, matching, rank-in order) or by writing an appropriate answer (fill-in-the-blank or other short answer format). The questions may be given with related visual or auditory material.

Examples:
1. Given a partially completed statement table, the student completes the table.
2. Given a tape-recording of heart sounds, the student answers a series of multiple-choice questions regarding what he/she hears.
3. Given a medical history for a patient who has been admitted to a hospital with given symptoms and lab tests, the student selects the most likely diagnosis from a set of six possible alternative diagrams.

Figure 17:
Best Uses (X) Of Some Evaluation Methods