Creating Competency Performance Examinations and Assessments Concepts

Kirkwood (1981) observed: "Performance assessment has seldom been tried and found difficult; rather, it has been found difficult and seldom tried." The process of developing and implementing performance assessment methods, whether in academic or non-academic settings, or in didactic or clinical courses, is complex and has many essential components. Lenburg (1978, 1979, 1983, 1984, 1990, 1999), Lenburg & Mitchell (1991), and Luttrell, et al (1999) have described these components in detail; others have applied or adapted them. Some of the basic concepts are summarized below, but the scope of this article precludes elaboration.

Some Essential Basics

Briefly, the evaluation of learners can be divided into either normative-referenced or criterion-referenced to indicate whether the learner is being compared to a group of others or to a set of standards or expectations. Evaluation also is divided into formative or summative types to indicate whether the focus is on the continuing learning period or is at the conclusion of learning time. Competency performance assessment or examinations, as described in the COPA Model, are criterion-referenced and summative methods, although formative examples can be used as well, similar to interim quizzes. The performance of learners (students, practitioners) is judged against a predetermined standard at the conclusion of a designated period of learning and practice. Moreover, performance examinations can be based on various types of simulations or actual situations likely to be encountered in practice. On a continuum from total simulation to total actual performance, various types of performance examinations can be constructed to correspond with the competencies to be assessed; other factors to consider include purpose, human and material resources, environment, persons being evaluated, and potential consequences of the outcomes. A combination of various types of simulation and/or actual performance examinations, along with written examinations usually allows for comprehensive assessment of performance competencies, clinical or otherwise.

Another basic principle is the need to separate learning time from assessment time, which inherently means different roles and functions for instructors and learners during learning and testing times. The learner uses the designated learning time to prepare for the specified time when verification of competence is required based on the established standard. This approach differs from the traditional practice of evaluating the learner during every practice session, resulting in a series of anecdotal notes that subsequently are summarized to determine the extent or quality of learning. In a competency-based system, the learner has a guided and un-judged time for learning followed by a scheduled time to demonstrate and confirm competence. Evaluation focused on helping the learner to become more competent during the learning time still is essential. This is the coaching, teaching, and consultation of expert to learner that promotes competence; it is not appropriate for determining the final grade or competence appraisal. Learners (of whatever type or level) need to know what time it is, and be held accountable accordingly. It is learning time or documentation time?

Essential Psychometric Concepts

Whether the competency performance assessments are actual or simulated, or used in didactic or clinical settings, they are most effective when designed and implemented based on a series of essential psychometric concepts. Faculty have used such concepts for decades to construct norm-referenced paper and pencil tests, but they rarely have used them in clinical evaluation. Lenburg created a constellation of ten basic concepts and adapted them for developing and implementing objective performance examinations (Lenburg, 1979, 1983, 1990, 1999; Lenburg & Mitchell, 1991). Space precludes elaboration here, but a few comments will highlight the usefulness of these concepts to develop more accurate assessment instruments.

Sampling is essential, as it is in paper/pencil testing, and follows similar patterns of selecting the most frequently encountered, most essential, and most critical skills for the testing period. Determination of these competencies influences the type of examination, the timing, process and other logistics. The level of acceptability is established by the number and type of skills, and 100% performance of the specified critical elements, no more and no less. Protocols are established to insure that each test episode for a given group, is comparable in extent, difficulty, and requirements. To be fair, each person must have equivalent situations; not the same but equivalent. Additional protocols are designed to insure that the process is implemented consistently, regardless of who administers the examination, or when it is conducted. This requires consensus of all those involved, which is based on some degree of involvement in the ongoing process and very specific orientations, monitoring and maintenance programs. When performance examinations are administered in actual clinical environments (not simulations) the concept of flexibility is essential, as each client (patient) is different. Moreover, protocols for using systematized conditions to determine what to do in case of unanticipated events promotes objectivity, comparability, and consistency, and prevents the erosion of the process when assessors simply respond in whatever way they choose at the time.

Two Types of Performance Evaluations

These concepts apply to development and implementation of various types of objective evaluations. Lenburg (1998, 1999; Luttrell et al, 1999) developed two variations of competency assessments to clarify levels and types of expected abilities, entitled Competency Performance Assessments (CPAs) and Competency Performance Examinations (CPEs). CPAs are designed for use in non-clinical, didactic, classroom-type situations and for related types of assignments, such as projects, poster presentations, analyses of reports, research articles, or writing manuals or reports. CPEs are used in clinical, client-related environments and corresponding critical elements are more exacting as they take into account the legal, ethical and professional components of responsible care of actual persons. Thus, the whole range of practice competencies can be objectively assessed using a similar set of psychometric concepts, protocols and policies, regardless of where they are learned, or the type of skills involved.

After any particular CPE or CPA is developed, it must be pilot tested to work out problems, to discover more effective and/or efficient methods, logistics, wording, instructions, or the like (Lenburg, 1979, 1998, 1999; Luttrell, et al, 1999). Logistics and policies also must be developed consistent with the type of assessments, conditions, resources, purposes and other factors. This essential work often is done while final versions of the assessment are completed and the pilot testing is being implemented. In educational settings the most troublesome and critical policies pertain to grading. Essential questions relate to determining how many exams to use, what grade to assign to the "bottom line" and how to use contract grading for students who want to excel. These and other questions influence progression or termination from the course or employment position.

Once the initial competency performance assessments and/or examinations are developed it is essential to establish ongoing responsibilities for orientations, revisions, updating and quality improvements; this usually is a standing committee related to curriculum or evaluation, or staff education or appraisals, or an institutional committee assigned to such activities. An ongoing program of evaluation of the COPA process, policies, consequences, and results needs to be implemented on a regular and systematic basis and results used for quality improvements.