Outcomes Measurement – What is It?

Measurement has been defined as the assignment of numbers to phenomena according to specified rules (7). Quantification of phenomena helps to determine variability among different subjects. Four different levels of measurement are commonly used: nominal, ordinal, interval, and ratio. Nominal refers to phenomena that are available only in distinct categories, such as gender. Ordinal data are those that are categorical, but, the categories can be ranked, such as clothes size. Interval level data are those represented by real numbers that have equal distances among the possible values, such as age. Finally, ratio level data are similar to interval level data, but, zero represents a real value, such as weight. The importance of understanding the levels of measurement is that certain statistical tests can only be run with specific levels of measurement. For example, we can determine a mean value for age since it is interval level, but, we cannot determine a mean gender. In healthcare, interval or ratio level data is preferred, but, many times it is unavailable.

Outcomes measurement refers to collecting and analyzing data using predetermined outcomes indicators for the purposes of making decisions about healthcare. Each specific indicator must be operationally defined very precisely so that all professionals can be consistent during data collection and analysis. The National Library of Healthcare Indicators (8) provides a comprehensive list of quality indicators with their associated operational definitions that are available for clinicians.

Definitions for indicators, such as patient satisfaction or skin integrity, are frequently operationalized through the use of scales or instruments. When selecting instruments to measure outcomes, it is important to consider a whole range of factors that include validity, reliability and feasibility for use in clinical settings (Table 2). Of utmost importance is the assurance of instrument validity and reliability. These qualities are referred to as psychometric properties and assure that the instruments provide data that can be objectively quantified. Whenever possible, it is preferable to select existing instruments that have established validity and reliability rather than spending the time to develop new ones. Using existing instruments also enables us to project the amount of time it will take to administer, score, and complete the actual data collection.

Validity refers to whether the instrument is measuring what it is supposed to measure and is generally determined by a panel of experts. One form of validity is called content validity where a panel of experts will reach consensus that the items on the instrument are measuring what they are supposed to measure.

Reliability refers to the consistency or dependability of the instrument. There are several forms of reliability that are appropriate for outcomes measurement. Inter-rater reliability is used to assure an acceptable level of consistency when multiple individuals are collecting data. In this form of reliability, all the data collectors are given the same instrument to score and then a percent agreement is calculated. The higher the agreement, the more reliability exists.

Test-retest reliability refers to the ability of the instrument to provide consistent results when administered multiple times. It is used when there are two or more instances where the same instrument will be used. For example, if we wanted to know whether a patient education program achieved its objectives, we might give patients a pretest, then provide the education, and follow it with the same test as a post test to determine how much they improved. For each set of test scores, a correlation coefficient is performed to test the relationship between the scores.

Internal consistency reliability refers to whether the individual items on an instrument all contribute positively to the concept being measured. For example, the items on a patient satisfaction instrument should all be measuring patient satisfaction. A reliability coefficient, typically called coefficient alpha is generated. The closer this number is to 1, the greater the reliability.

Sensitivity of an instrument refers to the amount of variation that can be detected among subjects. It is important for all measures, but, is especially critical when changes are anticipated or important decisions will be made based on the data. Several item analysis techniques presently exist to help researchers determine sensitivity.

The characteristics of efficiency/burden, simplicity, and interpretability are many times collectively grouped. They refer to such qualities as ease of reading, the length of the instrument, and ease of scoring. Instruments already in use will have guidelines that speak to these issues; however, those that are newly developed will require pilot study to determine these characteristics. Without consideration of ease of use (9), administering and scoring data forms can be extremely time consuming for the data collector and burdensome for the subject.

Clinical feasibility is applicable only in health care settings and refers to whether the actual collection of data can occur given the clinical condition of the subject. For example, certain patients are too acutely ill to participate in a paper and pencil questionnaire. Occasionally, the use of technology prevents appropriate participation in studies. Consideration of the clinical situation prior to implementation of data collection will prevent unnecessary pressures during the actual study period.