Expiration Date: December 31, 2002. No CE credit will be given after this date.
Table of Contents
Abstractby Joanne R. Duffy DNSc., RN, CCRN and
Nurse-sensitive outcomes indicators have been established by the American Nurses Association to help demonstrate nursing's unique contributions to healthcare. A beginning knowledge base in research methods, while not the typical domain of nurses, is necessary to implement this program. Levels of measurement, validity, and reliability are reviewed. Challenges inherent in data collection are described and strategies to overcome them are suggested. The nurses' role in maintaining data integrity as well as the focus of evidence based nursing practice are highlighted.
|
|||||||||||||||||||||||
|
Nursing Quality Indicators |
Operational Definitions |
| Nosocomial Infection Rate | The rate per 1000 patient acute care days at which patients develop clinically active bacteremia (as defined by CDC) in whom there is no evidence to suggest that infection was present or incubating at admission (using CDC differential criteria)*under development. |
| Patient Fall Rate | The rate at which patients fall during the course of their hospital stay per 1000 patient days. |
| Patient Satisfaction with Nursing Care | Patient opinion of care received from nursing staff during the hospital stay as determined by scaled responses to a uniform series of questions designed to elicit patient views regarding key elements of nursing care services. |
| Patient Satisfaction with Pain Management | Patient opinion of how well nursing staff managed their pain as determined by scaled responses to a uniform series of questions designed to elicit patient views regarding specific aspects of pain management. |
| Patient Satisfaction with Educational Information | Patient opinion of nursing staff efforts to educate them regarding their condition and care requirements as determined by scaled responses to a uniform series of questions designed to elicit patient views regarding specific aspects of patient education activities. |
| Patient Satisfaction with Care | Patient opinion of the care received during the hospital stay as determined by scaled responses to a uniform series of questions designed to elicit patient views regarding global aspects of care. |
| Nursing Job Satisfaction | Job satisfaction expressed by nurses working in hospital settings as determined by scaled responses to a uniform series of questions designed to elicit nursing staff attitudes toward specific aspects of their employment situation. |
| Maintenance of Skin Integrity | Rate per 1000 patient days at which patients
develop pressure ulcers (Grade I or greater)
during the course of their hospital stay, but,
72 hours or more following their admission.
Mix of RNs, LPNs, Unlicensed Staff Caring for Patients in Acute Care Settings: The ratios (expressed in FTEs) of registered nurses with direct patient care responsibilities to LPNs and unlicensed workers. |
| Total Nursing Care Hours Provided per Patient Day | Total number of hours worked by nursing staff with direct patient care responsibilities on acute care units per patient day. |
A national database has been formed through the ANA whereby acute care organizations can voluntarily report their specific results and compare these to a national benchmark for each indicator. Many acute care organizations throughout the United States are now participating in this database. Such participation, however, presents many challenges for the professional nurse. The purpose of this article is to provide the nurse with beginning knowledge in outcomes measurement to help support the ongoing evaluation of patient safety and quality.
Outcomes Measurement – What Is It?
Measurement has been defined as the assignment of numbers to phenomena according to specified rules (7). Quantification of phenomena helps to determine variability among different subjects. Four different levels of measurement are commonly used: nominal, ordinal, interval, and ratio. Nominal refers to phenomena that are available only in distinct categories, such as gender. Ordinal data are those that are categorical, but, the categories can be ranked, such as clothes size. Interval level data are those represented by real numbers that have equal distances among the possible values, such as age. Finally, ratio level data are similar to interval level data, but, zero represents a real value, such as weight. The importance of understanding the levels of measurement is that certain statistical tests can only be run with specific levels of measurement. For example, we can determine a mean value for age since it is interval level, but, we cannot determine a mean gender. In healthcare, interval or ratio level data is preferred, but, many times it is unavailable.
Outcomes measurement refers to collecting and analyzing data using predetermined outcomes indicators for the purposes of making decisions about healthcare. Each specific indicator must be operationally defined very precisely so that all professionals can be consistent during data collection and analysis. The National Library of Healthcare Indicators (8) provides a comprehensive list of quality indicators with their associated operational definitions that are available for clinicians.
Definitions for indicators, such as patient satisfaction or skin integrity, are frequently operationalized through the use of scales or instruments. When selecting instruments to measure outcomes, it is important to consider a whole range of factors that include validity, reliability and feasibility for use in clinical settings (Table 2). Of utmost importance is the assurance of instrument validity and reliability. These qualities are referred to as psychometric properties and assure that the instruments provide data that can be objectively quantified. Whenever possible, it is preferable to select existing instruments that have established validity and reliability rather than spending the time to develop new ones. Using existing instruments also enables us to project the amount of time it will take to administer, score, and complete the actual data collection.
Validity refers to whether the instrument is measuring what it is supposed to measure and is generally determined by a panel of experts. One form of validity is called content validity where a panel of experts will reach consensus that the items on the instrument are measuring what they are supposed to measure.
Reliability refers to the consistency or dependability of the instrument. There are several forms of reliability that are appropriate for outcomes measurement. Inter-rater reliability is used to assure an acceptable level of consistency when multiple individuals are collecting data. In this form of reliability, all the data collectors are given the same instrument to score and then a percent agreement is calculated. The higher the agreement, the more reliability exists.
Test-retest reliability refers to the ability of the instrument to provide consistent results when administered multiple times. It is used when there are two or more instances where the same instrument will be used. For example, if we wanted to know whether a patient education program achieved its objectives, we might give patients a pretest, then provide the education, and follow it with the same test as a post test to determine how much they improved. For each set of test scores, a correlation coefficient is performed to test the relationship between the scores.
Internal consistency reliability refers to whether the individual items on an instrument all contribute positively to the concept being measured. For example, the items on a patient satisfaction instrument should all be measuring patient satisfaction. A reliability coefficient, typically called coefficient alpha is generated. The closer this number is to 1, the greater the reliability.
Sensitivity of an instrument refers to the amount of variation that can be detected among subjects. It is important for all measures, but, is especially critical when changes are anticipated or important decisions will be made based on the data. Several item analysis techniques presently exist to help researchers determine sensitivity.
The characteristics of efficiency/burden, simplicity, and interpretability are many times collectively grouped. They refer to such qualities as ease of reading, the length of the instrument, and ease of scoring. Instruments already in use will have guidelines that speak to these issues; however, those that are newly developed will require pilot study to determine these characteristics. Without consideration of ease of use (9), administering and scoring data forms can be extremely time consuming for the data collector and burdensome for the subject.
Clinical feasibility is applicable only in health care settings and refers to whether the actual collection of data can occur given the clinical condition of the subject. For example, certain patients are too acutely ill to participate in a paper and pencil questionnaire. Occasionally, the use of technology prevents appropriate participation in studies. Consideration of the clinical situation prior to implementation of data collection will prevent unnecessary pressures during the actual study period.
Table 2. Example Criteria for Selecting Instruments (7)
|
Challenges Inherent in Outcomes Measurement
Collection of outcomes data requires time. With the present workload of nursing staff, it seems obvious that collecting data does not rank highly on the priority list of nursing activities. Yet, without evidence that nursing is indeed impacting the quality of care, decisions will be made that may be detrimental to quality patient care. In addition, linking outcomes data to nursing structural variables such as staffing levels could have an immediate impact on health and institutional policy. Huston (1) advocates that outcomes measurement begin at the unit level and actively involve all nurses. This may require adjustment of budgets to include both direct and indirect costs associated with outcomes tracking (10). The challenge for both nursing administrators and staff nurses is to change their focus from tasks to evidence-based practice.
Data Collection.
The specific procedures for collecting data must be planned in advance, written, and understood by all involved to ensure consistency during the process. Copies of a written data collection plan should be available for reference in all areas where data collection is occurring. To complete a written plan, decisions that address the who, what, when, where, and how of data collection must be resolved prior to beginning the process.
Who refers to the data collectors. Will they be staff nurses, quality improvement specialists, clinical specialists, other clinicians, or non clinicians? Who will they report to for questions and who will make decisions about the process? Data collectors need to have a knowledge base about the topic, process, and how the varied instruments are used. They also need to be flexible yet precise. In clinical settings, last minute changes frequently occur. Consistency and integrity in the process of data collection, however, is critical. Training will have to be available for all data collectors so that questions can be answered and interrater reliability can be assured..
What refers to the specific research question/s, unit of analysis, data definitions, the forms and protocols required, and the sample. The specific question/s to be answered provides the foundation for the data collection effort and guides the entire process. The unit of analysis is the basic unit of an investigation (7). In healthcare, it often is the patient, although sometimes it can be the department or the provider. Operational definitions that are readily available will assist the data collectors when questions occur about specific variables. One way to provide consistency is to include a data dictionary for all staff involved in the data collection process. A data dictionary provides definitions for the study variables and examples for the data collectors to review. All required forms and specific procedures pertinent to the data collection process should be included with the data dictionary. This assures a complete set of reference materials that will guide the collectors through the process.
The sample from which data are collected must be considered prior to the implementation of data collection. When investigating specific questions, researchers consider the type of sample, sample criteria, and the desired sample size. In clinical situations, we rarely have the opportunity to prospectively consider these factors. With a little preplanning, however, clinicians can use the sampling principles from research to derive a powerful sample from an established clinical service (11). The goal is to determine a subset of patients that will allow us to learn about the care they received without needless data collection on all patients. Consultation with a statistician can be helpful in determining sample size and methods.
When is the frequency of data collection. Prior to implementation, the responsible individual must decide how frequently the data will be collected. There is no right way or magic associated with this decision. It depends on the measure being collected and its importance to the overall performance improvement plan. Many process indicators are collected concurrently, but, are analyzed quarterly. An example of this includes a "skin integrity" risk assessment. Some indicators, such as patient falls, are dependent on additional data collection forms (the incident report). Others still are only collected at discharge or even post discharge (patient satisfaction). Nonetheless, decisions about frequency of data collection will need to be made for each indicator.
Where refers to the actual place or setting of data collection. Will it be on the patient unit, in the patient records department, or in the patient's home? If patients are required to directly answer questionnaires, those who are acutely ill will require more time compared to those who are already discharged. Abstracting data from patient records, while not involving interaction with patients, may present a variety of problems. Often, the data may be missing from the patient record or not legible. Data may be recorded in the chart, but, not be consistent with the operational definition. This requires the data collector to clinically interpret and make decisions about indicators.
How
How refers to the actual administration of an instrument. Several methods are used to collect data such as questionnaires, interviews, or observation. Knowledge of the instrument and its intended administration process is essential since the data must be accurately recorded. Participant problems such as patient refusal or patient absence and external influences such as family members can influence the administration of an instrument (12). The data collectors themselves may find difficulties when patient needs appear more crucial than the data collection process or they are asked to stop data collecting and provide patient care. Finally, data entry, lost forms, or not enough forms can complicate the implementation of data collection.
Using a step by step thoughtful approach prior to implementing data collection procedures will prevent errors, minimize burden, and lead to successful program implementation. As answers to these five questions become evident, a plan for data collection should be written and serve as the basis for training.
Data versus Information
Within the healthcare industry, much activity surrounds data collection; yet, most of it fails to yield good information. Nurses collect data every day and sometimes hourly. Examples of data include vital signs, weight, and relevant assessment parameters. Information or knowledge, however, provides answers to questions that guide clinicians to change their practices. For example, the trending of vital signs over time provide a pattern that may lead to certain clinical decisions. Many times healthcare professionals get caught up in the data gathering process rather than focusing on generating information that can be used to improve care (13). Keeping the goal/s of outcomes measurement as well as the quality of the data in mind helps to alleviate this common trap.
Data Integrity
The concept of "dirty" data is important to consider when implementing outcomes measurement. Data elements that are missing or incomplete, transcribed incorrectly, or inaccurately collected are considered dirty and lead to credibility problems. In essence, data are worthless unless clinicians believe they are accurate. Outcomes measurement is so important to healthcare decision making that ensuring data integrity must remain a top priority and everyone's responsibility.
Methods for ensuring data integrity include:
- creating a flow chart to depict the flow of data from pre to post analysis (Figure 1)
- reviewing data collection forms prior to data entry
- assuming that data entry is error prone
- providing a written data dictionary with specific operational definitions
- implementing a thorough orientation program for data collectors
- following a predetermined data audit and data cleaning process (inspect for outliers, missing
- values; perform accuracy checks including ranges and transposition of digits)
- completing descriptive analysis and checking for frequency distributions
- assessing the data for systematic bias; evaluating the need for recoding
- comparing results to historical data and clinical experience
When reporting the data it is important to state the exact mechanisms for ensuring data integrity so that readers are assured of the objectivity, validity and reliability of the data. Clinicians who can understand and trust the data are more prone to use the data in decision making!
Data Entry
According to Jennings and Staggers (12), "fundamental to successful performance measurement and other data-driven initiatives is information system support" (p. 25). While the healthcare industry is behind others in information system support, the availability of such systems is changing. Many organizations are integrating clinical databases and designing electronic patient records. Healthcare continues to lag behind, however, which places a strain on the system when trying to measure outcomes. Healthcare professionals often resort to manual data collection and then request non clinical personnel to enter the values. While this approach is efficient, it is danger-prone in terms of data integrity. It also eliminates the need for clinicians to learn more about information systems. As a consequence, many healthcare professionals are not up to date with basic computer applications. Continuing education and systems designed for direct data entry will help alleviate this frustration.
Data Analysis
The approach to data analysis should be determined prior to data collection and includes consideration of the specific questions being answered as well as the levels of measurement used in the data collection process. Statistical consultation is necessary to accurately perform the analysis and interpret the results. Typically, the sample characteristics are analyzed using descriptive statistics including frequency distributions, means, and standard deviations. Analyzing demographic variables, such as age, gender, and educational level descriptively provides valuable insights into the sample's attributes. Further descriptive statistics are used to analyze the specific study variables. Finally, inferential statistics are employed to investigate for relationships or comparisons. Statistical procedures such analysis of variance and regression analysis allow researchers to test hypotheses. Findings from the data analysis should be interpreted in light of current clinical practice and prior research.
Data Reporting
Results of data collection and analysis should be disseminated to all those involved in clinical practice. Regular feedback about performance has been shown to facilitate outcomes measurement (14). Outcomes reports should be distributed quarterly and in a simple format so that all those involved can comprehend the information. The use of colorful graphs and charts will add clarity to textual information. Internal thresholds and national benchmarks add clinical relevance to the results. Any limitations of the results should be clearly described. Setting aside time at staff meetings for discussion of outcomes data stresses its importance and offers staff the opportunity to ask questions for a better overall understanding of the data. Educating clinicians about the interpretation of results and allowing them to provide feedback about the report design assists in a team approach to outcomes measurement. Clinical specialists enjoy a distinct role in this aspect of improving quality.Outcomes Measurement as Evidence of Quality Nursing Care
The contribution that nursing makes to healthcare is well known, but, has not been clearly demonstrated. Little objective evidence presently exists establishing linkages between nursing and health outcomes. Most of present day nursing care is still based on intuition or trial and error practices. Although it presents challenges, and sometimes confusion and chaos, the profession of nursing must join its medical colleagues in the routine investigation of its practices for the purpose of generating EVIDENCE (Figure 2).
This EVIDENCE can then be used to improve care, justify staffing levels and other structural variables, and revise institutional, state, and national policy.
The ANA Safety and Quality Indicator Project has provided the impetus for these linkages through the assessment of structural, process, and outcomes indicators. This national database enjoys a growing pool of participating acute care institutions and is beginning to show evidence that nursing does indeed impact health outcomes (15).
The Nursing Care Report Card for Acute Care (4) as this project is known provides acute care institutions with the framework for collecting, analyzing, and comparing specific nurse-sensitive indicators. The ANA views these data as "a tool for protecting our patients" (p. E-3), part of our professional accountability, and an agenda for future research. Overcoming the challenges of measurement, data collection, and analysis will provide the missing EVIDENCE that demonstrates nursing as a strong link to patient safety and healthcare quality.
References
1. Huston, CJ. 1999. "Outcomes Measurement In Healthcare: New Imperatives For Professional Nursing Practice". Nursing Case Management, 4(4): 188-195.
2. Ree, L, Blegen, M, Goode, C. 1998. "Adverse Patient Occurrences As A Measure Of Nursing Care Quality". Journal of Nursing Administration, 28(5): 62-69.
3. Irvine,D, Sidani, S, Hall, L. 1998. Finding value in nursing care: A Framework for Quality Improvement and Clinical Evaluation. Nursing Economics, 16(3): 110-116.
4. American Nurses Association 1995. Nursing Care Report Card For Acute Care. Washington, DC: American Nurses Publishing.
5. Moore, C, Lynn, M, McMillan, B, Evan, S. 1999. Implementation of the Ana Report Card. Journal of Nursing Administration, 29(6):48-54.
6. American Nurses Association. 1999. Nursing Fact Sheet on Quality. Washington, DC: American Nurses Publishing.
7. Polit, D, Hungler, B. 1995. Nursing Research: Principles and Methods. Philadelphia: JB Lippincott company.
8. Joint Commission on Accreditation of Healthcare Organizations 1995. National Library of Healthcare Indicators: author.
9. Huber, D. 1998. Facilitating Instrument Evaluation. Nursing Economics, 16,1: 27-32.
10. Byers, V 1995. Overview of the Data Collection Process. Journal of Neuroscience Nursing, 27(3): 188-193.
11. Plsek, P. 1994. Tutorial: Planning for Data Collection Part III-Sample Size. Quality Management in Health Care, 3(1): 78-92.
12. Jennings, B, Staggers, N 1999. A Provocative Look at Performance Measurement. Nursing Administration Quarterly, 24(1): 17-30.
13. Plsekk, P. 1994. Tutorial: Planning for Data Collection Part I: Asking the Right Question. Quality Management in Health Care, 2(2): 76-81.
14. Duffy, J. 2000. Cardiovascular Outcomes Initiative: Case studies in Performance Improvement. Outcomes Management for Nursing Practice, 4(3).
15. American Nurses Association 1997. Implementing Nursing's Report Card. Washington, DC: American Nurses Publishing
© 2000 American Nurses Association