The development and use of quality indicators (QIs) requires understanding certain methodological challenges. These include issues of dealing with different assessment types, measuring and adjusting for differences in risk, identifying and applying performance standards or thresholds, and ensuring a balance between sensitivity and specificity, or what we have called "target efficiency". The following sections briefly describe the primary methodological challenges and our current approaches to each.

Assessment Type

MDS resident assessment data are collected at several points in time for all residents: at initial admission to the facility; quarterly, after admission; annually (fourth quarter) after admission; upon significant change in health or functional status; and at readmission following a temporary discharge to a hospital, other treatment facility, or home; a small number of assessments also are performed for other reasons. It should be noted that the distinction between quarterly and annual assessments is trivial, as each annual assessment obviously is also a quarterly assessment. However, in some states, there is a difference between the amount of required information that is collected on the quarterly and annual assessments.

Information collected at these different times has different relationships to the care provided in the facility. Information collected at initial admission provides baseline information, but does not represent outcomes or processes of care provided in the facility. Information collected at readmission can represent outcomes of care provided while outside the facility, or outcomes of care within that facility that may have resulted in the need for transfer to another facility. For these assessments, the relationship between a QI and the quality of care provided by the facility is unclear. Information collected both at admission and at readmission also can provide insight into a facility's admission practices. Assessments conducted quarterly (annually) or upon significant change in resident status can be more safely assumed to reflect the quality of care provided by facility staff. Given these differences among the reasons for assessments, it is not surprising that the occurrence of a QI varies in frequency by assessment type.

In measuring the prevalence of a QI for a facility, then, we have had to consider whether or not to include data from all types of resident assessments. The issue of which assessments to include in inter-facility comparisons is a controversial one. We have elected to take the conservative approach of including in the comparisons only those assessment types that unambiguously reflect care provided in the facility (see Table 1, Assessments Used for QI Reports).

Table 1
Assessments Used for QI Reports

		QI Reports
MDS 2.0 Section A8a Reasons for Assessment		Facility Characteristics	Facility QI Profile	Resident Level Summary
1.	Admission Assessment	X	Excluded	X
2.	Annual Assessment	X	X	X
3.	Significant Change in status assessment	X	X	X
4.	Significant correction of prior assessment	X	X	X
5.	Quarterly review assessment	X	X	X
6.	Discharged - Return not anticipated	Excluded	Excluded	Excluded
7.	Discharged - Return anticipated	Excluded	Excluded	Excluded
8.	Discharged prior to completing initial assessment	Excluded	Excluded	Excluded
9.	Reentry	Excluded	Excluded	Excluded
10.	Significant correction of prior quarterly assessment	X	X	X
0.	NONE OF THE ABOVE	Excluded	Excluded	Excluded

This means that, for purposes of inter-facility comparison, we have chosen to exclude admission and readmission assessments from the calculation of facility QI scores. By excluding readmission assessment data from our calculations of the QIs, we increase the likelihood that the potential problem captured by the QI is rooted in care provided within the facility. The tradeoff is that we may miss cases of poor care that result in conditions necessitating treatment in a hospital (i.e., false negative). It is our belief that, where poor care does result in need for hospitalization, we also will observe its consequences among residents who also experience the condition, but do not require hospitalization for treatment.

Identification of Risk Factors

"Risk factors" are health or functional conditions that either increase or decrease the resident's probability of having a specific quality indicator. In developing our approach to risk adjustment, we have had to distinguish between two very important uses for risk information: (1) the identification of clinical risk factors, to facilitate the provision of appropriate and high quality care, and (2) the identification of differences in facility populations that could result in different rates of QI occurrence, where there is no difference in the quality of care provided. With regard to the use of QIs, we are interested in the second purpose.

In developing a method for inter-facility comparisons of quality, and the identification of facilities with potential quality of care problems, we have attempted to avoid using risk factors that are directly related to the quality of care. A system of risk adjustment for purposes of measurement of facility quality must exclude, as much as possible, the use of risk factors that the facility reasonably can be expected to identify and treat, to avoid the outcome of the QI. Risk factors used for facility comparison must instead focus on issues that differentiate the populations, but where the ability of the facility staff to intervene is believed to be minimal. This concept can be expressed in the following way:

QI = quality of care + risk + error

Given this purpose for the QIs, we have revised the original, RAPs-driven set of risk factors, to focus on those issues that we believe are not easily amenable to clinical interventions.

Risk Adjustment Procedures

Implicit in the preceding discussion is the idea that risk adjustment factors can be used to "level the playing field" when comparing quality across facilities. The purpose for which the QIs are intended must be a major consideration in choosing an approach. For purposes of quality monitoring and quality improvement, we have chosen a stratification approach. For each QI that has a risk adjustment factor, we have created what are essentially three separate measures. The first is the occurrence of the QI without regard to risk. The second and third measures are the QIs measured separately for those people who have the risk factors and for those who do not. For the sake of simplicity, we have referred to these groups as "high risk" and "low risk," respectively. By creating separate measures for the populations defined by risk, regulatory surveyors and quality assurance teams can (1) determine the relative sizes of the high and low risk populations for a facility; (2) identify whether the facility has a potential quality of care problem for either or both risk groups; (3) identify whether the facility has a potential quality of care problem for the resident population as a whole.

This approach also allows us to set separate thresholds for the high and low risk groups. This is important when we believe that the occurrence of a problem is more or less acceptable in these different groups. For instance, we may be willing to accept some (low) level of occurrence of pressure ulcers among the high risk group. On the other hand, our tolerance for pressure ulcers among the low risk group may be much less. We believe that the occurrence of pressure ulcers among this group is much more likely to be an indication of a problem with the quality of care, and so have treated it as a sentinel event which requires follow-up investigation for even one resident case.

Quality Performance Thresholds or Standards

Performance thresholds or standards are used to identify facilities with potential quality of care problems, by setting a level above which a facility's performance is considered suspect. Thresholds can be either absolute or relative. Absolute thresholds define a single number, above which facility QI scores are considered suspect. Absolute thresholds can be developed based on review of the literature or on a consensus of the experts. These standards may be as low as zero, so that any occurrence of a QI signals a potential problem for the facility. Sometimes these cases are called "sentinel events" in quality assurance parlance. Relative thresholds are peer-group based. A peer group is the comparison group of facilities, and may be defined on the basis of geography or facility characteristic (e.g., ownership type, certification). Relative thresholds are set relative to the distribution across the peer group facilities, e.g., the 75th percentile, the 90th percentile, the mean plus two standard deviations. The selection of peer group can have a dramatic impact on the setting of a threshold, and the consequent likelihood that a facility will be identified as having a potential quality problem related to any given QI.

The choice of a threshold affects the number of QIs for which a facility exceeds the threshold, the resources required to investigate potential quality problems, and the comparative standing of different facilities. Regardless of how the threshold is determined, it has implications for the cost and resources required of the regulatory survey process. The lower the threshold, the greater the number of facilities that will be identified for review. The implications are similar when QIs are used for internal facility quality improvement.

Based on our original QI development and validation, we have used a state (peer group) specific threshold of the 90th percentile, for most QIs. A few QIs are treated as sentinel events, so that any occurrence is cause for investigation. These include the prevalence of fecal impaction, the prevalence of dehydration, and the prevalence of pressure ulcers among residents at low risk of pressure ulcers. The establishment of thresholds is the subject of continuing analysis. For example CMS currently uses the 75th percentile as a threshold in its long term care survey process.

Target Efficiency

A fourth methodological concern addresses what we have called the "target efficiency" of the QI. This issue involves the specificity and sensitivity of the QI, in particular the likelihood of a false positive, i.e., that the QI will identify a resident or a facility for whom the QI flag is not ultimately found to represent a problem with the quality of care. Minimizing the number of false positives and false negatives is a critical concern, since each one decreases both the effectiveness and the efficiency of the quality monitoring process. False positives also may promote an erroneous perception of a quality of care problem for a facility, where no such problem exists. Using too strict a QI definition, however, may result in the opposite problem, failing to identify quality problems that in fact exist.

The target efficiency of the QIs varies with the extent to which the QIs (1) are prevalence versus incidence measures; (2) include both process and outcome measures; and (3) can be risk adjusted. QIs can be made more target efficient by combining consideration of risk, process and outcomes into a single indicator. An example of such a QI is the prevalence of pressure ulcers (outcome) among people who are at high risk (risk factor) of developing a pressure ulcer at a prior point in time and did not receive any special skin care program (process).

As indicated by this example, it is possible to define QIs that would have strong "target efficiency." Based on our development and validation work however, we have chosen to use fairly simple measures, rather than those that we believe have the greatest target efficiency. This decision is based on several considerations. First, the more target efficient QIs are often difficult to interpret, due to their complex definitions. Second, use of more target efficient QIs may result in an exclusion of cases that are a result of poor quality of care, but that do not meet all of the conditions set forth in the complex QI definitions, thereby resulting in an increase in false negatives. Third, the use of complex definitions to increase target efficiency also may result in increased error. Specifically, any error that results from the first component of a complex definition can be multiplied as the remainder of the definition compounds the error. Finally, the use of the QIs in the monitoring process can take advantage of the regulatory survey or internal facility review as a source of immediate verification, detecting false positives. The important general point with respect to target efficiency is that the more likely the case that the indicator itself is to be used to render decisions on quality of care without follow-up or verification, the more important is the target efficiency of that indicator.