Introduction

Peer Review Organizations (PROs) were established as part of the Tax Equity and Fiscal Responsibility Act of 1982.1 The Department of Health and Human Services, through the Health Care Financing Administration (HCFA), contracts with state PROs to do utilization and quality control reviews for Medicare and other federally sponsored medical programs. They review hospitals, nursing homes, outpatient facilities, home health agencies, and the practices of physicians. In doing so, they make judgments on whether the delivery of care meets "professionally recognized standards" and is timely and appropriate.

For physicians, the most important review is of the quality of care delivered to patients. A nurse, employed by the PRO, randomly selects a group of records in a hospital to determine whether there have been failures to meet "screens," pre-determined and pre-set by an agency of the Department of Health and Human Services. If a record fails to meet performance criteria, it is referred to physician reviewers, also employed by the PRO. Upon failure to meet quality care guidelines, sanctions may be invoked according to the estimated severity of the infraction.

Three levels of severity have been defined, and weights are assigned to each. Level I is a confirmed quality problem without the potential for significant adverse patient effects, level II is a confirmed quality problem with the potential for significant adverse effects, and level III is a confirmed quality problem with a significant adverse effect and a weight of 25.2 For example, a level III problem may be one that required the readmission of the patient to the hospital within 31 days, or resulted in the patient's death. PROs profile all physicians quarterly. Interventions of several types are initiated, depending on the total weights assessed during the quarter or in two consecutive quarters. Any level III infraction is reported to the state medical board for further review and action.

PRO sanctions place physicians in jeopardy emotionally and professionally. Their clinical judgment is questioned. They find themselves in the awkward position of being criticized if they give treatment irrespective of cost and censured or liable if they do not. A clear understanding of this dilemma requires a sound application of clinical judgment,3 which cannot be achieved without the physician having freedom to think and act in the best interest of patients, unencumbered by a maze of bureaucratic pronouncements.

The charge given to PROs presumes that medical criteria have been developed and defined with adequate specificity to expect uniform physician decisions. This is clearly not true. Therefore the ability of the PROs to impose sanctions on the basis of inadequately defined guidelines is a major factor in causing physician discomfort with the entire PRO review process. These imperfections in the guidelines are well described in a recent publication by Kellie and Kelly.4

No doubt, significant numbers of physician delinquencies occur. However, as we will discuss later, an HFCA report shows that a significant number of PRO errors also occur. According to PRO methodology, the physician will be notified that an infraction appears to have occurred. The physician must then review the complaint and the records and submit an explanation that the PRO may or may not accept.

This seems fair. Yet, physicians relate that, even when the explanation fully accounts for the problem, the PRO has frequently agreed but nevertheless assigned a weight to the "infraction." In this environment, physicians may well pay more attention to a potential PRO inquiry than to logical medical reasoning.

Case

A 69 year-old woman with a history of hypertension and anxiety came to the emergency room with a complaint of chest pain. She had a number of such episodes in the past 15 years. The present episode was described as a "grabbing sensation" mid-sternal in location, followed by episodes of chest "heaviness" that radiated to both arms with a cold, clammy, and sweaty sensation. She took four nitroglycerin tablets about 10 minutes apart with relief from the fourth tablet. The entire episode lasted about 40 minutes. She came to the emergency room for further evaluation, even though she was then free of pain.

Of note is the fact that a stress thallium test performed nine months earlier was found to be normal. She did not smoke or drink. Following evaluation in the emergency room with an electrocardiogram (ECG), which showed no Q waves, no ST segment changes and normal T waves, and a measurement of creatine phosphokinase (CPK) which was normal, she was admitted to a general floor. Subsequent heart catheterization showed normal coronary arteries. Further examination indicated that she had an esophageal motility disorder, a condition which is helped by coronary dilator drugs such as nitroglycerine.

Our Evaluation

We were assigned to review that case. The physician admitted the patient to the hospital, although his clinical judgment was that the patient did not require hospitalization. Yet, he feared criticism if he did not admit. Accordingly, the patient was admitted to a general floor with the notation, "Rule out myocardial infarction," placed in the medical record. Isoenzymes were drawn, and a coronary dilator drug was ordered. These actions lead to a PRO level II citation indicating medical mismanagement, "with potential for significant adverse effect."5

We employed medical decision-making techniques to determine whether the physician's admitting the patient to a general floor was justified by the data available at the time of admission. As part of our evaluation, specifically we attempted to assess the probability of myocardial infarction (MI) and, from this, the level of care required.

In our assessment, several questions were posed. First, given the history of the patient, what was the probability of MI? Second, what was the probability of MI after the findings on the ECG? Third, when a normal CPK test report is added to the first two questions, what was the likelihood of an MI in this patient? Fourth, at what level of probability should a patient be placed in a monitored unit? Finally, should patient outcome be considered in determining censure?

Several studies6 have addressed the issue of chest pain as it appears in the patient in an emergency room or a coronary care unit. Different parameters were set by various investigators in viewing the problem. The range of probability, considering the patient's findings, varied widely from 2.2% to 24%. The higher value included an abnormal ECG. None of these studies assessed probability without an ECG initially. We accepted the highest probability without an ECG (24%), arguing that an abnormal ECG would further increase the probability of a myocardial infarction. Thus, we adopted a worst case scenario.

Figure 1\7
Test threshold probability and probability of disease before ECG [Omitted]

Further, sensitivity analysis was performed on a computer model8 to determine when an ECG was appropriate. The data in this case indicated that an ECG was appropriate when the probability of MI exceeded 5.5% (Figure 1). Given the sensitivity of the ECG of 0.85 and a specificity of 0.96, the negative ECG here reduced the probability of MI from 24% to 4.7% (Table 1).

Table 1

Probability of disease before and after obtaining ECG, given positive and negative results [Omitted]

Given the negative ECG result, the need to test further was evaluated. "When a second independent test is used, the post-test likelihood determined from the first test becomes the pretest likelihood for the second test."9 Using the 4.7% probability after the negative ECG, the threshold for further testing had risen to 38% (Figure 2).

Figure 2\10
Test threshold probability and probability of disease after ECG [Omitted]

Thus, no test was necessary unless the probability of MI was 38% or better. Clearly this patient was below this threshold. Yet the physician performed a CPK test. This test also was negative.

Table 2

Effect of CPK test on the probability of disease following an ECG [Omitted]

At the time it was performed (within four hours of the pain), the CPK test has low sensitivity (0.4), but much higher specificity (0.8), and further reduced the probability of MI to 3.6% (Tables 2 and 3).11

Table 3

Tabular expression of Bayesian analysis of data [Omitted]

Policy Implications

In this era of rising medical costs, it is incumbent upon practicing physicians to provide the most cost-effective care. It has been estimated that placing a patient in a coronary care unit when there is a 5% probability of MI would cost $2.04 million per life saved; compared to $0.84 million per life saved for the same patients given routine medical ward care.12 Under the same conditions, the costs per year of life saved would be $139,000 and $61,000, respectively. In the case we have presented, monitored bed care would have been much more expensive than general floor care, roughly $800 a day versus $300 a day before add-ons. Knowingly or not, the physician was being cost-effective and could have been even more so without the confused fear of "big brother" watching.

Let us examine another aspect of physician quandary. In assessing the most appropriate action for the case presented, we employed a utility value, which is a measure of outcome. It is important that we look at whether outcome evaluation is appropriate or inappropriate. If appropriate, did the PRO use it in making its determination of wrongdoing in this case? In a recent paper, Kellie and Kelly stressed that "review criteria be based on reliable synthesis of current professional criteria ... and, when possible, be outcome validated."13 In commenting on Kellie and Kelly's paper, Wennberg went further by stating "the strategies of micro-managed care have led to the development of a plethora of rules to govern clinical practice."14 He pointed out that patients should be informed of their options and involved in decisions. Patient input is central to the assignment of utility values since outcome is the most important issue for the patient. Therefore outcome must be a consideration in PRO determinations. In the case presented, the PRO argued that if one states "rule out myocardial infarction," one has the obligation to place the patient in a monitored area since this is a standard of practice, citing Brush et al.15 What the PRO did not state was the content of the paragraph that followed the sentence cited. Clearly Brush et al.16 made a case for using various parameters to help decrease admissions in high cost, limited facilities. Our assessment followed the basic ideas that these authors set forth. In the case cited, had the PRO evaluated the final outcome, an esophageal motility disorder, they could have backed down gracefully and reversed their decision to sanction.

The use of outcome, whenever possible, and the attempts to determine cost-effectiveness in medical care, by and large is not considered by this PRO. But this PRO only reflects on the activities of all PROs. Criteria for determining when and how physicians should be sanctioned are neither uniformly nor objectively applied. Neither are there standard criteria that would indicate some form of malfeasance. Kellie and Kelly17 cite that the criteria to determine when a patient should have carotid artery surgery, or cataract removal, or a heart pacemaker implanted vary among PROs. If one accepts the variability among PROs in these conditions, it is easy to expect variability in other medical conditions. Variability is clearly demonstrated in a recent publication by HCFA.18

In the notes for interpreting the data in this report we find the following significant statement. "In no case is it possible to use this data to reach a conclusion about the accuracy of PRO clinical decisions or the overall quality of PRO review."19 It seems to us that if HCFA were confident of the actions of the organizations they supervise and regulate, such a disclaimer would never be made. To find support for this interpretation we examined the report of interventions (sanctions) taken against physicians. Of 48 PROs reporting "23 PROs exceed the national average of 11.6% errors, and 18 PROs have less than a 5% error rate." This leaves 7 PROs between the extremes. From a statistical standpoint, the error rate does not follow a normal distribution and is skewed toward making errors. This error prone state is not conducive to generating physician confidence in the PRO process. One error in nine by the scrutinizers is more than can be tolerated, when one's professional performance is so subject to scrutiny.

It has been stated that quality control is an important function assigned to the PROs. One does not get a clear sense that this, in fact, is a goal. In some states, it can be argued, that functions other than quality control are more likely the goal. In a 20 month period, one PRO referred 37 instances of physician/hospital deficiency to the licensing board out of a national total of 104 reports. One other PRO reported 28 cases. No other PRO reported more than 7 cases, and 32 PROs reported none during the period noted. The notion that quality control is an important function comes into question because of the marked variability with which different PROs approach reporting major deficiencies to licensing bodies. Such variability does not speak well for how criteria are applied, or whether the same criteria are being used evenly.

This is true even when we acknowledge that there have been some reversals of earlier decisions. What is telling is the need for HFCA to make extensive disclaimers regarding the activities of PROs. In a covering letter,20 the Director of the Office of Peer Review states:

There should be no inference that HFCA sets numeric goals for PRO denials or for the identification of problems. ... Therefore these numbers are simply data to be used in the further analysis of PRO performance which HFCA conducts through a number of mechanisms.

The mechanisms are not elucidated. Also, it should be noted that PRO contracts are issued for a three year period. Meanwhile PROs continue to function, can make errors and can level sanctions without immediate corrective action by the regulatory agency.

In no way do the actions of PROs generate comfort in this environment. Even more important, when one does not feel free to act, one cannot act in the best interest of the patient. Errors are to be expected, but they must be minimized so that they do not infringe the need and freedom to think. When one must be concerned with what a PRO might do, instead of the needs of the patient, then one is diverted from fulfilling one's primary responsibility, and the freedom to think about that responsibility is thwarted. This has the clear potential for decreasing, not increasing, the quality of medical care people receive. Neither does it improve cost-effectiveness. If the quality of medical care or cost-effectiveness are important issues to HCFA and the PROs, the present approaches need drastic revision.

Notes

* Dr. Caccamo is Director, Medical Education, Emeritus at St. Elizabeth Hospital Medical Center, Youngstown, OH and Professor Emeritus, Internal Medicine, Northeastern Ohio Universities College of Medicine [NEOUCOM] in Rootstown, OH.

Dr. Carter is Chair, Department of Pathology at St. Elizabeth and Assistant Professor, Pathology, NEOUCOM.

Dr. Erickson (Ph.D. RN, CCRN) is a Research Analyst in the Department of Medicine, Education and Research, Inc. in Youngstown.

Dr. Johnson, is Assistant Professor, Internal Medicine, NEOUCOM.

Dr. Kessler is Chief, Nephrology at St. Elizabeth and Professor, Internal Medicine, NEOUCOM.

1 Codified at 42 U.S.C. Sec. 1320c (1988).

2 The weights are respectively 1, 5 and 25. The "quality point system is a federally mandated condition of participation for all PRO contractors." OHIO STATE MEDICAL ASSOCIATION, PHYSICIANS' MEDICARE PEER REVIEW HANDBOOK, 12 and 15 (Revised ed. 1991).

3 A. FEINSTEIN, CLINICAL JUDGEMENT 12 (1967).

4 Kellie & Kelly, Medicare Peer Review Organization Preprocedure Review Criteria, 265 J.A.M.A. 1265 (1991).

5 Supra note 2, at 12 and 15.

6 Goldman, Weinberg, Weisberg, et al., A Computer-Derived Protocol to Aid in the Diagnosis of Emergency Room Patients with Acute Chest Pain, 307 NEW ENG. J. MED. 588 (1982); Tierney, Roth, Psaty, et al., Predictors of Myocardial Infarction in Emergency Room Patients, 13 CRITICAL CARE MED. 526 (1985); Fuchs & Scheidt, Improved Criteria for Admission to Cardiac Care Units, 246 J.A.M.A. 2037 (1981); Pozen, D'Agostina, Selker, Sytkowski & Hood, A Predictive Instrument to Improve Coronary-Care-Unit Admission Practices in Acute Ischemic Heart Disease, 310 NEW ENG. J. MED. 1273 (1984); Diamond & Forrester, Analysis of Probability as an Aid in the Clinical Diagnosis of Coronary-Artery Disease, 300 NEW ENG. J. MED. 1350 (1979); Goldman, Cook, Brand, et al. A Computer Protocol to Predict Myocardial Infarction in Emergency Department Patients with Chest Pain, 318 NEW ENG. J. MED. 797 (1988); Mulley, Thibault, Hughes, Barnett, Reder & Sherman, The Course of Patients with Suspected Myocardial Infarction, 302 NEW ENG. J. MED. 943 (1980); Fineberg, Scadden & Goldman, Care of Patients with a Low Probability of Myocardial Infarction. Cost-Effectiveness of Alternatives to Coronary-Care-Unit Admission, 310 NEW ENG. J. MED. 1301 (1984) [hereinafter Care of Patients]; Lee, Rouan, Weisberg, et al., Sensitivity of Routine Clinical Criteria for Diagnosing Myocardial Infarction Within 24 Hours of Hospitalization, 106 ANN. INTERNAL MED. 181 (1987).

7 The ECG should be obtained here when the probability of disease exceeds 5.5%. Below the testing threshold, only observation is warranted.

8 Chalmers, Blum, Buyse, Hewitt & Koch, DATA ANALYSIS FOR CLINICAL MEDICINE. THE QUANTITATIVE APPROACH TO PATIENT CARE IN GASTRENTEROLOGY. (Software diskette with manual. International Univ. Press 1988).

9 Diamond & Forester, supra note 6, at 1353.

10 The testing threshold for preforming a creatine phosphokinase test (CPK) has risen to 38%. The probability of disease has fallen to 4.7%, and, again, below the testing threshold, only observation is warranted.

11 From this objective data, it is apparent that the potential for adverse effects was very small at 3.6% probability of MI. Given this low probability, the patient did not require monitored care, as the PRO claimed, and care on a general floor was appropriate. Indeed with such a low probability of MI, the patient could safely have been sent home without a "potential for significant adverse effect." This interpretation is within the guidelines developed by Goldman in that he proposed that patients should be monitored when the probability of MI reaches 15%. See Goldman, Acute Chest Pain: Emergency Room Evaluation, Hospital Practice 94A (Jul. 15, 1986).

It is also important to realize that there are no guarantees regarding outcomes when using medical decision analysis. Complete certainty never has been, nor ever will be, possible in making medical decisions. What decision makers try to do is to maximize the outcomes in favor of those that are the most desirable.

12 Care of Patients, supra note 6.

13 Supra note 4, at 1270.

14 Wennberg, Unwanted Variation in the Rules of Practice, 265 J.A.M.A. 1306, 1306 (1991).

15 Brush, Brand, Acampora, Chalmer & Wackers, Use of the Initial Electrocardiogram in Predicting In-Hospital Complications of Acute Myocardial Infarction, 312 NEW ENG. J. MED. 1137 (1985).

16 Id.

17 Supra note 4.

18 Results of Peer Review Organization, Review for the Third Scope of Work. (Internally compiled report based upon reports covering the period April 1, 1989 through December 31, 1990).

19 Id., at i, Notes for Interpretation of Data.

20 Husk, Director, Office of Peer Review, HCFA, in memorandum attached to supra note 18, Spring 1991 (undated).