Statistics. One of the most significant and memorable things I learned in college statistics was that one could use statistics to prove - or attempt to prove - almost anything. A few years ago, a friend from the bar, who knew I was interested in judicial evaluation, inquired what I thought of analyzing judicial performance based on appellate reversal rates. I explained to him the challenges of getting a meaningful analysis, based on the vagaries of judicial rotations, the nuances of different types of cases and the definition of reversal rate. Apparently, my friend never actively pursued the project.
So, it was with significant interest that I came across Fred Corbit's article on judicial reversal rates, published in the October Bar Bulletin, with a link to a full analysis. As judicial elections and judicial performance come more into the public eye, Mr. Corbit is to be commended for his proposal that reversal rates may be a factor to be considered.
However, the validity of Mr. Corbit's study - and its implied relevance to judicial evaluations - must be scrutinized. In fairness to Mr. Corbit, he issues the disclaimer that reversal rates cannot be used solely to evaluate judges. However, the author's conclusions imply a correlation between the reversal rate and King County bar poll ratings.1 The analysis gives raw data, but can it accurately tell the bench, the bar and the electorate how an individual judge is performing?
One of the shortcomings of the analysis is the brevity of review (volumes 125 through 129 of Washington Appellate Reports), resulting in only a snapshot of a judge's performance as s/he rotates through the Superior Court system. Part of evaluating judges may be comparing the judge to an average statistic ("the norm," as characterized by Mr. Corbit) or comparing "significant differences between (sic) the various Superior Court judges." However, in statistics, the differences must be significant and validated to be useful.
The challenges to validating an analysis based on such a snapshot are self-evident when considering how reversals are reported, what they actually signify and whether reversal rates can be extrapolated properly to evaluate judges. Other organizations and study groups have recognized the challenges in evaluating judicial reversal rates and their correlation to judicial performance.
In a National Center for State Courts study reviewing criminal case reversals, the organization cautioned "to take care that appeals are equivalent." For example, the study noted that reversal rates for government-based appeals differed significantly from defendant-based appeals. The same study also cautioned "not to categorize all modifications together and call them reversal."2
Along similar lines, an Alaska Judicial Council report found inherent problems associated with evaluating judges based on analyzing reversal rates. The Alaska Judicial Council retains a staff that reviews every published appellate decision and trial court memorandum opinion. Staff members analyze each of the trial court's decisions, identifying each of the issues on appeal and then determining the appellate court resolution of that issue. In spite of relatively sophisticated methodology and the retention of professional staff to analyze appellate information and to input it into databases, the Judicial Council identified seven areas of concern that it recognized could skew the accuracy of analyzing reversal rates.3
Below is an outline of just some of the concerns of the methodology employed by Mr. Corbit and the conclusions he reached.
1. Judicial rotation. King County Superior Court judges typically are assigned to a primary rotation, typically for two to three years. These rotations include: civil, criminal, Unified Family Court, drug diversion court, juvenile court, dependency court and administrative (presiding judge, chief civil, chief criminal, chief UFC). Each judge is thus responsible for overseeing principal types of cases.
Statistically, it has been shown that different types of cases have different rates of appeal and different rates of reversal. For example, criminal cases have a much higher rate of appeal than civil cases (because defendants have a right to appeal; at public expense if they cannot afford counsel). By contrast, civil cases can be extremely expensive to appeal because of the cost of transcripts, records and fees.
However, the rate of reversal for criminal cases, because of the large number of appeals, is much lower than civil cases. Thus, a "civil" judge during the snapshot period presided over cases with a different reversal rate than her "criminal" judge counterpart. Cases in family court, with its large number of pro se litigants, generally have a much lower appeal rate, again because of costs.
Drug diversion court is considered a therapeutic court; the judge presiding over that court may have few, if any, appealable decisions. As a result, a comparatively brief snapshot captures the judge's performance only during that specific rotation that has its own reversal rate for those types of cases. Comparing judges to a "norm" or to one another is comparing the proverbial apples and oranges.
2. Identifying the "erring" judge. Another problem with the methodology is determining "whose case it is." The appellate court reports the name of the judge who entered judgment regardless of who tried the case or entered pre-trial rulings that may be reversed on appeal. Pre-trial rulings that are later reversed may be made on criminal motions calendars or civil motions calendars in non-pre-assigned cases with trials presided over and judgment entered by other judges.
Even in a pre-assigned (individually calendared) case, the case may well be brokered to another judge for trial and judgment entered by that trial judge. Similarly, a judge might try a criminal case, but due to unavailability that judge cannot enter the sentence. Another judge enters judgment and sentence, which is appealed. It is the sentencing judge whose name is reflected in the reporter.
Thus, Washington Appellate Reports will tell the reader who entered judgment but might not (and often does not) identify to the reader which judge made which ruling at issue. Given that a reversal on any issue is treated as a reversal by the author, the methodology is suspect in that it cannot accurately link an appellate ruling on a discrete issue to the judge who entered final judgment.
3. Defining a "reversal." Another shortcoming in the study is the omission of any discussion of the distinction in the types of reversals. For example, in a civil case, the trial judge may be appealed on 14 assignments of error. All of the judge's rulings are upheld, except for the amount of the attorney's fees award. This would show up in the study as a "reverse and remand," even though the substance of the decision was affirmed.
Another considerable basis for reversal is a change in the law - not properly a reflection of a trial judge's legal ability, whose charge is to follow the existing law. This actually occurs more frequently than practitioners might realize.4
Finally, cases are occasionally reversed on issues not even before the trial court, particularly those of constitutional magnitude. By way of example, claims of ineffective assistance of counsel or prosecutorial misconduct may be grounds for reversal, even if never raised before the trial judge.
4. Defining the "rate of reversal." One of the more concerning aspects of the study is the lack of a definition of "reversal rate" and the relevance of the rate used. Mr. Corbit uses statistics such as a 50% reversal rate or 65% reversal rate that appear to be a simple fraction of number of reversals over the total number of appeals. The problem with this approach is that it omits the voluminous judicial decisions that are never appealed. This creates a paradoxical skew against judges who make good decisions that are not appealed. The problems associated with the "reversal rate" employed in the study are illustrated below.
Judge 1 is a well-respected, experienced trial judge. In a six-month period, Judge 1 tries 18 civil cases, grants seven summary judgments of dismissal, enters 93 judgments and sentences and enters 286 orders on contested motions. Counsel are highly selective on appealing Judge 1's decisions. Of these decisions, 10 are appealed. Five are reversed, three on the basis of State v. Jones, the community custody case.5 Judge 1's reversal rate is 50%.
Judge 2 is new to the bench and has never sat on a criminal rotation before. In the same six-month period, he presides over 24 criminal cases (two weeks were spent filling in on drug diversion court), grants one summary judgment of dismissal, enters 74 judgments and sentences and enters 175 orders on contested motions. Of these decisions, 42 are appealed; 14 are reversed. Judge 2's reversal rate is 33%.
Judge 3 is a judge on the Unified Family Court. In a six-month period, she tries 20 cases, half of which involve at least one party who is a pro se litigant. She enters 86 family law orders. As a UFC judge, Judge 3 does not have a regular civil caseload or have any criminal matters. Judge 3 has six decisions appealed; one is reversed. Her reversal rate is 17%.
Looking at the raw data of these examples of "reversal rates," as defined by Mr. Corbit, what do they tell us about the quality of a judge's decision making? I would submit, really nothing. The three judges are in entirely different areas of the law, they have greatly varied quantities of decisions and there is significant variance in the number of decisions appealed. Further, the reasons for appeal are not analyzed nor is the substance of any appeal.
I am not a statistician, but perhaps one could analyze how these variations could be weighted, if at all, to give meaningful analysis to this data. The bar, the public and the electorate strive to obtain whatever information is available about judges and judicial candidates. As judicial campaigning becomes more politicized, the need for qualitative information about the judiciary becomes more critical. However, we owe it to the public, and ourselves, to ensure that such information is as accurate - and valid - as possible. n
Judge John P. Erlick was elected to King County Superior Court in 2000 and currently serves as the Chief Civil Judge. He is chair of the Superior Court Judges' Association Ethics Committee and serves on the State Commission on Judicial Conduct and on the SCJA Committee on Judicial Selection. The views expressed in this article are those of the author and do not reflect those of the court or of other individual judges.
1 In an analysis conducted on Dallas judges, the authors reached the opposite result from Mr. Corbit. In their study, the authors concluded that reversal rates did not correlate with bar results, implying that the lack of correlation was the result of bias, discrimination and other factors in bar polling. See Ray and McClellan, Misjudging the Judge - the Dallas Bar Poll, Full Report, at 4-5, http://www.sallymontgomery.com/documents/MisjudgingTheJudgeReportFull.pdf.
2National Center for State Courts, Appellate Court: Criminal Reversal, at 5. In general, judicial reversal rates have been neither an accepted nor utilized method for judicial performance evaluations. Rather, evaluating organizations have routinely considered other factors such as communication skills, legal ability, demeanor, administrative skills and integrity. See generally, National Center for State Courts, Judicial Performance Evaluation, Resource Guide; American Bar Association, "Judicial Performance Evaluation Guidelines," February 2005; Brody, David C., "A Report on the Washington State Judicial-Performance Evaluation Pilot Project," Washington State Bar Association, September 2002; National Center for State Courts, Commission on Trial Court Performance Standards, Trial Court Performance Standards (Williamsburg, Va., National Center for State Courts, 1990) (factors for performance evaluation should include: access to justice; expeditiousness and timeliness; equality, fairness, and integrity; independence and accountability; and public trust and confidence; Brody, David C., "Judicial Performance Evaluations by State Governments: Informing the Public While Avoiding the Pitfalls," Justice System Journal 21 (2000): 333-47.
3 The seven areas of concern noted by the Alaska Judicial Council in determining the accuracy of judicial reversal are: 1) subjectivity of dividing opinions into separate issues; 2) each issue is weighted equally, regardless of its impact on the case outcome, its legal importance, the applicable standard of review or whether the issue was raised in the trial court; 3) some types of cases are affirmed more often than other types of cases (for example, criminal cases are affirmed at a higher rate than civil cases); 4) information available about appellate decisions is incomplete; 5) administrative appeals pose a problem (Administrative decisions are appealed first to the superior court, which acts as an intermediate appellate court. Those cases may then be appealed to the supreme court, which gives no deference to the superior court's decision and takes up the case de novo.); 6) the present (Alaska) analysis involves only a relatively small number of cases for some judges (The fewer cases in a sample, the less reliable the analysis. "We find that affirmance rates for judges having fewer than ten cases reviewed on appeal is more misleading than helpful."); and 7) change in personnel in the reviewing courts. See Alaska Judicial Council, Appellate Affirmance Rate for Judges on the Ballot in 2002, June 11, 2002.
4 See, e.g., Crawford v. Washington, 541 U.S. 36, 53-54, 124 S. Ct. 1354, 158 L.Ed.2d 177 (2003) (holding that confrontation clause bars admission of testimonial statements of unavailable witness not previously cross-examined by defendant; overturning 25 years of precedent under Ohio v Roberts, 448 U.S. 56, 100 S. Ct. 2531, 65 L.Ed.2d 597 (1979)); Blakely v. Washington, 542 U.S. 296, 124 S. Ct. 2531, 159 L.Ed.2d 403 (2004) (holding provisions of Washington's Sentencing Reform Act unconstitutionally delegated authority to sentencing judge to impose exceptional sentence based on judicial finding of aggravating circumstances); McClarty v. Totem Elec., 157 Wn.2d 214, 231-32 (2006) (adopting federal ADA standards for disability discrimination under state law and rejecting standard set out in Pulcino v. Federal Express Corp., 141 Wn.2d 629, 640, 9 P.3d 787 (2000)); State v. Jones, 126 Wn. App. 136, 144, 107 P.3d 755 (2005), review granted, 155 Wn.2d 1017, 124 P.3d 659 (2005) (holding that in order to comply with Blakely, a jury must determine beyond a reasonable doubt whether a defendant is on community custody at the time of an offense).
5 126 Wn. App. 136.
