First Results—Assessment of Predictions

The Court released decisions in three cases today, in addition to one from earlier in January. Time to perform an initial assessment of the model’s predictions in this live experiment. Overall, the model performed quite well: it correctly predicted the outcome in all four cases. The vote-level accuracy was 67 percent (69 percent if you count Barrett’s Bowe vote differently). It’s a small sample, so time will tell, but that’s better than expected at the case level and slightly worse than expected at the vote level. Also notable, of the twelve incorrectly predicted votes, the model acknowledged nine as sufficiently uncertain that they could have gone the other way. Estimates of uncertainty were designed to capture the correct result 90 percent of the time. So far, 8 percent (3/36) of estimates failed that test—i.e., they were confident errors. So again, in line with expected parameters based on validation runs. Bowe v. United States. (Repeat habeas claims.) The Court reversed/remanded 5-4, with dissenting votes by Gorsuch, Thomas, Alito, and Barrett (in part). The model predicted the Court would reverse 7-2, with dissents by Alito and Roberts. The Roberts dissent was estimated to be uncertain, as were the predicted reverse votes by Thomas and Barrett.

Barrett v. United States. (Double jeopardy.) The Court reversed, 9-0, with a concurring opinion by Gorsuch. The model predicted the Court would reverse 7-2, with dissents by Alito and Kavanaugh.

Case v. Montana. (Emergency aid entry without a warrant.) The Court affirmed 9-0, with concurring opinions by Sotomayor and Gorsuch. The model predicted the Court would affirm 5-4. Three of the predicted dissents, however, were uncertain (Kagan, Barrett, Jackson).

Bost v. Illinois State Board of Elections. (Federal candidates' standing to challenge state election laws.) The Court reversed 7-2, with Jackson and Sotomayor dissenting. The model predicted the Court would reverse 5-4, with dissents by Sotomayor, Jackson, Thomas, and Barrett (the latter two of which were uncertain).