Last term, I experimented with using a customized transformer model to predict Supreme Court votes. I learned and thought a lot about prediction. You can read a term post mortem here and here, and about some of the more general lessons on this paper. Overall, last term the model performed well by algorithmic standards, but not so well relative to human predictions—humans had a terrific year.
Over the summer, I reviewed my modeling approach, from data ingestion to inference. It’s all revised and updated, and I’m pleased to say that I believe I have a much more powerful and robust model than I fielded last term. I call it experience-6, in my nod to Holmes, and it involves major changes in training and architecture.
Against the validation term for this model, OT25, the model scores vote-level accuracy of 89 percent, and case-level accuracy of 91 percent. I'd expect performance in OT26 to be a bit lower due to drift and the like, but in that general range. The model remains about as well calibrated as before, and as in last term, I endeavor to provide principled quantification of uncertainty for the predictions (in general, confidence bars cover the true outcome 90 percent of the time—in other words, confident errors should occur in no more than 10 percent of votes/cases).
One lesson I learned last term is that keeping up with the Court is shreds my attention. I enjoyed reading every case, at least in part, to develop a holistic sense of the docket, but that takes a lot of time. I also found that I did not comment enough on the cases that I do care quite a lot about, again because I was analyzing cases I care less about. So this term, though I will predict every argued case, I will only post in any depth about those close to my interests. All predictions can be viewed here post argument.
Another lesson is that early term predictions tend to be pretty terrible. I made some progress on that during training, but still I will only post post-argument predictions, again mostly to preserve bandwidth for where it is most useful.
More soon!