supremecourt.report · methods note

Predicting the Probability of Certiorari

Three calibrated models estimating whether a paid petition will be granted plenary review.

The models

Baseline — grant probability from case structure known at filing (shown on the daily petition dashboards). Enhanced — adds docket-development signals (conference reports). GVR — the companion “hold” risk of a grant-vacate-remand. IFP petitions (grant rate ~0.1%) are a separate regime and excluded.

Data & target

Eight Terms, OT2017–OT2024: 11,368 paid petitions, of which 496 were granted. Target is plenary granted vs. denied; GVRs, dismissals, and pending petitions are excluded from training. Every docket-development feature is snapshotted strictly before the decision date — leakage-safe.

Method

Logistic regression, Platt-calibrated. Validated leave-one-term-out: each Term is scored by a model trained on the other seven, so every statistic below is out-of-time, never in-sample.

Validation (out-of-time)

Model	Base rate	AUC	Avg. prec.	Brier
Baseline (daily)	4.36%	0.804	0.234	0.0374
Enhanced (conference)	4.36%	0.930	0.484	0.0291
GVR companion	2.38%	0.866	0.126	0.0220

AUC — chance the model ranks a random grant above a random denial. Avg. precision is the imbalance-aware metric (baseline 0.234 vs. a 4.36% base rate ≈ 5.4× chance). Brier — a constant-base-rate forecast scores 0.042; all three beat it. Accuracy is meaningless at a 4% base rate.

Calibration plot: predicted vs observed grant rate by decile — Predicted vs. observed grant rate by risk decile. Point size ∝ petitions in the bin.

What drives the estimate

U.S. as petitioner (the Solicitor General): ~43% granted vs. 3.7% — the largest structural cue.
Relists (enhanced): non-monotonic — ~1% at zero, ~20% at one, ~44% at two, falling to ~19% at 5+ (the “hold” zone). Modeled as a bucket, not a line.
Rule 10 dissent / circuit split, parsed from the petition PDF: lifts baseline AUC 0.720 → 0.804.
Court below: federal circuits far above state courts.

Calibration

Predicted probabilities track observed frequencies across deciles: the enhanced model’s top decile predicts 31.9% and observes 33.0%; the baseline’s predicts 19.2% and observes 17.7%. A “17%” means about 17%.

Limitations

Rare outcome — only 496 grants across eight Terms.
OT2024 is right-censored (late petitions undecided): a pessimistic test Term.
The dissent signal defaults to “absent” for the ~9% of petitions with no parseable PDF, conservatively understating it.
Entity typing and counsel matching are heuristic (regex).