supremecourt.report · methods note

Predicting the Probability of Certiorari

Three calibrated models estimating whether a paid petition will be granted plenary review.

The models

Baseline — grant probability from case structure known at filing (shown on the daily petition dashboards). Enhanced — adds docket-development signals (conference reports). GVR — the companion “hold” risk of a grant-vacate-remand. IFP petitions (grant rate ~0.1%) are a separate regime and excluded.

Data & target

Eight Terms, OT2017–OT2024: 11,806 paid petitions, of which 497 were granted. Target is plenary granted vs. denied; GVRs, dismissals, and pending petitions are excluded from training. Every docket-development feature is snapshotted strictly before the decision date — leakage-safe.

Method

Logistic regression, Platt-calibrated. Validated leave-one-term-out: each Term is scored by a model trained on the other seven, and the calibration map is fitted out-of-fold. That is out-of-fold, not out-of-time — a rolling-origin check (train only on earlier Terms) reproduces it to within 0.003 AUC.

Validation (leave-one-term-out)

AUC — chance the model ranks a random grant above a random denial. Avg. precision is the imbalance-aware metric (baseline 0.313 vs. a 4.13% base rate ≈ 7.6× chance). Brier — the reference is a constant forecast at each model’s own base rate, 0.0396, 0.0720 and 0.0515; all three beat it. Accuracy is meaningless at a 4.1% base rate.

What drives the estimate

• U.S. as petitioner (the Solicitor General): ~43% granted vs. 3.7% — the largest structural cue.
• Relists (enhanced): non-monotonic — ~1% at zero, ~20% at one, ~44% at two, falling to ~19% at 5+ (the “hold” zone). Modeled as a bucket, not a line.
• Rule 10 dissent / circuit split, parsed from the petition PDF.
• Counsel track record — prior petitions and prior wins, counted strictly before this petition was docketed.
• Court below: federal circuits far above state courts.

Calibration

Predicted probabilities track observed frequencies across deciles: the enhanced model’s top decile predicts 38.0% and observes 37.4%; the baseline’s predicts 22.8% and observes 21.8%. A “17%” means about 17%.

Limitations

• Rare outcome — only 497 grants across eight Terms.
• OT2024 is right-censored (late petitions undecided): a pessimistic test Term.
• The dissent signal defaults to “absent” for the ~9% of petitions with no parseable PDF, conservatively understating it.
• Entity typing and counsel matching are heuristic (regex).

Coefficients

Every term in every deployed model, from the artifacts themselves. β is on the log-odds scale; the odds ratio is e^β, the multiplier on the odds of the outcome for a one-unit change, holding the rest fixed. A • marks p < 0.05 — a reading aid, not a finding: this is eight Terms of observational data, nothing is corrected for the 112 tests reported here, and a coefficient is not a cause.