Statistics Quest Part of Formula Quest

Logistic regression and odds ratios

Logistic regression models the log-odds of a binary outcome as a linear function of predictors, so each exponentiated coefficient is an odds ratio.

\ln\frac{\pi(\mathbf{x})}{1-\pi(\mathbf{x})} = \beta_0 + \beta_1 x_1 + \dots + \beta_k x_k,\qquad \pi(\mathbf{x}) = \frac{1}{1+e^{-(\beta_0+\beta_1x_1+\dots)}}

Logit model and predicted probability

$\pi(\mathbf{x})$: P(Y = 1) for predictor values x
$\pi/(1-\pi)$: odds of the outcome
$\beta_j$: change in log-odds per unit of xⱼ, others fixed (fitted by maximum likelihood)

\text{OR}_j = e^{\beta_j},\qquad z = \frac{b_j}{\SE(b_j)},\qquad \text{CI: } e^{\,b_j \pm z_{1-\alpha/2}\SE(b_j)}

Odds ratio, Wald test and Wald interval

$\text{OR}_j$: multiplicative change in the odds per unit increase of xⱼ
$z$: Wald statistic, compared with the standard normal

G = D_0 - D,\qquad df = p-1

Likelihood-ratio test of all slopes = 0

$D,\ D_0$: deviance (−2 log-likelihood) of the model and of the intercept-only model
$p$: number of coefficients including the intercept

When to use

A binary (0/1) outcome — default or not, converted or not — predicted from quantitative and/or categorical predictors.

Assumptions

Independent observations; log-odds linear in the (quantitative) predictors.
Enough events: few outcomes per coefficient give unstable estimates.
No (quasi-)complete separation: if a predictor splits the outcomes perfectly, the maximum likelihood estimate does not exist and the engine stops with an explanation.

How to interpret

OR = 1.5: each extra unit of x multiplies the odds (not the probability) by 1.5, i.e. +50% odds.
Effects on the probability scale depend on where you start; report predicted probabilities for meaningful profiles.
McFadden's pseudo-R² = 1 − ℓ/ℓ₀ is not comparable with linear-regression $R^2$ .

Common mistakes

Calling an odds ratio a risk ratio ("1.65 times as likely") — they agree only when the outcome is rare.
Reading β on the probability scale.
Ignoring separation warnings or huge standard errors.
Using a 0.5 cut-off by default without considering costs — see classification metrics.

Example

Ten learners: hours of study 1, 2, 2, 3, 4, 4, 5, 6, 6, 7; passed 0, 0, 1, 0, 0, 1, 1, 0, 1, 1.

Fit: logit(π) = −2.001 + 0.500·hours. OR = $e^{0.500} = 1.65$ per hour (95% CI 0.76 to 3.60; Wald p = 0.21).
Predicted pass probability after 5 hours: $1/(1+e^{-0.500}) = 0.62$ .
With ten learners the interval includes OR = 1, so the data are consistent with no association as well as a strong one.