Named Distributions

One trial becomes a count.

A Bernoulli variable is a single yes/no event. A binomial is the sum of $n$ independent Bernoulli trials with the same success probability $p$.

Drag $n$ and $p$: the bars are $P(X=k)$ for $X\sim\mathrm{Binomial}(n,p)$. The center tracks $np$ and the spread tracks $np(1-p)$.

From one $K$-sided draw to a vector of counts.

A categorical variable is a single draw over $K$ outcomes with probabilities $(p_1,\dots,p_K)$. The multinomial is the vector of category counts after $n$ independent categorical draws. With $K=2$ these reduce to Bernoulli and binomial; each marginal count is $\mathrm{Binomial}(n, p_k)$, but counts share negative covariance because the total must equal $n$.

Bars show the expected counts $np_k$ with $\pm 2$ standard-deviation whiskers from the binomial marginals.

Waiting for the first success.

The geometric distribution counts trials until the first success. After failures, the remaining wait still has the same distribution: discrete memoryless waiting time.

The expectation follows from the tail sum:

\[\mathbb{E}X=\sum_{k\ge0}P(X>k)=\sum_{k\ge0}(1-p)^k=1/p.\]

Many tiny chances become a rate.

If $X_n\sim\mathrm{Binomial}(n,\lambda/n)$, then as $n\to\infty$ the mass approaches $\mathrm{Poisson}(\lambda)$. This is the count model for many independent rare opportunities.

Poisson processes are the process-level version: counts over time windows plus exponential waiting times.

The plot overlays the binomial bars (blue, wider) with the limiting Poisson distribution (red, narrower).

Waiting for several successes, or overdispersed counts.

A negative binomial can be seen as a sum of geometric waits. As a count model, it is what you reach for when Poisson variance is too small for the data.

The gray curve is a Poisson with the same mean. The wider bars show the extra dispersion.

The random-number source behind the others.

If $U\sim\mathrm{Uniform}(0,1)$, then $F^{-1}(U)$ has CDF $F$. This is the inverse-CDF method behind exact one-dimensional sampling.

The left axis is uniform probability. The curve is $F^{-1}(u)$ for the chosen family, turning evenly spaced $u$ values into nonuniform samples.

Monte Carlo & MCMC builds this into rejection and importance sampling.

Memoryless waiting in continuous time.

The exponential distribution is the continuous waiting time with no aging:

\[P(T>s+t\mid T>s)=P(T>t).\]

It is also the interarrival-time distribution in a Poisson process.

The minimum of independent exponentials is exponential again: $\min(T_1,T_2)\sim\mathrm{Exp}(\lambda_1+\lambda_2)$. Competing clocks add rates.

The shape sums with finite variance converge to.

Gaussian distributions are closed under sums: independent normals add to a normal. More broadly, normalized sums of many finite-variance variables drift toward a bell curve.

The bars show the sum of $m$ centered uniform variables rescaled to variance $\sigma^2$. The curve is the matching normal approximation.

A ratio distribution with no mean.

If $Z_1,Z_2$ are independent standard normals, then $Z_1/Z_2$ is Cauchy. Its tails are so heavy that the mean does not exist.

The plot shows the theoretical Cauchy density against a normal density. The Cauchy peak is lower and its tails decay much more slowly.

The double exponential: a sharp peak with exponential tails.

If $E_1,E_2$ are independent $\mathrm{Exp}(1/b)$, then $E_1-E_2\sim\mathrm{Laplace}(0,b)$. The density is the symmetric exponential $\frac{1}{2b}e^{-|x-\mu|/b}$, so it has a cusp at the mean and decays linearly on a log scale — heavier tails than a Gaussian, but far lighter than a Cauchy.

It is the maximum-entropy distribution on $\mathbb{R}$ for a fixed mean absolute deviation $\mathbb E|X-\mu|=b$, the same way Gaussian is max-entropy for fixed variance. As a noise model, $-\log p(x\mid\mu,b)\propto|x-\mu|/b$, so MLE under Laplace noise is median regression (L1 loss); as a prior on regression coefficients it gives the lasso.

Adding independent variables convolves their densities.

The density of $X+Y$ is $f_X*f_Y$. Some families are closed under addition: Gaussian plus Gaussian is Gaussian; Poisson plus Poisson is Poisson; Cauchy plus Cauchy is Cauchy.

The plot shows two draggable family choices and their sum.

Extremes act on the CDF, not the density.

If $M_n=\max(X_1,\dots,X_n)$, then:

\[P(M_n\le x)=F(x)^n.\]

If $m_n=\min(X_1,\dots,X_n)$, then:

\[P(m_n\le x)=1-(1-F(x))^n.\]

The plot uses a $\mathrm{Uniform}(0,1)$ base distribution and shows how increasing $n$ pushes mass toward the right edge for maxima and the left edge for minima.

Changing variables reshapes density by the derivative.

For a monotone transform:

\[f_Y(y)=f_X(g^{-1}(y))\left|\frac{d}{dy}g^{-1}(y)\right|.\]

Non-monotone transforms sum over preimages.

Two canonical cases: $Y=X^2$ turns $N(0,1)$ into $\chi^2_1$; $Y=e^X$ turns a normal into a log-normal.

Division creates heavy tails.

The Cauchy distribution is what you get when you divide one standard normal by another. Near-zero denominators create enormous ratios.

This is the ratio story behind the $t$ and $F$ sampling distributions too: normalize by an estimated scale, and tail weight appears.

Sum squared standard normals.

If $Z_i\sim N(0,1)$ independently, then $\sum_{i=1}^k Z_i^2\sim\chi^2_k$. It is the distribution of squared Gaussian length in $k$ dimensions.

A normal divided by estimated scale.

If $Z\sim N(0,1)$ and $V\sim\chi^2_\nu$, then $T=Z/\sqrt{V/\nu}$ has Student's $t_\nu$ distribution. As $\nu$ grows, the denominator stabilizes and $t$ becomes Gaussian.

Ratio of scaled chi-squares.

If $U\sim\chi^2_{d_1}$ and $V\sim\chi^2_{d_2}$ independently, then $(U/d_1)/(V/d_2)$ has an $F_{d_1,d_2}$ distribution. It appears in variance comparisons and ANOVA-style ratios.

One over a chi-square; the conjugate prior for normal variance.

If $X\sim\chi^2_\nu$ then $Y=1/X$ has the inverse chi-squared distribution with $\nu$ degrees of freedom. The scaled inverse chi-squared, $\mathrm{Scale}$-$\mathrm{Inv}\chi^2(\nu,\tau^2)$, multiplies by $\nu\tau^2$ and is the standard conjugate prior for the variance $\sigma^2$ of a normal with known mean: $n$ observations with sample variance $s^2$ update the posterior to $\mathrm{Scale}$-$\mathrm{Inv}\chi^2(\nu+n,\,(\nu\tau^2+n s^2)/(\nu+n))$.

For both unknown mean and unknown variance, the conjugate prior is Normal–scaled-Inv-$\chi^2$ (equivalently Normal–Gamma on the precision, via the $1/X$ edge). Marginalising out the variance turns the Gaussian posterior predictive into Student's $t$ with degrees of freedom equal to the posterior $\nu_n$ — heavy tails when $n$ is small, Gaussian as $n\to\infty$. This is the single derivation that ties Chi², Inv-$\chi^2$, and Student-$t$ together in Bayesian inference.

Slow tail decay changes which moments exist.

The Pareto tail has $P(X>x)\propto x^{-\alpha}$. On log-log axes it becomes a straight line, while exponential and Gaussian tails curve downward much faster. Smaller $\alpha$ means heavier tails and fewer finite moments.

Cauchy has no mean or variance. Pareto has finite moments only for orders below $\alpha$: mean needs $\alpha>1$, variance needs $\alpha>2$.