Study notes

Connected notes on probability, information theory, random processes, and Bayesian inference.

• Named distributionsA visual map of the standard distribution families.
• Entropy and mutual informationSurprise, uncertainty, and shared information.
• Choosing a priorThree principled routes and where they disagree.

Notes on transformer internals and interpretability methods for representations, circuits, probes, and interventions.

• AttentionA weighted sum, adaptive statistic, and kernel regression.
• Probes and validityWhy a probe that decodes a property does not show the model uses it.
• BindingHow a use finds its nonlocal source.