What Is Entropy?
Central Question
What properties should a numerical measure of uncertainty satisfy, and why do logarithms enter?
Why This Matters
Entropy becomes meaningful when its formula is motivated by how uncertainty should combine and change, rather than treated as an isolated computational rule.
Learning Prompt
Motivate entropy by asking how to compare uncertainty in fair, biased and deterministic experiments. First derive why the information associated with independent outcomes should add, leading to a logarithm of reciprocal probability. Then ask why averaging this quantity is a natural measure of uncertainty. Explain the assumptions needed for a Shannon-style characterization and do not claim that the logarithmic form follows without them. Make me predict inequalities and compare distributions before calculating. Separate outcome information from average uncertainty, and clarify logarithm bases, units and the convention for zero probability.
Ideas
Examples
Questions
Connections
- Combinatorics Behind the Gaussian
- Shannon Information
- Statistical Mechanics Connection
- Week overview