Shannon Information
Central Question
How does uncertainty become a statement about communicating and compressing messages?
Why This Matters
Information theory connects a measure of uncertainty with operational limits on description length and communication, giving entropy a concrete interpretation.
Learning Prompt
Develop Shannon entropy as the expected information of a discrete random variable, building from surprising outcomes and repeated messages rather than announcing a formula. Work through a biased binary source and motivate the connection with lossless coding and typical sequences, stating the assumptions and meaning of an asymptotic limit. Introduce conditional entropy and mutual information through a concrete dependent pair. Make me distinguish entropy from semantic meaning and from the length of one particular message. Use conceptual predictions to connect uncertainty, compression and dependence.