Information Theory

Founded by Claude Shannon, this theory quantifies the concept of information and forms the foundation of modern communication, data compression, and cryptography.

Formal Introduction

Information Theory was founded by Claude Shannon in 1948. Its core is the mathematical quantification of information. Shannon Entropy measures information uncertainty.

H(X) = -Σ p(x) log₂ p(x)

其中 p(x) 是事件 x 发生的概率。熵越大,不确定性越高,信息量越大。

Plain Language Introduction

Information content = degree of surprise. "The sun rises in the east" has little information (you already know it), while "It will snow tomorrow" has high information content (uncertain).

Examples:

  • Coin flip: 50% heads/tails, entropy = 1 bit (maximum uncertainty)
  • Certain event: 100% probability, entropy = 0 bit (no uncertainty)
  • Weather forecast: More possibilities mean higher entropy

Information Entropy Calculator

Enter text to calculate its information entropy (average information per character)

通信理论

香农定理:信道容量 = 带宽 × log₂(1 + 信噪比)

🗜️ 数据压缩

ZIP、MP3、JPEG等压缩算法都基于信息论原理

🔐 密码学

完美保密需要密钥熵 ≥ 消息熵(一次性密码本)