Library / Thermodynamics / THE-23
Thermodynamics
Entropy of a four-outcome distribution (Gibbs and Shannon)
Uncertainty of a system that can be found in one of four states with relative weights w₁ to w₄.
- Ref
- THE-23
- Rev
- v1
- Field
- Thermodynamics
- Result
- H [bit]
w₁ to w₄ are relative weights, such as counts or any non-negative numbers; they are scaled so the probabilities p = w/Σw add to 1. The Shannon entropy H = −Σ p log₂ p is in bits. The Gibbs entropy of statistical mechanics is the same sum with the natural log times k_B: S = k_B ln 2 × H in joules per kelvin.
Uncertainty of a system that can be found in one of four states with relative weights w₁ to w₄. The weights are scaled to probabilities p = w/Σw. The result is in bits (Shannon); multiplying the natural-log version by Boltzmann’s constant gives the Gibbs thermodynamic entropy in J/K. It is largest when all four states are equally likely.
Calculator
- Probability of state 10.5
- Largest possible value (all equal), log₂ 42 bit
- Thermodynamic entropy, k_B ln 2 × H1.675×10−23 J/K
What drives the result
Each variable is moved across its typical range while the others stay at the values above. The longest bar is the key driver.
- These weights give 1.75 bits, out of a maximum of 2 bits for four equally likely states. It is as uncertain as a choice among about 3.36 equally likely outcomes.
- Entropy is highest when all states are equally likely and falls to zero when one state is certain.
Chart data as tables
Ranked variables
| Variable | Typical range | Lowest result | Highest result | Swing (bit) | Local step | Change in result |
|---|---|---|---|---|---|---|
| w1 Weight of state 1 | 0 to 100 | 0.2929 | 1.812 | 1.519 | +10% | −2.13 % |
| w3 Weight of state 3 | 0 to 100 | 0.4388 | 1.841 | 1.402 | +10% | +0.839 % |
| w4 Weight of state 4 | 0 to 100 | 0.4388 | 1.841 | 1.402 | +10% | +0.839 % |
| w2 Weight of state 2 | 0 to 100 | 0.3847 | 1.75 | 1.365 | +10% | +0.276 % |
Sweep of Weight of state 1
| Weight of state 1 | Entropy (bit) |
|---|---|
| 0 | 1.5 |
| 12.5 | 1.163 |
| 25 | 0.7857 |
| 37.5 | 0.602 |
| 50 | 0.4921 |
| 62.5 | 0.4183 |
| 75 | 0.365 |
| 87.5 | 0.3247 |
| 100 | 0.2929 |
Sweep one variable
Variables and how each one contributes
Entropy
What the equation gives you. Everything else feeds it.
Weight of state 1
The relative weight of state 1. Changes the entropy through the probabilities: raising the weight of an already likely state lowers the entropy. Raising it by 10 % (to 4.4) moves the result by −2.13 %, from 1.75 to 1.713 bit.
Weight of state 2
The relative weight of state 2. Changes the entropy through the probabilities: it raises the entropy when it moves the state closer to equal chance. Raising it by 10 % (to 2.2) moves the result by +0.276 %, from 1.75 to 1.755 bit.
Weight of state 3
The relative weight of state 3. Raises the entropy when this state is less likely than average, because the distribution moves towards equal chance. Raising it by 10 % (to 1.1) moves the result by +0.839 %, from 1.75 to 1.765 bit.
Weight of state 4
The relative weight of state 4. Raises the entropy when this state is less likely than average, because the distribution moves towards equal chance. Raising it by 10 % (to 1.1) moves the result by +0.839 %, from 1.75 to 1.765 bit.
Worked example with the values above
- Total weight: 8, so p = (0.5, 0.25, 0.125, 0.125).
- Terms −p log₂p: (0.5, 0.5, 0.375, 0.375).
- H = 1.75 bit.
Assumptions and limits
- Four distinct states with fixed probabilities (the page scales the weights to sum to 1).
- Independent draws; correlations reduce the true entropy rate.
Common mistakes
- Expecting weights that do not sum to 1 to be used unscaled (the page rescales them).
- Using ln and calling the answer bits.
- Forgetting that 0 log 0 is taken as 0.
Statistics and uncertainty
Entropy estimated from observed counts is biased low when the counts are small: the true entropy is usually higher than the plug-in value, by roughly (number of states − 1)/(2N ln 2) bits for N samples.
Where it is used
- Information content of a symbol source and data compression limits.
- Entropy of a system with unequal state populations in statistical mechanics.
Maths you need
- Probability: values from 0 to 1
- Logarithms: log base 2