Quantification of Information

Simple Examples of Self Information - "Entropy" - Calculations - Page 1 reference

Recall that entropy is given by the general formula:          
 

Example 1:

Consider a random variable uniformly distributed over 32 = (2)5  outcomes.



Therefore, we need all five identifying binary digits - i.e.    - to represent the outcome message! 

Example 2:

Consider a horse race with eight horses - i.e. 8 = (2)3 - taking part. Suppose that the probabilities of each horse winning are given by
           

Therefore


 
Therefore, we do not need all three identifying binary digits to represent the outcome message!

This observation opens up the opportunity for code cleverness.  Consider the following: 

ALTERNATE CODING SCHEMES
Output
Binary
Code
Huffman
Code
1
000
0
 2
001
10
 3
010
110
 4
011
1110
 5
100
111100
 6
101
111101
 7
110
111110
 8
111
111111
Average number 
of bits used
3 bits
2 bits
 
 
The entropy of a random variable is a lower bound on the number of bits required to represent the random variable and on the average number of questions needed to identify the variable in a game of "twenty questions."


reference
The discussion here is drawn from examples used in Elements of Information Theory, Thomas M. Cover and Joy A. Thomas, John Wiley & Sons,  New York, (1991), ISBN 0-471-06259-6


This page was prepared and is maintained by R. Victor Jones
Comments to: jones@deas.harvard.edu. 
Last updated
February 8, 2004