Quantification of Information
Simple Examples of Self Information - "Entropy" - Calculations -
Page 2 reference
Recall
that entropy is given by the general
formula: 
Example 3:
Consider a discrete memoryless information source with
binary output alphabet with the
respective probabilities p and (1 - p):
Note the maximum
entropy
occurs when the probabilities are equal!
Example 4 :
Consider another
source sampled at a rate of 8,000
samples per second (with implies that the
souce has a bandwidth of 4 kHz and, thus,
is sampled at the Nyquist rate).
The resulting sample sequence can be approximated as a discrete
memoryless
information source with a five element output alphabet with output
probabilities 
.
Therefore the source
produces information at a rate of 15 kbits per second.
reference
Some
of
this discussion is drawn from Communication Systems Engineering,
John G. Proakis and Masoud Salehi, Prentice-Hall (1994), ISBN
0-13-158932-6.
This
page was
prepared and is maintained by R. Victor Jones
Comments to: jones@deas.harvard.edu.
Last updated
February 8, 2004