Quantification of Information
Self-Information - "Entropy"
As the first step
in finding a measure of information, consider an
information
source with a series of ordered outputs: reference
where
the output is 
most-likely and 
is least-likely -- e.g. 
might be, for example, the weather condition and air pollution level in
a given city
and on a certain day or, perhaps, the outcome of a particular athletic
event or ……
A measure of
"information" should satisfy the following
conditions
- The information content of an output
depends only on the probability of
occurring -- i.e.
-- and not on the value of
. We denote this function by 
and call it the self-information
of the output.
Note that 
- Self-information
is a
continuous function of
- Self-information
is a
decreasing function of
- If
, then 
.
Only the "logarithmic" function definition satisfies
these essential properties and thus self-information
may be written
Therefore, the information revealed by a particular source output is
the "weighted' average of the self-information of each of the various
outputs
--
which is usually called (but be careful see caution ) the entropy
of the source.
Definition
of the logarithmic function:
By way of an introduction to logarithms, you
may or
may
not recall, that if we take
this means
Thus, logarithms have the following important
property:
Examples
of logarithms:
-
-
(or a value of 3
bels = 30dB)
-
(or a value of
2.30 bels =
23.0dB)
-
the
"base-changing" rule
-
an application of
the base
changing rule
reference
Some
of this discussion is taken from Communication Systems
Engineering,
John G. Proakis and Masoud Salehi, Prentice-Hall (1994), ISBN
0-13-158932-6.
Comments to: jones@deas.harvard.edu.
Last updated November 1, 2005
