Basics of Information Theory

According to Shannon, the entropy of an information source S is defined as:

$H(S) = \eta = \sum_i ~ p_i \log_2 \frac{{\textstyle 1}}{{\textstyle p_i}}$

where p_i is the probability that symbol S_i in S will occur.

• $\log_2 \frac{{\textstyle 1}}{{\textstyle p_i}}$indicates the amount of information contained in S_i, i.e., the number of bits needed to code S_i.

• For example, in an image with uniform distribution of gray-level intensity, i.e. p_i = 1/256, then the number of bits needed to code each gray level is 8 bits. The entropy of this image is 8.

• Q: How about an image in which half of the pixels are white (I = 220) and half are black (I = 10)?