Information Theory_Entropy

Definition : The average information per message is called entropy. It is represented in bits / message.

Mathematical derivation of Entropy
Consider that we have M-different messages. Let these messages be m1, m2, m3, … and they have probabilities of occurrence as p1, p2, p3, … . Suppose that a sequence of L messages is transmitted. Then if L is very large, then we may say that,
p1L number of m1 message are transmitted,
p2L number of m2 message are transmitted,
p3L number of m3 message are transmitted,
. .
. .
. .
L number of message are transmitted.

Hence the information due to message m1 will be,

Since there are p1L number of m1 messages, the total information due to all m1 will be,

Similarly the total information due to all messages m2 will be
and so on
Thus the total information carried due to the sequence of L messages will be,
...eq.5
) + ) + …..+ …eq.6
The average information per message will be,
Average information = Total information / number of messages
= …eq.7
Average information is represented by Entropy. It is represented by H. Thus,
Entropy (H) = …eq.8
From equation 6, we can write the above equation as,
Entropy (H) = …eq.9
We can write the above equation using sign as follows:

….eq.10