information Theory_Information

An Introduction to information theory
• The performance of the communication system is measured in terms of its error probability. An errorless transmission is possible when probability of error at the receiver approaches zero.
• The performance of the system depends upon available signal power, channel noise and bandwidth. Based on these parameters it is possible to establish the condition for errorless transmission. These conditions are referred as Shannon’s theorems.
• The information theory is related to the concepts of statistical properties of messages/sources, channel , noise interference etc. The information theory is used for mathematical modeling and analysis of the communication systems.
• With the information theory and its modeling for communication systems, following two main points are resolved:
i) the irreducible complexity below which the signal cannot be compressed.
ii) The transmission rate for reliable communication over a noisy channel.

In this chapter we will study the concepts of information , entropy, channel capacity, information rate etc. and some source coding techniques.

Uncertainty
Consider the source which emits the discrete symbols randomly from the set of fixed alphabet i.e.
= ( , , , …., ) …eq.1
The various symbols in X have probabilities of which can be written as ,
P(X= k= 0, 1, 2, …… k-1 …eq.2