what is channel capacity in information theory | channel capacity is exactly equal to | formula theorem and unit ?

**THE CHANNEL CAPACITY**

** **In this section, let us discuss various aspects regarding channel capacity.

**9.12.1. Channel Capacity Per Symbol C _{s}.**

The channel capacity per symbol of a discrete memoryless channel (DMC) is defined as

C

_{s}= I (X;Y) b/symbol …(9.35)

where the maximization is over all possible input probability distributions {P(x

_{i})} on

*X*. Note that the channel capacity C

_{s}is a function of only the channel transition probabilities which define the channel.

**9.12.2. Channel Capacity Per Second C**

**If**

*r*symbols are being transmitted per second, then the maximum rate of transmission of information per second is

*r*C

_{s}. This is the channel capacity per second and is denoted by C(b/s), i.e.,

C =

*r*C

_{s}b/s …(9.36)

**9.12.3. Capacities of Special Channel**

**In this subsection, let us discuss capacities of various special channel.**

**9.12.3.1. Lossless Channel**

**For a lossless channel, H(X|Y) = 0, and**

*I*(X;Y) = H(X) …(9.37)

Thus, the mutual information (information transfer) is equal to the input (source) entropy, and no source information is lost in transmission.

Consequently, the channel capacity per symbol will be

C

_{s}= H(X) = log

_{2}

*m*…(9.38)

Where m is the number of symbols in

*X*.

**9.12.3.2. Deterministic Channel**

For a deterministic channel, H(Y|X) = 0 for all input distributions P(x

_{i}), and

I(X;Y) = H(Y) …(9.39)

Thus, the information transfer is equal to the output entropy. The channel capacity per symbol will be

C

_{s}= H(Y) log

_{2}

*n*…(9.40)

where

*n*is the number of symbols in Y.

**9.12.3.3. Noiseless Channel**

**Since a noiseless channel is both lossless and deterministic, we have**

I(X; Y) = H(X) = H(I’) …(9.41)

and the channel capacity per symbol is

C

_{s}= log

_{2}

*m*= log

_{2}

*n*…(9.42)

**9.12.3.4. Binary Symmetric Channel (BSC)**

For the binary symmetric channel (BSC), the mutual information is

*I*(X;Y) = H(Y) + p log

_{2}p + (1 – p) log

_{2}(1 – p) …(9.43)

and the channel capacity per symbol will be

C

_{s}= 1 + p log

_{2}p + (1- p) log

_{2}(1 -p) …(9.44)

**EXAMPLE 9.29. Verify the following expression:**

**C**

_{s}= log_{2}*m***where C**

_{s}is the channel capacity of a lossless channel and*m*is the number of symbols in*X*.**Solution:**For a lossless channel, we have

H(X|Y) = 0

Then, by equation (9.30), we have

*I*(X;Y) = H(X) – H(X|Y) = H(X)

Hence, by equations (9.35) and (9.9), we have

C

_{s}= I(X;Y)

or C

_{s}= H(X) = log

_{2}

*m*

**Hence proved.**

**EXAMPLE 9.30. Verify the following expression:**

**C**

_{s}= 1 + p log_{2}p + (1 – p) log_{2}(1 – p)**where C**

_{s}is the channel capacity of a BSC (figure 9.12)**Solution:**We know that the mutual information /(X: Y) of a BSC is given by

*I*(X;Y) = H(Y) + p log

_{2}p + (1 – p) log

_{2}(1 -p)

which is maximum when H(Y) is maximum. Since, the channel output is binary, H(Y) is maximum when each output has a probability of 0.5 and is achieved for equally likely inputs.

For this case H(Y) = 1, and the channel capacity is

**equation**

**EXAMPLE 9.31. Find the channel capacity of the binary erasure channel of figure 9.13.**

**Solution:**Let P(x

_{1}) = α. Then P(x

_{2}) = 1 – α.

**diagram**

**FIGURE 9.13**

We have

**EQUATION**

Using equation (9.17), we

[P(Y)] = [α 1 – α]

= [α(1 – p)] p (1 – α) (1 – p)] = [P(y

_{1}) P(y

_{2}) P(y

_{3})]

By using equation (9.19), we have

[P(X,Y)] =

or [P(X, Y)] =

In addition, from equations (9.24) and (9.26), we can calculate

**equation**

= – α(1 – p) log

_{2}α(1 – p) – p log

_{2}p – (1 – α)(1 – p) log

_{2}[(1 – α)(1 – p)]

= (1- p)[- α log

_{2}α – (1 – α) log

_{2}(1- α)] – p log

_{2}p – (1 -p) log

_{2}(1 -p)

Also, we have

**equation**

= – a(1 – p) log

_{2}(1 – p) – αp log

_{2}p – (1 – α) p log

_{2}p

– (1 – α)(1 -p) log

_{2}(1 -p)

= – p log

_{2}p-(1-p) log

_{2}(1 -p)

Thus, by equations (9.33) and (9.57), we have

*I*(X; Y) = I(Y) – H(Y|X) = (1 – p)[- α log

_{2}α – (1 – α) log

_{2}(1 – α)] = (1 – p)H(X)

And by equations (9.35) and (9.58), we have

**equation**

**9.13 ENTROPY RELATIONS FOR A CONTINUOUS CHANNEL**

In a Continuous channel, an information source produces a continuous signal x(t). The set of possible signals is considered as an ensemble of waveforms generated by some ergodic random process. It is further assumed that x(t) has a finite bandwidth so that x(t) is completely characterized by its periodic sample values. Hence, at any sampling instant, the collection of possible sample value constitutes a continuous random variable X descrbed by it probability density function f

_{X}(x).

The average amount of information per sample value of x(t) (i.e., entropy of a continuous source) is measured by

**equation**

The entropy H(X) defined by equation (9.45) is known as the differential entropy of X.

Also, the average mutual information in a continuous channel is defined (by analogy with the discrete case) as

I(X; Y) = H(X) H(X|Y) = H(Y) – H(Y|X)

where

**equation**…(9.46)

**equation**…(9.47)

**equation**…(9.48)

**9.14 CAPACITY OF AN ADDITIVE WHITE GAUSSIAN NOISE (AWGN) CHANNEL: SHANNON-HARTLEY LAW**

In an additive white Gaussian noise (AWGN) channel, the channel output

*Y*is given by

Y = X + n …(9.48)

where X is the channel input and n is an additive bandlimited white Gaussian noise with zero mean and variance .

The capacity C

_{s}of an AWGN channel is given by

**EQUATION**

where S/N is the signal-to-noise ratio at the channel output. If the channel bandwidth B Hz is fixed, then the output y(t) is also a bandlimited signal completely characterized by its periodic sample values taken at the Nyquist rate 2B samples/s.

Then the capacity C(b/s) of the AWGN channel is given by

C = 2B x C

_{s}= B log

_{2}b/s …(9.50)

Equation (9.50) is known as the

**Shannon-Hartley law.**

**The Shannon-Hartley law underscores the fundamental role of bandwidth and signal-to-noise ratio in communication. It also shows that we can exchange increased bandwidth for decreased signal power for a system with given capacity**

*C*.

**9.15 CHANNEL CAPACITY : A DETAILED STUDY**

**We know that the bandwidth and the noise power place a restriction upon the rate of information that can is expressed as be transmitted by a channel. It may be shown that in a channel which is disturbed by a white Gaussian noise, one can transmit information at a rate of**

*C*bits per second, where

*C*is the channel capacity and is expressed as

C =

*B*log

_{2}…(9.51)

In this expression, B = channel bandwidth in Hz

S = Signal power

N = Noise power

It may be noted that the expression (equation 9.50) for channel capacity is valid for white Gaussian not However, for other types of noise, the expression is modified.

**Proof:**Let us present a proof of channel capacity formula based upon the assumption that if a signal is mixed with noise, the signal amplitude can be recognized only within the root main square noise voltage. In other words, we can say that the uncertainty in recognizing the exact signal amplitude is equal to the root mean square noise voltage.

Again, let us assume that the average signal power and the noise power are

*S*watts and

*N*watts respectively. This means that the root mean square value of the received signal is volts and the root mean square value of the noise volt volts.

Now, we have to distinguish the received signal of the amplitude volts in the presence of the noise amplitude volts.

As a matter of fact, the input signal variation of less than volts will not be distinguished at the receiver end.

Therefore, the number of the distinct levels that can be distinguished without error can be expressed as

M = …(9.51)

Thus, equation (9.51) expresses the maximum value of

*M*.

Now, the maximum amount of information carried by each pulse having distinct levels is given by

I = log2 = log

_{2}bits …(9.52)

Now, after establishing expression in equation (8.15), we can determine the channel capacity. In fact, the channel capacity is the maximum amount of information that can be transmitted per second by a channel.

If a channel can transmit a maximum of

*K*pulses per second, then, the channel capacity

*C*is given by

C = log

_{2}bits per second …(9.53)

DO YOU KNOW? |

The communication system is designed to reproduce at the receiver either exactly or approximately the message emitted by the source. |

Recall that for bandwidth requirements of PAM signals, it has been shown that a system of bandwidth nf* _{m}* Hz can transmit 2n f

*, independent pulses per second. Further, under these conditions, the received signal will yield the correct values of the amplitudes of the pulses but will not reproduce the details of the pulse shapes.*

_{m}Now, since, we are interested only in the pulse amplitudes and not their shapes, it is concluded that a system with bandwidth B Hz can transmit a maximum of 2B pulses per second. Further, since, each pulse can carry a maximum information of log

_{2}bits, if follows that a system of bandwidth B can transmit the information at a following maximum rate:

C = B log

_{2}bits per second …(9.54)

Therefore, the channel capacity

*C*is limited by the bandwidth of the channel (or system) and noise signal.

For a noiseless channel, N = 0 and the channel capacity will be infinite. However, practically,

*N*always finite and therefore, the channel capacity is finite.*

The expression in equation (9.54) is also known as the

**Hartley-Shannon law**and is treated as the central theorem of information theory.

From Hartley-Shannon law, it is obvious that the bandwidth and the signal power can be exchanged for one another. To transmit the information at a given rate, we may reduce, the signal power transmitted provided that the bandwidth is increased correspondingly. In a similar manner, o increase the signal power.

As a matter of fact, the process of modulation is actually a means of effecting this exchange between the bandwidth and the signal-to-noise ratio.**

**NOTE:**It may be noted that the channel capacity represents the maximum amount of information that can be transmitted by a channel per second. To achieve this rate of transmission, the information has to be processed properly or coded in the most efficient manner.