Re: Question

[Mark H. Butler]

RE: Shannon's information theory

Information theory seems to be an area fraught with misunderstandings. This
is surprising as it is actually fairly simple - most of the confusion stems
from the fact Shannon called it "information" theory.

Essentially,  it is concerned with the efficiency of an encoding system. If
we are sending information over a channel using an alphabet U containing
letters u1, u2 .. uM, where each letter has an associated probability p(u)
of occurance then we define the entropy of the channel as

H(U) = - (sum: T = 1 to m) P(uT) log P(uT)

So if we have an alphabet of 4 letters of equal probability then H(u) = 2
bits. For an alphabet of 4 letters with probabilities 0.5, 0.25, 0.125 and
0.125 H(u) = 1.75 bits. If all the letters have an equal probability of
occurance entropy is maximal. As variation in probability increases so does
redundancy as certain letters become likely. This means entropy decreases.
By comparing the actual entropy of an encoding with the maximal entropy
(i.e. with equal probabilites of each letter) we get an idea of the
redundancy in the system. Therefore as well as applying these techniques to
communication systems and data compression, people have applied them to DNA
sequences or emergent behaviour. However they are not some kind of "strong
Church thesis detector test" (this states that all processes process
information therefore we can simulate any process on a universal
information processor) i.e. just because we measure the entropy of a system
and we get a value it doesn't prove that the system is processing
information. The entropy value is a measure of the amount of information
conveyed in a single symbol we use to represent the system. 

As for the units of measurement, that depends on what base are logs are in.
We use base 2 and hence bits because most modern communication systems are
digital.

As for references I'd recommend undergraduate textbooks on communication
theory:

"Principles of digital and analog communications", Jerry D. Gibson
(MacMillan 1993)
"Telecommunications Engineering", J Dunlop & D G Smith (Chapman and Hall
1989)

To see examples of how this is applied to biosystems try

"Distributivity, a general information theoretic network measure or why the
whole is more than the sum of its parts", Roland Somoygi
(address@hidden) and Stefanie Fuhrman, Information Processing in
Cells and Tissues (ed. M. Holcombe and R. Paton, Plenum 1998)

Mark H. Butler
address@hidden                  http://www.csc.liv.ac.uk/~mhb
L'pool Biocomputation Group   http://www.csc.liv.ac.uk/~biocomp
Postgrad/Mature Student Soc http://www.csc.liv.ac.uk/~mhb/postgrads.html



                  ==================================
   Swarm-Modelling is for discussion of Simulation and Modelling techniques
   esp. using Swarm.  For list administration needs (esp. [un]subscribing),
   please send a message to <address@hidden> with "help" in the
   body of the message.
                  ==================================