I too would like to butt in here with the observation that both of the
two entropies listed above are really just two different applications of
the *same* underlying general abstract concept applied to two different
situations. The entropies of Gibbs and of Shannon are both special cases
of the entropy concept of Bayesian probability theory. That is, entropy
is a measure of the uncertainty about the outcome of a random process
characterized by a given probability distribution. Each probability
distribution possesses an entropy and entropy is a functional on the space
of probability distributions, and, as such, is, therefore, a statistic.
The explicit formula for the entropy S of any distribution {p_r} is:
S = SUM_r{p_r*log(1/p_r)}, where the sum is over all allowed values of
the index r labeling the outcomes of the distribution whose probability
for the rth outcome is p_r. The base to which the logarithm is taken in
this formula determines the units in which the entropy is to be measured.
(If the entropy is to be measured in bits then the base of the
logarithms is 2; for bytes the base is 256.) The general interpretation of
this formula is that the entropy of a given distribution is: *the expected
minimal amount of further information necessary to determine, with
certainty, the exact outcome of the random process characterized by that
distribution, given only the initial information about the outcome
contained in that distribution*.
In the special case of Shannon's communication theory the probability
distribution in question is for the probability of each message to be
sent from an ensemble of possible messages. In the special case of
statistical thermodynamics the meaning of the relevant probability
distribution {p_r} is such that p_r is the probability that a given
macroscopic thermodynamic system is in microscopic state r when the
system has a given specified macroscopic state. Each macroscopic
state of the system has its own distribution of possible microscopic states
consistent with that macroscopic state, and thus the system's entropy is
a function of the system's macro(scopic) state (because S is determined
by {p_r} and {p_r} is determined by the system's macroscopic description).
In the special case of statistical mechanics the underlying meaning of the
entropy of a macroscopic physical system is that it is *the average minimal
amount (in appropriate units) of information necessary to determine the
exact microscopic state that the system is in given only that system's
macroscopic description*. This is the essence of the quantity that
Clausius first coined the term entropy for when it was recognized that
that for reversible quasistatic changes some function of the macroscopic
state changes by the integral over the path of changes the amount of heat
absorbed divided by the system's absolute temperature. The 2nd law of
thermodynamics only necessarily applies as a law of nature (rather than
some kind of metaphor) to the special case the entropy of Clausius/Maxwell/
Boltzmann/Gibbs. Other entropy measures which are not defined on the space
of probability measures on the space microscopic states of, usually
isolated, (unless further qualifications are made) physical macroscopic
systems do not necessarily have to have the nondecreasing-with-time
property.
Most other entropy measures used in physics (such as the Sinai-Kolmogorov
entropy in dynamical systems, and various entropies used in quantum
measurement theory) are various special-case applications of the general
SUM{plog(1/p)} formula applied to the special purposes at hand. Each
application has its own way of defining its own relevant probability
distributions on its own relevant probability space, however. Other
'entropies' (found in the literature) which are not reducible to versions
of this above formula are, IMO, unfortunate abuses of the term 'entropy'.
(I think John von Neumann would agree with me here.)
> In a
>discussion on bionet.info-theory, one of the regulars
>did a quick literature search and found literally hundreds
>of entropies.
I hope that most of these bio-info-theory entropy definitions are
reducible as special cases of the above formula applied to just
different probability distributions on different probability spaces.
>Now to the point, Tom Ray uses an entropy to provide
>an objective measure of diversity in his Tierra World
>simulation of Darwinian evolution:
>... <SNIP Ray's abstract>
>The entropy referred to above has the same form as the
>thermodynamic and Shannon entropies, i.e. it is the
>negative sum of p log(p). In this case p refers to
>the proportion of the total community occupied by
>each genotype.
I'm glad to see that Tom Ray is using the term entropy appropriately. :)
> Ray points out that this entropy is
>a measure of the diversity of the community. To explain
>why this is the case, consider first the case where
>there are 100 genotypes but only one of these occurs
>say 80% of the time. This would correspond to low
>diversity and also, according to the formula
>- sum [p log(p)], to low entropy. The case of highest
>diversity, and also highest entropy, would be where
>all possibilities occurred at the same frequency.
>
>Thus, increasing diversity leads to increasing entropy.
>
>So, here we have scientific proof that diversity leads
>to disorder. This is based on one of the most fundamental
>laws of science, the second law of thermodynamics.
No. Its true, but its based on the meaning of the definition of the term
'entropy'. The results of Ray's computer simulations of evolution may
indicate that for a given simulated ecosystem the diversity (as measured by
his entropy) tend to increase with time. But this result is *analogous to*
the 2nd law; it is not based on it.
David Bowman
dbowman@gtc.georgetown.ky.us