Understanding entropy and its connection to probability distributions

Question

Entropy tells us about the "uncertainty" of a probability distribution, i.e. roughly how much information is needed to describe an event that is described by a probability distribution. With this view it is natural that the uniform distribution is the one that maximizes the entropy function. This is the information theory point of view of entropy.

On the other hand, in physics textbooks the second law of thermodynamics says that a system tends towards a a macrostate that maximizes the entropy. For a finite system this means choosing a macrostate whose corresponding microstates have the greatest multiplicity.

These two viewpoints should be in agreement, but I'm confused as they almost seem to be in contradiction with one another. For example, in physics the multiplicity graphs usually have a very sharp and localized peak, which is almost as far as a uniform distribution as you can get. How does the physics view of entropy agree with the idea of getting as closed to a uniform distribution as possible?

Answer 1

The difference comes from the distinction between microstates and macrostates. It is the distribution over the microstates that is uniform. However, macrostates encompass many microstates with varying degeneracy. This explains the large discrepancy in their probabilities and why the distribution is typically peaked at the expected value in an accordance to the law of large numbers.

A typical illustration of this is the Ehrenfest urn. Say you have $N$ particles that are in two possible urns (or abstractly, states). The $2^N$ detailed distribution of the $N$ particles among the two urns represent the possible microstates. You can assume to be equiprobable (simply by the microcanonical ensemble, appealing to max ent etc.), so with probability $2^{-N}$. But you have $N$ macrostates representing the global proportion of particles among the balls: namely $M$ balls in one urn and the $N-M$ in the other. These macrostates are not equiprobable. Their probability is now binomial: $2^{-N}\binom NM$. As $N\to\infty$, the distribution will therefore peak at $M=N/2$. You therefore have you two seemingly contradicting aspects: a uniform distribution of microstates, but a very peaked distribution of macrostates.

In fact, it turns out that for typical statistical models, the distribution of microstates is not exactly uniform (canonical ensemble, grand canonical ensemble etc.). However, in the thermodynamic limit, you have asymptotic equipartition. The distribution is essentially supported on a subset where at leading logarithmic behaviour the probability is uniform. This information alone is enough to calculate the "peakedness" (mathematically the large deviations) of the macrostates' distribution.