Why was the 'differential entropy' from information theory so named?

Question

The entropy of a distribution $p$ on a discrete set $\mathcal{X}$ is defined as $$H(p) = -\sum_{x \in \mathcal{X}} p_x \log p_x.$$ Shannon in his classic paper [1] defines the analogue for continuous distributions with density $p$ as (modern notation) $$h(p) = - \int_{\mathcal{X}} p(x) \log p(x).$$

However, Shannon never uses the term differential entropy in his paper. The earliest use of the term I could find was from this paper of Kolmogorov [2] from 1956, and I couldn't go further because the papers he cites are hard to find (which seems to be true for most of the old papers of the Russian school, not that I can read Russian :-/). I also couldn't just start going through the ISIT proceedings from '48-56, because IEEExplore doesn't make them easily accessible. I haven't attempted to check the Trans. Inf. Th. volumes from the early '50s yet. In a different direction, standard texts like Cover and Thomas don't mention the etymology either.

I have two questions:

1. When was the term differential entropy first used in an academic publication (or technical report etc.)?
2. Why is the term called differential entropy?

2 above is the main reason that I'm asking this question. From a discussion with a colleague today, we concluded that the term could be differential entropy merely because it involves densities (which are derivatives of distributions) and integration and all this calculus-y talk is enough to draw in a term like differential (which seems like an imperfect reason to us), or that it is because the way $h$ rigorously arieses is as the difference between the entropy of a quantised version of the random variable in question and a term that can be viewed as a (metric) entropy of the quantising mesh, taken in the limit as the mesh size goes to $0$ - the key point being it is a difference. However, I'm not sure when this argument arose, although I think this might be from some work of Khintchine in the late '50s, which would be around the right time for the nomenclature.

I'm interested, thus, in if the name arises from either of the above reasons, or from another one altogether.

---

[1] http://math.harvard.edu/~ctm/home/text/others/shannon/entropy/entropy.pdf

[2] https://pdfs.semanticscholar.org/0e02/b535c582f43140d34c5322d117f89be25799.pdf

P.S. - I'm not too familiar with the tags on this stackexchange, and relevant terms such as etymology and electrical-engineering were no goes, please feel free to edit the tags.

Answer 1

There is a curious discrepancy between the Russian and the English pages of Wikipedia on "differential entropy". The English page is rather vague on the origin, but the Russian page states definitively (in Google's translation):

"In information theory, the functional was heuristically introduced by C. Shannon, but he is not the author of the term "differential entropy". The term itself was introduced by A. N. Kolmogorov together with I. M. Gelfand and A. M. Yaglom and underlines the fact that this concept has a different meaning than the entropy of discrete distributions. They also obtained a rigorous derivation of differential entropy as the first term of the asymptotic expansion of entropy, in which the dependence on the distribution of a random variable is manifested."

Aside from the difficulty of understanding how one can introduce a term "together with", their source is a Kolmogorov-Gelfand-Yaglom report dated from 1958, while the OP quoted report is from 1956, and by Kolmogorov alone. Unfortunately, the English translation under the link is, shall we say, messed up. In the book Kolmogorov, Theory of Information and Theory of Algorithms (Google translating the Russian original) we read in a footnote on p.29:

"The more detailed exposition of the mathematical side of questions to which was dedicated my report in 1947 is separated into an article [2]. Now the analogous role is played by chapter II of this work, which reproduces, with small changes, the report read by B.V. Gnedenko on my behalf at the symposium on the information theory of the American institute of radioengineers (MIT, September 1956) and based in many parts on a report, read by me together with I.M. Gelfand and A.M. Yaglom at the All-Union mathematical congress in June 1956".

It is the congress that the Kolmogorov-Gelfand-Yaglom report is from, the USSR Academy of Sciences only got around to publishing it in 1958. Nonetheless, Yaglom in his book Probability and Information (p.230) does not use "differential entropy" at all, he calls it "$\epsilon$-entropy". The text on p. 40 of the Kolmogorov's book gives us the answer: it is to Kolmogorov that the naming credits go, and Kolmogorov alone (compare to the "translation" on p.102 of the linked 1956 report):

"In the proper sense of the word the entropy of an object $\xi$ with continuous distribution is always infinite. If the continuous signals can not nevertheless serve to transmit infinitely large information, it is only because they are always observed with limited accuracy. It is natural therefore, by specifying the accuracy of observation $\epsilon$, to define the corresponding to it “$\epsilon$-entropy” $H_{\epsilon}(\xi)$ of the object $\xi$. This is what was done by Shannon under the name of "speed of creating messages". Although the choice of a new name does not change the essence of the matter, I decided to offer such a renaming, which underlines the more broad interest of the notion and its deep analogy with the ordinary exact entropy.

Unfortunately, Kolmogorov is not terribly specific on the exact reason for choosing "differential". A more explicit explanation, to the effect that it reflects the "differential" nature of probability densities, is given, for example, in Tarasenko's Russian textbook Introduction to the Course on the Theory of Information (1963, p.77):

"In this notation $\epsilon$ does not appear explicitly, however, that does not mean that $H_{\epsilon}(x)$ has ceased to be a relative quantity. The number $H_{\epsilon}(x)$ is usually called the entropy of a continuous random variable. The difference with the discrete entropy values is underlined in the title: due to the connection of $H_{\epsilon}(x)$ to the differential law of probability distribution it is often referred to as differential entropy; the sometimes used term relative entropy indicates the conventionality, relativity of this characteristic.

Let us also point out that the "non-absoluteness" of the quantity $H_{\epsilon}(x)$ is compounded by the specificity of its calculation: generally speaking, one can set such a law of quantization and such a way of shrinking the quantization intervals to zero, that the limit of the corresponding sums will be different from $H_{\epsilon}(x)$. However, by setting once a way of calculating some characteristic and finding out its meaning, in the future one can use it for comparisons of various distributions. Therefore, the quantity $H_{\epsilon}(x)$, despite its relativity, is very important in the theory of information."