Entropy and information

expected value of the amount of information delivered by a message

measurement of how one probability distribution is different from a second, reference probability distribution

measure of dependence between two variables

physical principle that erasing one bit of information at temperature 𝑇 requires energy 𝑘𝑇ln(2)

In information theory and statistics, negentropy is used as a measure of distance to normality. It is also known as negative entropy or syntropy.

statistics

In information theory, the cross-entropy between two probability distributions p and q, over the same underlying set of events, measures the average number of bits needed to identify an event drawn from the set when the coding scheme used for the set is optimized for an estimated probability distribution q, rather than the true distribution p.

measure of relative information in probability theory and information theory

In information theory, perplexity is a measure of uncertainty for a discrete probability distribution. The perplexity of a fair coin toss is , and that of a fair die roll is ; and generally, for a probability distribution with exactly outcomes each having a probability of exactly , the perplexity is simply . But perplexity can also be applied to unfair dice, and to other non-uniform probability distributions. It can be defined as the exponentiation of the information entropy. The larger the perplexity, the less likely it is that an observer can guess the value which will be drawn from the dist

lossless data compression scheme that is independent of the specific characteristics of the medium

principle in Bayesian statistics

measure of information in probability and information theory

logarithmic quantity derived from the probability of a particular event

family of diversity measures generalising Shannon entropy

concept in information theory

generalization of Boltzmann-Gibbs entropy

information Theory

nonnegative extended real number that is a measure of the complexity of a topological dynamical system

subject of study in ergodic theory

entropy of a process with only two probable values

measure the amount of directed (time-asymmetric) transfer of information