The Berger-Tung inner and outer bounds give a approximation to the rate-distortion region in lossy distributed source coding. The regions are not tight in general. Their tightness is discussed throughout Chapter 12 of Network Information Theory by El Gamal and Kim (in particular Section 12.5), but even so I …
Source Encoders as Channels
It is well known that a rate-distortion-optimal source encoder's output generally doesn't match its source's distribution. This can make some analyses a pain in the neck. For example, say you want to investigate the relationship between a signal that appears in a source, and that signal's appearance in an encoding …
Some Useful Multivariate Gaussian Information Quantities
Many information quantities for multivariate normal distributions have nice closed forms. Their essential parts usually reduce to logs of minor determinant quotients so they combine nicely. Here's a list of them. All of them are somewhere in Adaptive Wireless Communications by Bliss and Govindasamy.
Notation
- [n]=\{1,\dots,n\}
Combinatoral Iteration in Matlab
Often we need a to write a program that iterates over all combinations of k elements from some set (elements chosen without replacement). It isn't obvious how to efficiently traverse through all these sets, although if bitwise arithmetic is readily available one can use Gosper's Hack. This kind of …
Some Interaction Information Examples
Interaction information between three random variables has a much less immediate interpretation compared to other information quantities. This makes it more tricky to work with. An i.i. term I(X;Y;Z) between X,\ Y and Z, could be either negative:
If …