Target audience:

Theorists (computer scientists, statisticians, physicists) working on algorithmic and conceptual questions related to unbiased estimation of entropies and similar information-theoretic quantities in undersampled regimes; neuroscientists and other experimentalists interested in learning about these methods and in applying them to the analysis of experimental data.

Workshop description:

In the middle of the twentieth century it was noted that simple attempts to estimate Shannon's entropy and other information-theoretic quantities using samples from unknown probability distributions lead to strongly biased results. Now, fifty years after these observations, the problem is not resolved, and it has become clear that, at least in the case of severely undersampled distributions, no general way to precisely estimate the entropy exists. Much of the effort has been focused on finding the gentlest assumptions about the underlying probability distributions that will allow one to make unbiased guesses regarding their information characteristics. In the recent years with the proliferation of information-theoretic methods in life sciences, notably in neuroscience, existence of such methods and understanding their potentialities and limitations has become crucial. As a result, many new and interesting entropy estimators appeared in the past decade.

The aim of this workshop is to bring together scientists working on theoretical aspects of entropy and information estimation and those experimentalists who may benefit from using these methods. We expect to have a broad range of approaches discussed, including (a) traditional compression-based (using repetitions in the spirit of Lempel-Ziv), (b) frequentist statistics, (c) various Bayesian approaches, (d) methods based on empirically observed properties of various probability distributions. Further, we will also discuss (e) negative results for entropy and information estimation, (f) PAC-type bounds on the estimates, (g) computational complexity of entropy approximation. Finally, some of the speakers will present results of applications of various methods to different neurophysiological data and compare the methods against each other.

• Morning section (7:30 - 10:30am).
  • 7:30 - 7:55 Ilya Nemenman, UCSB. Entropy estimation: A general overview.
  • 7:55 - 8:20 Liam Paninski, NYU. Estimating entropy on m bins given fewer than m samples.
  • 8:20 - 8:45 Jose Costa, UMich, Ann Arbor. Applications of entropic graphs in nonparametric estimation.
  • 8:45 - 8:55 Coffee break
  • 8:55 - 9:20 Jonathan Victor, Cornell. Metric-space approach to information calculations, with application to single-neuron and multineuronal coding in primary visual cortex.
  • 9:20 - 9:45 Jon Shlens and Matt Kennel, UCSD. Estimating Entropy Rates and Information Rates in Retinal Spike Trains.
  • 9:45 - 10:30 Discussion.
• Midday (10:30am - 4:00pm). Skiing, snowboarding, and other recreational activities at the Whistler-Blackcomb resort.
• Afternoon section (4:00 - 7:00pm).
  • 4:00 - 4:25 William Bialek, Princeton. Entropy and information in spike trains: Why are we interested and where do we stand?
  • 4:25 - 4:50 Ronitt Rubinfeld, NEC Laboratories America. The computational complexity of approximating the entropy.
  • 4:50 - 5:15 Yun Gao, Brown University. Lempel-Ziv Entropy Estimators and Spike Trains.
  • 5:15 - 5:25 Coffee break.
  • 5:25 - 5:50 Pamela Reinagel, UCSD. Application of some entropy estimation methods to large experimental data sets from LGN neurons.
  • 5:50 - 6:15 Gal Chechik, Stanford. Information bearing elements and redundancy reduction in the auditory pathway.
  • 6:15 - 7:00 Discussion.