Quantifying the Flow of Information

Understanding and quantifying information transmission is crucial for improving and analyzing biological and engineered systems. Most, if not all, information-processing systems process signals that vary in time. Quantifying information transmission in time-varying signals is challenging due to the high dimensionality of trajectory space. This thesis introduces Path Weight Sampling (PWS), a novel Monte Carlo framework to address this problem. PWSmakes it possible to exactly calculate trajectory mutual information for any system described by a dynamical stochastic model.
To demonstrate the power of PWS we applied it to a widely used model of the bacterial chemotaxis system. We compared the information rate as computed via PWS to a recent experimental work which computed the information rate using an approximation and found significant discrepancies. These discrepancies were resolved by adapting our model to the experimental data. The revised model suggests a different receptor clustering structure in E. coli than previously believed. The study also found that for bacteria swimming in shallow gradients, the PWS result matches the Gaussian approximation of the information rate.
The Gaussian approximation for the mutual information rate is widely used in practice, yet it relies on assumptions of linear dynamics and additive Gaussian noise which are often violated in physical or biological systems. To assess the accuracy of the Gaussian approximation, we performed two case studies with a discrete linear system and a continuous diffusive system with nonlinearity. Our results present instances where the Gaussian approximation fails, emphasizing the need for exact methods like PWS.
While PWS does not suffer from the limitations of the Gaussian approximation, it requires a stochastic model of the system of interest, which is often unavailable. To address situations where a stochastic model is unavailable, the thesis proposes ML-PWS, a variant of PWS which learns a data-driven stochastic model from experimental time-series data. ML-PWS enables accurate mutual information rate computation for nonlinear systems directly from data, outperforming the Gaussian approximation in such cases.