
Nonlinear feedforward networks with stochastic outputs: infomax implies redundancy reduction.
We prove that maximization of mutual information between the output and the input of a feedforward neural network leads to full redundancy reduction under the following sufficient conditions: (i) the input signal is a (possibly nonlinear) invertible mixture of independent components; (ii) there is no input noise; (iii) the activity of each output neuron is a (possibly) stochastic variable with a probability distribution depending on the stimulus through a deterministic function of the inputs (where both the probability distributions and the functions can be different from neuron to neuron); (iv) optimization of the mutual information is performed over all these deterministic functions. This result extends that obtained by Nadal and Parga (1994) who considered the case of deterministic outputs.
Duke Scholars
Published In
ISSN
Publication Date
Volume
Issue
Start / End Page
Location
Related Subject Headings
- Stochastic Processes
- Nonlinear Dynamics
- Neurons
- Neurology & Neurosurgery
- Neural Networks, Computer
- Information Theory
- Feedback
- 4601 Applied computing
- 3209 Neurosciences
- 3202 Clinical sciences
Citation

Published In
ISSN
Publication Date
Volume
Issue
Start / End Page
Location
Related Subject Headings
- Stochastic Processes
- Nonlinear Dynamics
- Neurons
- Neurology & Neurosurgery
- Neural Networks, Computer
- Information Theory
- Feedback
- 4601 Applied computing
- 3209 Neurosciences
- 3202 Clinical sciences