Huctuatiolls in the test error are important in thelaming theory of finite-dimensional systems as... more Huctuatiolls in the test error are important in thelaming theory of finite-dimensional systems as they represent how well the test error matches the average test emr. By explicitly finding the variance of the test ermr due to randomness present in both the data set and dgonthm for a linear perceptron of dimension n. we are able to address such questions as the optimal test set size. Where exact results were not tractable, a good approximation is given to the variance. We find thaf the optimal lest set size possesses a phase transition between linearmd 3 power-law scaling in the system size n.
In supervised learning, learning from queries rather than from random examples can improve genera... more In supervised learning, learning from queries rather than from random examples can improve generalization performance significantly. We study the performance of query learning for problems where the student cannot learn the teacher perfectly, which occur frequently in practice. As a prototypical scenario of this kind, we consider a linear perceptron student learning a binary perceptron teacher. Two kinds of queries for maximum information gain, i.e., minimum entropy, are investigated: Minimum student space entropy (MSSE) queries, which are appropriate if the teacher space is unknown, and minimum teacher space entropy (MTSE) queries, which can be used if the teacher space is assumed to be known, but a student of a simpler form has deliberately been chosen. We nd that for MSSE queries, the structure of the student space determines the e cacy of query learning, whereas MTSE queries lead to a higher generalization error than random examples, due to a lack of feedback about the progress of the student in the way queries are selected.
Huctuatiolls in the test error are important in thelaming theory of finite-dimensional systems as... more Huctuatiolls in the test error are important in thelaming theory of finite-dimensional systems as they represent how well the test error matches the average test emr. By explicitly finding the variance of the test ermr due to randomness present in both the data set and dgonthm for a linear perceptron of dimension n. we are able to address such questions as the optimal test set size. Where exact results were not tractable, a good approximation is given to the variance. We find thaf the optimal lest set size possesses a phase transition between linearmd 3 power-law scaling in the system size n.
We analyse natural gradient learning in a two-layer feed-forward neural network using a statistic... more We analyse natural gradient learning in a two-layer feed-forward neural network using a statistical mechanics framework which is appropriate for large input dimension. We find significant improvement over standard gradient descent in both the transient and asymptotic phases of learning.
Abstract The choice of training parameters and training rules is of great significance in on-line ... more Abstract The choice of training parameters and training rules is of great significance in on-line training of neural networks. We employ a variational method for determining globally optimal learning parameters and learning rules for on-line gradient descent training of multi-layer neural networks. The approach is based on maximizing the total decrease in generalization error over a fixed time-window, using a statistical mechanics description of the learning process. The method is employed for obtaining optimal learning rates in both ...
We analyse the matrix momentum algorithm, which provides an efficient approximation to on-line Ne... more We analyse the matrix momentum algorithm, which provides an efficient approximation to on-line Newton's method, by extending a recent statistical mechanics framework to include second order algorithms. We study the efficacy of this method when the Hessian is available and also consider a practical implementation which uses a single example estimate of the Hessian. The method is shown to provide excellent asymptotic performance, although the single example implementation is sensitive to the choice of training parameters. We conjecture that matrix momentum could provide efficient matrix inversion for other second order algorithms.
Philosophical Magazine B: Physics of Condensed Matter
A method for calculating the globally optimal learning rate in on-line gradient-descent training ... more A method for calculating the globally optimal learning rate in on-line gradient-descent training of multilayer neural networks is presented. The method is based on a variational approach which maximizes the decrease in generalization error over a given time frame. We demonstrate the method by computing optimal learning rates in typical learning scenarios. The method can also be employed when di erent learning rates are allowed for di erent parameter vectors as well as to determine the relevance of related training algorithms based on modi cations to the basic gradient descent rule.
General, theoretical, and mathematical biophysics (including logic of biosystems, quantum biology... more General, theoretical, and mathematical biophysics (including logic of biosystems, quantum biology, and relevant aspects of thermodynamics, information theory, cybernetics, and bionics). PACS. 02.50−r -Probability theory, stochastic processes, and statistics.
Abstract Using a statistical mechanical formalism we calculate the evidence, generalisation error... more Abstract Using a statistical mechanical formalism we calculate the evidence, generalisation error and consistency measure for a linear perceptron trained and tested on a set of examples generated by a non linear teacher. The teacher is said to be unrealisable ...
We show that in supervised learning from a particular data set Bayesian model selection, based on... more We show that in supervised learning from a particular data set Bayesian model selection, based on the evidence, does not optimise generalization performance even for a learnable linear problem. This is achieved by examining the finite size effects in hyperparameter ...
Many natural, technological and social systems are inherently not in equilibrium. We show, by det... more Many natural, technological and social systems are inherently not in equilibrium. We show, by detailed analysis of exemplar models, the emergence of equilibrium-like behavior in localized or nonlocalized domains within non-equilibrium systems as conjectured in some real systems. Equilibrium domains are shown to emerge either abruptly or gradually depending on the system parameters and disappear, becoming indistinguishable from the remainder of the system for other parameter values. The models studied, defined on densely and sparsely connected networks, provide a useful representation of many real systems.
We study noisy computation in randomly generated k-ary Boolean formulas. We establish bounds on t... more We study noisy computation in randomly generated k-ary Boolean formulas. We establish bounds on the noise level above which the results of computation by random formulas are not reliable. This bound is saturated by formulas constructed from a single majority-like gates.
Proceedings of the 6th Intl Symposium on Modeling and Optimization, 2008
Code Division Multiple Access (CDMA) in which the signature code assignment to users contains a r... more Code Division Multiple Access (CDMA) in which the signature code assignment to users contains a random element has recently become a cornerstone of CDMA research. The random element in the construction is particularly attractive in that it provides robustness and flexibility in application, whilst not making significant sacrifices in terms of multiuser efficiency. We present results for sparse random codes of two types, with and without modulation. Simple microscopic consideration on system samples would suggest differences in the phase space of the two models, but we demonstrate that the thermodynamic results and metastable states are equivalent in the minimum bit error rate detector. We analyse marginal properties of interactions and also make analogies to constraint satisfiability problems in order to understand qualitative features of the decoding and metastable states. This may have consequences for developing algorithmic methods to escape metastable states, thus improving decoding performance.
Natural gradient descent is a principled method for adapting the parameters of a statistical mode... more Natural gradient descent is a principled method for adapting the parameters of a statistical model on-line using an underlying Riemannian parameter space to redefine the direction of steepest descent. The algorithm is examined via methods of statistical physics which accurately characterize both transient and asymptotic behavior. A solution of the learning dynamics is obtained for the case of multilayer neural network training in the limit of large input dimension. We find that natural gradient learning leads to optimal asymptotic performance and outperforms gradient descent in the transient, significantly shortening or even removing plateaus in the transient generalization performance which typically hamper gradient descent training. 87.10.+e, 02.50.-r, 05.20.-y
Huctuatiolls in the test error are important in thelaming theory of finite-dimensional systems as... more Huctuatiolls in the test error are important in thelaming theory of finite-dimensional systems as they represent how well the test error matches the average test emr. By explicitly finding the variance of the test ermr due to randomness present in both the data set and dgonthm for a linear perceptron of dimension n. we are able to address such questions as the optimal test set size. Where exact results were not tractable, a good approximation is given to the variance. We find thaf the optimal lest set size possesses a phase transition between linearmd 3 power-law scaling in the system size n.
In supervised learning, learning from queries rather than from random examples can improve genera... more In supervised learning, learning from queries rather than from random examples can improve generalization performance significantly. We study the performance of query learning for problems where the student cannot learn the teacher perfectly, which occur frequently in practice. As a prototypical scenario of this kind, we consider a linear perceptron student learning a binary perceptron teacher. Two kinds of queries for maximum information gain, i.e., minimum entropy, are investigated: Minimum student space entropy (MSSE) queries, which are appropriate if the teacher space is unknown, and minimum teacher space entropy (MTSE) queries, which can be used if the teacher space is assumed to be known, but a student of a simpler form has deliberately been chosen. We nd that for MSSE queries, the structure of the student space determines the e cacy of query learning, whereas MTSE queries lead to a higher generalization error than random examples, due to a lack of feedback about the progress of the student in the way queries are selected.
Huctuatiolls in the test error are important in thelaming theory of finite-dimensional systems as... more Huctuatiolls in the test error are important in thelaming theory of finite-dimensional systems as they represent how well the test error matches the average test emr. By explicitly finding the variance of the test ermr due to randomness present in both the data set and dgonthm for a linear perceptron of dimension n. we are able to address such questions as the optimal test set size. Where exact results were not tractable, a good approximation is given to the variance. We find thaf the optimal lest set size possesses a phase transition between linearmd 3 power-law scaling in the system size n.
We analyse natural gradient learning in a two-layer feed-forward neural network using a statistic... more We analyse natural gradient learning in a two-layer feed-forward neural network using a statistical mechanics framework which is appropriate for large input dimension. We find significant improvement over standard gradient descent in both the transient and asymptotic phases of learning.
Abstract The choice of training parameters and training rules is of great significance in on-line ... more Abstract The choice of training parameters and training rules is of great significance in on-line training of neural networks. We employ a variational method for determining globally optimal learning parameters and learning rules for on-line gradient descent training of multi-layer neural networks. The approach is based on maximizing the total decrease in generalization error over a fixed time-window, using a statistical mechanics description of the learning process. The method is employed for obtaining optimal learning rates in both ...
We analyse the matrix momentum algorithm, which provides an efficient approximation to on-line Ne... more We analyse the matrix momentum algorithm, which provides an efficient approximation to on-line Newton's method, by extending a recent statistical mechanics framework to include second order algorithms. We study the efficacy of this method when the Hessian is available and also consider a practical implementation which uses a single example estimate of the Hessian. The method is shown to provide excellent asymptotic performance, although the single example implementation is sensitive to the choice of training parameters. We conjecture that matrix momentum could provide efficient matrix inversion for other second order algorithms.
Philosophical Magazine B: Physics of Condensed Matter
A method for calculating the globally optimal learning rate in on-line gradient-descent training ... more A method for calculating the globally optimal learning rate in on-line gradient-descent training of multilayer neural networks is presented. The method is based on a variational approach which maximizes the decrease in generalization error over a given time frame. We demonstrate the method by computing optimal learning rates in typical learning scenarios. The method can also be employed when di erent learning rates are allowed for di erent parameter vectors as well as to determine the relevance of related training algorithms based on modi cations to the basic gradient descent rule.
General, theoretical, and mathematical biophysics (including logic of biosystems, quantum biology... more General, theoretical, and mathematical biophysics (including logic of biosystems, quantum biology, and relevant aspects of thermodynamics, information theory, cybernetics, and bionics). PACS. 02.50−r -Probability theory, stochastic processes, and statistics.
Abstract Using a statistical mechanical formalism we calculate the evidence, generalisation error... more Abstract Using a statistical mechanical formalism we calculate the evidence, generalisation error and consistency measure for a linear perceptron trained and tested on a set of examples generated by a non linear teacher. The teacher is said to be unrealisable ...
We show that in supervised learning from a particular data set Bayesian model selection, based on... more We show that in supervised learning from a particular data set Bayesian model selection, based on the evidence, does not optimise generalization performance even for a learnable linear problem. This is achieved by examining the finite size effects in hyperparameter ...
Many natural, technological and social systems are inherently not in equilibrium. We show, by det... more Many natural, technological and social systems are inherently not in equilibrium. We show, by detailed analysis of exemplar models, the emergence of equilibrium-like behavior in localized or nonlocalized domains within non-equilibrium systems as conjectured in some real systems. Equilibrium domains are shown to emerge either abruptly or gradually depending on the system parameters and disappear, becoming indistinguishable from the remainder of the system for other parameter values. The models studied, defined on densely and sparsely connected networks, provide a useful representation of many real systems.
We study noisy computation in randomly generated k-ary Boolean formulas. We establish bounds on t... more We study noisy computation in randomly generated k-ary Boolean formulas. We establish bounds on the noise level above which the results of computation by random formulas are not reliable. This bound is saturated by formulas constructed from a single majority-like gates.
Proceedings of the 6th Intl Symposium on Modeling and Optimization, 2008
Code Division Multiple Access (CDMA) in which the signature code assignment to users contains a r... more Code Division Multiple Access (CDMA) in which the signature code assignment to users contains a random element has recently become a cornerstone of CDMA research. The random element in the construction is particularly attractive in that it provides robustness and flexibility in application, whilst not making significant sacrifices in terms of multiuser efficiency. We present results for sparse random codes of two types, with and without modulation. Simple microscopic consideration on system samples would suggest differences in the phase space of the two models, but we demonstrate that the thermodynamic results and metastable states are equivalent in the minimum bit error rate detector. We analyse marginal properties of interactions and also make analogies to constraint satisfiability problems in order to understand qualitative features of the decoding and metastable states. This may have consequences for developing algorithmic methods to escape metastable states, thus improving decoding performance.
Natural gradient descent is a principled method for adapting the parameters of a statistical mode... more Natural gradient descent is a principled method for adapting the parameters of a statistical model on-line using an underlying Riemannian parameter space to redefine the direction of steepest descent. The algorithm is examined via methods of statistical physics which accurately characterize both transient and asymptotic behavior. A solution of the learning dynamics is obtained for the case of multilayer neural network training in the limit of large input dimension. We find that natural gradient learning leads to optimal asymptotic performance and outperforms gradient descent in the transient, significantly shortening or even removing plateaus in the transient generalization performance which typically hamper gradient descent training. 87.10.+e, 02.50.-r, 05.20.-y
Uploads
Papers by David Saad