Content deleted Content added
No edit summary |
m Dating maintenance tags: {{Cn}} |
||
| (45 intermediate revisions by 33 users not shown) | |||
Line 1:
{{Not to be confused with|Stochastic gradient descent#RMSProp}}
{{Short description|Learning heuristic for supervised learning}}
'''Rprop''', short for resilient [[backpropagation]], is a learning [[heuristics|heuristic]] for [[supervised learning]] in [[Feedforward neural network|feedforward]] [[artificial neural network]]s. This is a [[First-order approximation|first-order]] [[optimization (mathematics)|optimization]] [[algorithm]]. This algorithm was created by Martin Riedmiller and Heinrich Braun in 1992.<ref>Martin Riedmiller und Heinrich Braun: Rprop - A Fast Adaptive Learning Algorithm. Proceedings of the International Symposium on Computer and Information Science VII, 1992</ref>
Next to the [[Cascade correlation algorithm]] and the [[Levenberg-Marquardt algorithm]], Rprop is one of the fastest weight update mechanisms. ▼
Similarly to the [[Manhattan update rule]], Rprop takes into account only the [[Sign (mathematics)|sign]] of the [[partial derivative]] over all patterns (not the magnitude), and acts independently on each "weight". For each weight, if there was a sign change of the partial derivative of the total error function compared to the last iteration, the update value for that weight is multiplied by a factor ''η''<sup>−</sup>, where ''η''<sup>−</sup> < 1. If the last iteration produced the same sign, the update value is multiplied by a factor of ''η''<sup>+</sup>, where ''η''<sup>+</sup> > 1. The update values are calculated for each weight in the above manner, and finally each weight is changed by its own update value, in the opposite direction of that weight's partial derivative, so as to minimise the total error function. ''η''<sup>+</sup> is empirically set to 1.2 and ''η''<sup>−</sup> to 0.5.{{cn|date=August 2022}}
Rprop can result in very large weight increments or decrements if the gradients are large, which is a problem when using mini-batches as opposed to full batches. [[RMSprop]] addresses this problem by keeping the moving average of the squared gradients for each weight and dividing the gradient by the square root of the mean square.{{cn|date=June 2024}}
▲RPROP is a [[batch update algorithm]]. Next to the [[
==Variations==
Martin Riedmiller developed three algorithms, all named RPROP. Igel and Hüsken assigned names to them and added a new variant:<ref name="improving">Christian Igel and Michael Hüsken. [http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.17.1332 Improving the Rprop Learning Algorithm]. Second International Symposium on Neural Computation (NC 2000), pp. 115-121, ICSC Academic Press, 2000</ref>
<ref name="evaluation">Christian Igel and Michael Hüsken. [http://citeseerx.ist.psu.edu/viewdoc/download?rep=rep1&type=pdf&doi=10.1.1.102.1273 Empirical Evaluation of the Improved Rprop Learning Algorithm]. Neurocomputing 50:105-123, 2003</ref>
# RPROP+ is defined at [http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.21.1417 A Direct Adaptive Method for Faster Backpropagation Learning: The RPROP Algorithm].<ref>Martin Riedmiller and Heinrich Braun. [http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.21.1417 A direct adaptive method for faster backpropagation learning: The Rprop algorithm]. Proceedings of the IEEE International Conference on Neural Networks, 586-591, IEEE Press, 1993</ref>
# RPROP− is defined at [http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.27.7876 Advanced Supervised Learning in Multi-layer Perceptrons – From Backpropagation to Adaptive Learning Algorithms]. Backtracking is removed from RPROP+.<ref>Martin Riedmiller. [http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.27.7876 Advanced supervised learning in multi-layer perceptrons - From backpropagation to adaptive learning algorithms]. Computer Standards and Interfaces 16(5), 265-278, 1994</ref>
# iRPROP− is defined in [http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.21.3428 Rprop – Description and Implementation Details]<ref>Martin Riedmiller. [http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.21.3428 Rprop – Description and Implementation Details]. Technical report, 1994</ref> and was reinvented by Igel and Hüsken.<ref name=evaluation /> This variant is very popular and most simple.
# iRPROP+ is defined at [http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.17.1332 Improving the Rprop Learning Algorithm] and is very robust and typically faster than the other three variants.<ref name="improving"/><ref name="evaluation"/>
== References ==
{{Reflist}}
==External links==
*[http://www.ias.informatik.tu-darmstadt.de/Research/RpropToolbox Rprop Optimization Toolbox]
*[http://www.mathworks.com/matlabcentral/fileexchange/32445-rprop Rprop training for Neural Networks in MATLAB]
[[Category:
[[Category:
| |||