[SwarmFest2004] My submission (Can swarm based systems outperform other methods i n training neural networks?)

[Prokhorov, Danil (D.V.)]

Type: Research or Application
Format: Abstract (Poster is fine too).

Title: Can swarm based systems outperform other methods in training neural 
networks?

Danil V. Prokhorov
Research and Advanced Engineering
Ford Motor Company
2101 Village Rd., MD 2036
Dearborn, MI 48124
address@hidden

Particle swarm optimization (PSO) has been applied to a wide variety of 
problems since its inception in 1995 [1].  Yet, it seems to be a deficit of 
applications of PSO to neural network training problems, especially in cases of 
medium- and large-size networks (more than 1000 weights), large training data 
sets (more than 100,000 data vectors) and recurrent neural networks.

We are interested in efficient training methods for neural networks, especially 
those methods that scale well to problems requiring large data sets and 
recurrent neural networks.  We have developed the training methods based on the 
extended Kalman filter (EKF) algorithm and applied them successfully to many 
problems in system modeling and control using neural networks [2]-[4].  The EKF 
methods operate fundamentally in the pattern-by-pattern mode of data 
presentation (as opposed to the PSO which operates in the batch mode), although 
presenting training data in mini-batches (streams) has been found to be very 
effective [2].  The EKF training complexity scales roughly as O(square of 
number of weights).

Recently, there have been claims of superior behavior of PSO applied to simple 
neural network training problems (see, e.g., [5]).  On the contrary, our own 
research demonstrates that, while the PSO may be effective in comparison with 
simple gradient based algorithms like the standard gradient descent and other 
first-order techniques, it is substantially inferior to the EKF and, possibly, 
other more advanced methods, especially when dealing with complex problems like 
ones discussed in [3].  Having much more experience with the KF based 
techniques than with the PSO, we might well be unaware of the right set of 
tricks swarm researchers employ to deal with large-scale optimization problems. 
 However, it is also possible that that, at least partially, the reason behind 
the observed PSO disadvantage lies in its batch mode of operation and poorly 
understood initialization of particles for large optimization problems.

We wish to discuss our comparative results with those presented in [5] and 
offer to future PSO benchmark studies a couple of challenging problems for 
training recurrent neural networks already efficiently solved by the EKF.


[1]     J. Kennedy, RC Eberhart, and Y. Shi. Swarm Intelligence. San Francisco, 
Morgan
Kaufmann, 2001.

[2]     Feldkamp and Puskorius, "A Signal Processing Framework Based on Dynamic 
Neural Networks with Application to Problems in Adaptation, Filtering and 
Classification," Proc. IEEE, Vol. 86, No. 11, pp. 2259-2277, 1998.

[3]     Prokhorov, D., Feldkamp, L., and I. Tyukin, "Adaptive Behavior with 
Fixed Weights in Recurrent Neural Networks: An Overview," Proc. of 
International Joint Conference on Neural Networks (IJCNN), WCCI'02, Honolulu, 
Hawaii, May 2002.

[4]     D. Prokhorov, G. Puskorius, and L. Feldkamp, "Dynamical Neural Networks 
for Control," in J. Kolen and S. Kremer (Eds.) A Field Guide to Dynamic 
Recurrent Networks, IEEE Press, 2001. 

[5]     Gudise, V. G. and Venayagamoorthy, G. K. "Comparison of particle swarm 
optimization and backpropagation as training algorithms for neural networks." 
Proceedings of the IEEE Swarm Intelligence Symposium 2003 (SIS 2003), 
Indianapolis, Indiana, USA. pp. 110-117, 2003.