1

Analysis of Steady-States of Continuous-Time

Impulse K-Winners-Take-All Neural Network

Pavlo Tymoshchuk
L’viv Polytechnic National University
pavlo.v.tymoshchuk@lpnu.ua

I. INTRODUCTION
K-winners-take-all (KWTA) neural networks (NNs) define the K
maximal out of N inputs, where 1  K  N is a positive integer [1] – [3]. In
the particular case when K = 1, the KWTA network becomes a winner-
takes-all (WTA) network [4], [5]. WTA and KWTA neural networks are
related with simultaneous recurrent NNs which require fulfilling iterations to
converge to steady-states [6].
KWTA NNs are used in different applications, especially, for pattern
recognition, in processing data and signals, in making decisions, in
competitive learning, and in parallel sorting and filtering. The KWTA
networks are used in telecommunications and vision systems, for image
processing, classifying, clustering, decoding, mobile robot navigation, and
feature extraction. KWTA tools are applied for networking cognitive
phenomena and impulse neural networks [3].
Various NNs have been proposed for obtaining solutions the WTA
and KWTA problems. For instance, a design, implementation and
verification of WTA network in a number of CMOS integrated circuits is
presented in [7]. Continuous-time KWTA NNs realized in analog hardware
are faster, have smaller size and larger power-efficiency comparatively to
digital counterparts [8], [9].
Neurons can be modelled based on impulse networks. An impulse
can be described by Dirac delta function or by exponential decay. Trains of
impulses and exponential decays are employed to model the potential of a
neuron membrane. The sum of exponential decays between spikes and delta
functions are used to describe a level of activity that averages in time the
influences of postsynaptic spikes on the neuron. Impulse neurons are applied
to model psychophysical data with a laminar cortical network. Impulse-
based computation, caused by networks of computation in the central
nervous system, are capable to provide essential performance advantages

CADMD 2018, October 19-20, 2018, Lviv, UKRAINE

 2

comparatively to traditional methods for specific kinds of large-scale

problems [10].
In this paper, a train of impulses defined by the sum of Dirac delta
functions is applied in continuous-time KWTA NN. For this reason, in
contrast to other competitors for which a trajectory of the state variable of
the network to the KWTA operation has continuous nonlinear, continuous
piecewise-linear or continuous linear shape, a state variable trajectory of the
network to the KWTA operation has stepping form. Therefore, the
theoretical speed of convergence of state variable trajectories of the network
to the KWTA operation goes to infinity if the period of firing impulse
approximates zero. This provides a capability of the NN to choose
instantaneously without transient modes, the K maximal out of N inputs.
This is the principal advantage of the network. There is analyzed an
existence and uniqueness of the network steady-states. Computer
simulations illustrating and confirming theoretical results are given.

II. PROBLEM FORMULATION


Let us consider a vector of inputs a  an1 , an2 ,...,anN T  n ,
1  N   elements of which are unknown but have finite values. The
inputs have different values and can be disposed in a descending order of
quantity meeting the inequalities
  an1  an2  ...  anN   , (1)
where n1 , n2 ,...,n N are the unknown numbers of the first maximal input, the
second maximal input and so on up to the Nth maximal input. It is required
to investigate existence and uniqueness of steady-states of the network that
can define instantaneously without transient modes, the K maximal of these
inputs, which are referred to as the winners. The designed network should
process the vector of inputs a to obtain a vector of outputs

b  bn1 ,bn2 ,...,bnN T such that the following KWTA property holds [1], [3]:
bni  0 ,i  1,2,...,K ; bn j  0 , j  K  1, K  2,...,N . (2)
It should be also possible to determine the KWTA operation as
follows [11]:
dni  1,i  1,2,...,K; dn j  0, j  K  1,K  2,...,N. (3)
Note that in the case of using outputs (3) of the network only K
winners out of N inputs will be determined. No information is obtained on
the ordering of inputs by value which can be employed further, for example,

CADMD 2018, October 19-20, 2018, Lviv, UKRAINE

 3

for obtaining solutions of such problems as clustering, classification, etc.

[3].

III. IMPULSE CONTINUOUS-TIME KWTA NEURAL NETWORK

Consider the continuous-time KWTA NN described in [3] that is
governed by the following state equation:
dx m
 rD( x )   ( t  tl ) (4)
dt l 0
and by output equation
bnk  ank  x, k  1,2,...,N , (5)
where
N
D( x )   S k ( x )  K (6)
k 1
is a function of difference between obtained and required quantity of
positive outputs,
1, if ank  x  0;
Sk ( x )   (7)
0 , otherwise
is a step activation function,
  , if t  tl ;
 t  tl    (8)
 0 , if t  tl
m
is an impulse shaped as the Dirac delta function,   ( t  tl ) is a train of
l 1
impulses, tl is a lth time instant of firing impulses, m is a number of
impulses necessary to reach a convergence of search process to the KWTA
operation, and r is a resolution of the network.
For the continuous-time KWTA NN described by state equation (4)
and by output equation (5), the KWTA operation (3) can be defined as
follows:
d k ( x )  Sk ( x ) , (9)
where k=1,2,…,N.
In contrast to other competitors with continuous nonlinear trajectory
of the state variable x, the network described by state equation (4) and by
output equation (5) presents stepping state trajectory of x. Therefore, it can
reach theoretically any finite speed of processing inputs defined by the
period of firing impulses. If this period tends to zero, a processing time of
inputs of the network also goes to zero. This is main advantage of the

CADMD 2018, October 19-20, 2018, Lviv, UKRAINE

 4

network. A practical speed of processing input vector of the network is

limited by limitations of its software or hardware realization. In particular, a
software realization of the network has limited computational accuracy. On
the other hand, a hardware realization of the network, the restrictions are
finite speed of comparator, non-idealities of integrator, mismatch, etc.

IV. EXISTENCE AND UNIQUENESS OF STEADY-STATES

OF THE NETWORK
We analyze the existence and uniqueness of the KWTA steady-states
of the state equation (4) solutions. Integrating both sides of this equation
gives
t m
x( t )   rD( x )   (   tl )d  x0  r  D( xl ) H (t  tl ),
m
(10)
t0 l 0 l 0

where
 1, if t  tl  0;

H ( t  tl )  [ 0 ,1 ], if t  t1  0;
 0, if t  tl  0

is a Heaviside step activation function. As the terms x0 , r , D( xl ) and
H (t  tl ) of the solution (10) are finite, x(t ) is finite as well. In the KWTA
steady-states the solution (10) should meet the inequalities
m
aK 1  x0  r  D( xl )H (t  tl )  aK . (11)
l 0
Let us rewrite the inequalities (11) in the following shape:
aK 1  x0  rI  aK , (12)
m
where I   D( xl )H( t  tl ) is an integer number. There exists an integer I
l 0

such that inequalities (12) hold under the bounds 0  r  min ani  an j ,
i  j  1,2,...,N for arbitrary distinct inputs (1) and for any finite initial
condition x0 . This means that a constant number x*   exists, such that
N
aK 1  x*  aK . For x* , equality D( x ) 
*
 S k ( x* )  K  0 means that
k 1

x* is a steady-state solution of the equation (4). Taking into account (2) and
(5), it follows that steady-state KWTA outputs bnk  ank  x* , k=1,2,3,…,N

CADMD 2018, October 19-20, 2018, Lviv, UKRAINE

 5

of the network described by state equation (4) and by output equation (5)
exist. This implies that x* is a KWTA steady-state solution of the state
equation (4).
It is necessary to note that the steady-state solution of the state
equation (4) can accept arbitrary finite value in the region aK 1  x*  aK
N
meeting the equality D( x* )   S k ( x* )  K  0 . The solution x* is not
k 1

unique as this equality is satisfied for each x*  aK 1 , aK  . It follows from

this fact that the outputs bnk  ank  x* , k=1,2,…,N are also not unique.
However, the KWTA property (2) of the network described by state
equation (4) band output equation (5) is defined by signs of outputs bnK
rather than by their values. These signs are unique for any aK 1  x*  aK .
Therefore, the network described by state equation (4) and by output
equation (5) has a unique KWTA property (2).

V. COMPUTER SIMULATIONS
Let us consider example with corresponding computer simulations
which illustrate the performance of the continuous-time impulse KWTA
neural network described by state equation (4) and by output equation (5).
We set the input vector a= 1.4 ,3.1,2.3,9.2,10,7.6 , 5.7 ,4.8 ,6.9 ,8.5 T , i. e.
N=10, K=7, r  0.5  min ani  an j , i  j  1,2,...,10 and the initial value
of the state variable x0  0 . A 1.81 GHz desktop PC and Euler solver of
non-stiff ordinary differential equations in the Simulink environment of
Matlab program (ODE1) of the fixed-step size 1  10 11 are employed.
Impulse source is realized by sequential connection of pulse generator,
differentiator and absolute value blocks.
Fig. 1 shows, in normalized units, the train of m=4 impulses fired
with the period τ=0.15 ms. In Fig. 2, the dynamics of elements of output
vector b of the KWTA NN with impulse train for   0.5 ns are presented.
As one can see from this figure, the trajectories of the network outputs
converge to unique KWTA steady-states starting from their initial values by
using four impulses after 2 ns.

CADMD 2018, October 19-20, 2018, Lviv, UKRAINE

 6

Fig. 1. Train of m=4 impulses (8) fired with the period τ=0.15 ms of the KWTA NN
described by state equation (4) and by output equation (5).

Fig. 2. Dynamics of elements of output vector b of the network for   0.5 ns .

VI. CONCLUSION
This paper describes a continuous-time KWTA NN that contains
impulse train. The network is capable of defining K maximal among
arbitrary finite value N unknown different inputs positioned in an unknown
region, where 1  K  N . There is analyzed existence and uniqueness of
steady-states of the network. In contrast to other comparable competitors,
state variable trajectories of the network have piecewise-constant, i. e.
stepping shape. Therefore, steady-state KWTA operation of the network can
be reached theoretically instantaneously as the period of firing impulses
approximates zero. Thus, the network can define the K maximal among N
inputs without transient dynamics, i.e. instantaneously.

REFERENCES
[1] E. Majani, R. Erlanson, and Y. Abu-Mostafa, “On the k-winners-take-all
network,” in Advances in Neural Information Processing Systems 1, R.

CADMD 2018, October 19-20, 2018, Lviv, UKRAINE

 7

P. Lippmann, J. E. Moody, and D. S. Touretzky, Eds. San Mateo, CA:

Morgan Kaufmann, 1989, pp. 634-642.
[2] J. Wang, “Analysis and design of a k-winners-take-all network with a
single state variable and the Heaviside step activation function,” IEEE
Trans. Neural Networks, vol. 21, no. 9, pp. 1496-1506, Sept. 2010.
[3] P. Tymoshchuk, “Stability of impulse K-Winners-Take-All neural
network,” Computer Systems of Design. Theory and Practice, No 882,
pp. 90-98, 2017.
[4] R. P. Lippmann, “An introduction to computing with neural nets,” IEEE
Acoustics, Speech and Signal Processing Magazine, vol. 3, no. 4, pp. 4-
22, Apr. 1987.
[5] P.Tymoshchuk and E.Kaszkurewicz, ”A winner-take all circuit using
neural networks as building blocks,” Neurocomputing, vol. 64, pp.
375-396, Mar. 2005.
[6] X. Cai, D. Prokhorov, and D. Wunsch, “Training winner-take-all
simultaneous recurrent neural networks,” IEEE Trans. Neural Networks,
vol. 18, no. 3, pp. 674-684, May 2007.
[7] J. Lazzaro, S. Ryckebusch, M. A. Mahowald, and C. A. Mead, “Winner-
take-all networks of O(N) complexity,” in Advances in Neural
Information Processing Systems 1, R. P. Lippmann, J. E. Moody, and D.
S. Touretzky, Eds. San Mateo, CA: Morgan Kaufmann, 1989, pp. 703-
711.
[8] B. Sekerkiran and U. Cilingiroglu, “A CMOS K-winners-take-all circuits
with 0(N) complexity,” IEEE Trans. Circuits Systems II, vol. 46, no. 1,
pp. 1-5, Jan. 1999.
[9] A. Cichocki and R. Unbehauen, Neural Networks for Optimization and
Signal Processing. New York, NY, USA: Wiley, 1993.
[10] R. C. O’Reilly and Y. Munakata, Computational Explorations in
Cognitive Neuroscience: Understanding the Mind by Simulating the
Brain. Cambridge, MA: MIT Press, 2000.
[11] W. Maass, “Neural computation with winner-take-all as the only
nonlinear operation,” in Advances in Information Processing Systems,
vol. 12, S. A. Solla, T. K. Leen, and K.-R. Mueller, Eds. Cambridge,
MA: MIT Press, 2000, pp. 293–299.

CADMD 2018, October 19-20, 2018, Lviv, UKRAINE