Recurrent Neural Networks with Word Embeddings

**********************************************

Code - Citations - Contact

++++++++++++++++++++++++++

Directly running experiments is also possible using this `github repository <https://github.com/mesnilgr/is13>`_.

If you use this tutorial, cite the following papers:

Thank you!

Please email to `Grégoire Mesnil <http://www-etud.iro.umontreal.ca/~mesnilgr/>`_ for any

problem report or feedback. We will be glad to hear from you.

Recurrent Neural Network Model

++++++++++++++++++++++++++++++

A token corresponds to a word. Each token in the ATIS vocabulary is associated to an index. Each sentence is a

array of indexes (``int32``). Then, each set (train, valid, test) is a list of arrays of indexes. A python

'O', 'B-arrive_time.time_relative', 'B-arrive_time.time',

'I-arrive_time.time', 'I-arrive_time.time']

Given a sentence i.e. an array of indexes, and a window size i.e. 1,3,5,..., we

need to convert each word in the sentence to a context window surrounding this

To summarize, we started with an array of indexes and ended with a matrix of

indexes. Each line corresponds to the context window surrounding this word.

Once we have the sentence converted to context windows i.e. a matrix of indexes, we have to associate

these indexes to the embeddings (real-valued vector associated to each word).

sentence) of **context window word embeddings** which is easy to feed to a simple

recurrent neural network to iterate with.

The followin (Elman) recurrent neural network (E-RNN) takes as input the current input

(time ``t``) and the previous hiddent state (time ``t-1``). Then it iterates.

Training

++++++++

For stochastic gradient descent (SGD) update, we consider the whole sentence as a mini-batch

and perform one update per sentence. It is possible to perform a pure SGD (contrary to mini-batch)

where the update is done on only one single word at a time.

After each iteration/update, we normalize the word embeddings to keep them on a unit sphere.

Early-stopping on a validation set is our regularization technique:

the training is run for a given number of epochs (a single pass through the

whole dataset) and keep the best model along with respect to the F1 score

computed on the validation set after each epoch.

Although there is interesting research/`code

<https://github.com/JasperSnoek/spearmint>`_ on the topic of automatic

...

('BEST RESULT: epoch', 57, 'valid F1', 97.23, 'best test F1', 94.2, 'with the model', 'rnnslu')

Running experiments on ATIS using this `repository <https://github.com/mesnilgr/is13>`_

will run one epoch in less than 40 seconds on i7 CPU 950 @ 3.07GHz using less than 200 Mo of RAM::

[learning] epoch 44 >> 100.00% completed in 35.31 (sec) <<

[...]

We can check the k-nearest neighbors of the learned embeddings. L2 and

cosine distance gave the same results so we plot them for the cosine distance.