pytorch/sequence_models_tutorial.ipynb

{
  "cells": [
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "%matplotlib inline"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "\nSequence Models and Long-Short Term Memory Networks\n===================================================\n\nAt this point, we have seen various feed-forward networks. That is,\nthere is no state maintained by the network at all. This might not be\nthe behavior we want. Sequence models are central to NLP: they are\nmodels where there is some sort of dependence through time between your\ninputs. The classical example of a sequence model is the Hidden Markov\nModel for part-of-speech tagging. Another example is the conditional\nrandom field.\n\nA recurrent neural network is a network that maintains some kind of\nstate. For example, its output could be used as part of the next input,\nso that information can propogate along as the network passes over the\nsequence. In the case of an LSTM, for each element in the sequence,\nthere is a corresponding *hidden state* $h_t$, which in principle\ncan contain information from arbitrary points earlier in the sequence.\nWe can use the hidden state to predict words in a language model,\npart-of-speech tags, and a myriad of other things.\n\n\nLSTM's in Pytorch\n~~~~~~~~~~~~~~~~~\n\nBefore getting to the example, note a few things. Pytorch's LSTM expects\nall of its inputs to be 3D tensors. The semantics of the axes of these\ntensors is important. The first axis is the sequence itself, the second\nindexes instances in the mini-batch, and the third indexes elements of\nthe input. We haven't discussed mini-batching, so lets just ignore that\nand assume we will always have just 1 dimension on the second axis. If\nwe want to run the sequence model over the sentence \"The cow jumped\",\nour input should look like\n\n\\begin{align}\\begin{bmatrix}\n   \\overbrace{q_\\text{The}}^\\text{row vector} \\\\\n   q_\\text{cow} \\\\\n   q_\\text{jumped}\n   \\end{bmatrix}\\end{align}\n\nExcept remember there is an additional 2nd dimension with size 1.\n\nIn addition, you could go through the sequence one at a time, in which\ncase the 1st axis will have size 1 also.\n\nLet's see a quick example.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "# Author: Robert Guthrie\n\nimport torch\nimport torch.autograd as autograd\nimport torch.nn as nn\nimport torch.nn.functional as F\nimport torch.optim as optim\n\ntorch.manual_seed(1)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "lstm = nn.LSTM(3, 3)  # Input dim is 3, output dim is 3\ninputs = [autograd.Variable(torch.randn((1, 3)))\n          for _ in range(5)]  # make a sequence of length 5\n\n# initialize the hidden state.\nhidden = (autograd.Variable(torch.randn(1, 1, 3)),\n          autograd.Variable(torch.randn((1, 1, 3))))\nfor i in inputs:\n    # Step through the sequence one element at a time.\n    # after each step, hidden contains the hidden state.\n    out, hidden = lstm(i.view(1, 1, -1), hidden)\n\n# alternatively, we can do the entire sequence all at once.\n# the first value returned by LSTM is all of the hidden states throughout\n# the sequence. the second is just the most recent hidden state\n# (compare the last slice of \"out\" with \"hidden\" below, they are the same)\n# The reason for this is that:\n# \"out\" will give you access to all hidden states in the sequence\n# \"hidden\" will allow you to continue the sequence and backpropogate,\n# by passing it as an argument  to the lstm at a later time\n# Add the extra 2nd dimension\ninputs = torch.cat(inputs).view(len(inputs), 1, -1)\nhidden = (autograd.Variable(torch.randn(1, 1, 3)), autograd.Variable(\n    torch.randn((1, 1, 3))))  # clean out hidden state\nout, hidden = lstm(inputs, hidden)\nprint(out)\nprint(hidden)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Example: An LSTM for Part-of-Speech Tagging\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nIn this section, we will use an LSTM to get part of speech tags. We will\nnot use Viterbi or Forward-Backward or anything like that, but as a\n(challenging) exercise to the reader, think about how Viterbi could be\nused after you have seen what is going on.\n\nThe model is as follows: let our input sentence be\n$w_1, \\dots, w_M$, where $w_i \\in V$, our vocab. Also, let\n$T$ be our tag set, and $y_i$ the tag of word $w_i$.\nDenote our prediction of the tag of word $w_i$ by\n$\\hat{y}_i$.\n\nThis is a structure prediction, model, where our output is a sequence\n$\\hat{y}_1, \\dots, \\hat{y}_M$, where $\\hat{y}_i \\in T$.\n\nTo do the prediction, pass an LSTM over the sentence. Denote the hidden\nstate at timestep $i$ as $h_i$. Also, assign each tag a\nunique index (like how we had word\\_to\\_ix in the word embeddings\nsection). Then our prediction rule for $\\hat{y}_i$ is\n\n\\begin{align}\\hat{y}_i = \\text{argmax}_j \\  (\\log \\text{Softmax}(Ah_i + b))_j\\end{align}\n\nThat is, take the log softmax of the affine map of the hidden state,\nand the predicted tag is the tag that has the maximum value in this\nvector. Note this implies immediately that the dimensionality of the\ntarget space of $A$ is $|T|$.\n\n\nPrepare data:\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "def prepare_sequence(seq, to_ix):\n    idxs = [to_ix[w] for w in seq]\n    tensor = torch.LongTensor(idxs)\n    return autograd.Variable(tensor)\n\n\ntraining_data = [\n    (\"The dog ate the apple\".split(), [\"DET\", \"NN\", \"V\", \"DET\", \"NN\"]),\n    (\"Everybody read that book\".split(), [\"NN\", \"V\", \"DET\", \"NN\"])\n]\nword_to_ix = {}\nfor sent, tags in training_data:\n    for word in sent:\n        if word not in word_to_ix:\n            word_to_ix[word] = len(word_to_ix)\nprint(word_to_ix)\ntag_to_ix = {\"DET\": 0, \"NN\": 1, \"V\": 2}\n\n# These will usually be more like 32 or 64 dimensional.\n# We will keep them small, so we can see how the weights change as we train.\nEMBEDDING_DIM = 6\nHIDDEN_DIM = 6"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Create the model:\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "class LSTMTagger(nn.Module):\n\n    def __init__(self, embedding_dim, hidden_dim, vocab_size, tagset_size):\n        super(LSTMTagger, self).__init__()\n        self.hidden_dim = hidden_dim\n\n        self.word_embeddings = nn.Embedding(vocab_size, embedding_dim)\n\n        # The LSTM takes word embeddings as inputs, and outputs hidden states\n        # with dimensionality hidden_dim.\n        self.lstm = nn.LSTM(embedding_dim, hidden_dim)\n\n        # The linear layer that maps from hidden state space to tag space\n        self.hidden2tag = nn.Linear(hidden_dim, tagset_size)\n        self.hidden = self.init_hidden()\n\n    def init_hidden(self):\n        # Before we've done anything, we dont have any hidden state.\n        # Refer to the Pytorch documentation to see exactly\n        # why they have this dimensionality.\n        # The axes semantics are (num_layers, minibatch_size, hidden_dim)\n        return (autograd.Variable(torch.zeros(1, 1, self.hidden_dim)),\n                autograd.Variable(torch.zeros(1, 1, self.hidden_dim)))\n\n    def forward(self, sentence):\n        embeds = self.word_embeddings(sentence)\n        lstm_out, self.hidden = self.lstm(\n            embeds.view(len(sentence), 1, -1), self.hidden)\n        tag_space = self.hidden2tag(lstm_out.view(len(sentence), -1))\n        tag_scores = F.log_softmax(tag_space)\n        return tag_scores"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Train the model:\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "model = LSTMTagger(EMBEDDING_DIM, HIDDEN_DIM, len(word_to_ix), len(tag_to_ix))\nloss_function = nn.NLLLoss()\noptimizer = optim.SGD(model.parameters(), lr=0.1)\n\n# See what the scores are before training\n# Note that element i,j of the output is the score for tag j for word i.\ninputs = prepare_sequence(training_data[0][0], word_to_ix)\ntag_scores = model(inputs)\nprint(tag_scores)\n\nfor epoch in range(300):  # again, normally you would NOT do 300 epochs, it is toy data\n    for sentence, tags in training_data:\n        # Step 1. Remember that Pytorch accumulates gradients.\n        # We need to clear them out before each instance\n        model.zero_grad()\n\n        # Also, we need to clear out the hidden state of the LSTM,\n        # detaching it from its history on the last instance.\n        model.hidden = model.init_hidden()\n\n        # Step 2. Get our inputs ready for the network, that is, turn them into\n        # Variables of word indices.\n        sentence_in = prepare_sequence(sentence, word_to_ix)\n        targets = prepare_sequence(tags, tag_to_ix)\n\n        # Step 3. Run our forward pass.\n        tag_scores = model(sentence_in)\n\n        # Step 4. Compute the loss, gradients, and update the parameters by\n        #  calling optimizer.step()\n        loss = loss_function(tag_scores, targets)\n        loss.backward()\n        optimizer.step()\n\n# See what the scores are after training\ninputs = prepare_sequence(training_data[0][0], word_to_ix)\ntag_scores = model(inputs)\n# The sentence is \"the dog ate the apple\".  i,j corresponds to score for tag j\n#  for word i. The predicted tag is the maximum scoring tag.\n# Here, we can see the predicted sequence below is 0 1 2 0 1\n# since 0 is index of the maximum value of row 1,\n# 1 is the index of maximum value of row 2, etc.\n# Which is DET NOUN VERB DET NOUN, the correct sequence!\nprint(tag_scores)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Exercise: Augmenting the LSTM part-of-speech tagger with character-level features\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nIn the example above, each word had an embedding, which served as the\ninputs to our sequence model. Let's augment the word embeddings with a\nrepresentation derived from the characters of the word. We expect that\nthis should help significantly, since character-level information like\naffixes have a large bearing on part-of-speech. For example, words with\nthe affix *-ly* are almost always tagged as adverbs in English.\n\nTo do this, let $c_w$ be the character-level representation of\nword $w$. Let $x_w$ be the word embedding as before. Then\nthe input to our sequence model is the concatenation of $x_w$ and\n$c_w$. So if $x_w$ has dimension 5, and $c_w$\ndimension 3, then our LSTM should accept an input of dimension 8.\n\nTo get the character level representation, do an LSTM over the\ncharacters of a word, and let $c_w$ be the final hidden state of\nthis LSTM. Hints:\n\n* There are going to be two LSTM's in your new model.\n  The original one that outputs POS tag scores, and the new one that\n  outputs a character-level representation of each word.\n* To do a sequence model over characters, you will have to embed characters.\n  The character embeddings will be the input to the character LSTM.\n\n\n"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.1"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}