pytorch/word_embeddings_tutorial.ipynb

{
  "cells": [
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "%matplotlib inline"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "\nWord Embeddings: Encoding Lexical Semantics\n===========================================\n\nWord embeddings are dense vectors of real numbers, one per word in your\nvocabulary. In NLP, it is almost always the case that your features are\nwords! But how should you represent a word in a computer? You could\nstore its ascii character representation, but that only tells you what\nthe word *is*, it doesn't say much about what it *means* (you might be\nable to derive its part of speech from its affixes, or properties from\nits capitalization, but not much). Even more, in what sense could you\ncombine these representations? We often want dense outputs from our\nneural networks, where the inputs are $|V|$ dimensional, where\n$V$ is our vocabulary, but often the outputs are only a few\ndimensional (if we are only predicting a handful of labels, for\ninstance). How do we get from a massive dimensional space to a smaller\ndimensional space?\n\nHow about instead of ascii representations, we use a one-hot encoding?\nThat is, we represent the word $w$ by\n\n\\begin{align}\\overbrace{\\left[ 0, 0, \\dots, 1, \\dots, 0, 0 \\right]}^\\text{|V| elements}\\end{align}\n\nwhere the 1 is in a location unique to $w$. Any other word will\nhave a 1 in some other location, and a 0 everywhere else.\n\nThere is an enormous drawback to this representation, besides just how\nhuge it is. It basically treats all words as independent entities with\nno relation to each other. What we really want is some notion of\n*similarity* between words. Why? Let's see an example.\n\nSuppose we are building a language model. Suppose we have seen the\nsentences\n\n* The mathematician ran to the store.\n* The physicist ran to the store.\n* The mathematician solved the open problem.\n\nin our training data. Now suppose we get a new sentence never before\nseen in our training data:\n\n* The physicist solved the open problem.\n\nOur language model might do OK on this sentence, but wouldn't it be much\nbetter if we could use the following two facts:\n\n* We have seen  mathematician and physicist in the same role in a sentence. Somehow they\n  have a semantic relation.\n* We have seen mathematician in the same role  in this new unseen sentence\n  as we are now seeing physicist.\n\nand then infer that physicist is actually a good fit in the new unseen\nsentence? This is what we mean by a notion of similarity: we mean\n*semantic similarity*, not simply having similar orthographic\nrepresentations. It is a technique to combat the sparsity of linguistic\ndata, by connecting the dots between what we have seen and what we\nhaven't. This example of course relies on a fundamental linguistic\nassumption: that words appearing in similar contexts are related to each\nother semantically. This is called the `distributional\nhypothesis <https://en.wikipedia.org/wiki/Distributional_semantics>`__.\n\n\nGetting Dense Word Embeddings\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nHow can we solve this problem? That is, how could we actually encode\nsemantic similarity in words? Maybe we think up some semantic\nattributes. For example, we see that both mathematicians and physicists\ncan run, so maybe we give these words a high score for the \"is able to\nrun\" semantic attribute. Think of some other attributes, and imagine\nwhat you might score some common words on those attributes.\n\nIf each attribute is a dimension, then we might give each word a vector,\nlike this:\n\n\\begin{align}q_\\text{mathematician} = \\left[ \\overbrace{2.3}^\\text{can run},\n   \\overbrace{9.4}^\\text{likes coffee}, \\overbrace{-5.5}^\\text{majored in Physics}, \\dots \\right]\\end{align}\n\n\\begin{align}q_\\text{physicist} = \\left[ \\overbrace{2.5}^\\text{can run},\n   \\overbrace{9.1}^\\text{likes coffee}, \\overbrace{6.4}^\\text{majored in Physics}, \\dots \\right]\\end{align}\n\nThen we can get a measure of similarity between these words by doing:\n\n\\begin{align}\\text{Similarity}(\\text{physicist}, \\text{mathematician}) = q_\\text{physicist} \\cdot q_\\text{mathematician}\\end{align}\n\nAlthough it is more common to normalize by the lengths:\n\n\\begin{align}\\text{Similarity}(\\text{physicist}, \\text{mathematician}) = \\frac{q_\\text{physicist} \\cdot q_\\text{mathematician}}\n   {\\| q_\\text{\\physicist} \\| \\| q_\\text{mathematician} \\|} = \\cos (\\phi)\\end{align}\n\nWhere $\\phi$ is the angle between the two vectors. That way,\nextremely similar words (words whose embeddings point in the same\ndirection) will have similarity 1. Extremely dissimilar words should\nhave similarity -1.\n\n\nYou can think of the sparse one-hot vectors from the beginning of this\nsection as a special case of these new vectors we have defined, where\neach word basically has similarity 0, and we gave each word some unique\nsemantic attribute. These new vectors are *dense*, which is to say their\nentries are (typically) non-zero.\n\nBut these new vectors are a big pain: you could think of thousands of\ndifferent semantic attributes that might be relevant to determining\nsimilarity, and how on earth would you set the values of the different\nattributes? Central to the idea of deep learning is that the neural\nnetwork learns representations of the features, rather than requiring\nthe programmer to design them herself. So why not just let the word\nembeddings be parameters in our model, and then be updated during\ntraining? This is exactly what we will do. We will have some *latent\nsemantic attributes* that the network can, in principle, learn. Note\nthat the word embeddings will probably not be interpretable. That is,\nalthough with our hand-crafted vectors above we can see that\nmathematicians and physicists are similar in that they both like coffee,\nif we allow a neural network to learn the embeddings and see that both\nmathematicians and physicisits have a large value in the second\ndimension, it is not clear what that means. They are similar in some\nlatent semantic dimension, but this probably has no interpretation to\nus.\n\n\nIn summary, **word embeddings are a representation of the *semantics* of\na word, efficiently encoding semantic information that might be relevant\nto the task at hand**. You can embed other things too: part of speech\ntags, parse trees, anything! The idea of feature embeddings is central\nto the field.\n\n\nWord Embeddings in Pytorch\n~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nBefore we get to a worked example and an exercise, a few quick notes\nabout how to use embeddings in Pytorch and in deep learning programming\nin general. Similar to how we defined a unique index for each word when\nmaking one-hot vectors, we also need to define an index for each word\nwhen using embeddings. These will be keys into a lookup table. That is,\nembeddings are stored as a $|V| \\times D$ matrix, where $D$\nis the dimensionality of the embeddings, such that the word assigned\nindex $i$ has its embedding stored in the $i$'th row of the\nmatrix. In all of my code, the mapping from words to indices is a\ndictionary named word\\_to\\_ix.\n\nThe module that allows you to use embeddings is torch.nn.Embedding,\nwhich takes two arguments: the vocabulary size, and the dimensionality\nof the embeddings.\n\nTo index into this table, you must use torch.LongTensor (since the\nindices are integers, not floats).\n\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "# Author: Robert Guthrie\n\nimport torch\nimport torch.autograd as autograd\nimport torch.nn as nn\nimport torch.nn.functional as F\nimport torch.optim as optim\n\ntorch.manual_seed(1)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "word_to_ix = {\"hello\": 0, \"world\": 1}\nembeds = nn.Embedding(2, 5)  # 2 words in vocab, 5 dimensional embeddings\nlookup_tensor = torch.LongTensor([word_to_ix[\"hello\"]])\nhello_embed = embeds(autograd.Variable(lookup_tensor))\nprint(hello_embed)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "An Example: N-Gram Language Modeling\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nRecall that in an n-gram language model, given a sequence of words\n$w$, we want to compute\n\n\\begin{align}P(w_i | w_{i-1}, w_{i-2}, \\dots, w_{i-n+1} )\\end{align}\n\nWhere $w_i$ is the ith word of the sequence.\n\nIn this example, we will compute the loss function on some training\nexamples and update the parameters with backpropagation.\n\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "CONTEXT_SIZE = 2\nEMBEDDING_DIM = 10\n# We will use Shakespeare Sonnet 2\ntest_sentence = \"\"\"When forty winters shall besiege thy brow,\nAnd dig deep trenches in thy beauty's field,\nThy youth's proud livery so gazed on now,\nWill be a totter'd weed of small worth held:\nThen being asked, where all thy beauty lies,\nWhere all the treasure of thy lusty days;\nTo say, within thine own deep sunken eyes,\nWere an all-eating shame, and thriftless praise.\nHow much more praise deserv'd thy beauty's use,\nIf thou couldst answer 'This fair child of mine\nShall sum my count, and make my old excuse,'\nProving his beauty by succession thine!\nThis were to be new made when thou art old,\nAnd see thy blood warm when thou feel'st it cold.\"\"\".split()\n# we should tokenize the input, but we will ignore that for now\n# build a list of tuples.  Each tuple is ([ word_i-2, word_i-1 ], target word)\ntrigrams = [([test_sentence[i], test_sentence[i + 1]], test_sentence[i + 2])\n            for i in range(len(test_sentence) - 2)]\n# print the first 3, just so you can see what they look like\nprint(trigrams[:3])\n\nvocab = set(test_sentence)\nword_to_ix = {word: i for i, word in enumerate(vocab)}\n\n\nclass NGramLanguageModeler(nn.Module):\n\n    def __init__(self, vocab_size, embedding_dim, context_size):\n        super(NGramLanguageModeler, self).__init__()\n        self.embeddings = nn.Embedding(vocab_size, embedding_dim)\n        self.linear1 = nn.Linear(context_size * embedding_dim, 128)\n        self.linear2 = nn.Linear(128, vocab_size)\n\n    def forward(self, inputs):\n        embeds = self.embeddings(inputs).view((1, -1))\n        out = F.relu(self.linear1(embeds))\n        out = self.linear2(out)\n        log_probs = F.log_softmax(out)\n        return log_probs\n\n\nlosses = []\nloss_function = nn.NLLLoss()\nmodel = NGramLanguageModeler(len(vocab), EMBEDDING_DIM, CONTEXT_SIZE)\noptimizer = optim.SGD(model.parameters(), lr=0.001)\n\nfor epoch in range(10):\n    total_loss = torch.Tensor([0])\n    for context, target in trigrams:\n\n        # Step 1. Prepare the inputs to be passed to the model (i.e, turn the words\n        # into integer indices and wrap them in variables)\n        context_idxs = [word_to_ix[w] for w in context]\n        context_var = autograd.Variable(torch.LongTensor(context_idxs))\n\n        # Step 2. Recall that torch *accumulates* gradients. Before passing in a\n        # new instance, you need to zero out the gradients from the old\n        # instance\n        model.zero_grad()\n\n        # Step 3. Run the forward pass, getting log probabilities over next\n        # words\n        log_probs = model(context_var)\n\n        # Step 4. Compute your loss function. (Again, Torch wants the target\n        # word wrapped in a variable)\n        loss = loss_function(log_probs, autograd.Variable(\n            torch.LongTensor([word_to_ix[target]])))\n\n        # Step 5. Do the backward pass and update the gradient\n        loss.backward()\n        optimizer.step()\n\n        total_loss += loss.data\n    losses.append(total_loss)\nprint(losses)  # The loss decreased every iteration over the training data!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Exercise: Computing Word Embeddings: Continuous Bag-of-Words\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nThe Continuous Bag-of-Words model (CBOW) is frequently used in NLP deep\nlearning. It is a model that tries to predict words given the context of\na few words before and a few words after the target word. This is\ndistinct from language modeling, since CBOW is not sequential and does\nnot have to be probabilistic. Typcially, CBOW is used to quickly train\nword embeddings, and these embeddings are used to initialize the\nembeddings of some more complicated model. Usually, this is referred to\nas *pretraining embeddings*. It almost always helps performance a couple\nof percent.\n\nThe CBOW model is as follows. Given a target word $w_i$ and an\n$N$ context window on each side, $w_{i-1}, \\dots, w_{i-N}$\nand $w_{i+1}, \\dots, w_{i+N}$, referring to all context words\ncollectively as $C$, CBOW tries to minimize\n\n\\begin{align}-\\log p(w_i | C) = -\\log \\text{Softmax}(A(\\sum_{w \\in C} q_w) + b)\\end{align}\n\nwhere $q_w$ is the embedding for word $w$.\n\nImplement this model in Pytorch by filling in the class below. Some\ntips:\n\n* Think about which parameters you need to define.\n* Make sure you know what shape each operation expects. Use .view() if you need to\n  reshape.\n\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "CONTEXT_SIZE = 2  # 2 words to the left, 2 to the right\nraw_text = \"\"\"We are about to study the idea of a computational process.\nComputational processes are abstract beings that inhabit computers.\nAs they evolve, processes manipulate other abstract things called data.\nThe evolution of a process is directed by a pattern of rules\ncalled a program. People create programs to direct processes. In effect,\nwe conjure the spirits of the computer with our spells.\"\"\".split()\n\n# By deriving a set from `raw_text`, we deduplicate the array\nvocab = set(raw_text)\nvocab_size = len(vocab)\n\nword_to_ix = {word: i for i, word in enumerate(vocab)}\ndata = []\nfor i in range(2, len(raw_text) - 2):\n    context = [raw_text[i - 2], raw_text[i - 1],\n               raw_text[i + 1], raw_text[i + 2]]\n    target = raw_text[i]\n    data.append((context, target))\nprint(data[:5])\n\n\nclass CBOW(nn.Module):\n\n    def __init__(self):\n        pass\n\n    def forward(self, inputs):\n        pass\n\n# create your model and train.  here are some functions to help you make\n# the data ready for use by your module\n\n\ndef make_context_vector(context, word_to_ix):\n    idxs = [word_to_ix[w] for w in context]\n    tensor = torch.LongTensor(idxs)\n    return autograd.Variable(tensor)\n\n\nmake_context_vector(data[0][0], word_to_ix)  # example"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.1"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}