notebooks/MNIST-torchvision.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# MNIST with torchvision and skorch\n",
    "\n",
    "This notebooks shows how to define and train a simple Neural-Network with PyTorch and use it via skorch with the help of torchvision.\n",
    "\n",
    "<table align=\"left\"><td>\n",
    "<a target=\"_blank\" href=\"https://colab.research.google.com/github/dnouri/skorch/blob/master/notebooks/MNIST.ipynb\">\n",
    "    <img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" />Run in Google Colab</a>  \n",
    "</td><td>\n",
    "<a target=\"_blank\" href=\"https://github.com/dnouri/skorch/blob/master/notebooks/MNIST.ipynb\"><img width=32px src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" />View source on GitHub</a></td></table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note**: If you are running this in [a colab notebook](https://colab.research.google.com/github/dnouri/skorch/blob/master/notebooks/MNIST.ipynb), we recommend you enable a free GPU by going:\n",
    "\n",
    "> **Runtime**   →   **Change runtime type**   →   **Hardware Accelerator: GPU**\n",
    "\n",
    "If you are running in colab, you should install the dependencies and download the dataset by running the following cell:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "! [ ! -z \"$COLAB_GPU\" ] && pip install torch scikit-learn==0.21.* skorch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from itertools import islice\n",
    "\n",
    "from sklearn.model_selection import train_test_split\n",
    "import torch\n",
    "import torchvision\n",
    "from torchvision.datasets import MNIST\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading Data\n",
    "\n",
    "Use torchvision's data repository to provide MNIST data in form of a torch `Dataset`. Originally, the `MNIST` dataset provides 28x28 `PIL` images. To use them with PyTorch, we convert those to tensors by adding the `ToTensor` transform."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "mnist_train = MNIST('datasets', train=True, download=True, transform=torchvision.transforms.Compose([\n",
    "    torchvision.transforms.ToTensor(),\n",
    "]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "mnist_test = MNIST('datasets', train=False, download=True, transform=torchvision.transforms.Compose([\n",
    "    torchvision.transforms.ToTensor(),\n",
    "]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Taking a look at the data\n",
    "\n",
    "Each entry in the `mnist_train` and `mnist_test` Dataset instances consists of a 28 x 28 images and the corresponding label (numbers between 0 and 9). The image data is already normalized to the range [0; 1]. Let's take a look at the first 5 images of the training set:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_example, y_example = zip(*islice(iter(mnist_train), 5))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor(0.), tensor(1.))"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_example[0].min(), X_example[0].max()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Print a selection of training images and their labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_example(X, y, n=5):\n",
    "    \"\"\"Plot the images in X and their labels in rows of `n` elements.\"\"\"\n",
    "    fig = plt.figure()\n",
    "    rows = len(X) // n + 1\n",
    "    for i, (img, y) in enumerate(zip(X, y)):\n",
    "        ax = fig.add_subplot(rows, n, i + 1)\n",
    "        ax.imshow(img.reshape(28, 28))\n",
    "        ax.set_xticks([])\n",
    "        ax.set_yticks([])\n",
    "        ax.set_title(y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWQAAABbCAYAAABEQP/sAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAEnFJREFUeJztnXl0VFW2h7+TATIwBk0Am5mEMIiooKIC2iJqNw+lARFweLQuG3jQiqK0LG1bxecsjYjiBEHpVp8z3bZDo8hyQBQFtJFJMBEkRAIkjAlJ5bw/dt1KKgZIIKl7iuxvrayq3Dq3atete8/9nX323sdYa1EURVH8J8ZvAxRFURRBO2RFURRH0A5ZURTFEbRDVhRFcQTtkBVFURxBO2RFURRH0A5ZURTFEZztkI0xHxljiowxe4N/6/y2yW+MMSnGmDeMMfuMMTnGmNF+2+QKxpj04PmywG9b/MYYM9EYs9wYU2yMyfLbHlcwxnQ1xnxojCk0xnxvjBnqt02VcbZDDjLRWtso+NfFb2McYDZwEEgDxgBPGmO6+2uSM8wGvvTbCEfYCkwH5vptiCsYY+KAt4B/AinA9cACY0yGr4ZVwvUOWQlijEkGhgF3WGv3Wms/ARYCV/lrmf8YY64ACoAP/LbFBay1r1tr3wR2+G2LQ2QCrYEZ1tqAtfZD4FMcu35c75DvM8bkG2M+Ncac57cxPpMBBKy16ytsWwXUa4VsjGkC3A3c7LctitOYQ2zrEWlDDofLHfJUoCNwEvA08A9jTCd/TfKVRkBhpW2FQGMfbHGJe4DnrLWb/TZEcZq1wM/ALcaYeGPMIGAAkOSvWeE42yFba5dZa/dYa4uttfOR4cVv/LbLR/YCTSptawLs8cEWJzDG9AIGAjP8tkVxG2ttCXAZ8FtgGzKi+j9gi592VSbObwNqgKXqYUd9YT0QZ4xJt9ZuCG47BVjto01+cx7QHvjRGAMyiog1xnSz1p7mo12Kg1hrv0FUMQDGmM+A+f5Z9EucVMjGmGbGmIuMMQnGmDhjzBigP/Ce37b5hbV2H/A6cLcxJtkYcw5wKfCCv5b5ytNAJ6BX8G8O8DZwkZ9G+U3wmkkAYpEbVEIwyqBeY4zpGTwWScaYKUArIMtns8JwskMG4pGwne1APjAJuMxaW99jkScAiYgv7EVgvLW23ipka+1+a+027w9x6xRZa7f7bZvP3A4cAP4EXBl8fruvFrnBVUAucv1cAFxorS3216RwjBaoVxRFcQNXFbKiKEq9QztkRVEUR9AOWVEUxRG0Q1YURXEE7ZAVRVEcoUaxiQ1MQ5tAcl3Z4gRF7OOgLa52Akp9OCYAe9iVb609sTpt9ZhUTX04Lnr9VE11z5UadcgJJHOmueDorYoCltmaFQyrD8cEYJF9Nae6bfWYVE19OC56/VRNdc8VdVkoiqI4gnbIiqIojqAdsqIoiiNoh6woiuII2iEriqI4Qr0vyRfNlP76dAByJ0jBqlV9pbTrKUuvAaD17AYAxC7+2gfrFEWpKaqQFUVRHME5hWzixKTYE0+o8vV1U9oDEEgqA6Bdp58BSJogsejbHhVV+HXvl0P75Af2AXDmK7IOZuebPq9lqyNL2YBTAXhs7uMAdI6XY1YWfH1F33kArOsdAOCW9mdF1sAoYN/wMwF44MEnQ9vuufxqAOzy//hikx9sfKgvAGtGy7kUb2IB6D/h+lCbxDe/iLxh9RRVyIqiKI4QcYUc2zUdANswHoCtA5oBcOAsUbEpTeXx41NermLvX/LOfll0+YHHLwZg2cl/B+CHkgOhNvfnXQhA64+juxh/yaDeANz6hKzalBEvo4GyoDbeVFICQGFZQwBOlQeKL+kDQOLib0PvVVZUVPcGH4IDl54hjy1EjaXMXRpxG37uLVrknuz/ivhnu8C2yWcD8NHIBwEosQ3CG0T3pRK1qEJWFEVxhIgo5MB55QsAP5o1GyhXd0dLiRX/6J9n/TcAcfvklt73lYkANP6pNNS2Yb6o5aTly47pMyNNbJMmAOzrnwnA5Bmi/s9P3BtsEX4/zdolqueDJ8Qv+OlfHgPg38/OAaDbgomhth2nRl6VemztL3YndSqQDXMj+OExosptWzknLkhdG3rpA3N2BA3xl71tZFSVEnNs12E0cPAiGVnmjJHvPP60JQDc2Hx9WLuTn50EQFKu9CUFZ0v0Uru/yfna4L3ldW6rKmRFURRH0A5ZURTFESLismi4bmvo+VdFbQDIiM+r1r4350rI1qa9EgaX1elVAArLZFiR9thnR3yPaJ2f2PL8SQB82Wd2tdrfnfolAO82kqH32OxBAMxvvwiAJt121LaJR8Vdg18B4IE1gyL+2bGd2gGwdoD4SXp9cWXotdZfflvlPscTe0dIuN9rQ2cGt0i46JwCcYstulyG98k5q0P7lBGdbB8nrrtZt8r107uhuDljgjr0muyBAJza9EcAVl03M2x/r93ZKaMASHmvjg1GFbKiKIozREQhl+ZuCz2f9cAIAO69WMLbYr9pBMCqCbPC9pme3xOA7wcmARAoyAVgdN8JAGT/Udp1YFUdWe0fXkr0i70kWD+G8ImXsTlS0Hv5oq4AfHuttFt8IAGA1OUyYfX9LlE98f+7WN6n2us41C3xpvTIjeqIuGf3h/1/YGMTnyyJLEWDJdTwzvtkZJARH34yzH9GwkZbfnfkEaermGCgQNHAUwB47baHAGgdJ/Gf1+ZI+GvOw10ASH57JQCLk9oCsOSNDNkvfWHY++5e2QKAlDqzvBxVyIqiKI4Q8cSQlHkSbnXiP+SuE9ixE4DuPX4PwOr+cgdf+PQAAFILwu/YZqko4g7+RW3VGYdOiRYv3pC1QwGIHS6ji2a/Fe94txcknC1j9mYAYjavAKD5x/K+JfeK7+y1nuXxZb8/X4YYkSw8VHZuLwD6JXwSsc+sTPvkcD96m0UBnyyJLLlXSiLQ+YleQpCE/3l+1JYzo1cZe+ROFP/3F1M8X7Ao4xHfS/JP6TBJnErKl/BXb25p6/UyIl2WHu5D9pLOOj8l11UkxnWqkBVFURzBt+JCgfxwpVKyO9xP2n3MdwBsf1Lu5JQdv0rGnN4dgPybxPfrJc18JXHpfLi3GwA7XpIIlRa7ZHjQdIEUSWoafJ8j3cHTYhuGnu+4UXypqYuPyfQakTM4UT4zNilyHxokrr34CYenhPsHE3/YFXp+PJ5hcb+SSJ3V/aTglJdQtUbEIj8+Kn7TZKIraaoiG2ZJ5Mi638k8lBcV0vXf4wDInJIN/LLP8Rg3/q0qt0+/V8rYNt8cueG4KmRFURRHcKb8ZtepksY49mSJIJjXTpYTHzDifwBo/HJ0l8ysTExSuUosfXA3AJ9nvg7AD6UHAbhpmpQLbf6xxEmmJkup0dpQcme0klXJs2vhvapLXOc9Yf8XrW0Wsc/e/NdkAM5pKPrpud2/khcKdkfMhkgS210iCXr/vepSoiNflzmETq9F53W18ZHykrLrfidxxoVl4h8fsXY0AF0mSZ8S2BN+3sUky7mwY7hEcl3aSKIxYpARXOYr0ud0zor8RJUqZEVRFEdwRiEHCgoB2DFeYmt/XCj+1D9Nfx6A2y6XCAO7Qjymbe4N3r1sdObhHRjQPfT8vcwnwl677obJADR+U9SLf1G7dUvq8trPAYs9QaJ38oaJbzTl8i0ALMl4LthCYrWfnH2Z2JAX/dEFVZEzRI7Dqy1WBLfIXMzojRJxkHH/RiD6/OaxaakAzB9afs14UUieMm5wYU5wezgxvWQupsfcNQBMT3ss+IrMrZyz8goAuvxFXvfj2KhCVhRFcQRnFLJH2Sq5O11x1y0A/O3OhwFYeZYoZYKuo+7JEnub/oxk8JVuyo6ckbVAz3tWhp57OfNeBl5tL5njLctTUmEwEWv8H1kcSJHvnXyYNmX9JDbbxkpm2eaBomYOtpYwgZgGomPe7ycz7F4C2raAtLtjk4ysdpaJXkqKkfZpy8Sv6P9RqF12jpX6DW+Meyi4RRaCGLdZ4vpLrpHjEtj+Y8Rtqw1Mgtjv1aWoSOIfJTrJtJNopA3jZJ5g0ECJtZ+c+jQAbePEV+wp6EBwlG1elno5gYINdWB59VCFrCiK4gjOKWQPb1mfietkxrPJ/eILfLGjlFxafbVks2W2uQ6ALnfJvSWwYVNE7awpBVeJgrk97eHQtrJgrYqv3hcfV1tq16/pxZ6WVfCqvbtGPiudyGXqFRfFB+0QRTJv2gwAFk7sdch9prZ4FoCYYFWyA1YiULYG5Ds9vv08AAYuuhGAZivkWLZ6X6oJmhw5b7avEVWUFivK2h5nld28qIrPpj8e3JIQ9vrSLe0BaJMd3Qu42iIJzl9WHB/admZD+U3fWvQSEH6eV2TRAVHAG4JDRW+hh+UH5Zxp9rz/6b+qkBVFURzBWYXsYT4VX+v+4TK72mekLLOybKrkna89XxTUmPZSW7fw3EhbWDNKRajRtMLSOUuLxC/W8XmpG32sURVejPPah3sEt3wFwJhNl4TaZN7wAxDZmeTOV8qMf/f7xP/fps9PR9xn8c8SLbH9HfEHtlgtaqjBu18GW8j/GYQvr+N9r5+mSm3oPg1F/by096SjM95x1k+T39wbDVWm7f3yGO0+80CexOLfOf660LaH50jERc/gJbVgt/iQpy8ZAkBGlsQnx+VJJFfqi1I/5/w2HwJwzWJ5r8rnkB+oQlYURXEE5xWyh3dnTHtMHotuFR2ZZOS2+Ez7fwIweKj4EpPeiJ7c/B0BqQl9rJEinjJed//JAKy9VPyJ7+yX2O2tszuH2jbe5V+GVofbau6ra8XRRQUk9d8e9v/ti4cBkEHtRrL4hVchcHrvN6t8/cL/SGxto+XR7TuuTMUFR6d1OKPKNpV/4z2XSru320rtihIrejQx252FXlUhK4qiOILzCtmrobtxhMwa9+iVDZQrY49ZO0UpJL3lvx+opkz5VFZRyQj6emuKp5J+DlaLW9NblPEF344EIPliiTxpTHTWLahN2r0V7V7UcO7NktjaHvHh32tKbn8Amo6SanbRlpFXF5Qmiv6sHHXUIUtGXy5kxKpCVhRFcQTnFLLpLZEB64NZN8+cMx+A/gkHq2xfbGWW/fOdHWRDWW4dW3iMBDPJYircC2ee+yIAs8mo0Vvl3C0xza9d/ShQXkf5tC+kjmvrod8dk6mK+5zaIFz1eSyddxoAqbuOz1odR0Pjl4IjxEf8teNwqEJWFEVxBN8VclyHdgBsHNsagL+MlGybYY3yD7vftDxZP2vJTClu0Xy+/1k21SLo6quYTTQgUVYyuDFL1vbqNE9ei98m9RbyBpwIQMpIyTqb1FZqRV+SJD7nhfvSALj6W1k5+ISnDlcdon4Sa0R77MqQDK+W7/hpzbGz+VUZScablVW+3uojuX7Ud1zOniu8GspHN1cTCVQhK4qiOELEFbK3tlnh6a0AGHn3uwCMa/b6Yfe7OVfubkufEGWckiUxhs3LokQZH4YEIz/DmgvnAPBJP4ko2VDcEoCxTbOr3O+Grf0AePcziURJv0GjKA5FwAZHJFEuQbyImr/2WgCU+4691TL6vCNx+Jk5On9QmcKO7v/47luoKIpST9AOWVEUxRHq1GUR10qG3Dvnlk8yje+wBIBRjfMOu+/En6RK0NdPynD8hFcl9TNlT3S7KNI+ktTvqX/oG9r2QMvw7+SF+J2bkB22fUWx3D9HLbkegIyxMjmRrgkf1WZ/n/1+m3BMFKVIaOO5CfuCW2Txgff2iysw43opulT7i2NFPyctkd8+fuIvF2xwBVXIiqIojlCrCvngRTLhdnCylLeb1vlfAAxK3HfIfTzyApL223/hzQBk3r4WgJQCUY/Hyx0/sF4Wl9wwon1oW7dJUlL0u8tnVblP5r8mANDlCbnDZ6xwN2zHVbywN6X+4pXyzdotpXxHNZbyr/u7S4BBg81b/DGsAnqWKoqiOEKtKuTsy6R/X3/yK4dsM7ugEwAzl0hBeROQXOLM6VIwPT1PymYe7wHtFUttdp4sz4dM7lNl2wzEL+igy8t5ihdJUk2g1/ExxmqychsAk7b8GoA5bZb4aU5UMuOp4QCMmiKLXLS643sAdhT0lAaff+OLXaAKWVEUxRlqVSFnjJdkjcHjTz9y20rFo493Raz4Q8sZUlznNzOk2E5Hqk41jhZKf8gBYEswC3gwR77WlHBOemEdACMvGwzAy51lcYsBfx4FQMpoWdAhUFAYcdtUISuKojiC78WFFEVRIkkgX4p5HRzWAoCuj/wBgDUDnwJgSOa10tAHX7IqZEVRFEdQhawoSr3EU8rp18jjELwoJ42yUBRFqfcYa6sf3WqM2Q7k1J05TtDOWntidRvXk2MCNTguekyqpp4cFz0mVVOt41KjDllRFEWpO9RloSiK4gjaISuKojiCdsiKoiiOoB2yoiiKI2iHrCiK4gjaISuKojiCdsiKoiiOoB2yoiiKI2iHrCiK4gj/D02WALQRBlEZAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f4cf07e1f60>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_example(torch.stack(X_example), y_example, n=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Preparing a validation split\n",
    "\n",
    "skorch can split the data for us automatically but since we are using `Dataset`s for their lazy-loading property there is no way skorch can do a stratified split automatically without exploring the data completely first (which it doesn't). \n",
    "\n",
    "If we want skorch to do a validation split for us we need to retrieve the `y` values from the dataset and pass these values to `net.fit` later on:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_train = np.array([y for x, y in iter(mnist_train)])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Build Neural Network with PyTorch\n",
    "\n",
    "Simple, fully connected neural network with one hidden layer. Input layer has 784 dimensions (28x28), hidden layer has 98 (= 784 / 8) and output layer 10 neurons, representing digits 0 - 9."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "from torch import nn\n",
    "import torch.nn.functional as F"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "device = 'cuda' if torch.cuda.is_available() else 'cpu'\n",
    "mnist_flat_dim = 28 * 28"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A simple neural network classifier with linear layers and a final softmax in PyTorch:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "class ClassifierModule(nn.Module):\n",
    "    def __init__(\n",
    "            self,\n",
    "            input_dim=mnist_flat_dim,\n",
    "            hidden_dim=98,\n",
    "            output_dim=10,\n",
    "            dropout=0.5,\n",
    "    ):\n",
    "        super(ClassifierModule, self).__init__()\n",
    "        self.dropout = nn.Dropout(dropout)\n",
    "\n",
    "        self.hidden = nn.Linear(input_dim, hidden_dim)\n",
    "        self.output = nn.Linear(hidden_dim, output_dim)\n",
    "\n",
    "    def forward(self, X, **kwargs):\n",
    "        X = X.reshape(-1, self.hidden.in_features)\n",
    "        X = F.relu(self.hidden(X))\n",
    "        X = self.dropout(X)\n",
    "        X = F.softmax(self.output(X), dim=-1)\n",
    "        return X"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "skorch allows to use PyTorch with an sklearn API. We will train the classifier using the classic sklearn `.fit()`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "from skorch import NeuralNetClassifier\n",
    "from skorch.dataset import CVSplit"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "torch.manual_seed(0)\n",
    "\n",
    "net = NeuralNetClassifier(\n",
    "    ClassifierModule,\n",
    "    max_epochs=10,\n",
    "    iterator_train__num_workers=4,\n",
    "    iterator_valid__num_workers=4,\n",
    "    lr=0.1,\n",
    "    device=device,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  epoch    train_loss    valid_acc    valid_loss     dur\n",
      "-------  ------------  -----------  ------------  ------\n",
      "      1        \u001b[36m0.7886\u001b[0m       \u001b[32m0.8987\u001b[0m        \u001b[35m0.3618\u001b[0m  2.5069\n",
      "      2        \u001b[36m0.4238\u001b[0m       \u001b[32m0.9213\u001b[0m        \u001b[35m0.2811\u001b[0m  2.3977\n",
      "      3        \u001b[36m0.3555\u001b[0m       \u001b[32m0.9301\u001b[0m        \u001b[35m0.2418\u001b[0m  2.3908\n",
      "      4        \u001b[36m0.3192\u001b[0m       \u001b[32m0.9379\u001b[0m        \u001b[35m0.2139\u001b[0m  2.4211\n",
      "      5        \u001b[36m0.2881\u001b[0m       \u001b[32m0.9434\u001b[0m        \u001b[35m0.1944\u001b[0m  2.3268\n",
      "      6        \u001b[36m0.2663\u001b[0m       \u001b[32m0.9464\u001b[0m        \u001b[35m0.1804\u001b[0m  2.3883\n",
      "      7        \u001b[36m0.2526\u001b[0m       \u001b[32m0.9503\u001b[0m        \u001b[35m0.1700\u001b[0m  2.3774\n",
      "      8        \u001b[36m0.2416\u001b[0m       \u001b[32m0.9524\u001b[0m        \u001b[35m0.1618\u001b[0m  2.3192\n",
      "      9        \u001b[36m0.2300\u001b[0m       \u001b[32m0.9549\u001b[0m        \u001b[35m0.1535\u001b[0m  2.4264\n",
      "     10        \u001b[36m0.2193\u001b[0m       \u001b[32m0.9562\u001b[0m        \u001b[35m0.1477\u001b[0m  2.4252\n"
     ]
    }
   ],
   "source": [
    "net.fit(mnist_train, y=y_train);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prediction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import accuracy_score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred = net.predict(mnist_test)\n",
    "y_test = np.array([y for x, y in iter(mnist_test)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9581"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "accuracy_score(y_test, y_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An accuracy of about 96% for a network with only one hidden layer is not too bad.\n",
    "\n",
    "Let's take a look at some predictions that went wrong.\n",
    "\n",
    "We compute the index of elements that are misclassified and plot a few of those to get an idea\n",
    "of what went wrong."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "error_mask = y_pred != y_test"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have the mask we need a way to access the images from the `mnist_test` dataset. Luckily, skorch provides a helper class that lets us slice arbitrary `Dataset` objects, `SlicedDataset`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "from skorch.helper import SliceDataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "mnist_test_sliceable = SliceDataset(mnist_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_pred = torch.stack(list(mnist_test_sliceable[error_mask]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWQAAABbCAYAAABEQP/sAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAE9hJREFUeJztnXl0VEX2xz+VEBL2fd/CHkAHUBHBBRGQMxLR8QiogMu4DYwwijouMCOLC24oAuIgiAq4IAjKIuCgDArDIvwUkbBEgiyyBIY1gUCS9/vjvtexk0YIdOdVyP2ck9OvX1X3u13prvetW7duGcdxUBRFUfwnym8DFEVRFEE7ZEVRFEvQDllRFMUStENWFEWxBO2QFUVRLEE7ZEVRFEvQDllRFMUSrO6QjTG3GWOSjDFpxpifjTFX+22T32ibhMYY09gYc8IYM9VvW/zGGLPEbYtj7t8mv23ym9+0hfeXZYwZ47dduSnmtwGnwxjTBXgR6AWsAmr4a5H/aJv8LuOA1X4bYREPOY4z0W8jbMFxnNLesTGmFLAX+MQ/i0JjbYcMDAOGO46zwn2+y09jLEHbJATGmNuAQ8ByoJHP5ij2cyuwD/jGb0NyY6XLwhgTDVwGVDHGJBtjdhpjxhpjSvhtm19om4TGGFMWGA486rctlvGCMWa/MWaZMeZav42xjLuA9x0L80ZY2SED1YAY5E52NdAKaA0M8dMon9E2Cc0IYJLjODv8NsQingAaALWACcAcY0xDf02yA2NMXaAD8J7ftoTC1g75uPs4xnGc3Y7j7AdGATf4aJPfaJvkwhjTCugMvOa3LTbhOM5Kx3GOOo6T4TjOe8AyivD3JBd3At86jpPityGhsNKH7DjOQWPMTsC6IYVfaJuE5FogHthujAEoDUQbY5o7jnOJj3bZhgMYv42whDuBkX4bcTpsVcgAk4EBxpiqxpgKwMPAXJ9t8httk2AmAA0R900r4C1gHtDVT6P8xBhT3hjT1RgTZ4wpZozpDVwDLPTbNr8xxrRH3DjWRVd4WKmQXUYAlYHNwAlgOvCcrxb5j7bJb3AcJx1I954bY44BJxzHSfXPKt+JAZ4FEoAsYCNws+M4RT4WGZnM+9RxnKN+G3I6jIUTjYqiKEUSm10WiqIoRQrtkBVFUSxBO2RFURRL0A5ZURTFErRDVhRFsYR8hb0VN7FOHKUiZYsVnCCNk07GWQfRF4U2ATjKwf2O41Q5m7raJqEpCu2iv5/QnO13JV8dchylaGs6nbtVhYCVzuJ81S8KbQLwb2fGL2dbV9skNEWhXfT3E5qz/a6oy0JRFMUStENWFEWxBO2QFUVRLEE7ZEVRFEvQDllRFMUSbM729rsUq14NgJONa4Ysj9mcs93cpqcaAFB+g0TjVEw6AUDUN/8XSRPPmehqVQHISj0gJ7KzfLRGKQqYmOIAZHRqCcC+1jGBssxWxwDo1GAzAIu3NgGgznjpPqKXrC0wOy90VCEriqJYQqFRyIf7XAHAgRtE3T7ZegEAd5adH7L+pMN1A8e3lJkFQIUecUF1EmtdGnY7w0HpmaKIU4/XBiD9fRkFlJ/y37Bfq1i9OgBk/boXAOfUybBfIxJ4ig4nWx4yM320pvBhYmMBONHpDwBk/20/AEsvfvvML64lm54PS2gOwPKWxSNgYdFEFbKiKIolWKeQo1o2A2DjAFlO+c31rwNQJXq1lJ/lPeTectt/8yzutPVsZPV62SA4+ca3AGjR6CEAykfgWkl/F/XtFKsOQJMHV0fgKuHjUN92ALwzYhQAa06Iwn9lQk8Aary+Uiqq3z0k5rKLAIh7VTZVWdD4zIp4TYaMml7bfX3Q+dRB3ih0XfgMDCMH7pPvyuFGwefbXpMEwNT4JQA8tKstAAsXB2/D2GTCbgAyt26LnJG5UIWsKIpiCdYp5LT6ZQDY/Mfx7pkS+Xr9W4ckomLaL23OWLccyfl674KixM7I/1uO9RSf/JruojTLRsko4gbs3qy50tr/AZC4aCAAoztOA2DNo2MA6N/rGgB29I8HwFnzUwFbaBeer33T6FYAfNVN/t/1Y0oH1dt8Kg2ArvMeAaDRBzlzCcV3HQQgMyV3OoZDYbf3bDFtLg4c3zHlCwCuKBFsX5Wo5QCUjIohFKcc0aOv1vxWHvt+G1TeOu1vANQZse38DT5LVCEriqJYgnbIiqIollDgLotitWsBkPSEhHRVWy6LNcp+KKE0URmyC/ZmN/xqR6ZMZdUpJsOju9ffBcDBpEry+tVSv/zyHQA4xySIvdwhO90RZ0O/2+dF/Bq7r5F281wVz6S2jPg1w0HWT7KbfZMH5Pk4ZJHCyF7ignn7xdcAKDNLJvX+0ulOed2WrQVppjVsGiv/15QbJ7hngl0VN2/pCkD6U+6k7vJVed7DxoDCbu8uDRz3KrPbPQpv+N09vRYCsGSauEcKYnJPFbKiKIolFIhCji5fLnB8+bwUAGZX/hyAK797KKhu7BcSdvV4t7uBHEUU3awxABU3/SyP2ZuDXmfjXTy/OO1FzXQt/aZ7Jn8Tmvmhx5Urg54veu0qACoQ/sUnBUGZj2WEdVeZQQAsG/YGAAcvk2XoZYuIQg5M4o2RSbzkxLfcEtFex7JlYdXFC+R3l/CI/L7M0R8K0Mrzp0Hs3ohfY2CFjXKtBRIi+OSs3oGyxu/KQpqspC1hvaYqZEVRFEuIqEKOihP/ZMaMHIX8dOWvAGj6aX8AEmZJWFLuMH5PGQeeh/lOZCO7r5LFMA2LBSvjYunhef+okiUDx2WiJXxsX5a8eeUFMvIo7MspKk0UhT9xkIQ/7m/pzVH4ZlKB4injlO6ez1g01+w08R3/c7z87pqMkpCw7II1L2x8nHp54Lhz3eBto17/nyzpXviIhECeLC/dXL/nZgAwc6+Edv64QlaM1FkUnC4gvZqEyT05dCoAiaUkyVdinzcCddrukpC4aqqQFUVRLkwiopCjK1QAYOMImQHf1OzNQNmaDHlMGC4+vawjRyJhQqHCa68B98wOOt/hx1sBqPXi8rBcZ1/fnEiKpyqNAyDhP+JLbLD3+7BcI9xENxalu7WvpFut1V7Squ45LAuIysySx4pfyvcps0ENAOKLfwxAw5kSdeMUkL0FTcBn7EZT5PYZH84+DsDoAfcDUGNBeL5LfnOgd8XA8dwFEnHlKdkoI7q/xPqdAMTsEX/z1C2y9Dv7B1k63YDUoPf0IsAqDxU/+x9LyoKYnZnSaT2w5Y5A3Vpz5HsY7rkrVciKoiiWEBGF/GsfSRC06U+ynPXztAqBskmJXQDISv05EpculCSPlwQ595YN9oWVeK5cqOrnTJ077I40iK6Uo3p23p0AwKyBLwE5ymF1hqiYUlGiWrq1EzWz5LjUmHdYlOKzW7oBUHb1j5E12md+HXgZACk3eqPQYI3VvZ/4OuMW5I0vLsz8NiZ4yNQ+AGT1kWX0XnTE/CluIqXHxaec/f2GkO/lbXZR7mOZT5kcvyiofHRqR6nXOSdhWaSiulQhK4qiWEJEFPLRtseDno9O6RQ4LrFZlTHAwbvaBY7ntnvFPZLoipnuiKLYWom1Pt+Z8OgqVQBoXX7Heb5TZNmf2DRwHHed+Peun/UYAE3/Jf5BL9rGi+AZ3rM1ACtHSjKqa0vItlw900VtH2su8xhZG4Lj1gs7Gd0kedaMgS+7ZyRCx4szbve6xGPXnFs448rzQ93h4hcft7YXANXfkAiTBc0lquLXz2Q01WP44wBUmhTcJpVmSX/1dq5ojYG7JEpj3esy6irLirDbnhtVyIqiKJYQEYX84ZXBMZAzmk8NlLUb9SgA9T+X2L+itkFidGWZEe788LLAudxxx2/ffwsAUWnh2YT1VHPJGzKk8sI8ZXXe9T8Dq6f2Phz+cuBc78GijBtNFVWSOz46+4QowZOlTdD5j47K6OLt+LkArJ8rWxUN6SfJL4ov/C6MlvuAkc+b1u8wAE1iSgUV/5Ip8STZblqH6HJlAcg6ItEmF3Li/ri54id/cX13AOp/ItEVQ6vL2of5Q2UkesXVkrq1ylfSSNNqeyNUeZ6efQqAn5+QubCyX0deGXuoQlYURbGEiMijy2NlpcspR+7GFaJytlDa2EviX0/1lLKLFv8FgHKrpc6x2m4WMjcgoPK6tKD33v8HUQTVluwDIKuQ+aSd2jKjO6Lql3nKvLjj0sskMqAgYmdj90r7+rli69LhawCYeihn9VXFL8Tnezo9d+QOye72yqP/AmBYqsykr7pJVl+N6iDRGI8N/gCATyaOBqDPLfJ9cwpp9IWneFdfMj1keYviMtr66SE36sJNFdNg0b0ANBsuvviC3JaooMncJtEQW9w9Krr8VXzHK56W70BSF4nVjuoiejTbVcb/3CcvWDxW5ncqfV3w/ndVyIqiKJYQEYVcf46sCtocWDWUlxgTDcCmzu4mi53zd41VT4ov7eENtwFQMdHuWfSjt4miu+7JZXnKphyVXLTl+otOzfS2tHf9hdHlQ29v6mTI7LG3pbtH1iF3ax3n9BrbU+Ol1m08G/MjyorUeAAOfls9cK7OgdAryrysf8OGTwIgNVMU46qb5XzmNtnGp4Krkib/IPHITJcc07e8LzPps2+5Eih8OVKSXm7iHv0nX6/ber2015g29QBY0E0iBzw1eSFTY7r8j4fdd6k8VpW5Ga8POuX+TH7sUR+ASsn+RaaoQlYURbGEiCjkpn+VO1DXT2Rm+86xcwJlJd0VVoklJc7Uu0vll8tj5bb2bWtZndPiZZk5bfi4nXGXexPlcw+rkjfvbMVomQFPGuqtVJPHqGj5jJuunRRUP9rIffSZ1BYA/LNysD/0sufFcVjjI1G/KV3jyM3+I+KLL/U7KrqgiH1FIiOeH/9+4NwL2/sCUP59+X8e7iMjjD8PljzaS4/JSr41d8tuDtkpoVdheauzJvcUpXyPq5Rv/lRGKrPbNQ7UzTp0+Hw/SsQZ0C44VjbllHx3bn3u8ZD1D3eQaJTkjpPl9RVkBDH2/hsAiB984SvkrFTpaz76QXzEz3SROQtPGWdblPNOFbKiKIolREQhO64PNObfcif6MKFmnjpv3Cq+36wY8ZO2f0xiCEdWX52va0W595TaLXefoaa/TGs/0T0yecq6lRSV063TxDxlochy5I5eP1bu/PPSJddtcob4YL97eiwA/7hPcuM+UDpvHoOaE8K7/9j54H1PBs3vEzi35QWJxrnmDvF1f9FCtq//8rhkc5t9a1sAspNDK+Pc5FbKb34msfKffto6UCfqJmnX7KNHz+FT+EOnryVXReMJoUeG5ZPFb0rH4POZ9U5E0iyrKNYgHoAvO452z8SGrJc0WEamTe5JKQCrQqMKWVEUxRJ8W6ZVakbwnm5zWkrs38i+opDTHVnJd+nSfgDUmyi+5v0DJSPTd22mUpj486QBAPzYf2yesjnpEimw7Kj4M3cdD46qWL1M/KVV1gb7eysskTu5U0b8wc6uPQDMTJS8ryUflJytI6pKruMXDjQPvDbuB/Ed2rRuK2Fozi4x97XtAMDNtcXnfu0oWblXa4rUydp/birGU8q9n5D3m/7SK4GyactFLS9pK7k/stPDtFVLBHHSf38OJuPvBwvIEns52EZGjrWLiTL2dhR59xPJPNn1JhlBftZRRmVPXHQ3ANnrCz4CSRWyoiiKJfifyMCl7kJ3KxGZXKekER9nUgeJMOhbT+5m8+O9fAzB95Lte8T/05htEbXzXKn7kuRQ6LLy/jxlcdvduOE94hPOOhKsahqcZifoQE7WPcHnS0+XtfdbEmXvMERgM3ldToa5RqnhyZMRTrIO5nzuPZ1E9e/JErVa/YTEJYdL0Zf5SNro3q39AuemzJS4+TWLJBfC4Wvc72Qhyv/gxaRvGifRJ8kXB+eVWXxcFHXT52WFZuH5ZPknuoVkDxzlzkd4OSo+flMWPdQdL9+pRVntAXixv/zOnDj/ukVVyIqiKJZgjUKO+U5W01yx9nYAVlwSvE3wlHgv94PcQzIcudsluiv1EgbavWuyc0p84l5EwW+JlM3NBku+j8RXewPQNCUn74et7eSRnZZ25krhYFVODHfXkcE5D1oNkQgGL9+uDYxZLrnFByVKspfRXWQuZcZ/ZeeQisXF772ghhexE6y5Bo15EIDqG+z5TJEiaYDsuNM6VqJnEubJ/7PJ+OARZ4+e+Vv1GElUISuKoliCdsiKoiiWYI3LwgvGrz5AltHe+I5MrDwdL0td28XKIHvmscoADJ4v27U0eiR0AnMFMnfINujYvXOTNVQdJ8P4VpVlaPv9g+K6uGqnLMuv+I7/y/KbDlgHwJDLZdLu2ariculeamnI+kvd9R8DxvQHoPpo/z9DQVErfn/Q86g0mdDc/oxM4lVpL4vJHq4o6Rfu2pYo9VJ+BfzpU1QhK4qiWII1CtkjkA7wOnkYOFDu7EfbyEaECUPkrtfol4LbVkUpWtR7ThYKPJJ4NQD/eOo9AMa90+S0rykovJSrn30gtmXfLkvxn6/mKud9opynb5RFLo2Gye+metKFP4l3Jjb0HBP03Eu7kI1sqLFyUwMAmhzwb5svVciKoiiWYJ1Czk21N+TOXs19nnn6qooSFrzkWD+3laXqb0Zf5Jac9MmivNR8SX4Xa14STdWVVkHl9RHFrHMrZ6ZH8o0ANBskobd+tpkqZEVRFEuwXiErim+4S6adQrR0Wsnh0BJ3S7CLg8//afNNACSvlO2sGg6VNALZJ/xPSaoKWVEUxRJUISuKckFSa6T42buPbJOrROKM67uP9mzgpApZURTFGrRDVhRFsQTtkBVFUSzBOPnYBt4Ykwr8EjlzrKCe4zhVzrZyEWkTyEe7aJuEpoi0i7ZJaM6qXfLVISuKoiiRQ10WiqIolqAdsqIoiiVoh6woimIJ2iEriqJYgnbIiqIolqAdsqIoiiVoh6woimIJ2iEriqJYgnbIiqIolvD/3XuWRaBdCd0AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f4ce66baa58>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_example(X_pred[:5], y_pred[error_mask][:5])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Convolutional Network\n",
    "\n",
    "Next we want to turn it up a notch and use a convolutional neural network which is far better\n",
    "suited for images than simple densely connected layers.\n",
    "\n",
    "PyTorch expects a 4 dimensional tensor as input for its 2D convolution layer. The dimensions represent:\n",
    "\n",
    "* Batch size\n",
    "* Number of channels\n",
    "* Height\n",
    "* Width\n",
    "\n",
    "MNIST data only has one channel since there is no color information. As stated above, each MNIST vector represents a 28x28 pixel image. Hence, the resulting shape for the input tensor needs to be `(x, 1, 28, 28)` where `x` is the batch size and automatically provided by the data loader.\n",
    "\n",
    "Luckily, our data is already formated that way:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([1, 28, 28])"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_example[0].shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let us define the convolutional neural network module using PyTorch:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Cnn(nn.Module):\n",
    "    def __init__(self, dropout=0.5):\n",
    "        super(Cnn, self).__init__()\n",
    "        self.conv1 = nn.Conv2d(1, 32, kernel_size=3)\n",
    "        self.conv2 = nn.Conv2d(32, 64, kernel_size=3)\n",
    "        self.conv2_drop = nn.Dropout2d(p=dropout)\n",
    "        self.fc1 = nn.Linear(1600, 100) # 1600 = number channels * width * height\n",
    "        self.fc2 = nn.Linear(100, 10)\n",
    "        self.fc1_drop = nn.Dropout(p=dropout)\n",
    "\n",
    "    def forward(self, x):\n",
    "        x = torch.relu(F.max_pool2d(self.conv1(x), 2))\n",
    "        x = torch.relu(F.max_pool2d(self.conv2_drop(self.conv2(x)), 2))\n",
    "        \n",
    "        # flatten over channel, height and width = 1600\n",
    "        x = x.view(-1, x.size(1) * x.size(2) * x.size(3))\n",
    "        \n",
    "        x = torch.relu(self.fc1_drop(self.fc1(x)))\n",
    "        x = torch.softmax(self.fc2(x), dim=-1)\n",
    "        return x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As before we can wrap skorch's `NeuralNetClassifier` around our module and start training it like every other sklearn model using `.fit`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "torch.manual_seed(0)\n",
    "\n",
    "cnn = NeuralNetClassifier(\n",
    "    Cnn,\n",
    "    max_epochs=10,\n",
    "    lr=0.0002,\n",
    "    optimizer=torch.optim.Adam,\n",
    "    device=device,\n",
    "    iterator_train__num_workers=4,\n",
    "    iterator_valid__num_workers=4,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  epoch    train_loss    valid_acc    valid_loss     dur\n",
      "-------  ------------  -----------  ------------  ------\n",
      "      1        \u001b[36m0.9299\u001b[0m       \u001b[32m0.9306\u001b[0m        \u001b[35m0.2432\u001b[0m  6.2281\n",
      "      2        \u001b[36m0.3104\u001b[0m       \u001b[32m0.9537\u001b[0m        \u001b[35m0.1506\u001b[0m  6.2632\n",
      "      3        \u001b[36m0.2221\u001b[0m       \u001b[32m0.9665\u001b[0m        \u001b[35m0.1166\u001b[0m  6.4261\n",
      "      4        \u001b[36m0.1783\u001b[0m       \u001b[32m0.9715\u001b[0m        \u001b[35m0.0983\u001b[0m  6.3131\n",
      "      5        \u001b[36m0.1543\u001b[0m       \u001b[32m0.9747\u001b[0m        \u001b[35m0.0877\u001b[0m  6.2015\n",
      "      6        \u001b[36m0.1416\u001b[0m       \u001b[32m0.9754\u001b[0m        \u001b[35m0.0799\u001b[0m  6.2380\n",
      "      7        \u001b[36m0.1297\u001b[0m       \u001b[32m0.9776\u001b[0m        \u001b[35m0.0727\u001b[0m  6.3239\n",
      "      8        \u001b[36m0.1146\u001b[0m       \u001b[32m0.9794\u001b[0m        \u001b[35m0.0674\u001b[0m  6.2880\n",
      "      9        \u001b[36m0.1067\u001b[0m       \u001b[32m0.9815\u001b[0m        \u001b[35m0.0632\u001b[0m  6.2531\n",
      "     10        \u001b[36m0.1032\u001b[0m       \u001b[32m0.9823\u001b[0m        \u001b[35m0.0603\u001b[0m  6.2440\n"
     ]
    }
   ],
   "source": [
    "cnn.fit(mnist_train, y=y_train);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred_cnn = cnn.predict(mnist_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9853"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "accuracy_score(y_test, y_pred_cnn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An accuracy of >98% should suffice for this example!\n",
    "\n",
    "Let's see how we fare on the examples that went wrong before:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7255369928400954"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "accuracy_score(y_test[error_mask], y_pred_cnn[error_mask])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Great success! The majority of the previously misclassified images are now correctly identified."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_pred_cnn = torch.stack(list(mnist_test_sliceable[error_mask]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWQAAABbCAYAAABEQP/sAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAFH9JREFUeJztnXl4FFXWh9+bhYR93wQk7JsOoCLihgjIp0R0eBQXwGXcBkYYxV1wZHHBDUU2B0VUwAVBUAEBR0VHGBBx1xCIBAUECQxrgECS+/1xqjp0EgYi3akbct7nydPVVbe7T1eqb/3uueeca6y1KIqiKMETE7QBiqIoiqAdsqIoiiNoh6woiuII2iEriqI4gnbIiqIojqAdsqIoiiNoh6woiuIIznbIxpglxpgDxpi93l9q0Da5gjGmmXdupgdtS5AYYxKMMVOMMb8YY/YYY742xlwctF0uYIy52hiTYozJNMb8bIw5L2ibgsYYk2SMWWCM2WGM2WKMGW+MiQvarsNxtkP2uN1aW8H7axG0MQ4xAVgZtBEOEAdsADoDlYGHgJnGmKQAbQocY0x34AngRqAicD6wLlCj3GAisBWoC7RDrpuBgVqUD9c7ZCUfxpirgZ3AR0HbEjTW2kxr7XBr7Xprba61dh6QDpwetG0BMwIYaa1d7p2XTdbaTUEb5QCNgJnW2gPW2i3AQqBNwDaF4XqH/LgxZpsxZqkx5oKgjQkaY0wlYCRwV9C2uIgxpjbQHPgxaFuCwhgTC5wB1DTGpBljNnpD87JB2+YAY4GrjTHljDH1gIuRTtkZXO6Q7wMaA/WAycD7xpgmwZoUOKOAKdbaDUEb4hrGmHhgBvCqtXZ10PYESG0gHrgCOA8ZmrcHhgVplCN8iiji3cBG4EtgbqAW5cPZDtlau8Jau8dam2WtfRVYClwStF1BYYxpB3QDng3aFtcwxsQA04CDwO0BmxM0+73HcdbazdbabcAYSvFvB0LXyCLgHaA8UAOoivjancGpGcajYAETtBEBcgGQBPxqjAGoAMQaY1pba08L0K5AMXIypiDK8BJr7aGATQoUa+0OY8xG5Pei5FENaACMt9ZmAVnGmKnAI8C9gVp2GE4qZGNMFWNMD2NMojEmzhjTF5kpXhS0bQEyGWiCDEHbAS8A84EeQRrlAJOAVsCl1tr9R2tcSpgKDDLG1DLGVAXuAOYFbFOgeCOFdGCA16dUAa4Hvg3WsnCc7JARH9gjQAawDRgEXG6tLbWxyNbafdbaLf4fsBc4YK3NCNq2oDDGNARuQ25QWw6LWe8bsGlBMwoJi1wDpABfA48GapEb9Ab+D+lX0oBs4M5ALcqH0QL1iqIobuCqQlYURSl1aIesKIriCNohK4qiOIJ2yIqiKI6gHbKiKIojFCkxpIxJsImUj5YtTnCATA7arGNOQCkN5wRgDzu2WWtrHktbPSeFUxrOi/5+CudYr5UidciJlKej6frHrSoBrLBFK6JWGs4JwL/srF+Ota2ek8IpDedFfz+Fc6zXirosFEVRHEE7ZEVRFEfQDllRFMURtENWFEVxBO2QFUVRHKEk1UMOI65ObQAONjup0OPxa/KWEEt9oDEAVX6SaJxqKQcAiPn319E08Q8TW7sWADkZ22VHbk6A1iilARNfBoCsrm0B2No+PnQsu91eALo2XgPAR+uaA9BgknQfsUu+KjY7T3RUISuKojhCiVHIu/qdBcD2S0Td3t9e1ia8rtKCQttP2XVyaLt3xTkAVL0yMaxNcj03FyeuMFsUccb++gDse01GAVWm/SfinxXXsAEAOb/9DoA9dDDinxENfEWHzZWH7OwArSl5mIQEAA50/RMAuX/fBsBnp7549BfXWw7AiJatAVjWtkwULCydqEJWFEVxBOcUckzbVgCsHiTplP++6DkAasaulOPHeA+5qfKvhz1LPGI7F1n5gyyunXbpCwC0aSrrdlaJwmel3Cvq28bVAaD5bSuj8CmRY2f/TgC8PGoMAKsOiMJ/enIfAOo+t0Iaqt+9UMwZpwCQ+IwsNLOw2dEV8aosGTU9u/misP0ZQ/xR6HeRMzCCbL9ZrpVdTcP3dzw/BYDpSUsAuH1TRwAWfRS+NGXzyZsByF63PnpG5kMVsqIoiiM4p5AzG1UEYM3Fk7w9ZYv0+hd2SkTFjF86HLVtZdKK9N7FRdmN0f+37O0jPvlVvURpVoqRUcQluL2AdfWv/gtA8uLBAIztMgOAVXeNA2DgVecDsGFgEgB21Y/FbKFb+L721LHtAPi4p/y/G8VXCGu35lAmAD3myxJzTV/Pm0sos2kHANnp+csx7Iy4vceK6XBqaPvaaR8AcFbZcPtqxiwDoFxMPIVxyIoefeakz+Wx/+dhx9tn/h2ABqPWH7/Bx4gqZEVRFEfQDllRFMURit1lEVe/HgAp90lIV+1lkqxR6Q0JpYnJklWw13jhVxuyZSqrQZwMj2744XoAdqRUl9evlPZVlm0AwO6VIPbKO910RxwLA66ZH/XP2Hy+nDffVfFwRtuof2YkyPkxFYDmt8rzCUiSwuirxAXz4hPPAlBxjkzq/bXrdfK6teuK00xnSB0v/9f0Syd7e8JdFZev7QHAvge8Sd1lXxR4DxcDCnu+8llo+6qKm72tyIbf3XjVIgCWzBD3SHFM7qlCVhRFcYRiUcixVSqHts+cnw7A3BrvAXDOl7eHtU34QMKu7ul5A5CniGJbNQOgWurP8pi7Jux1Lt7Fi4o9W9RMjwoTvT1Fm9AsCleesyLs+eJnzwWgKpFPPikOKr4lI6zrKw4BYOmI5wHYcYakoVcqJQo5NIk3Tibx0pJf8I6I9tqbK4lVpy6U313LO+X3ZfZ8W4xWHj+NE36P+mcMrrpaPmuhhAjeP6dv6FizVySRJidlbUQ/UxWyoiiKI0RVIcckin8ya1aeQn6wxscAtHhnIAAt50hYUv4wfl8Zh55H+E7kIpvPlWSYJnHhyjhuX2TeP6ZcudB2xVgJH9uaI29eY6GMPEp6OkX1l0ThvzREwh+3tfXnKAIzqVjxlXF6L99nLJprbqb4jv8xSX53zcdISFhu8ZoXMd7KODO03e3k8GWjnvuvpHQvulNCIA9WkW5uwKOzAJj9u4R2fr9cMkYaLA4vF7CvtoTJ3T98OgDJ5aXIV3K/50NtOm6SkLjaqpAVRVFOTKKikGOrVgVg9SiZAU9tNTF0bFWWPLYcKT69nN27o2FCicI/X4NunBu2v/P3VwBQ74llEfmcrf3zIikeqD4BgJafii+x8e/fROQzIk1sM1G66/pLudV6Z0tZ1S27JIGo4hx5rPahXE/ZjesCkFTmLQCazJaoG1tM9hY3IZ+xF02R32e8K3c/AGMH3QJA3YWRuZaCZnvfaqHteQsl4spXsjFGdH/ZHzYCEL9F/M3T10rqd+63kjrdmIyw9/QjwGoMFz/7xeUkIWZjtnRat669NtS23vtyHUZ67koVsqIoiiNERSH/1k8KBKX+WdJZ38usGjo2Jbk7ADkZP0fjo0skaZOkQM5NlcJ9YWUfrVxY8z9Mg2vdjjSIrZ6nejbe0BKAOYOfBPKUw8osUTHlY0S19OwkambJfmkxf5coxUfW9gSg0srvo2t0wPw2+AwA0i/1R6HhGqvXAPF1Ji4sGF9ckjk8JnjY9H4A5PSTNHo/OmLBNK+Q0j3iU8795qdC38tf7KLyWzKfMjVpcdjxsRldpF23vIJl0YrqUoWsKIriCFFRyHs67g97Pja9a2i77BpVxgA7ru8U2p7X6WlvS6IrZnsjirivJNb6eGfCY2vWBKB9lQ3H+U7RZVtyi9B24oXi37tozt0AtPin+Af9aBs/gmdkn/YArBgtxaguKCvLcvXZJ2p7b2uZx8j5KTxuvaST1VOKZ80a/JS3RyJ0/DjjTs9JPPZJ80pmXHlROHmk+MUnfHUVAHWelwiTha0lquK3d2U0deXIewCoPiX8nFSfI/3Vi/miNQZvkiiN756TUVcllkfc9vyoQlYURXGEqCjkN84Jj4Gc1Xp66FinMXcB0Og9if0rbQskxtaQGeFudywN7csfd/ziLb0BiMmMzCKsh1pL3ZBhNRYVONbgleArsPpq742RT4X29R0qyrjpdFEl+eOjcw+IEjxYwYTtf3OPjC5eTJoHwA/zZKmiYQOk+EWZRV9G0PIAMPJ9MwfsAqB5fPmww79kSzxJrlfWIbZyJQBydku0yYlcuD9xnvjJn/ihFwCN3pboiuF1JPdhwXAZiZ51npRurfmxnKQZ9f0Rqjzfl3sIgJ/vk7mwSp9EXxn7qEJWFEVxhKjIozMTJNPlkJW7cdWYvCWUVl8l8a+H+sixUz76KwCVV0qbvfW9KmReQECN7zLD3nvbn0QR1F6yFYCcEuaTtvVlRndUrQ8LHPPjjisslciA4oidTfhdzm+QGVunj1wFwPSdedlX1T4Qn++R9Nzua6W629N3/ROAERkyk/7FZZJ9NaazRGPcPfR1AN5+aSwA/XrL9WZLaPSFr3hXnjaz0ONtysho68fbvagLr1RM48U3AdBqpPjii3NZouIme71EQ6z11qjo/jfxHS9/UK6BlO4Sqx3TXfRorqeM/7FVXvDReJnfqf5J8fvfVSEriqI4QlQUcqP3JStoTShrqCDxJhaA1G7eIovdivYZX9wvvrQ7froagGrJbs+i77laFN2F9y8tcGzaHqlFW3mg6NRsf0l7z18YW6Xw5U1tlswe+0u6++Ts9JbWsUfW2L4aL//d6mMxP6osz0gCYMfndUL7GmwvPKPMr/o3YuQUADKyRTF+cbnsz14vy/hU9VTS1G8lHpmZUmO692sykz639zlAyauRkvJUc2/r0yK9bt1Fcr7GdWgIwMKeEjngq8kTmboz5X884ubT5bGWzM34fdAh72fy/ZWNAKieFlxkiipkRVEUR4iKQm7xN7kD9XhbZravG/9+6Fg5L8MquZzEmfp3qaJyZoLc1j5vL9k5bZ6SmdMm97gZd/l7snzvETUL1p2tFisz4CnD/Uw1eYyJle+YesGUsPaxRu6jD2e0AeAfNcL9oWc8Jo7Dum+K+k3vkUh+tu0WX3z5/6Gii4uEpyUy4rFJr4X2Pf5rfwCqvCb/z139ZITxl6FSR/uzvZLJt+oGWc0hN73wLCw/O2tqH1HKN3pK+fJ3ZKQyt1OzUNucnbuO96tEnUGdwmNl0w/JtXPFo/cU2n5XZ4lGSesyVV5fVUYQ42+5BICkoSe+Qs7JkL7mzW/FR/xwd5mz8JVxrkM171QhK4qiOEJUFLL1fKDx/5I70RstTyrQ5vkrxPebEy9+0rPvlhjC0XVWFumzYrx7Sv22m4/SMlhmnP2St2UKHOtZTlROz64vFThWGDlW7uiNEuTOP3+f1LpNyxIf7JcPjgfgoZulNu6tFQrWMThpcmTXHzse/OtkyIJ+oX1rH5donPOvFV/3B21k+foP90s1t7lXdAQgN61wZZyf/Ep54rsSK//OO+1DbWIuk/Oau2fPH/gWwdD1E6lV0Wxy4SPDKmniN6VL+P7shgeiaZZTxDVOAuDDLmO9PQmFtksZKiPT5jemF4NVhaMKWVEUxRECS9MqPyt8Tbf320rs3+j+opD3WcnkO/2zAQA0fEl8zdsGS0WmLztMpyTxlymDAPh+4PgCx97fJ5ECS/eIP3PT/vCoipVLxV9a86twf2/VJXIntxXFH2w3bQFgdrLUfS13m9RsHVVLah0/vr116LWJ34rv0KW8rZbD81aJubljZwAury8+9wvGSOZevWnSJmfbH1MxvlLue5+838wnnw4dm7FM1PKSjlL7I3dfhJZqiSJ23/+eg8m6d0cxWeIuOzrIyLF+nChjf0WRV96WypM9LpMR5LtdZFR23yk3AJD7Q/FHIKlCVhRFcYTgCxl4nLzIW0pEJtcpZ8THmdJZIgz6N5S72YIkvx5D+L3k1y3i/2nG+qja+Uc5+UmpodB9xS0FjiX+6sUNbxGfcM7ucFXT+AgrQYdqsm4J319hpuTer02WtcMQgc3U7/IqzDXNiEydjEiSsyPve2/pKqp/S46o1ToHJC45Uoq+4ptyjm5aNyC0b9psiZtftVhqIew637smS1D9Bz8mPXWCRJ+knRpeV+aj/aKoWzwmGZol55sVndg2Uj1wjDcf4deoeGuiJD2cPEmuqcU5ZwPwxED5ndnE4LpFVciKoiiO4IxCjv9SsmnO+uoaAJafFr5M8LQkv/aD3EOyrNztkr1MvZaD3V412R4Sn7gfUXA40bK51VCp95H8TF8AWqTn1f1w9Tz55GZmHr1RJPgiL4a7x+jwmgfthkkEg19v1wXGLZPa4kOSpdjL2O4ylzLrP7JySLUy4vdeWNeP2AnXXEPG3QZAnZ/c+U7RImWQrLjTPkGiZ1rOl/9n80nhI84r+xQt6zGaqEJWFEVxBO2QFUVRHMEZl4UfjF9nkKTRXvqyTKw8mCSprp0SZJA9e28NAIYukOVamt5ZeAFzBbI3yDLouL1ykzPUmiDD+HY1ZGj7zW3iujh3o6TlV3s5+LT8FoO+A2DYmTJp90gtcbn0Kv9Zoe0/8/I/Bo0bCECdscF/h+KiXtK2sOcxmTKh+evDMolX82xJJrujmpRfuH59srRL/w0Ipk9RhawoiuIIzihkn1A5wAvlYfBgubPv6SALEbYcJne9pr8U37IqSumi4aOSKHBn8nkAPPTAqwBMeLn5EV9TXPglV999XWzLvUZS8R+r7SnnraKcZ66WJJemI+R3UyflxJ/EOxo/9RkX9twvu5CLLKixIrUxAM23B7fMlypkRVEUR3BOIeen9vNyZ6/tPc8+clNFiQh+cayfO0qq+sTYU7wjBwOyqCAnPSm/i1VPiqbqQbuw440QxaxzK0fnyrRLAWg1REJvgzxnqpAVRVEcwXmFrCiB4aVM2xKUOq3ksXOJtyTYqeH7/7zmMgDSVshyVk2GSxmB3APBlyRVhawoiuIIqpAVRTkhqTda/Oy9RnfId0TijBt5j+4s4KQKWVEUxRm0Q1YURXEE7ZAVRVEcwdgiLANvjMkAfomeOU7Q0Fpb81gbl5JzAkU4L3pOCqeUnBc9J4VzTOelSB2yoiiKEj3UZaEoiuII2iEriqI4gnbIiqIojqAdsqIoiiNoh6woiuII2iEriqI4gnbIiqIojqAdsqIoiiNoh6woiuII/w9B7MkNsw2uWAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f4ce62afd30>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_example(X_pred_cnn[:5], y_pred_cnn[error_mask][:5])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Grid searching parameter configurations\n",
    "\n",
    "Finally we want to show an example of how to use sklearn grid search when using torch `Dataset` instances.\n",
    "\n",
    "When doing k-fold validation grid search we have the same problem as before that sklearn is only able to do (stratified) splits when the data is sliceable. While skorch knows how to deal with PyTorch `Dataset` objects and only needs `y` to be known beforehand, sklearn doesn't know how to deal with `Dataset`s and needs a wrapper that makes them sliceable.\n",
    "\n",
    "Fortunately, we already know that skorch provides such a helper: `SliceDataset`.\n",
    "\n",
    "What is left to do is to define our parameter search space and run the grid search with a sliceable instance of `mnist_train`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import GridSearchCV"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "params = {\n",
    "    'module__dropout': [0, 0.5, 0.8],\n",
    "    'max_epochs': [2],\n",
    "    'verbose': [False],\n",
    "    'train_split': [False],\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The parameter we are interested in here is the dropout rate. We want to see which of the values (no dropout, 50%, 80%) is the best choice for our network.\n",
    "\n",
    "Additionally:\n",
    "\n",
    "- We use only two epochs (`max_epochs: [2]`) for each `.fit` (only to reduce execution time, normally we wouldn't change this and possibly add an `EarlyStopping` callback).\n",
    "- Disable the network output (`verbose: [False]`)\n",
    "- Disable the internal train/validation split (`train_split: [False]`) since the grid search will do k-fold validation anyway"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Re-initializing module.\n",
      "Re-initializing optimizer.\n"
     ]
    }
   ],
   "source": [
    "cnn.initialize();"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "gs = GridSearchCV(cnn, param_grid=params, scoring='accuracy', verbose=1, cv=3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "mnist_train_sliceable = SliceDataset(mnist_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 3 folds for each of 3 candidates, totalling 9 fits\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.\n",
      "[Parallel(n_jobs=1)]: Done   9 out of   9 | elapsed:  1.6min finished\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=3, error_score='raise-deprecating',\n",
       "             estimator=<class 'skorch.classifier.NeuralNetClassifier'>[initialized](\n",
       "  module_=Cnn(\n",
       "    (conv1): Conv2d(1, 32, kernel_size=(3, 3), stride=(1, 1))\n",
       "    (conv2): Conv2d(32, 64, kernel_size=(3, 3), stride=(1, 1))\n",
       "    (conv2_drop): Dropout2d(p=0.5)\n",
       "    (fc1): Linear(in_features=1600, out_features=100, bias=True)\n",
       "    (fc2): Linear(in_features=100, out_features=10, bias=True)\n",
       "    (fc1_drop): Dropout(p=0.5)\n",
       "  ),\n",
       "),\n",
       "             iid='warn', n_jobs=None,\n",
       "             param_grid={'max_epochs': [2], 'module__dropout': [0, 0.5, 0.8],\n",
       "                         'train_split': [False], 'verbose': [False]},\n",
       "             pre_dispatch='2*n_jobs', refit=True, return_train_score=False,\n",
       "             scoring='accuracy', verbose=1)"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gs.fit(mnist_train_sliceable, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After running the grid search we now know the best configuration in our search space:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'max_epochs': 2, 'module__dropout': 0, 'train_split': False, 'verbose': False}"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gs.best_params_"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}