tutorials/Titanic_Basic_Interpret.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Getting started with Captum - Titanic Data Analysis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this notebook, we will demonstrate the basic features of the Captum interpretability library through an example model trained on the Titanic survival data. We will first train a deep neural network on the data using PyTorch and use Captum to understand which of the features were most important and how the network reached its prediction.\n",
    "  \n",
    "  **Note:** Before running this tutorial, please install the scipy, pandas, and matplotlib packages."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initial imports\n",
    "import numpy as np\n",
    "\n",
    "import torch\n",
    "\n",
    "from captum.attr import IntegratedGradients\n",
    "from captum.attr import LayerConductance\n",
    "from captum.attr import NeuronConductance\n",
    "\n",
    "import matplotlib\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "from scipy import stats\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will begin by importing and cleaning the dataset. Download the dataset from http://biostat.mc.vanderbilt.edu/wiki/pub/Main/DataSets/titanic3.csv and update the cell below with the path to the dataset csv."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Download dataset from: http://biostat.mc.vanderbilt.edu/wiki/pub/Main/DataSets/titanic3.csv\n",
    "# Update path to dataset here.\n",
    "dataset_path = \"titanic3.csv\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Read dataset from csv file.\n",
    "titanic_data = pd.read_csv(dataset_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With the data loaded, we now preprocess the data by converting some categorical features such as gender, location of embarcation, and passenger class into one-hot encodings (separate feature columns for each class with 0 / 1). We also remove some features that are more difficult to analyze, such as name, and fill missing values in age and fare with the average values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "titanic_data = pd.concat([titanic_data,\n",
    "                          pd.get_dummies(titanic_data['sex']),\n",
    "                          pd.get_dummies(titanic_data['embarked'],prefix=\"embark\"),\n",
    "                          pd.get_dummies(titanic_data['pclass'],prefix=\"class\")], axis=1)\n",
    "titanic_data[\"age\"] = titanic_data[\"age\"].fillna(titanic_data[\"age\"].mean())\n",
    "titanic_data[\"fare\"] = titanic_data[\"fare\"].fillna(titanic_data[\"fare\"].mean())\n",
    "titanic_data = titanic_data.drop(['name','ticket','cabin','boat','body','home.dest','sex','embarked','pclass'], axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After processing, the features we have are:\n",
    "\n",
    "* Age - Passenger Age\n",
    "* Sibsp - Number of Siblings / Spouses Aboard\n",
    "* Parch - Number of Parents / Children Aboard\n",
    "* Fare - Fare Amount Paid in British Pounds\n",
    "* Female - Binary variable indicating whether passenger is female\n",
    "* Male - Binary variable indicating whether passenger is male\n",
    "* EmbarkC - Binary variable indicating whether passenger embarked at Cherbourg\n",
    "* EmbarkQ - Binary variable indicating whether passenger embarked at Queenstown\n",
    "* EmbarkS - Binary variable indicating whether passenger embarked at Southampton\n",
    "* Class1 - Binary variable indicating whether passenger was in first class\n",
    "* Class2 - Binary variable indicating whether passenger was in second class\n",
    "* Class3 - Binary variable indicating whether passenger was in third class\n",
    "\n",
    "(Reference: http://campus.lakeforest.edu/frank/FILES/MLFfiles/Bio150/Titanic/TitanicMETA.pdf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now convert the data to numpy arrays and separate the training and test sets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set random seed for reproducibility.\n",
    "np.random.seed(131254)\n",
    "\n",
    "# Convert features and labels to numpy arrays.\n",
    "labels = titanic_data[\"survived\"].to_numpy()\n",
    "titanic_data = titanic_data.drop(['survived'], axis=1)\n",
    "feature_names = list(titanic_data.columns)\n",
    "data = titanic_data.to_numpy()\n",
    "\n",
    "# Separate training and test sets using \n",
    "train_indices = np.random.choice(len(labels), int(0.7*len(labels)), replace=False)\n",
    "test_indices = list(set(range(len(labels))) - set(train_indices))\n",
    "train_features = data[train_indices]\n",
    "train_labels = labels[train_indices]\n",
    "test_features = data[test_indices]\n",
    "test_labels = labels[test_indices]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are now ready to define the neural network architecture we will use for the task. We have defined a simple architecture using 2 hidden layers, the first with 12 hidden units and the second with 8 hidden units, each with Sigmoid non-linearity. The final layer performs a softmax operation and has 2 units, corresponding to the outputs of either survived (1) or not survived (0)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "torch.manual_seed(1)  # Set seed for reproducibility.\n",
    "class TitanicSimpleNNModel(nn.Module):\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "        self.linear1 = nn.Linear(12, 12)\n",
    "        self.sigmoid1 = nn.Sigmoid()\n",
    "        self.linear2 = nn.Linear(12, 8)\n",
    "        self.sigmoid2 = nn.Sigmoid()\n",
    "        self.linear3 = nn.Linear(8, 2)\n",
    "        self.softmax = nn.Softmax(dim=1)\n",
    "\n",
    "    def forward(self, x):\n",
    "        lin1_out = self.linear1(x)\n",
    "        sigmoid_out1 = self.sigmoid1(lin1_out)\n",
    "        sigmoid_out2 = self.sigmoid2(self.linear2(sigmoid_out1))\n",
    "        return self.softmax(self.linear3(sigmoid_out2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can either use a pretrained model or train the network using the training data for 200 epochs. Note that the results of later steps may not match if retraining. The pretrained model can be downloaded here: https://github.com/pytorch/captum/blob/master/tutorials/models/cifar_torchvision.pt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model Loaded!\n"
     ]
    }
   ],
   "source": [
    "net = TitanicSimpleNNModel()\n",
    "USE_PRETRAINED_MODEL = True\n",
    "\n",
    "if USE_PRETRAINED_MODEL:\n",
    "    net.load_state_dict(torch.load('models/titanic_model.pt'))\n",
    "    print(\"Model Loaded!\")\n",
    "else:\n",
    "    criterion = nn.CrossEntropyLoss()\n",
    "    num_epochs = 200\n",
    "\n",
    "    optimizer = torch.optim.Adam(net.parameters(), lr=0.1)\n",
    "    input_tensor = torch.from_numpy(train_features).type(torch.FloatTensor)\n",
    "    label_tensor = torch.from_numpy(train_labels)\n",
    "    for epoch in range(num_epochs):    \n",
    "        output = net(input_tensor)\n",
    "        loss = criterion(output, label_tensor)\n",
    "        optimizer.zero_grad()\n",
    "        loss.backward()\n",
    "        optimizer.step()\n",
    "        if epoch % 20 == 0:\n",
    "            print ('Epoch {}/{} => Loss: {:.2f}'.format(epoch+1, num_epochs, loss.item()))\n",
    "\n",
    "    torch.save(net.state_dict(), 'models/titanic_model.pt')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now evaluate the training and test accuracies of our model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train Accuracy: 0.8493449781659389\n"
     ]
    }
   ],
   "source": [
    "out_probs = net(input_tensor).detach().numpy()\n",
    "out_classes = np.argmax(out_probs, axis=1)\n",
    "print(\"Train Accuracy:\", sum(out_classes == train_labels) / len(train_labels))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test Accuracy: 0.816793893129771\n"
     ]
    }
   ],
   "source": [
    "test_input_tensor = torch.from_numpy(test_features).type(torch.FloatTensor)\n",
    "out_probs = net(test_input_tensor).detach().numpy()\n",
    "out_classes = np.argmax(out_probs, axis=1)\n",
    "print(\"Test Accuracy:\", sum(out_classes == test_labels) / len(test_labels))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Beyond just considering the accuracy of the classifier, there are many important questions to understand how the model is working and it's decision, which is the purpose of Captum, to help make neural networks in PyTorch more interpretable."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first question we can ask is which of the features were actually important to the model to reach this decision? This is the first main component of Captum, the ability to obtain **Feature Attributions**. For this example, we will apply Integrated Gradients, which is one of the Feature Attribution methods included in Captum. More information regarding Integrated Gradients can be found in the original paper here: https://arxiv.org/pdf/1703.01365.pdf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To apply integrated gradients, we first create an IntegratedGradients object, providing the model object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "ig = IntegratedGradients(net)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To compute the integrated gradients, we use the attribute method of the IntegratedGradients object. The method takes tensor(s) of input examples (matching the forward function of the model), and returns the input attributions for the given examples. For a network with multiple outputs, a target index must also be provided, defining the index of the output for which gradients are computed. For this example, we provide target = 1, corresponding to survival. \n",
    "\n",
    "The input tensor provided should require grad, so we call requires\\_grad\\_ on the tensor. The attribute method also takes a baseline, which is the starting point from which gradients are integrated. The default value is just the 0 tensor, which is a reasonable baseline / default for this task. \n",
    "\n",
    "The returned values of the attribute method are the attributions, which match the size of the given inputs, and delta, which approximates the error between the approximated integral and true integral."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_input_tensor.requires_grad_()\n",
    "attr, delta = ig.attribute(test_input_tensor,target=1, return_convergence_delta=True)\n",
    "attr = attr.detach().numpy()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To understand these attributions, we can first average them across all the inputs and print / visualize the average attribution for each feature."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average Feature Importances\n",
      "age :  -0.454\n",
      "sibsp :  -0.119\n",
      "parch :  -0.056\n",
      "fare :  0.175\n",
      "female :  0.154\n",
      "male :  -0.359\n",
      "embark_C :  0.086\n",
      "embark_Q :  -0.001\n",
      "embark_S :  -0.082\n",
      "class_1 :  0.062\n",
      "class_2 :  0.021\n",
      "class_3 :  -0.159\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAssAAAGECAYAAADA7m3/AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAgAElEQVR4nO3de5hsVX3m8e8rBwgKgsgRUC4nMsSIRkk8EU28EIFEZUYw4i1EYaIh6BCTJ2MSMhhCNHEwxjEX40SiBrwL3sBLFDyKUUYNBwQEvIB4EATloqgIEtHf/LFXS9H26tOHqu463Xw/z1NP7dp71V5r7aruemvVqtqpKiRJkiT9tHtMuwGSJEnS5sqwLEmSJHUYliVJkqQOw7IkSZLUYViWJEmSOgzLkiRJUodhWZIkSeowLEtacknOTvLtJFtPuy2T0PrzgyQ3j1wePeY+1ySpJKsm1c4F1HlCkrcsVX3zSXJkkk9Nux2SZFiWtKSSrAEeCxTwlEWqY8kC5ohjqmrbkcunp9CGn8hgWf6Pn9LjJ0lzWpb/SCUta88FPgOcDBwxszLJo5J8I8kWI+uemuSitnyPJMcm+UqSG5OcmmTHtm1mFPZ5Sb4GfKytP63t8ztJ/j3JQ0b2fd8k70/y3STnJvmr0ZHMJD+f5Kwk30rypSTPuCudnW8/SQ5O8rnWhquSnDBy139v1zfNjFTPHvmdPfrcRrj/Osk5wC3AA5Nsn+QNSa5N8vXWzy1YgLbvFya5LMn3krwsyV5JPt3afGqSrVrZ/ZNcneR/JbkhyYYkh4/sa/skb0pyfZIrk7xkJsy3UeRzkrw6ybeAdwL/DDy69f2mjR2vkWNxRJKvtTYcN7J9i9a2r7S+nJdk9wU8Rk9Ocmm7z9eTvHghx07SymFYlrTUngu8tV1+I8nOAFX1GeD7wBNGyv4W8La2/CLgUODxwP2BbwP/NGvfjwceDPxGu/1vwN7A/YDzW50z/qnVtwtDaB8N7vcCzmp13w94NvDa0bC9EAvYz/fb8dgBOBh4QZJD27bHtesdNnGk+jnAUcB2wJXAKcDtwH8BfhH4deD5m9CNJwKPAB4F/AlwEnA4sDvw0NanGbsAOwEPYDieJyV5UNv2j8D2wAMZHqfnAv995L77AVcwHKffBo4GPt36vkMrM9/xmvEY4EHAAcDxSR7c1v9Ra+uTgXsDvwPcsoDH6A3A71XVdq2/H1vQUZO0YhiWJS2ZJI8B9gROrarzgK8wBOIZb6eFryTbMQSbt7dtvwccV1VXV9VtwAnAYbM+sj+hqr5fVbcCVNUbq+p7I+Uf3kY4twCeBvxFVd1SVZcyhMoZ/xXYUFX/WlW3V9X5wLuBw+bp3j8kualdzl/Ifqrq7Kr6fFX9uKouan19/IIOZt/JVXVJVd0O7Ag8CfjDdlyuA14NPGsT9veKqvpuVV0CXAycWVVXVNV3GN6M/OKs8n9eVbdV1SeADwLPaMf7mcCftcdjA/AqhmA/45qq+sd2nG6dqyELPF5/WVW3VtWFwIXAw9v65wMvqaov1eDCqrqRjT/WPwT2SXLvqvp22y7pbsSwLGkpHcEQtm5ot9/GyIhuu/2bGb7495vA+VV1Zdu2J/DemUAKfAH4EbDzyP2vmlloH7uf2D52/y6woW3aCVgNrBotP2t5T2C/kfB7E8No6i7z9O1FVbVDu/zSQvaTZL8kH29TE77DMJq60zx1LMTsfmwJXDtS/+sYRlAX6psjy7fOcXvbkdvfrqrvj9y+kuFTgJ2Ardrt0W0P6LR7Tgs8Xt8YWb5lpH27M7w5m21jj/XTGN60XZnkExnzi5uSlh+/RCFpSSTZBngGsEWSmUCzNbBDkoe3kb5Lk1zJMBo6OgUDhjD1O1V1zhz7XtMWa2T1bwGHAAcyBOXtGaZuBLieYWrCbsCXW/ndZ9X1iao66C51duH7eRvwGuBJVfWDJH/HHeGv5ij/feCeI7fnCu+j97sKuA3YqY00L7b7JLnXSGDeg2E0+gaGEdo9gUtHtn195L6z+ztX/+c7XhtzFbBXa8/s9d3HqKrOBQ5JsiVwDHAqd36uSFrhHFmWtFQOZRgJ3gfYt10eDHySYR7qjLcxzE9+HHDayPp/Bv46yZ4ASVYnOWSe+rZjCIo3MgTMl89sqKofAe8BTkhyzyQ/P6sNHwB+LslzkmzZLr88Mv91oTa2n+2Ab7Xg90juPCXleuDHDHN8Z1wAPC7JHkm2B/5svsqr6lrgTOBVSe6d4UuSeyUZd6rHfP4yyVZJHsswxeG0drxPZXj8tmuP4R8B8/1M3TeB3dK+QNjMd7w25vXAy5LsncHDktyXeR6j1o/Dk2xfVT8EvsvwHJZ0N2JYlrRUjgD+taq+VlXfmLkwjBQePjL3+O3A/sDHRqZrAPw9cAZwZpLvMfyixn7z1Pcmho/6v84wmvmZWduPYRht/gbw5lbvbQBV9T2GL8I9C7imlXkFw0j4gi1gPy8EXtr6czxDoJy57y3AXwPntOkBj6qqsxh+KeIi4DyGoLcxz2WYAnEpw8j6u4BdN6Ufm+AbrY5rGL5MeXRVfbFt+32GkfErgE8xvCl64zz7+hhwCfCNJDPPg+7xWoD/08qfyRB63wBss4DH6DnAhjaV52iGLx9KuhtJ1VyfdEnS3UuSVwC7VNURGy2sn5Jkf+AtVbXbtNsiSZPkyLKku6X227oPax/JPxJ4HvDeabdLkrR58Qt+ku6utmOYenF/4DqGnzI7faotkiRtdpyGIUmSJHU4DUOSJEnqmEhYTvLEJF9KcnmSY+fY/kdJLk1yUZJ1Mz/9JEmSJG3Oxp6G0U5j+mXgIOBq4Fzg2e30sTNlfg34bFXdkuQFwP5V9cz59rvTTjvVmjVrxmqbJEmStDHnnXfeDVW1eq5tk/iC3yOBy6vqCoAk72A4a9ZPwnJVfXyk/GdYwO9UrlmzhvXr10+geZIkSVJfO3vsnCYxDeMBDKcLnXF1W9fzPODfJlCvJEmStKgmMbKcOdbNObcjyW8Da4E5T7Wa5CjgKIA99thjAk2TJEmS7rpJjCxfDew+cns3hlOG3kmSA4HjgKdU1W1z7aiqTqqqtVW1dvXqOaeNSJIkSUtmEmH5XGDvJD+bZCvgWcAZowWS/CLwOoagfN0E6pQkSZIW3dhhuapuB44BPgJ8ATi1qi5J8tIkT2nFXglsC5yW5IIkZ3R2J0mSJG02JnK666r6EPChWeuOH1k+cBL1SJIkSUvJM/hJkiRJHYZlSZIkqcOwLEmSJHUYliVJkqQOw7IkSZLUYViWJEmSOgzLkiRJUsdEfmdZ0l2z5tgPTrsJG7XhxIOn3QRJkqbGkWVJkiSpw7AsSZIkdRiWJUmSpA7DsiRJktRhWJYkSZI6DMuSJElSh2FZkiRJ6jAsS5IkSR2GZUmSJKnDM/hJmojlcDZC8IyEkqRN48iyJEmS1GFYliRJkjoMy5IkSVKHYVmSJEnqMCxLkiRJHYZlSZIkqcOwLEmSJHUYliVJkqQOw7IkSZLUYViWJEmSOgzLkiRJUodhWZIkSeowLEuSJEkdhmVJkiSpw7AsSZIkdRiWJUmSpA7DsiRJktRhWJYkSZI6DMuSJElSh2FZkiRJ6jAsS5IkSR0TCctJnpjkS0kuT3LsHNsfl+T8JLcnOWwSdUqSJEmLbeywnGQL4J+AJwH7AM9Oss+sYl8DjgTeNm59kiRJ0lJZNYF9PBK4vKquAEjyDuAQ4NKZAlW1oW378QTqkyRJkpbEJKZhPAC4auT21W3dJktyVJL1SdZff/31E2iaJEmSdNdNIixnjnV1V3ZUVSdV1dqqWrt69eoxmyVJkiSNZxJh+Wpg95HbuwHXTGC/kiRJ0lRNIiyfC+yd5GeTbAU8CzhjAvuVJEmSpmrssFxVtwPHAB8BvgCcWlWXJHlpkqcAJPnlJFcDTwdel+SSceuVJEmSFtskfg2DqvoQ8KFZ644fWT6XYXqGJEmStGx4Bj9JkiSpYyIjy5KkzdeaYz847SYsyIYTD552EyTppziyLEmSJHUYliVJkqQOw7IkSZLUYViWJEmSOgzLkiRJUodhWZIkSeowLEuSJEkdhmVJkiSpw7AsSZIkdRiWJUmSpA5Pdy1JkiZiOZxa3dOqa1M5sixJkiR1GJYlSZKkDsOyJEmS1GFYliRJkjoMy5IkSVKHYVmSJEnqMCxLkiRJHYZlSZIkqcOwLEmSJHUYliVJkqQOw7IkSZLUYViWJEmSOgzLkiRJUodhWZIkSeowLEuSJEkdhmVJkiSpw7AsSZIkdRiWJUmSpI5V026AJEnS5mjNsR+cdhM2asOJB0+7CSueI8uSJElSh2FZkiRJ6jAsS5IkSR2GZUmSJKnDsCxJkiR1GJYlSZKkjomE5SRPTPKlJJcnOXaO7VsneWfb/tkkayZRryRJkrSYxg7LSbYA/gl4ErAP8Owk+8wq9jzg21X1X4BXA68Yt15JkiRpsU1iZPmRwOVVdUVV/SfwDuCQWWUOAU5py+8CDkiSCdQtSZIkLZpU1Xg7SA4DnlhVz2+3nwPsV1XHjJS5uJW5ut3+Sitzw6x9HQUcBbDHHns84sorrxyrbXfVSjpjz3LoC6ys/ng2pZXB55qWis81LYXl8DyD6T3XkpxXVWvn2jaJkeW5RohnJ/CFlKGqTqqqtVW1dvXq1RNomiRJknTXTSIsXw3sPnJ7N+CaXpkkq4DtgW9NoG5JkiRp0UwiLJ8L7J3kZ5NsBTwLOGNWmTOAI9ryYcDHatz5H5IkSdIiWzXuDqrq9iTHAB8BtgDeWFWXJHkpsL6qzgDeALw5yeUMI8rPGrdeSZIkabGNHZYBqupDwIdmrTt+ZPkHwNMnUZckSZK0VDyDnyRJktRhWJYkSZI6DMuSJElSh2FZkiRJ6jAsS5IkSR2GZUmSJKnDsCxJkiR1GJYlSZKkDsOyJEmS1GFYliRJkjoMy5IkSVKHYVmSJEnqMCxLkiRJHYZlSZIkqcOwLEmSJHUYliVJkqQOw7IkSZLUYViWJEmSOgzLkiRJUodhWZIkSepYNe0GSJtiw4kHT7sJkiTpbsSRZUmSJKnDsCxJkiR1GJYlSZKkDsOyJEmS1GFYliRJkjoMy5IkSVKHYVmSJEnqMCxLkiRJHYZlSZIkqcOwLEmSJHUYliVJkqQOw7IkSZLUsWraDZAk6e5sw4kHT7sJkubhyLIkSZLUYViWJEmSOgzLkiRJUodhWZIkSeoYKywn2THJWUkua9f36ZT7cJKbknxgnPokSZKkpTTuyPKxwLqq2htY127P5ZXAc8asS5IkSVpS44blQ4BT2vIpwKFzFaqqdcD3xqxLkiRJWlLjhuWdq+pagHZ9v3F2luSoJOuTrL/++uvHbJokSZI0no2elCTJR4Fd5th03KQbU1UnAScBrF27tia9f0mSJGlTbDQsV9WBvW1Jvplk16q6NsmuwHUTbZ0kSZI0ReNOwzgDOKItHwGcPub+JEmSpM3GuGH5ROCgJJcBB7XbJFmb5PUzhZJ8EjgNOCDJ1Ul+Y8x6JUmSpEW30WkY86mqG4ED5li/Hnj+yO3HjlOPJEmSNA2ewU+SJEnqMCxLkiRJHYZlSZIkqWOsOcva/G048eBpN0GSJGnZcmRZkiRJ6jAsS5IkSR2GZUmSJKnDsCxJkiR1GJYlSZKkDsOyJEmS1GFYliRJkjoMy5IkSVKHYVmSJEnqMCxLkiRJHYZlSZIkqcOwLEmSJHUYliVJkqQOw7IkSZLUYViWJEmSOgzLkiRJUodhWZIkSeowLEuSJEkdq6bdAEmSJC2uDScePO0mLFuOLEuSJEkdhmVJkiSpw7AsSZIkdRiWJUmSpA7DsiRJktRhWJYkSZI6DMuSJElSh2FZkiRJ6jAsS5IkSR2GZUmSJKnDsCxJkiR1GJYlSZKkDsOyJEmS1GFYliRJkjoMy5IkSVLHWGE5yY5JzkpyWbu+zxxl9k3y6SSXJLkoyTPHqVOSJElaKuOOLB8LrKuqvYF17fZstwDPraqHAE8E/i7JDmPWK0mSJC26ccPyIcApbfkU4NDZBarqy1V1WVu+BrgOWD1mvZIkSdKiGzcs71xV1wK06/vNVzjJI4GtgK90th+VZH2S9ddff/2YTZMkSZLGs2pjBZJ8FNhljk3HbUpFSXYF3gwcUVU/nqtMVZ0EnASwdu3a2pT9S5IkSZO20bBcVQf2tiX5ZpJdq+raFoav65S7N/BB4CVV9Zm73FpJkiRpCY07DeMM4Ii2fARw+uwCSbYC3gu8qapOG7M+SZIkacmMG5ZPBA5KchlwULtNkrVJXt/KPAN4HHBkkgvaZd8x65UkSZIW3UanYcynqm4EDphj/Xrg+W35LcBbxqlHkiRJmgbP4CdJkiR1GJYlSZKkDsOyJEmS1GFYliRJkjoMy5IkSVKHYVmSJEnqMCxLkiRJHYZlSZIkqcOwLEmSJHUYliVJkqQOw7IkSZLUYViWJEmSOgzLkiRJUodhWZIkSeowLEuSJEkdhmVJkiSpw7AsSZIkdRiWJUmSpA7DsiRJktRhWJYkSZI6DMuSJElSh2FZkiRJ6jAsS5IkSR2GZUmSJKnDsCxJkiR1GJYlSZKkDsOyJEmS1GFYliRJkjoMy5IkSVKHYVmSJEnqMCxLkiRJHYZlSZIkqcOwLEmSJHUYliVJkqQOw7IkSZLUYViWJEmSOgzLkiRJUodhWZIkSeoYKywn2THJWUkua9f3maPMnknOS3JBkkuSHD1OnZIkSdJSGXdk+VhgXVXtDaxrt2e7FviVqtoX2A84Nsn9x6xXkiRJWnTjhuVDgFPa8inAobMLVNV/VtVt7ebWE6hTkiRJWhLjBtedq+pagHZ9v7kKJdk9yUXAVcArquqaMeuVJEmSFt2qjRVI8lFglzk2HbfQSqrqKuBhbfrF+5K8q6q+OUddRwFHAeyxxx4L3b0kSZK0KDYalqvqwN62JN9MsmtVXZtkV+C6jezrmiSXAI8F3jXH9pOAkwDWrl1bG2ubJEmStJjGnYZxBnBEWz4COH12gSS7JdmmLd8H+FXgS2PWK0mSJC26ccPyicBBSS4DDmq3SbI2yetbmQcDn01yIfAJ4G+r6vNj1itJkiQtuo1Ow5hPVd0IHDDH+vXA89vyWcDDxqlHkiRJmgZ/xk2SJEnqMCxLkiRJHYZlSZIkqcOwLEmSJHUYliVJkqQOw7IkSZLUYViWJEmSOgzLkiRJUodhWZIkSeowLEuSJEkdhmVJkiSpw7AsSZIkdRiWJUmSpA7DsiRJktRhWJYkSZI6DMuSJElSh2FZkiRJ6jAsS5IkSR2GZUmSJKnDsCxJkiR1GJYlSZKkDsOyJEmS1GFYliRJkjpWTbsBkrQ52nDiwdNugiRpM+DIsiRJktRhWJYkSZI6DMuSJElSh2FZkiRJ6jAsS5IkSR2GZUmSJKnDsCxJkiR1GJYlSZKkDsOyJEmS1GFYliRJkjoMy5IkSVKHYVmSJEnqMCxLkiRJHYZlSZIkqWOssJxkxyRnJbmsXd9nnrL3TvL1JK8Zp05JkiRpqYw7snwssK6q9gbWtds9LwM+MWZ9kiRJ0pIZNywfApzSlk8BDp2rUJJHADsDZ45ZnyRJkrRkxg3LO1fVtQDt+n6zCyS5B/Aq4I/HrEuSJElaUqs2ViDJR4Fd5th03ALreCHwoaq6KsnG6joKOApgjz32WODuJUmSpMWx0bBcVQf2tiX5ZpJdq+raJLsC181R7NHAY5O8ENgW2CrJzVX1U/Obq+ok4CSAtWvX1kI7IUmSJC2GjYbljTgDOAI4sV2fPrtAVR0+s5zkSGDtXEFZkiRJ2tyMO2f5ROCgJJcBB7XbJFmb5PXjNk6SJEmaprFGlqvqRuCAOdavB54/x/qTgZPHqVOSJElaKp7BT5IkSeowLEuSJEkdhmVJkiSpw7AsSZIkdRiWJUmSpA7DsiRJktRhWJYkSZI6DMuSJElSh2FZkiRJ6jAsS5IkSR2GZUmSJKnDsCxJkiR1GJYlSZKkDsOyJEmS1GFYliRJkjoMy5IkSVKHYVmSJEnqMCxLkiRJHYZlSZIkqcOwLEmSJHWsmnYDNkcbTjx42k2QJEnSZsCRZUmSJKnDsCxJkiR1GJYlSZKkDsOyJEmS1GFYliRJkjoMy5IkSVKHYVmSJEnqMCxLkiRJHYZlSZIkqcOwLEmSJHUYliVJkqQOw7IkSZLUYViWJEmSOgzLkiRJUkeqatptmFOS64Erp92OCdoJuGHajZiQldQXWFn9WUl9gZXVn5XUF7A/m7OV1BdYWf1ZSX2BldWfPatq9VwbNtuwvNIkWV9Va6fdjklYSX2BldWfldQXWFn9WUl9AfuzOVtJfYGV1Z+V1BdYef3pcRqGJEmS1GFYliRJkjoMy0vnpGk3YIJWUl9gZfVnJfUFVlZ/VlJfwP5szlZSX2Bl9Wcl9QVWXn/m5JxlSZIkqcORZUmSJKnDsKw5JXl9kn3a8s3Tbs+0JFmT5OJpt2NGkhcl+UKSt067LQux2O1NckKSFy/Gvhdbkv2TfGDa7ZAkzc+wrDlV1fOr6tJpt2OpJFk17TYs0AuBJ1fV4RsruJn0acHt1eYjyZFJXjPmPjYk2WmBZXdJ8o4kX0lyaZIPJfm5ceof2fdS92W3JKcnuSzJFUlek2Trceqftf+l7s9xSS5JclGSC5LsN2bdi/4GN8kbk1y3FAMdi92fJLsn+XgbdLgkyR8sYl2L3ZefSfIfSS5sffnLxapr0gzLE5LkfUnOa0+Ao9q65yX5cpKzk/zLzD+4JKuTvDvJue3yq1Nu+72SfLA9gS9O8szW5rUjZV6V5Pwk65Ksbute1F7YLkryjrbuhCRvTvKx9mLxu0vYjzVJvpjklNamdyW5Z5Lj23G+OMlJSdLKn53k5Uk+AfxBkp2TvLcdhwuT/Erb9Rbt8bskyZlJtlmqPs3q3z8DDwTOSPKnSf5fks+16we1MkcmOS3J+4Ez27o/bv2/aCn/Oc1q73HtBezc1uZDRtr7viTvT/LVJMck+aNW5jNJdmzlfrfd98L2t3PPOerbK8mH29/hJ5P8/BL0ceY59/r2/HprkgOTnNOe/49sl596rGbt515zHZ/lKskWm1A2wHuBs6tqr6raB/hfwM6L1b5NcRf68h7gfVW1N7A3sA3wN4vUvE22if15NPBfgV+qqocBBwJXLVbbJuhk4InTbsSE3A78z6p6MPAo4H+kfeq7DN0GPKGqHg7sCzwxyaOm3KaFqSovE7gAO7brbYCLgQcAG4AdgS2BTwKvaWXeBjymLe8BfGHKbX8a8C8jt7cHzgbWttsFHN6Wjx/pxzXA1m15h3Z9AnBhOw47Mfxjvf8S9WNNa+uvtttvBF4889i0dW8G/ltbPht47ci2dwJ/2Ja3aMdhDcM/q33b+lOB357iY7WhHdd7A6vaugOBd7flI4GrR56Pv87wbeUwvDn+APC4KbT35TPHDdgB+DJwr9bey4HtgNXAd4CjW7lXjzwe9x3Z518Bvz/yfHtxW14H7N2W9wM+tkTPuduBX2jH97z2vAtwCPC+eR6r/YEPtOU5j88E2vfbwH8AFwCva8/rm4FXtLZ+FHhk+1u4AnjKyPPodODDwJeAvxjZ5/vafS8BjhpZfzPwUuCzwGNGHvtt2n5+t9PGJwD/vkL6csDsvrTH/9vAtsuwP78JvH/M5+BzgYsYXhfezJ3/Zn8XOLdtezdwz7b+6QyvoxfOHE/gISPH6yLa3/pG/jYvXoS/+an0Z6T+04GDlntfgHsC5wP7TfoxWozL1BuwUi7cERIvZHjBPxY4ZWT7i7gjZF7XnlQzl68D202x7T8HfJXhn/Rj27qzuSMs/4g7XuwfCFzQlj8MvIvhn/62I8fhpSP7fhNw6BL1Yw3wtZHbT2B48Xgaw4vE59uxPnakj48fKX89LfzP2udlI7f/FHjJFB+rDQwvcrszjMZd3Pr1xbb9SOBfR8r/bbvPzHPtcuB5U2jv+tbWmXZ8DXhwa+/oG7WvAQ9oy78D/F1bfjzDG87Pt+fqP488314MbAvcOuvvatHfhM7x/HgTd7yxfGBrR++x2p87wvKcx2fMtj0YeD+wZbv9WoYXxwKe1Na9l+ETiC2Bh3PH3/aRwLXAfbljAGDm/8HsgYH7ttsFPGPWY7+GIfQ9d552vgh49UruC/A52hvuZdafbRmej19ubXz8Jj4HH8IQ6HeaaR93DmS9N8Gf547/AzMDMf/IHX9bWwHbLOBvc6JheZr9GenT14B7L9e+MLwpvID2xnCSj89iXjaHOY3LXpL9GUaMHl1VtyQ5m+FJ+ODOXe7Ryt66NC2cX1V9OckjgCcD/zvJmRu7S7s+GHgc8BTgz5M8ZNb22eWXwlx1v5bhxeSqJCcAPzOy/fsL2OdtI8s/YnghmraXAR+vqqcmWcMQ/GeM9inA/66q1y1d0+YU4GlV9aU7rRzmP44e3x+P3P4x/OR/1MkMb7ouTHIkQ9AcdQ/gpqrad7LNXpCNtX++x2rGnMdnTAcAjwDObTOPtmF4o/6fDG90YXjhu62qfpjk8wwvxjPOqqobAZK8h2FEcj3woiRPbWV2Z5hqcCPD38a7Z7XhdOBvqmrcL3gul76Euf/fZTn2p6pubq8NjwV+DXhnkmOr6uTefWZ5AvCuqrqh7e9brb8zHprkrxg+TdkW+Ehbfw5wcpJTGaa1AHwaOC7JbsB7quqyBbZhkqbWnyTbMjyGf1hV312ufamqHwH7JtkBeG+Sh1bVZvMl+h7nLE/G9sC3W1D+eYZ5RfcEHp/kPhm+aPW0kfJnAsfM3EgyjRf4n0hyf+CWqnoLw0jkL80qcg/gsLb8W8CnktwD2L2qPg78CXf8QQEckmEi/30ZQs25i9yFUXu0eXYAzwY+1ZZvaP9sDpv7bsDwMf4LYJjXl+Tei9fMsW3PMEoOw0hTz0eA32l9J8kDktxvkdvWa8fvtzmdJPnFTbz/dsC1SbYEfurLgu3F46tJnt72nyQPH7PNk7KQx2rc4zOXMHy6tW+7PKiqTgB+WG2Ih5FwX1Wjb05gjjeeswYGHs4wYjrz5vMH7YVw1DnAk2b61XEJQ3BcKX1ZO7qi/R/ZmWEAZbn1h6r6UVWdXVV/wfC69bT5ys/Se3cZq3EAAAVNSURBVPMw42TgmKr6BeAvZ9pbVUcDL2EI/BckuW9VvY1hYOZW4CNJnrAJ7ZiUqfSn/d97N/DWqnpPr9wmmupjU1U3MQwcLIu55YblyfgwsCrJRQyjSJ9heHF8OcPH/x8FLmWYngHDR3VrM3zh6lLg6KVv8p38AvAfSS4AjmP4yGXU94GHJDmP4d3oSxk+SnlLG/H4HMNHjze18v8BfJDhOLysqq5Zgj7M+AJwRHssdgT+L/AvDKM072P+4P4HwK+1Pp3H8DHV5upvGD4FOIfhsZhTVZ3JMEf+061f72IInkvtZQwfJ1+U4RvqL9vE+/85w9/SWcAXO2UOB56X5EKG0LK5fEluIY/VuMdnLuuAw2beHCXZMcmem3D/g9p9tgEOZQhXcw0MzOd4hpHN185T5mPA1hn5MnCSX07y+GXYl3XAPZM8t7VzC+BVDFPwbp1VbrPvT5IHJdl7ZNW+wJWb0M51wDPawAlpX9gdMeeb4CR7VdVnq+p44AZg9yQPBK6oqn8AzgAetgntmJQl7097M/MGhmll/2eZ92V1G1GmPXcPpP//fPMyrfkfd4cLd8zjXcUwP+2p027TEvT5BNqcpynUvYZF+EKHFy/L9QI8kzu+dHMeQ4C6eWT7nf5eZ7YxjICfyvCm9ydfIgO2Bv6t7e80hpGh/UfvO7KvDQzz1QP8K8NH/r123r/V9xWGNzofZNaXhJZRX3ZnCAyXATcBr1uujw3DiP//YxjsuYjhY/edNvE5eAR3fCHsZO48L/YFDN9BOJth3uvJbf17GAY4Lgb+vrXzz9pz4wKGAaod56nz7Qzzun/I8IXniX1PY6n7wzDFptrxn/lOw5OXaV8exjC4dlG7//GTelwW++LprhdRkr9leOf0MwxTL/6gVvgBzzAn+Oaq+tsp1L2G4QtTD13quiVptgw/P/l24Der6rxpt0fSXWNYliRJkjr8NQxJ0qJrcyPXzbHpgGq/7LBcrKS+wPLsz3Js83xWUn9WUl9mOLIsSZIkdfhrGJIkSVKHYVmSJEnqMCxL0pQk+VGSC0Yua+7CPnZI8sLJt06SBM5ZlqSpSXJzVW278ZLz7mMNd+EnE5NsUT99VjdJ0iyOLEvSZqSdav2VSc5tZ/n8vbZ+2yTrkpyf5PNJZs5QeCKwVxuZfmWS/ZN8YGR/r0lyZFvekOT4JJ8Cnp5kryQfTnJekk+2M7+R5OlJLk5yYZJ/X9ojIEmbF386TpKmZ5t2mnmAr1bVU4HnAd+pql9OsjVwTpIzgasYzgL63SQ7AZ9JcgZwLPDQqtoXIMn+G6nzB1X1mFZ2HXB0VV2WZD+GUx8/geFUyL9RVV+fOT2tJN1dGZYlaXpunQm5I34deFiSw9rt7YG9GU7b+/IkjwN+DDwA2Pku1PlOGEaqgV8BTksys23rdn0OcHKSUxlObytJd1uGZUnavAT4/ar6yJ1WDlMpVgOPqKofJtkA/Mwc97+dO0+xm13m++36HsBNc4R1quroNtJ8MHBBkn2X68kEJGlczlmWpM3LR4AXJNkSIMnPJbkXwwjzdS0o/xqwZyv/PWC7kftfCeyTZOsk2wMHzFVJVX0X+GqSp7d6kuThbXmvqvpsVR0P3ADsPvluStLy4MiyJG1eXg+sAc7PMD/ieuBQ4K3A+5OsBy4AvghQVTcmOSfJxcC/VdUft+kTFwGXAZ+bp67Dgf+b5CXAlsA7gAuBVybZm2GUe11bJ0l3S/50nCRJktThNAxJkiSpw7AsSZIkdRiWJUmSpA7DsiRJktRhWJYkSZI6DMuSJElSh2FZkiRJ6jAsS5IkSR3/H4t4oiHqK8DDAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 864x432 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Helper method to print importances and visualize distribution\n",
    "def visualize_importances(feature_names, importances, title=\"Average Feature Importances\", plot=True, axis_title=\"Features\"):\n",
    "    print(title)\n",
    "    for i in range(len(feature_names)):\n",
    "        print(feature_names[i], \": \", '%.3f'%(importances[i]))\n",
    "    x_pos = (np.arange(len(feature_names)))\n",
    "    if plot:\n",
    "        plt.figure(figsize=(12,6))\n",
    "        plt.bar(x_pos, importances, align='center')\n",
    "        plt.xticks(x_pos, feature_names, wrap=True)\n",
    "        plt.xlabel(axis_title)\n",
    "        plt.title(title)\n",
    "visualize_importances(feature_names, np.mean(attr, axis=0))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From the feature attribution information, we obtain some interesting insights regarding the importance of various features. We see that the strongest features appear to be age and being male, which are negatively correlated with survival. Embarking at Queenstown and the number of parents / children appear to be less important features generally."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An important thing to note is that the average attributions over the test set don't necessarilly capture all the information regarding feature importances. We should also look at the distribution of attributions for each feature. It is possible that features have very different attributions for different examples in the dataset. \n",
    "\n",
    "For instance, we can visualize the distribution of attributions for sibsp, the number of siblings / spouses."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAEICAYAAACktLTqAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAXOElEQVR4nO3dfbRldX3f8fdHQDRCBJwBYUAGlFiRFUc7oSxNVkzAiGgEu0SxPowGF7HRGI1tBTWJWqnYVE3SNrowWAd8AHwqRLGCKEXbCA4WeRCUERCGQWYQefIBBL79Y/+GHC73zj33aWbuz/drrbPOPr/923t/f2ff8zn77rPvuakqJEl9ecTWLkCSNP8Md0nqkOEuSR0y3CWpQ4a7JHXIcJekDhnuW0GSDyf5i3la1xOS3J1ku/b4giSvnY91t/V9Kcmq+VrfDLb7niS3JvnRLJZ9sOYkr07yjfmvcMvodf9OJ8n1SQ7b2nUsZob7PGs/lD9PcleS25P83ySvS/Lgc11Vr6uq/zjmujb7A15VN1TVTlV1/zzU/s4kH5+w/udV1eq5rnuGdewDvAU4sKoeP0WftyW5rgXfuiRnbJq3NWoeqeuCJD9JsuOE9o8lec+Eti73b5ITklw4SfuSJPcmOWg+t6fJGe4L4w+ramdgX+Ak4K3AKfO9kSTbz/c6txH7Aj+uqg2TzWxHmq8EDquqnYCVwPlbsL5JJVkO/A5QwAvnYX2Ldf+eBjwzyX4T2o8BLq+qK7ZCTb96qsrbPN6A6xlCZ7TtYOAB4KD2+GPAe9r0EuALwO3AbcDXGd50T2vL/By4G/gPwHKG4DgWuAG4cKRt+7a+C4D3AhcDdwBnAbu1ec8G1k1WL3A4cC/wy7a974ys77Vt+hHAO4AfAhuAU4HHtnmb6ljVarsVePtmnqfHtuU3tvW9o63/sDbmB1odH5tk2f8G/M1m1j1a86uB/wP81/Z8XA0cOtL31cC1wF3AdcDLx1luiu3+ZVvmA8AXRtqPa8/rvW1M//grsH/PBf5yQtvFwBvb9BOBrwI/buv6BLDLZK8jRl4vk40T2Av4LMPP0nWbtjHy2lsD3AncAnxga2fEFsuirV1AbzcmCffWfgPwb9v0gz+s7YX6YWCHdvsdIJOta+QFdirwGODRU7z4bwIOan0+C3y8zZvyxd+m37mp78j80Rf/HwFrgf2BnYDPAadNqO0jra6nAfcAT5nieTqVIZh2bst+Hzh2qjonLPsKhjfCf89w1L7dZmp+NXAf8Ob2/L6UIRR3a8/PncCTW989gadOt9xm6loL/AnwLxlCdI+ReQ/u86l+Vjrbvy8Hrhl5/GSGN5el7fGTgOcAOwJLGd7I/maKuh/y3I2Ok+EN6RKGN9ZHttqvBZ7b5v8T8Mo2vRNwyNbOiC1187TMlrOeIVAm+iVDqOxbVb+sqq9X+0ncjHdW1U+r6udTzD+tqq6oqp8CfwG8ZNMHcnP0coYjn2ur6m7gBOCYCacP3lVVP6+q7wDfYQiBh2i1vBQ4oaruqqrrgfcznGqZVlV9HPhT4LnA/wY2JDl+M4tsYAiOX1bVGcD3gOe3eQ8AByV5dFXdXFVXjrncxDH9NsPppDOr6hLgB8C/GWc8k1jU+7f5PLBHkme2x68CvlRVGwGqam1VnVdV97S2DwC/O4uaf4vhDePdVXVvVV3L8AZ0TJv/S+BJSZZU1d1V9c1ZbGNRMty3nGUMR5sT/TXD0dK5Sa6dJqQ2uXEG83/IcOS5ZKwqN2+vtr7RdW8P7DHSNnp1y88YjpYmWsJwlDVxXcvGLaSqPlFVhwG7AK8D3p3kuVN0v2nCG+YPgb1aOL60LX9zki8m+RfTLTfFNlYB51bVre3xJ1vbbCz2/UtV/Qz4NPCqJGF443jwg9skuyc5PclNSe4EPj7LMewL7NUuXrg9ye3A20ZqPhb4DeDqJN9K8oJZbGNRMty3gCS/xRBcD7skrx25vqWq9gf+EPjzJIdumj3FKqc7st9nZPoJDEcvtwI/BX5tpK7tGH4lHne96xleTKPrvo/hXOZM3Npqmrium2a4HtpR9aeByxhOVUxmWQuY0W2tb8t/uaqew/Db09UMR33TLjcqyaOBlwC/m+RH7fLNNwNPS7LpyHay57bX/bvJaobn5TkMp9++MDLvva2e36yqX2c41ZaHrWHwkHEBo1dQ3QhcV1W7jNx2rqojAKrqmqp6GbA78D7gM0keM8vxLCqG+wJK8uvtSOF0hnOdl0/S5wVJntRC5E7g/naD4UW1/yw2/YokByb5NeDdwGdquJTu+8Cjkjw/yQ4MH56NXrJ3C7B89LLNCT4FvDnJfkl2Av4TcEZV3TeT4lotZwInJtk5yb7AnzMcvU2rXbv+/LbsI5I8D3gqcNEUi+wOvDHJDkmOBp4CnJNkjyQvbC/2exg+aLx/uuUmWf9RbbkDgRXt9hSGD8df1fpMti+73L8jvs5wocDJwOlVde/IvJ0Znu/bkyxj+PxkKpcCRyTZLcnjgTeNzLsYuDPJW5M8Osl2SQ5qB1QkeUWSpVX1QKsFHrqPu2W4L4x/THIXw1HF2xnOJ75mir4HAF9h+EH/J+Dvq+qCNu+9wDvar5v/bgbbP43hQ6gfAY8C3ghQVXcwfOD3DwxHyT8F1o0s9+l2/+Mk355kvR9t676Q4aqEXzCc+56NP23bv5bhN5pPtvWP406GX71vYHjB/meGD6un+mOlixie51uBE4EXV9WPGX7+38JwxHobwznfPxljuYlWAf+jhmvSf7TpxnBVz8vbOetTgAPbvvyfbbme9y/tlNapDL8NnDph9ruAZzB8SP1Fhg9vp3Iaw/n96xmuwhn9m4b7GX7jXdFqvpVh/I9tXQ4HrkxyN/C3wDFV9YvZjmkx2XRVhqQRSV7NcBXJb2/tWqTZ8MhdkjpkuEtShzwtI0kd8shdkjq0TXwx0ZIlS2r58uVbuwxJWlQuueSSW6tq6WTztolwX758OWvWrNnaZUjSopLkh1PN87SMJHXIcJekDhnuktQhw12SOmS4S1KHDHdJ6pDhLkkdMtwlqUOGuyR1aJv4C1VJ+lWy/PgvPjh9/UmT/s/1OfPIXZI6ZLhLUocMd0nqkOEuSR0y3CWpQ4a7JHXIcJekDk0b7kkeleTiJN9JcmWSd7X2/ZJclOSaJGckeWRr37E9XtvmL1/YIUiSJhrnyP0e4Per6mnACuDwJIcA7wM+WFUHAD8Bjm39jwV+UlVPAj7Y+kmStqBpw70Gd7eHO7RbAb8PfKa1rwaOatNHtse0+YcmybxVLEma1ljn3JNsl+RSYANwHvAD4Paquq91WQcsa9PLgBsB2vw7gMdNss7jkqxJsmbjxo1zG4Uk6SHGCvequr+qVgB7AwcDT5msW7uf7Ci9HtZQdXJVrayqlUuXLh23XknSGGZ0tUxV3Q5cABwC7JJk0xeP7Q2sb9PrgH0A2vzHArfNR7GSpPGMc7XM0iS7tOlHA4cBVwFfA17cuq0CzmrTZ7fHtPlfraqHHblLkhbOOF/5uyewOsl2DG8GZ1bVF5J8Fzg9yXuA/wec0vqfApyWZC3DEfsxC1C3JGkzpg33qroMePok7dcynH+f2P4L4Oh5qU6SNCv+haokdchwl6QOGe6S1CHDXZI6ZLhLUocMd0nqkOEuSR0y3CWpQ4a7JHXIcJekDhnuktQhw12SOmS4S1KHDHdJ6pDhLkkdMtwlqUOGuyR1yHCXpA4Z7pLUIcNdkjpkuEtShwx3SeqQ4S5JHZo23JPsk+RrSa5KcmWSP2vt70xyU5JL2+2IkWVOSLI2yfeSPHchByBJerjtx+hzH/CWqvp2kp2BS5Kc1+Z9sKr+y2jnJAcCxwBPBfYCvpLkN6rq/vksXJI0tWmP3Kvq5qr6dpu+C7gKWLaZRY4ETq+qe6rqOmAtcPB8FCtJGs+MzrknWQ48HbioNb0hyWVJPppk19a2DLhxZLF1TPJmkOS4JGuSrNm4ceOMC5ckTW3scE+yE/BZ4E1VdSfwIeCJwArgZuD9m7pOsng9rKHq5KpaWVUrly5dOuPCJUlTGyvck+zAEOyfqKrPAVTVLVV1f1U9AHyEfz71sg7YZ2TxvYH181eyJGk641wtE+AU4Kqq+sBI+54j3V4EXNGmzwaOSbJjkv2AA4CL569kSdJ0xrla5lnAK4HLk1za2t4GvCzJCoZTLtcDfwxQVVcmORP4LsOVNq/3ShlJ2rKmDfeq+gaTn0c/ZzPLnAicOIe6JElz4F+oSlKHDHdJ6pDhLkkdMtwlqUOGuyR1yHCXpA4Z7pLUIcNdkjpkuEtShwx3SeqQ4S5JHTLcJalDhrskdchwl6QOGe6S1CHDXZI6ZLhLUocMd0nqkOEuSR0y3CWpQ4a7JHXIcJekDhnuktQhw12SOjRtuCfZJ8nXklyV5Mokf9bad0tyXpJr2v2urT1J/i7J2iSXJXnGQg9CkvRQ4xy53we8paqeAhwCvD7JgcDxwPlVdQBwfnsM8DzggHY7DvjQvFctSdqsacO9qm6uqm+36buAq4BlwJHA6tZtNXBUmz4SOLUG3wR2SbLnvFcuSZrSjM65J1kOPB24CNijqm6G4Q0A2L11WwbcOLLYutY2cV3HJVmTZM3GjRtnXrkkaUpjh3uSnYDPAm+qqjs313WStnpYQ9XJVbWyqlYuXbp03DIkSWMYK9yT7MAQ7J+oqs+15ls2nW5p9xta+zpgn5HF9wbWz0+5kqRxjHO1TIBTgKuq6gMjs84GVrXpVcBZI+2valfNHALcsen0jSRpy9h+jD7PAl4JXJ7k0tb2NuAk4MwkxwI3AEe3eecARwBrgZ8Br5nXiiVJ05o23KvqG0x+Hh3g0En6F/D6OdYlSZoD/0JVkjpkuEtShwx3SeqQ4S5JHTLcJalDhrskdchwl6QOGe6S1CHDXZI6ZLhLUocMd0nqkOEuSR0y3CWpQ4a7JHXIcJekDhnuktQhw12SOmS4S1KHDHdJ6pDhLkkdMtwlqUOGuyR1yHCXpA4Z7pLUoWnDPclHk2xIcsVI2zuT3JTk0nY7YmTeCUnWJvlekucuVOGSpKmNc+T+MeDwSdo/WFUr2u0cgCQHAscAT23L/H2S7earWEnSeKYN96q6ELhtzPUdCZxeVfdU1XXAWuDgOdQnSZqFuZxzf0OSy9ppm11b2zLgxpE+61rbwyQ5LsmaJGs2btw4hzIkSRPNNtw/BDwRWAHcDLy/tWeSvjXZCqrq5KpaWVUrly5dOssyJEmTmVW4V9UtVXV/VT0AfIR/PvWyDthnpOvewPq5lShJmqlZhXuSPUcevgjYdCXN2cAxSXZMsh9wAHDx3EqUJM3U9tN1SPIp4NnAkiTrgL8Cnp1kBcMpl+uBPwaoqiuTnAl8F7gPeH1V3b8wpUuSpjJtuFfVyyZpPmUz/U8ETpxLUZKkufEvVCWpQ4a7JHXIcJekDhnuktQhw12SOmS4S1KHDHdJ6pDhLkkdMtwlqUOGuyR1yHCXpA4Z7pLUIcNdkjpkuEtShwx3SeqQ4S5JHTLcJalDhrskdchwl6QOGe6S1CHDXZI6ZLhLUocMd0nq0LThnuSjSTYkuWKkbbck5yW5pt3v2tqT5O+SrE1yWZJnLGTxkqTJjXPk/jHg8AltxwPnV9UBwPntMcDzgAPa7TjgQ/NTpiRpJqYN96q6ELhtQvORwOo2vRo4aqT91Bp8E9glyZ7zVawkaTyzPee+R1XdDNDud2/ty4AbR/qta20Pk+S4JGuSrNm4ceMsy5AkTWa+P1DNJG01WceqOrmqVlbVyqVLl85zGZL0q2224X7LptMt7X5Da18H7DPSb29g/ezLkyTNxmzD/WxgVZteBZw10v6qdtXMIcAdm07fSJK2nO2n65DkU8CzgSVJ1gF/BZwEnJnkWOAG4OjW/RzgCGAt8DPgNQtQsyRpGtOGe1W9bIpZh07St4DXz7UoSdLc+BeqktQhw12SOmS4S1KHDHdJ6pDhLkkdMtwlqUOGuyR1yHCXpA4Z7pLUIcNdkjpkuEtShwx3SeqQ4S5JHTLcJalDhrskdchwl6QOGe6S1CHDXZI6ZLhLUocMd0nqkOEuSR0y3CWpQ4a7JHXIcJekDm0/l4WTXA/cBdwP3FdVK5PsBpwBLAeuB15SVT+ZW5mSpJmYjyP336uqFVW1sj0+Hji/qg4Azm+PJUlb0EKcljkSWN2mVwNHLcA2JEmbMddwL+DcJJckOa617VFVNwO0+90nWzDJcUnWJFmzcePGOZYhSRo1p3PuwLOqan2S3YHzklw97oJVdTJwMsDKlStrjnVIkkbM6ci9qta3+w3A54GDgVuS7AnQ7jfMtUhJ0szMOtyTPCbJzpumgT8ArgDOBla1bquAs+ZapCRpZuZyWmYP4PNJNq3nk1X1v5J8CzgzybHADcDRcy9TkjQTsw73qroWeNok7T8GDp1LUZKkufEvVCWpQ4a7JHXIcJekDhnuktQhw12SOmS4S1KHDHdJ6pDhLkkdMtwlqUOGuyR1yHCXpA4Z7pLUIcNdkjpkuEtShwx3SeqQ4S5JHTLcJalDhrskdchwl6QOGe6S1CHDXZI6ZLhLUocMd0nq0PZbuwBJ2lYtP/6LD05ff9Lzx543WZ8tbcHCPcnhwN8C2wH/UFUnLdS2JGkq44TwVP0XswUJ9yTbAf8deA6wDvhWkrOr6rsLsb2ezPQHcVvkGLbc+rfEcz2XbUy17FyOeudrnAsV4tvKm8NCHbkfDKytqmsBkpwOHAnMe7j3ECTj+FUZ50zNV/DMZvltzTihsiXeABbaOKE/VZ/eA31Uqmr+V5q8GDi8ql7bHr8S+FdV9YaRPscBx7WHTwa+N++FbF1LgFu3dhHzzDEtHj2OyzE93L5VtXSyGQt15J5J2h7yLlJVJwMnL9D2t7oka6pq5dauYz45psWjx3E5pplZqEsh1wH7jDzeG1i/QNuSJE2wUOH+LeCAJPsleSRwDHD2Am1LkjTBgpyWqar7krwB+DLDpZAfraorF2Jb27AeTzk5psWjx3E5phlYkA9UJUlbl18/IEkdMtwlqUOG+zxJ8tdJrk5yWZLPJ9llin7XJ7k8yaVJ1mzpOmdiBmM6PMn3kqxNcvyWrnMmkhyd5MokDySZ8hK0Rbafxh3TotlPAEl2S3Jekmva/a5T9Lu/7adLk2yTF25M99wn2THJGW3+RUmWz3Wbhvv8OQ84qKp+E/g+cMJm+v5eVa1YBNfsTjumka+aeB5wIPCyJAdu0Spn5grgXwMXjtF3seynace0CPcTwPHA+VV1AHB+ezyZn7f9tKKqXrjlyhvPmM/9scBPqupJwAeB9811u4b7PKmqc6vqvvbwmwzX9i9qY47pwa+aqKp7gU1fNbFNqqqrqqqrv4Yec0yLaj81RwKr2/Rq4KitWMtcjPPcj471M8ChSSb7Y9CxGe4L44+AL00xr4Bzk1zSvoJhsZhqTMuAG0cer2tti91i3U9TWYz7aY+quhmg3e8+Rb9HJVmT5JtJtsU3gHGe+wf7tAOqO4DHzWWjfp/7DCT5CvD4SWa9varOan3eDtwHfGKK1TyrqtYn2R04L8nVVTXOKYIFMQ9jmvarJra0ccY0hkW3n6ZbxSRtW/066M2NawareULbV/sDX01yeVX9YH4qnBfjPPfzvn8M9xmoqsM2Nz/JKuAFwKE1xR8QVNX6dr8hyecZfmXbaqExD2Pa5r5qYroxjbmORbWfxrDN7SfY/LiS3JJkz6q6OcmewIYp1rFpX12b5ALg6cC2FO7jPPeb+qxLsj3wWOC2uWzU0zLzpP1zkrcCL6yqn03R5zFJdt40DfwBw4dh26RxxkSHXzWx2PbTmBbjfjobWNWmVwEP+w0lya5JdmzTS4BnsQBfLT5H4zz3o2N9MfDVqQ4Qx1ZV3ubhBqxlOGd2abt9uLXvBZzTpvcHvtNuVzL8Sr3Va5/LmNrjIxiupvnBIhjTixiOku4BbgG+3MF+mnZMi20/tXofx3CVzDXtfrfWvpLhv7sBPBO4vO2ry4Fjt3bdU4zlYc898G6GAyeARwGfbq+5i4H957pNv35AkjrkaRlJ6pDhLkkdMtwlqUOGuyR1yHCXpA4Z7pLUIcNdkjr0/wGW9omx+5ZJMQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.hist(attr[:,1], 100);\n",
    "plt.title(\"Distribution of Sibsp Attribution Values\");"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We note that a vast majority of the examples have an attribution value of 0 for sibsp, which likely corresponds to having a value of 0 for the feature (IntegratedGradients would provide an attribution of 0 when the feature value matches the baseline of 0). More significantly, we see that although the average seems smaller in magnitude in the plot above, there are a small number of examples with extremely negative attributions for this feature."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To better understand this, we can bucket the examples by the value of the sibsp feature and plot the average attribution for the feature. In the plot below, the size of the dot is proportional to the number of examples with that value."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZAAAAEGCAYAAABLgMOSAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAgAElEQVR4nO3deZxcVZ338c83ewhJICRASIAQCDCgY8QWxWhkSdhGlihLcPRBB40LKujjgsvj4DaDPuPgzCgOYZGoCAxLBAVkk7CICJ0YshBDWALERAgkLEnI1v2bP+5pKDpV1dW3u7qq6O/79apX3Xvuuef+OvDqX99z7j1HEYGZmVln9al1AGZm1picQMzMLBcnEDMzy8UJxMzMcnECMTOzXPrVOoCeNHLkyBg3blytwzAzayhz5859LiJGtS/vVQlk3LhxNDc31zoMM7OGIunJYuXuwjIzs1ycQMzMLBcnEDMzy6VXjYF01iubW/jNgpX8/I/LeealTWzc0sLg/n0Zu+Ng/unde3HkAbsyoJ9zsJn1TupNc2E1NTVFJYPoz6/bxH/csYyrm1cgwYbNLdvUGTKwL30kPvzOPfn0Yfuw/UDnYjN7Y5I0NyKa2pf7t147j61ex/SZ9/PC+s1saS2dXNdvypLKJfc+wc2LVnHFxw9h1+GDeipMM7Oaq3n/i6SjJS2V9Kikc4ocHyjpqnT8T5LGFRz7aipfKumorsayYu0G3n/BfTy3blPZ5FFo09ZWnlrzCide8AfWrt/c1RDMzBpGTROIpL7AT4BjgAOA0yQd0K7aGcDaiNgHOB/4fjr3AGA6cCBwNHBBai+XrS2tnHbR/azbuJXO9uq1tAZr1m3io5c9mPfyZmYNp9Z3IAcDj0bE4xGxGbgSOKFdnROAWWn7GuAISUrlV0bEpoh4Ang0tZfL7//yLGvWbaYl55jQ5pZg6d9e5qGnX8gbgplZQ6l1AhkDPF2wvyKVFa0TEVuBF4GdKjwXSTMkNUtqXr16dclA/vuux1hfZLC8MzZtbeGiex7vUhtmZo2i1glERcra3wKUqlPJuUTEzIhoioimUaO2mcoFgCeeW8/ilS91FGuHWgNue/gZj4WYWa9Q6wSyAti9YH8ssLJUHUn9gOHAmgrPrcj9jz9PHxXLR53Xv28f5j21tlvaMjOrZ7VOIA8CEyTtJWkA2aD4De3q3ACcnrZPAn4f2csrNwDT01NaewETgAfyBPHChi1sbula91Wb1ghe2rilW9oyM6tnNX0PJCK2SvoMcAvQF7g0IhZL+jbQHBE3AJcAv5D0KNmdx/R07mJJ/wM8DGwFzoyIXFmgNaJI51d+ra3d15aZWb2q+YuEEXETcFO7sm8WbG8ETi5x7veA73U1hh2260//fn1o2dL13/x9JIYP7t/ldszM6l2tu7DqQtOeI7qtrS0trbxl9x26rT0zs3rlBALst+tQ9h61fZfbETB531GMGjqw60GZmdU5J5Dkk+/dmyEDc7/IDsDgAX2ZMXl8N0VkZlbfnECSow7clSED+hV9uaQSffuIsTsOpmnPHbs1LjOzeuUEkgzo14dfffwdbJfjLqSPYPig/vzijHegbnqfxMys3jmBFNhn56FcNeMQhg/uT58K80D/vmLU9gO57tPvYpdhns7dzHoPJ5B23jRmODed9R6OffNoBvbrw6D+xf+JthvQl0H9+zDtrWO4+ezJjBs5pIcjNTOrrZq/B1KPxuwwmB9/8CDWrt/MVQ8+xS//9BRr1m9m05ZWBvbvw67DBvHRSeOYdtBYr0RoZr2Wl7Q1M7OySi1p6y4sMzPLxQnEzMxycQIxM7NcnEDMzCwXJxAzM8vFCcTMzHJxAjEzs1ycQMzMLBcnEDMzy6UmCUTSCEm3SVqWvreZA13SREl/lLRY0gJJpxYcu0zSE5Lmp8/Env0JzMysVncg5wB3RMQE4I60394G4P9ExIHA0cCPJBWuFfuliJiYPvOrH7KZmRWqVQI5AZiVtmcBJ7avEBGPRMSytL0SeBYY1WMRmplZWbVKILtExCqA9L1zucqSDgYGAI8VFH8vdW2dL6nkIuSSZkhqltS8evXq7ojdzMyoYgKRdLukRUU+J3SyndHAL4CPRkRrKv4qsD/wdmAE8JVS50fEzIhoioimUaN8A2Nm1l2qtphFREwpdUzSM5JGR8SqlCCeLVFvGHAj8I2IuL+g7VVpc5OknwFf7MbQzcysArXqwroBOD1tnw5c376CpAHAbODnEXF1u2Oj07fIxk8WVTVaMzPbRq0SyHnAVEnLgKlpH0lNki5OdU4BJgMfKfK47uWSFgILgZHAd3s2fDMz84qEZmZWllckNDOzbuUEYmZmuTiBmJlZLk4gZmaWixOImZnl4gRiZma5VO1NdGtsC1a8wA9vfYSW1lY+e/gE3jF+p1qHZGZ1xgnEtvH0mg1Mn3k/Gza3AND85Atcf+Yk9tt1aI0jM7N64i4s28YDT6x53X5ra3Dvo8/VKBozq1dOILaN0cMHvW6/X19tU2Zm5gRi2zhk7504+W1j6d9X9O8rpv7dLhx94K61DsvM6oznwrKSXtywhdYIdhwyoNahmFkNlZoLy4PoVtLw7frXOgQzq2PuwjIzs1ycQMzMLBcnEDMzy8UJxMzMcnECMTOzXCp6CkvSGGDPwvoRcXe1gjIzs/rXYQKR9H3gVOBhoCUVB9ClBCJpBHAVMA5YDpwSEWuL1GsBFqbdpyLi+FS+F3AlMAKYB3w4IjZ3JSYzM6tcJV1YJwL7RcSxEXFc+hzfDdc+B7gjIiYAd6T9Yl6JiInpU3jd7wPnp/PXAmd0Q0xmZlahShLI40A13ig7AZiVtmeRJaqKSBJwOHBNnvPNzKzrKhkD2QDMl3QHsKmtMCI+18Vr7xIRq1JbqyTtXKLeIEnNwFbgvIj4NbAT8EJEbE11VgBjip0saQYwA2CPPfboYshmZtamkgRyQ/p0mqTbgWKz8H29E83sERErJY0Hfi9pIfBSkXpFJ/WKiJnATMjmwurEdc3MrIwOE0hEzJI0ANg3FS2NiC2VNB4RU0odk/SMpNHp7mM08GyJNlam78clzQHeClwL7CCpX7oLGQusrCQmMzPrHh2OgUg6FFgG/AS4AHhE0uRuuPYNwOlp+3Tg+iLX3lHSwLQ9EpgEPBzZFMJ3AieVO9/MzKqnkkH0HwJHRsR7I2IycBRwfjdc+zxgqqRlwNS0j6QmSRenOn8HNEt6iCxhnBcRD6djXwG+IOlRsjGRS7ohJjMzq1AlYyD9I2Jp205EPCKpy09lRcTzwBFFypuBj6Xt+4A3lzj/ceDgrsZhZmb5VJJAmiVdAvwi7f8jMLd6IZmZWSOoJIF8CjgT+BwgsjfQL6hmUGZmVv8qeQprE/Dv6WNmZgaUSSCS/iciTknvXWzz/kRE/H1VIzMzs7pW7g7krPT9vp4IxMzMGkvJx3jbphkBPh0RTxZ+gE/3THhmZlavKnkPZGqRsmO6OxAzM2ss5cZAPkV2p7G3pAUFh4YCf6h2YGZmVt/KjYH8CrgZ+Fdev1bHyxGxpqpRmZlZ3SuZQCLiReBFSV9pd2h7SdtHxFPVDc3MzOpZJS8S3kj2GK+AQcBewFLgwCrGZWZmda6SFwlfNxeVpIOAT1QtIjMzawiVPIX1OhExD3h7FWIxM7MG0uEdiKQvFOz2AQ4CVlctIjMzawiVjIEMLdjeSjYmcm11wjEzs0ZRyRjItwAkDct24+WqR2VmZnWvkiVtm9KEiguAhZIekvS26odmZmb1rJIurEvJ5sO6B0DSu4GfAZ6N18ysF6vkKayX25IHQETcC3SpG0vSCEm3SVqWvncsUucwSfMLPhslnZiOXSbpiYJjE7sSj5mZdV7JBCLpoPTOxwOSLpR0qKT3SroAmNPF654D3BERE4A7eP1UKQBExJ0RMTEiJgKHAxuAWwuqfKnteETM72I8ZmbWSeW6sH7Ybv+fC7a3WWCqk04ADk3bs8gSUvspUwqdBNwcERu6eF0zM+sm5ebCOqyK192lbb2RiFglaecO6k9n2yV1vyfpm6Q7mLT07jYkzQBmAOyxxx5di9rMzF5Vbjr3D0XEL9u9SPiqiCi7Rrqk24Fdixz6emcClDQaeDNwS0HxV4G/AQOAmWR3L98uEefMVIempqau3jmZmVlSrgtrSPoeWqZOSRExpdQxSc9IGp3uPkYDz5Zp6hRgdkRsKWi7bbXETZJ+BnwxT4xmZpZfuS6sCyX1BV6KiPO7+bo3AKcD56Xv68vUPY3sjuNVBclHwInAom6Oz8zMOlD2Md6IaAGOr8J1zwOmSlpGtmTuefDqS4sXt1WSNA7YHbir3fmXp5cbFwIjge9WIUYzMyujkhcJ75P0Y+AqYH1bYZqVN5eIeB44okh5M/Cxgv3lwJgi9Q7Pe20zM+selSSQd6XvwkHqIHs3w8zMeqlKEsgZEfF4YYGk8VWKx8zMGkQlU5lcU6Ts6u4OxMzMGku590D2J1v3fLik9xccGka2NrqZmfVi5bqw9gPeB+wAHFdQ/jLw8WoGZWZm9a/ceyDXA9dLmhwRdxcekzSp6pGZmVldq2QM5EdFyv6ruwMxM7PGUm4M5BCyR3hHtZsPaxjQt9qBmZlZfSs3BjIA2D7VKZwP6yWy6dXNzKwXKzcGchdwl6TLIuLJtnJJg8gG1Zf1QHxmZlanOhwDiYgnJfWVdIyknwPLgVOrHpmZmdW1sm+iS5oMfBD4B+ABYBIw3isDmplZuUH0FcBTwE/J1h9/WdITTh5mZgblu7CuJZsJ91TgOElD6Ppa6GZm9gZRMoFExFnAOLK1yA8DHiF7pPcUSdv3THhmZlavOlpQKiLi9xHxcbJk8kGyFQCXVz80MzOrZ5VM5w5AWpP8N8BvJA2uXkhmZtYIKpnKZBsR8Up3B2JmZo0lVwIxMzOrOIGkp7C6jaSTJS2W1CqpqUy9oyUtlfSopHMKyveS9CdJyyRdJWlAd8ZnZmbldZhAJL1L0sPAkrT/FkkXdMO1FwHvB+4uVUFSX+AnwDHAAcBpkg5Ih78PnB8RE4C1wBndEJOZmVWokjuQ84GjgOcBIuIhYHJXLxwRSyJiaQfVDgYejYjHI2IzcCVwgiQBh/PacruzyJ4OMzOzHlJRF1ZEPN2uqKUKsRQzBii89opUthPwQkRsbVe+DUkzJDVLal69enVVgzUz600qeYz3aUnvAiKNM3yO1J3VEUm3A7sWOfT1tOJhh00UKYsy5dsWRswEZgI0NTX5TXozs25SSQL5JPAfZH/hrwBuBc6spPGImJI/NEjX271gfyywEngO2EFSv3QX0lZuZmY9pMMEEhHPAf/YA7EU8yAwQdJewF+B6cAHIyIk3Um2sNWVwOlAJXc0ZiXd99hz/HbBKt4ydjinNO1ONtRmZqV0mEAk/WeR4heB5gq7oUq1O41sbfVRwI2S5kfEUZJ2Ay6OiGMjYqukzwC3kC2je2lELE5NfAW4UtJ3gT8Dl+SNxWzBihf4p8seZOOWVmbP68vLG7fysfeMr3VYZnWtki6sQcD+wNVp/wPAYuAMSYdFxNl5LhwRs4HZRcpXAscW7N8E3FSk3uNkT2mZddncJ9cSaYTslS0tzFm62gnErAOVJJB9gMPbnniS9FOycZCpwMIqxmbWY94+bgRtPVaD+/dlyt/tXNuAzBpAJQlkDDCErNuKtL1bRLRI2lS1yMx60JvGDOeXZ7yDmxf9jb8fO5zj37JbrUMyq3uVJJAfAPMlzSF7fHYy8C9papPbqxibWY9qGjeCpnEjah2GWcOo5CmsSyTdRDbeIOBraZwC4EvVDM7MzOpXpZMpbgRWAWuAfSR1eSoTMzNrbJU8xvsx4Cyyl/XmA+8E/kg2F5WZmfVSldyBnAW8HXgyIg4D3gp4Uikzs16ukgSyMSI2AkgaGBF/AfarblhmZlbvKnkKa4WkHYBfA7dJWovnnTIz6/UqeQprWto8N80/NRz4XVWjMjOzulc2gUjqAyyIiDcBRMRdPRKVmZnVvbJjIBHRCjwkaY8eisfMzBpEJWMgo4HFkh4A1rcVRsTxVYvKzMzqXiUJ5FtVj8LMzBpOJYPod0naE5gQEbdL2o5sbQ4zM+vFOnwPRNLHgWuAC1PRGLJHes3MrBer5EXCM4FJwEsAEbEM8GIJZma9XCUJZFNEbG7bkdQPiOqFZGZmjaCSBHKXpK8BgyVNJVva9jfVDcvMzOpdJQnkHLLJExcCnyBbn/wbXbmopJMlLZbUKqmpRJ3dJd0paUmqe1bBsXMl/VXS/PQ5tlgbZmZWPZU8xnsC8POIuKgbr7sIeD+vDcwXsxX4vxExT9JQYK6k2yLi4XT8/Ij4t26MyczMOqGSO5DjgUck/ULSP6QxkC6JiCURsbSDOqsiYl7afhlYQvYEmJmZ1YEOE0hEfBTYh2zs44PAY5IurnZghSSNI1uH5E8FxZ+RtEDSpZJ2LHPuDEnNkppXr/YyJmZm3aWiJW0jYgtwM3AlMJesW6ssSbdLWlTk0+G57drZHrgWODsiXkrFPwX2BiaSLbX7wzKxz4yIpohoGjVqVGcubWZmZVSypO3RwHTgMGAOcDFwSkfnRcSUrgYnqT9Z8rg8Iq4raPuZgjoXAb/t6rXMeqNXNrewbtNWRg0dWOtQrAFVMp7xEbI7j09ExKbqhvMaSQIuAZZExL+3OzY6Ilal3Wlkg/Jm1gmPPruOE3/yBzZtbWHG5PF86aj9ax2SNZhKxkCmR8Sv25KHpEmSftKVi0qaJmkFcAhwo6RbUvlukm5K1SYBHwYOL/K47g8kLZS0gOzO6PNdicesN7pp4So2bN7Klpbg5/c9WetwrAFV9ESVpIlkA+inAE8A15U/o7yImA3MLlK+Ejg2bd8LqMT5H+7K9c0M3jl+Jwb060MfiUkTRtY6HGtAJROIpH3Jxj5OA54HrgIUEYf1UGxmVkUH7zWCGz/3Hla9sJF3jh9R63CsAZW7A/kLcA9wXEQ8CiDJXUVmbyB7j9qevUdtX+swrEGVGwP5APA34E5JF0k6ghJdSmZm1vuUTCARMTsiTgX2J3t89/PALpJ+KunIHorPzMzqVCVPYa2PiMsj4n3AWGA+2QSLZmbWi1X0JnqbiFgTERdGxOHVCsjMzBpDpxKImZlZGycQMzPLxQnEzMxycQIxM7NcnEDMzCwXJxAzM8vFCcTMzHJxAjEzs1ycQMzMLBcnEDMzy8UJxMzMcnECMTOzXGqSQCSdLGmxpFZJTWXqLU9rn8+X1FxQPkLSbZKWpe8deyZyMzNrU6s7kEXA+4G7K6h7WERMjIjCRHMOcEdETADuwNPLm5n1uJokkIhYEhFLu9DECcCstD0LOLHrUZmZWWfU+xhIALdKmitpRkH5LhGxCiB971yqAUkzJDVLal69enWVwzUz6z36VathSbcDuxY59PWIuL7CZiZFxEpJOwO3SfpLRFTS7fWqiJgJzARoamqKzpxrZmalVS2BRMSUbmhjZfp+VtJs4GCycZNnJI2OiFWSRgPPdvVaZmbWOXXbhSVpiKShbdvAkWSD7wA3AKen7dOBSu9ozMysm9TqMd5pklYAhwA3Srolle8m6aZUbRfgXkkPAQ8AN0bE79Kx84CpkpYBU9O+mZn1IEX0nmGBpqamaG5u7riimZm9StLcdq9SAHXchWVmZvXNCcTMzHJxAjEzs1ycQMzMLBcnEDMzy8UJxMzMcnECMTOzXJxAzMwsFycQMzPLxQnEzMxycQIxM7NcnEDMzCwXJxAzM8vFCcTMzHJxAjEzs1ycQMzMLBcnEDMzy8UJxMzMcnECMTOzXGqSQCSdLGmxpFZJ26yzm+rsJ2l+weclSWenY+dK+mvBsWN79icwM7N+NbruIuD9wIWlKkTEUmAigKS+wF+B2QVVzo+If6tmkGZmVlpNEkhELAGQVOkpRwCPRcSTVQvKzMw6pVHGQKYDV7Qr+4ykBZIulbRjqRMlzZDULKl59erV1Y3SzKwXqVoCkXS7pEVFPid0sp0BwPHA1QXFPwX2JuviWgX8sNT5ETEzIpoiomnUqFE5fhIzs8a1dv1mHnhiDZu2tnR721XrwoqIKd3U1DHAvIh4pqDtV7clXQT8tpuuZWb2hvHihi0c/sM5bNrayoG7DePqT76rW9tvhC6s02jXfSVpdMHuNLJBeTMzK7D8+fVs3NLKhs0t/PmpF7q9/Vo9xjtN0grgEOBGSbek8t0k3VRQbztgKnBduyZ+IGmhpAXAYcDneyh0M7OGceBuwzhk/E4M6t+Hs6fs2+3tKyK6vdF61dTUFM3NzbUOw8ysoUiaGxHbvLPXCF1YZmZWh5xAzMwsFycQMzPLxQnEzMxycQIxM7NcnEDMzCwXJxAzM8vFCcTMzHLpVS8SSloN1HJK+JHAczW8fh6Ouec0YtyNGDM0Zty1jHnPiNhmNtpelUBqTVJzsbc565lj7jmNGHcjxgyNGXc9xuwuLDMzy8UJxMzMcnEC6Vkzax1ADo655zRi3I0YMzRm3HUXs8dAzMwsF9+BmJlZLk4gZmaWixNID5B0qaRnJTXM0ruSdpd0p6QlkhZLOqvWMXVE0iBJD0h6KMX8rVrHVClJfSX9WdJvax1LpSQtTyuDzpfUECu1SdpB0jWS/pL+3z6k1jF1RNJ+6d+47fOSpLNrHRd4DKRHSJoMrAN+HhFvqnU8lUjrzo+OiHmShgJzgRMj4uEah1aSJAFDImKdpP7AvcBZEXF/jUPrkKQvAE3AsIh4X63jqYSk5UBTRDTMC3mSZgH3RMTFkgYA20VE9y8WXiWS+gJ/Bd4REbV8KRrwHUiPiIi7gTW1jqMzImJVRMxL2y8DS4AxtY2qvMisS7v906fu/0KSNBb4B+DiWsfyRiZpGDAZuAQgIjY3UvJIjgAeq4fkAU4gVgFJ44C3An+qbSQdS11B84Fngdsiou5jBn4EfBlorXUgnRTArZLmSppR62AqMB5YDfwsdRdeLGlIrYPqpOnAFbUOoo0TiJUlaXvgWuDsiHip1vF0JCJaImIiMBY4WFJddxlKeh/wbETMrXUsOUyKiIOAY4AzU1dtPesHHAT8NCLeCqwHzqltSJVLXW7HA1fXOpY2TiBWUhpHuBa4PCKuq3U8nZG6JuYAR9c4lI5MAo5P4wlXAodL+mVtQ6pMRKxM388Cs4GDaxtRh1YAKwruSq8hSyiN4hhgXkQ8U+tA2jiBWFFpQPoSYElE/Hut46mEpFGSdkjbg4EpwF9qG1V5EfHViBgbEePIuid+HxEfqnFYHZI0JD1cQeoGOhKo66cMI+JvwNOS9ktFRwB1+1BIEadRR91XkN3SWZVJugI4FBgpaQXwzxFxSW2j6tAk4MPAwjSmAPC1iLiphjF1ZDQwKz2p0gf4n4homMdiG8wuwOzs7wz6Ab+KiN/VNqSKfBa4PHUHPQ58tMbxVETSdsBU4BO1jqWQH+M1M7Nc3IVlZma5OIGYmVkuTiBmZpaLE4iZmeXiBGJmZrk4gViPkjRNUkjav9axdETSdpIuTzPOLpJ0b3ozH0n3pe9Du3sG3dTmiwWzr96es52Jko7tztgK2v5Iejy9sGykpNWSBpY57zJJJ1UjJut5TiDW004jmyV3enc0lt75qJazgGci4s1pFuUzgC0AEfGuKl4XshljJ6bPlJxtTAQ6lUCUqeT3wnXA1PR+QpuTgBsiYlNnrmmNywnEekz6630S2S/i6QXlVxX+pZz+Sv1Amhjx/0t6UNICSZ9Ixw9Na5X8CliYyn6dJvVbXDixn6QzJD0iaY6kiyT9OJWPknRtavtBSZOKhDyabOpsACJiadsvR0nrCuoNkzRb0sOS/ltSnxT7ZenOZaGkz6fz5kj6kaT70rGKp/8oFbOkg1N7f07f+6UX5b4NnJruYk6VdK6kLxa0t0jSuPRZIukCYB6wu6QjJf1R0jxJV7fdeRX8W7wE3A0cV1D86kR/kr6ZYlwkaWaa2aD9z7Nc0si03SRpTtoeomwNnQfTz3RCpf9G1sMiwh9/euQDfAi4JG3fBxyUtqcBs9L2AOBpYDAwA/hGKh8INAN7kb3Vvx7Yq6DtEel7MNmUGjsBuwHLgRFkU7vfA/w41fsV8O60vQfZlC3t451INqvvH4HvAhMKjq1L34cCG8lmeu0L3Eb2l/jbyGYDbqu/Q/qeA1yUticDi4pc91DgRWB++ny9XMzAMKBf2p4CXJu2P9L286b9c4EvFuwvAsalTyvwzlQ+kiw5DEn7XwG+WSTOk4HZaXs3YCXQt/C/R9r+BXBc2r4MOCltLwdGpu0mYE7a/hfgQ23/bsAjbbH4U18fT2ViPek0sqnLIZs48DSyv3hvBv4z9Z0fDdwdEa9IOhL4+4I+8+HABGAz8EBEPFHQ9uckTUvbu6d6uwJ3RcQaAElXA/umOlOAAwr+MB4maWhka58AEBHzJY0nm+dpCvCgpEMiYkm7n+uBiHg8XeMK4N3AHcB4Sf8F3AjcWlD/itT+3ZKGSdohtl2X4p7YdmGpojGnf5dZkiaQTbHen857Ml5beOudwAHAH9K1BpAl0fZ+C1ygbJ2NU4BrIqIlHTtM0peB7cgS+GLgNxXGciTZBJNtd0uDSAmzcz+SVZsTiPUISTsBhwNvkhRkf62HpC9HxMbUfXEUcCqvTRgn4LMRcUu7tg4luwMp3J8CHBIRG1Jbg9L5pfRJ9V8pF3dkC1RdB1wnqZVsTKH9L7L28wFFRKyV9Jb0M51J9gv2n0rVLxdDRzGnJHVnRExTtnbLnBLnb+X13daDCrbXF2yL7O7ptHLBpCT/O7I7yOlAWzfdIOACstUKn5Z0brtrFYun8LiAD0TE0nLXt9rzGIj1lJPIlvTdMyLGRcTuwBNkf61DdkfyUeA9QFvCuAX4lLJp5ZG0r4ovADQcWJuSx/5kf0EDPAC8V9KOkvoBHyg451bgM207kia2b1TSJEk7pu0BZH+VF1sJ7mBJeykbfD4VuDf17feJiGuB/8frpw0/NbX5buDFiHixSJvFlIp5OK+N1XykoP7LwNCC/eVtcUg6iKw7sJj7gUmS9kl1t5O0b4m6VwBfIJtcse0Opi0ZPJfGTko9dbWcrKsPXv/f5hbgs23jJpLeWuJ8qzEnEOspp5GtGY8L0MYAAAEqSURBVFHoWuCDaftWsjGB2yNicyq7mGy67XmSFgEXUvyu+XdAP0kLgO+QfpFFxF/J+tP/BNye2mr7Zf05oEnZ4PzDwCeLtLs3cJekhcCfycZgri1S74/AeWRjCk+kn3MMMEfZTMaXAV8tqL9W2WPA/032QEGlSsX8A+BfJf2B7M6uzZ1kXV7zJZ2aYh+RYvoU2djCNiJiNVkiuiL9m94PlHrs+lay8Y+rIrJBi9QddxHZAw6/Bh4sce63gP+QdA/QUlD+HbJuuAXpv/t3SpxvNebZeO0NTdL2EbEu3YHMBi6NiPaJrCfjmUM2kN1cqxjMuovvQOyN7tz0F3fb3cGvaxyP2RuG70DMzCwX34GYmVkuTiBmZpaLE4iZmeXiBGJmZrk4gZiZWS7/C3EWt6NljJPsAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "bin_means, bin_edges, _ = stats.binned_statistic(test_features[:,1], attr[:,1], statistic='mean', bins=6)\n",
    "bin_count, _, _ = stats.binned_statistic(test_features[:,1], attr[:,1], statistic='count', bins=6)\n",
    "\n",
    "bin_width = (bin_edges[1] - bin_edges[0])\n",
    "bin_centers = bin_edges[1:] - bin_width/2\n",
    "plt.scatter(bin_centers, bin_means, s=bin_count)\n",
    "plt.xlabel(\"Average Sibsp Feature Value\");\n",
    "plt.ylabel(\"Average Attribution\");\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see that the larger magnitude attributions correspond to the examples with larger Sibsp feature values, suggesting that the feature has a larger impact on prediction for these examples. Since there are substantially fewer of these examples (compared to those with a feature value of 0), the average attribution does not completely capture this effect."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have a better understanding of the importance of different input features, the next question we can ask regarding the function of the neural network is how the different neurons in each layer work together to reach the prediction. For instance, in our first hidden layer output containing 12 units, are all the units used for prediction? Do some units learn features positively correlated with survival while others learn features negatively correlated with survival?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This leads us to the second type of attributions available in Captum, **Layer Attributions**. Layer attributions allow us to understand the importance of all the neurons in the output of a particular layer. For this example, we will be using Layer Conductance, one of the Layer Attribution methods in Captum, which is an extension of Integrated Gradients applied to hidden neurons. More information regarding conductance can be found in the original paper here: https://arxiv.org/abs/1805.12233. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To use Layer Conductance, we create a LayerConductance object passing in the model as well as the module (layer) whose output we would like to understand. In this case, we choose net.sigmoid1, the output of the first hidden layer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "cond = LayerConductance(net, net.sigmoid1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now obtain the conductance values for all the test examples by calling attribute on the LayerConductance object. LayerConductance also requires a target index for networks with mutliple outputs, defining the index of the output for which gradients are computed. Similar to feature attributions, we provide target = 1, corresponding to survival. LayerConductance also utilizes a baseline, but we simply use the default zero baseline as in integrated gradients."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "cond_vals = cond.attribute(test_input_tensor,target=1)\n",
    "cond_vals = cond_vals.detach().numpy()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can begin by visualizing the average conductance for each neuron."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average Neuron Importances\n",
      "0 :  -0.115\n",
      "1 :  -0.039\n",
      "2 :  0.027\n",
      "3 :  0.011\n",
      "4 :  -0.019\n",
      "5 :  -0.134\n",
      "6 :  0.006\n",
      "7 :  0.001\n",
      "8 :  -0.121\n",
      "9 :  -0.000\n",
      "10 :  -0.295\n",
      "11 :  0.014\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAtEAAAGDCAYAAADtZ0xmAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAgAElEQVR4nO3de7xdZX3n8c9XIniplksChGusUhU7Ndoj2Fqpw0WwWEOdqlBHU6tl7GitrbbiZdRa6YSxrdqOtkUUouIVtVBtFYii1hE02CAgYhCCRAJELoraouhv/thP6vZ4Tk6e7JPsc46f9+u1Xnut9Txrrd9zdpLzzdrP3jtVhSRJkqRtd49xFyBJkiTNN4ZoSZIkqZMhWpIkSepkiJYkSZI6GaIlSZKkToZoSZIkqZMhWpIkSepkiJY05yS5KMntSXYbdy2zoY3nP5IcOLTv6CQbxljWNkmyLEklWTTuWgBaLQ8adx2SZIiWNKckWQY8FijgSTvoGuMIhN8B/teOvshcCbuzbaGOS9L8ZYiWNNc8E7gYOAtYuWVnkkcnuSnJLkP7fjPJF9v6PZKckuSrSW5N8r4ke7a2LXdTn53ka8DH2/73t3N+M8mnkjxs6Nx7JfmnJN9K8vkkr03yr0PtD0lyQZLbklyd5KkzjOtvgJOmu4uaZL8kH0iyOcl1SV4w1HZWktcObT8uycah7Q1JXtJ+Ft9JsijJQ9sd8DuSXJnkSZPO96YkH0lyZ5JLkjxwhvqHj31zkn9J8u0kn0myb5I3tFcPvpzkEZNqe2mSL7X2M5Pca6j995Jc036O5yXZb6itkjwvyXpgfZJPtabL2rWflmSPJB9uP7fb2/oBQ+e4KMmftzrvTHJ+ksVD7b+a5P+1n9MNSX6n7d8tyV8m+VqSm5P8fZJ7t7bF7Tp3tLo/ncTfp9JPGf/SS5prngmc3ZZjk+wDUFUXM7ibe+RQ398G3tXWXwCcAPwasB9wO/CmSef+NeChwLFt+1+AQ4C9gS+0a27xpna9fRmE+eFAf1/ggnbtvYGTgDcPh/ApfB14C/DqyQ0tgP0TcBmwP3AU8MIkx07uuxUnAccDuwNp5zu/1fcHwNlJHjyp/58BewDXAKd2XOupwCuAxcBdwGcZ/PwWA+cAfz2p/9MZ/MwfCPx8O5YkRwL/u51vKXA98J5Jx54AHA4cWlVHtH0Pr6qfqar3Mvg9diZwMHAQ8O/A/510jt8GnsXgZ7Er8OJ2/YMY/Bn4W2AJsBxY1445rdW6HHgQg+flla3tRcDGdsw+wMsYvHIi6adJVbm4uLjMiQX4VeD7wOK2/WXgj4baXwu8ra3fj0HIPbhtXwUcNdR3aTvXImAZg5Dzc1u59u6tz88Cu7RjHzzp2v/a1p8GfHrS8f8AvGqac18EPIdB6Pom8DDgaGBDaz8c+NqkY14KnNnWzwJeO9T2OGDj0PYG4HeHth8L3ATcY2jfu4FXD53vjKG2Xwe+PE3tW352i4aOfctQ+x8AVw1t/xfgjkm1PXfStb7a1t8K/J+htp9pP/dlbbuAIyfVU8CDtvI8Lgdun/Szf8XQ9v8EPjr0M/7QFOdI+7P1wKF9vwxc19ZfA5y7tTpcXFwW/uKdaElzyUrg/Kr6Rtt+F0N3gNv2kzN4w+GTgS9U1fWt7WDgQ+0l9jsYhOofMLhTuMUNW1aS7JJkVZv+8S0GYQ8Gd1OXMAjfN0x1bLvW4Vuu1a73dAZ3radVVZsZ3CV9zaSmg4H9Jp3vZZNqn8lwffsBN1TVD4f2Xc/gbuoWNw2tf5dBgN1WNw+t//sU25PPNVzb9a2+LXVuef6oqm8Dt06qc/jYn5DkPkn+Icn17Xn8FLB7hqb9MP1YDwS+OsVplwD3AS4dej4+2vYDvI7B3fvzk1yb5JSt1ShpYfKNGpLmhDbf9KnALkm2hJ7dGASih1fVZVX1pSTXA0/gx6dywCBs/W5VfWaKcy9rq8Mvuf82sIJ2R5jBHejbGdyF3AzcDRwAfKX1P3Do2BuAT1bVMdsx1NcB1wKfm3S+66rqkGmO+Q6DULfFVGF9eGw3AgcmucdQkD6IH41lZxv+2R3EoD7a48FbGto0mb0YTH3ZYqZpEi8CHgwcXlU3JVkO/BuD53EmNwCHTbH/Gwz+M/Cwqvr65MaqurNd90VtCs8nkny+qtZswzUlLRDeiZY0V5zA4M7xoQxekl/OYP7ypxnMk97iXQzmPx8BvH9o/98DpyY5GCDJkiQrtnK9+zGYz3srg4D6F1saquoHwAeBV7c7nQ+ZVMOHgZ9P8owk92zLo5I8dKZBVtUdwF8Bfzq0+3PAt9qbA+/d7pL/QpJHtfZ1wK8n2TPJvsALZ7jMJQyC95+22h4H/AY/Od94Z3lekgMyeKPny4D3tv3vAp6VZHl7deEvgEuqasNWznUz8HND2/djEHjvaOd/VUddZwNHJ3lqBm/G3CvJ8vYfj7cAr0+yN0CS/bfMUU/yxCQPShLgWwz+3P6g47qSFgBDtKS5YiWDOcBfq6qbtiwMpj88PT/6iLN3M5gT/PGhaR8AbwTOY/AS+50MPuHj8K1c7+0MphJ8HfhS6z/s+QzuTt8EvKNd9y74zzuRjwdOZHA39SYGb0Tb1s+1fiNDoauF9t9g8B+H6xjcCT2jXZ92/csY3DE/nx+F0ClV1fcYfDzgE9q53gw8s6q+vI31zbZ3Maj72ra8ttW5hsHH/n0A2MTgjYcnznCuVwOr2zSLpwJvAO7NYJwXM5h2sU2q6msM5mi/CLiNwX9WHt6aX8JgysbFbZrIhQzueMPgzagXAt9m8KbKN1fVRdt6XUkLQ6p8Q7EkzSTJacC+VbVyxs76Txl8ocxzqurCcdciSbPJO9GSNIUMPgf6FzNwGPBs4EPjrkuSNDf4xkJJmtr9GEzh2A+4hcE85nPHWpEkac5wOockSZLUyekckiRJUidDtCRJktRpXs6JXrx4cS1btmzcZUiSJGkBu/TSS79RVUumapuXIXrZsmWsXbt23GVIkiRpAWvfkjslp3NIkiRJnQzRkiRJUidDtCRJktTJEC1JkiR1MkRLkiRJnQzRkiRJUidDtCRJktTJEC1JkiR1MkRLkiRJnQzRkiRJUidDtCRJktTJEC1JkiR1WjTuAqTZsOyUj4y7hG2yYdXx4y5BkiTNAu9ES5IkSZ0M0ZIkSVInQ7QkSZLUyRAtSZIkdTJES5IkSZ0M0ZIkSVInQ7QkSZLUyRAtSZIkdTJES5IkSZ0M0ZIkSVInQ7QkSZLUadG4C5AkSdL4LDvlI+MuYUYbVh0/7hJ+gneiJUmSpE6GaEmSJKmT0zmkOciX1iRJmtu8Ey1JkiR1MkRLkiRJnQzRkiRJUidDtCRJktTJNxZK0k8x38QqSdvHO9GSJElSp1kJ0UmOS3J1kmuSnDJF+25J3tvaL0mybKjtpW3/1UmOnY16JEmSpB1p5BCdZBfgTcATgEOBk5IcOqnbs4Hbq+pBwOuB09qxhwInAg8DjgPe3M4nSZIkzVmzcSf6MOCaqrq2qr4HvAdYManPCmB1Wz8HOCpJ2v73VNVdVXUdcE07nyRJkjRnzcYbC/cHbhja3ggcPl2fqro7yTeBvdr+iycdu/9UF0lyMnAywEEHHTQLZfebD2/AgW1/E858GM+2jmWhvfFooY1nIf1ZW0hj6e2rnWuh/VlbSONZSGPp7asfmY0QnSn21Tb22ZZjBzurTgdOB5iYmJiyjyRJmpsMalpoZmM6x0bgwKHtA4Abp+uTZBHws8Bt23isJEmSNKfMRoj+PHBIkgck2ZXBGwXPm9TnPGBlW/8t4ONVVW3/ie3TOx4AHAJ8bhZqkiRJknaYkadztDnOzwc+BuwCvK2qrkzyGmBtVZ0HvBV4R5JrGNyBPrEde2WS9wFfAu4GnldVPxi1JkmSJGlHmpVvLKyqfwb+edK+Vw6t/wfwlGmOPRU4dTbqkCRJknYGv7FQkiRJ6mSIliRJkjoZoiVJkqROhmhJkiSpkyFakiRJ6mSIliRJkjoZoiVJkqROhmhJkiSpkyFakiRJ6mSIliRJkjoZoiVJkqROhmhJkiSp06JxFyBp4duw6vhxlyBJ0qzyTrQkSZLUyRAtSZIkdTJES5IkSZ0M0ZIkSVInQ7QkSZLUyRAtSZIkdTJES5IkSZ0M0ZIkSVInQ7QkSZLUyRAtSZIkdTJES5IkSZ0M0ZIkSVInQ7QkSZLUyRAtSZIkdTJES5IkSZ0M0ZIkSVInQ7QkSZLUadG4C9D4bFh1/LhLkCRJmpe8Ey1JkiR1MkRLkiRJnQzRkiRJUidDtCRJktTJEC1JkiR1MkRLkiRJnUYK0Un2THJBkvXtcY9p+q1sfdYnWTm0/6IkVydZ15a9R6lHkiRJ2hlGvRN9CrCmqg4B1rTtH5NkT+BVwOHAYcCrJoXtp1fV8rbcMmI9kiRJ0g43aoheAaxu66uBE6bocyxwQVXdVlW3AxcAx414XUmSJGlsRg3R+1TVJoD2ONV0jP2BG4a2N7Z9W5zZpnL8rySZ7kJJTk6yNsnazZs3j1i2JEmStP1m/NrvJBcC+07R9PJtvMZUwbja49Or6utJ7gd8AHgG8PapTlJVpwOnA0xMTNRUfSRJkqSdYcYQXVVHT9eW5OYkS6tqU5KlwFRzmjcCjxvaPgC4qJ376+3xziTvYjBnesoQLUmSJM0Vo07nOA/Y8mkbK4Fzp+jzMeDxSfZobyh8PPCxJIuSLAZIck/gicAVI9YjSZIk7XCjhuhVwDFJ1gPHtG2STCQ5A6CqbgP+HPh8W17T9u3GIEx/EVgHfB14y4j1SJIkSTvcjNM5tqaqbgWOmmL/WuA5Q9tvA942qc93gF8a5fqSJEnSOPiNhZIkSVInQ7QkSZLUyRAtSZIkdTJES5IkSZ0M0ZIkSVInQ7QkSZLUyRAtSZIkdTJES5IkSZ0M0ZIkSVInQ7QkSZLUyRAtSZIkdTJES5IkSZ0M0ZIkSVInQ7QkSZLUyRAtSZIkdTJES5IkSZ0M0ZIkSVInQ7QkSZLUyRAtSZIkdTJES5IkSZ0M0ZIkSVInQ7QkSZLUyRAtSZIkdTJES5IkSZ0M0ZIkSVInQ7QkSZLUyRAtSZIkdTJES5IkSZ0M0ZIkSVInQ7QkSZLUyRAtSZIkdTJES5IkSZ0M0ZIkSVInQ7QkSZLUyRAtSZIkdTJES5IkSZ0M0ZIkSVKnkUJ0kj2TXJBkfXvcY5p+H01yR5IPT9r/gCSXtOPfm2TXUeqRJEmSdoZR70SfAqypqkOANW17Kq8DnjHF/tOA17fjbweePWI9kiRJ0g43aoheAaxu66uBE6bqVFVrgDuH9yUJcCRwzkzHS5IkSXPJqCF6n6raBNAe9+44di/gjqq6u21vBPafrnOSk5OsTbJ28+bN212wJEmSNKpFM3VIciGw7xRNLx/x2pliX03XuapOB04HmJiYmLafJEmStKPNGKKr6ujp2pLcnGRpVW1KshS4pePa3wB2T7Ko3Y0+ALix43hJkiRpLEadznEesLKtrwTO3dYDq6qATwC/tT3HS5IkSeMyaoheBRyTZD1wTNsmyUSSM7Z0SvJp4P3AUUk2Jjm2Nb0E+OMk1zCYI/3WEeuRJEmSdrgZp3NsTVXdChw1xf61wHOGth87zfHXAoeNUoMkSZK0s/mNhZIkSVInQ7QkSZLUyRAtSZIkdTJES5IkSZ0M0ZIkSVInQ7QkSZLUyRAtSZIkdTJES5IkSZ1G+rKVnzYbVh0/7hIkSZI0B3gnWpIkSepkiJYkSZI6GaIlSZKkToZoSZIkqZNvLJQkaQ7yzezS3OadaEmSJKmTIVqSJEnqZIiWJEmSOhmiJUmSpE6GaEmSJKmTIVqSJEnqZIiWJEmSOhmiJUmSpE6GaEmSJKmTIVqSJEnqZIiWJEmSOhmiJUmSpE6Lxl2AJM0nG1YdP+4SJElzgHeiJUmSpE6GaEmSJKmTIVqSJEnqZIiWJEmSOhmiJUmSpE6GaEmSJKmTIVqSJEnqZIiWJEmSOhmiJUmSpE6GaEmSJKnTSCE6yZ5JLkiyvj3uMU2/jya5I8mHJ+0/K8l1Sda1Zfko9UiSJEk7w6h3ok8B1lTVIcCatj2V1wHPmKbtT6pqeVvWjViPJEmStMONGqJXAKvb+mrghKk6VdUa4M4RryVJkiTNCaOG6H2qahNAe9x7O85xapIvJnl9kt2m65Tk5CRrk6zdvHnz9tYrSZIkjWzGEJ3kwiRXTLGsmIXrvxR4CPAoYE/gJdN1rKrTq2qiqiaWLFkyC5eWJEmSts+imTpU1dHTtSW5OcnSqtqUZClwS8/Ft9zFBu5Kcibw4p7jJUmSpHEYdTrHecDKtr4SOLfn4Ba8SRIG86mvGLEeSZIkaYcbNUSvAo5Jsh44pm2TZCLJGVs6Jfk08H7gqCQbkxzbms5OcjlwObAYeO2I9UiSJEk73IzTObamqm4Fjppi/1rgOUPbj53m+CNHub4kSZI0Dn5joSRJktTJEC1JkiR1MkRLkiRJnQzRkiRJUidDtCRJktTJEC1JkiR1MkRLkiRJnQzRkiRJUidDtCRJktTJEC1JkiR1MkRLkiRJnQzRkiRJUidDtCRJktTJEC1JkiR1MkRLkiRJnQzRkiRJUidDtCRJktTJEC1JkiR1MkRLkiRJnQzRkiRJUidDtCRJktTJEC1JkiR1MkRLkiRJnQzRkiRJUidDtCRJktTJEC1JkiR1MkRLkiRJnQzRkiRJUidDtCRJktTJEC1JkiR1MkRLkiRJnQzRkiRJUidDtCRJktTJEC1JkiR1MkRLkiRJnQzRkiRJUidDtCRJktRppBCdZM8kFyRZ3x73mKLP8iSfTXJlki8medpQ2wOSXNKOf2+SXUepR5IkSdoZRr0TfQqwpqoOAda07cm+Czyzqh4GHAe8Icnure004PXt+NuBZ49YjyRJkrTDjRqiVwCr2/pq4ITJHarqK1W1vq3fCNwCLEkS4EjgnK0dL0mSJM01o4bofapqE0B73HtrnZMcBuwKfBXYC7ijqu5uzRuB/bdy7MlJ1iZZu3nz5hHLliRJkrbfopk6JLkQ2HeKppf3XCjJUuAdwMqq+mG7Ez1ZTXd8VZ0OnA4wMTExbT9JkiRpR5sxRFfV0dO1Jbk5ydKq2tRC8i3T9Ls/8BHgFVV1cdv9DWD3JIva3egDgBu7RyBJkiTtZKNO5zgPWNnWVwLnTu7QPnHjQ8Dbq+r9W/ZXVQGfAH5ra8dLkiRJc82oIXoVcEyS9cAxbZskE0nOaH2eChwB/E6SdW1Z3tpeAvxxkmsYzJF+64j1SJIkSTvcjNM5tqaqbgWOmmL/WuA5bf2dwDunOf5a4LBRapAkSZJ2Nr+xUJIkSepkiJYkSZI6GaIlSZKkToZoSZIkqZMhWpIkSepkiJYkSZI6GaIlSZKkToZoSZIkqZMhWpIkSepkiJYkSZI6GaIlSZKkToZoSZIkqZMhWpIkSepkiJYkSZI6GaIlSZKkToZoSZIkqZMhWpIkSepkiJYkSZI6GaIlSZKkToZoSZIkqZMhWpIkSepkiJYkSZI6GaIlSZKkToZoSZIkqZMhWpIkSepkiJYkSZI6GaIlSZKkToZoSZIkqZMhWpIkSepkiJYkSZI6GaIlSZKkToZoSZIkqZMhWpIkSepkiJYkSZI6GaIlSZKkToZoSZIkqZMhWpIkSeo0UohOsmeSC5Ksb497TNFneZLPJrkyyReTPG2o7awk1yVZ15blo9QjSZIk7Qyj3ok+BVhTVYcAa9r2ZN8FnllVDwOOA96QZPeh9j+pquVtWTdiPZIkSdION2qIXgGsbuurgRMmd6iqr1TV+rZ+I3ALsGTE60qSJEljM2qI3qeqNgG0x7231jnJYcCuwFeHdp/apnm8PsluWzn25CRrk6zdvHnziGVLkiRJ22/GEJ3kwiRXTLGs6LlQkqXAO4BnVdUP2+6XAg8BHgXsCbxkuuOr6vSqmqiqiSVLvJEtSZKk8Vk0U4eqOnq6tiQ3J1laVZtaSL5lmn73Bz4CvKKqLh4696a2eleSM4EXd1UvSZIkjcGo0znOA1a29ZXAuZM7JNkV+BDw9qp6/6S2pe0xDOZTXzFiPZIkSdION2qIXgUck2Q9cEzbJslEkjNan6cCRwC/M8VH2Z2d5HLgcmAx8NoR65EkSZJ2uBmnc2xNVd0KHDXF/rXAc9r6O4F3TnP8kaNcX5IkSRoHv7FQkiRJ6mSIliRJkjoZoiVJkqROhmhJkiSpkyFakiRJ6mSIliRJkjoZoiVJkqROhmhJkiSpkyFakiRJ6mSIliRJkjoZoiVJkqROhmhJkiSpkyFakiRJ6mSIliRJkjoZoiVJkqROhmhJkiSpkyFakiRJ6mSIliRJkjoZoiVJkqROhmhJkiSpkyFakiRJ6mSIliRJkjoZoiVJkqROhmhJkiSpkyFakiRJ6mSIliRJkjoZoiVJkqROhmhJkiSpkyFakiRJ6mSIliRJkjoZoiVJkqROhmhJkiSpkyFakiRJ6mSIliRJkjoZoiVJkqROhmhJkiSp06JxFyBJkjSfbFh1/LhL0Bww8p3oJHsmuSDJ+va4xxR9Dk5yaZJ1Sa5M8tyhtl9KcnmSa5L8TZKMWpMkSZK0I83GdI5TgDVVdQiwpm1Ptgn4lapaDhwOnJJkv9b2d8DJwCFtOW4WapIkSZJ2mNkI0SuA1W19NXDC5A5V9b2quqtt7rblukmWAvevqs9WVQFvn+p4SZIkaS6ZjRC9T1VtAmiPe0/VKcmBSb4I3ACcVlU3AvsDG4e6bWz7pjr+5CRrk6zdvHnzLJQtSZIkbZ9temNhkguBfadoevm2XqiqbgB+sU3j+Mck5wBTzX+uaY4/HTgdYGJiYso+kiRJ0s6wTSG6qo6eri3JzUmWVtWmNj3jlhnOdWOSK4HHAp8BDhhqPgC4cVtqkiRJksZlNqZznAesbOsrgXMnd0hyQJJ7t/U9gMcAV7fpH3cmeXT7VI5nTnW8JEmSNJfMRoheBRyTZD1wTNsmyUSSM1qfhwKXJLkM+CTwl1V1eWv7feAM4Brgq8C/zEJNkiRJ0g6TwYdizC8TExO1du3acZchSZKkBSzJpVU1MVWbX/stSZIkdTJES5IkSZ0M0ZIkSVInQ7QkSZLUyRAtSZIkdZqXn86RZDNw/bjrmCWLgW+Mu4hZspDGAo5nLltIY4GFNZ6FNBZwPHPZQhoLLKzxLKSxHFxVS6ZqmJcheiFJsna6j06ZbxbSWMDxzGULaSywsMazkMYCjmcuW0hjgYU1noU0lq1xOockSZLUyRAtSZIkdTJEj9/p4y5gFi2ksYDjmcsW0lhgYY1nIY0FHM9ctpDGAgtrPAtpLNNyTrQkSZLUyTvRkiRJUidD9JgkOS7J1UmuSXLKuOsZRZK3JbklyRXjrmU2JDkwySeSXJXkyiR/OO6atleSeyX5XJLL2lj+bNw1zYYkuyT5tyQfHncto0iyIcnlSdYlWTvuekaVZPck5yT5cvv788vjrml7JXlwe162LN9K8sJx17W9kvxR+zfgiiTvTnKvcdc0iiR/2MZy5Xx8Xqb6vZlkzyQXJFnfHvcYZ43bapqxPKU9Nz9MsmA/pcMQPQZJdgHeBDwBOBQ4Kcmh461qJGcBx427iFl0N/Ciqnoo8GjgefP4+bkLOLKqHg4sB45L8ugx1zQb/hC4atxFzJL/WlXLF8jHQb0R+GhVPQR4OPP4Oaqqq9vzshz4JeC7wIfGXNZ2SbI/8AJgoqp+AdgFOHG8VW2/JL8A/B5wGIM/Z09Mcsh4q+p2Fj/5e/MUYE1VHQKsadvzwVn85FiuAJ4MfGqnV7MTGaLH4zDgmqq6tqq+B7wHWDHmmrZbVX0KuG3cdcyWqtpUVV9o63cyCAL7j7eq7VMD326b92zLvH4jRJIDgOOBM8Zdi34kyf2BI4C3AlTV96rqjvFWNWuOAr5aVfP5S74WAfdOsgi4D3DjmOsZxUOBi6vqu1V1N/BJ4DfHXFOXaX5vrgBWt/XVwAk7tajtNNVYquqqqrp6TCXtNIbo8dgfuGFoeyPzNKQtdEmWAY8ALhlvJduvTX1YB9wCXFBV83YszRuAPwV+OO5CZkEB5ye5NMnJ4y5mRD8HbAbObFNtzkhy33EXNUtOBN497iK2V1V9HfhL4GvAJuCbVXX+eKsayRXAEUn2SnIf4NeBA8dc02zYp6o2weBmDrD3mOvRDAzR45Ep9s3ru4MLUZKfAT4AvLCqvjXuerZXVf2gvSR9AHBYeyl0XkryROCWqrp03LXMksdU1SMZTO16XpIjxl3QCBYBjwT+rqoeAXyH+fNy9LSS7Ao8CXj/uGvZXm1u7QrgAcB+wH2T/PfxVrX9quoq4DTgAuCjwGUMpuFJO5Uhejw28uP/az6A+f3S2oKT5J4MAvTZVfXBcdczG9pL6xcxv+evPwZ4UpINDKZBHZnkneMtaftV1Y3t8RYG820PG29FI9kIbBx6peMcBqF6vnsC8IWqunnchYzgaOC6qtpcVd8HPgj8yphrGklVvbWqHllVRzCYSrB+3DXNgpuTLAVoj7eMuR7NwBA9Hp8HDknygHaX40TgvDHXpCZJGMzrvKqq/nrc9YwiyZIku7f1ezP4Zfrl8Va1/arqpVV1QFUtY/D35uNVNS/vqCW5b5L7bVkHHs/gZep5qapuAm5I8uC26yjgS2MsabacxDyeytF8DXh0kvu0f9+OYh6/6RMgyd7t8SAGb2Cb788RDHLAyra+Ejh3jLVoGywadwE/jarq7iTPBz7G4F3Sb6uqK8dc1nZL8m7gccDiJBuBV1XVW8db1UgeAzwDuLzNJQZ4WVX98xhr2l5LgdXtE2HuAbyvqub1x8ItIPsAHxpkGhYB76qqj463pJH9AXB2uzlwLfCsMdczkjbf9hjgf4y7llFU1SVJzgG+wGDaw78x/79R7gNJ9gK+Dzyvqm4fd0E9pvq9CawC3pfk2Qz+4/OU8VW47aYZy23A3wJLgI8kWVdVx46vyh3DbyyUJEmSOjmdQ5IkSepkiJYkSZI6GaIlSZKkToZoSZIkqZMhWpIkSepkiJakOSpJJfmroe0XJ3n1GEuSJDWGaEmau+4Cnpxk8WyeNAP++y9JI/AfUUmau+5m8KUYfzS5oTtZG30AAAHpSURBVH0b5QeSfL4tj2n7X53kxUP9rkiyrC1XJXkzgy/dODDJSUkub31OGzrm20lOTXJZkouT7NP2P6X1vSzJp3b04CVpLjNES9Lc9ibg6Ul+dtL+NwKvr6pHAf8NOGMbzvVg4O1V9QgG3/R2GnAksBx4VJITWr/7AhdX1cOBTwG/1/a/Eji27X/SCGOSpHnPr/2WpDmsqr6V5O3AC4B/H2o6Gji0fW04wP2T3G+G011fVRe39UcBF1XVZoAkZwNHAP8IfA/Y8vXwlzL46muAzwBnJXkf8MHtH5UkzX+GaEma+97AYArGmUP77gH8clUNB2uS3M2Pv8p4r6H17wx33cr1vl9V1dZ/QPtdUVXPTXI4cDywLsnyqrq1aySStEA4nUOS5riqug14H/Dsod3nA8/fspFkeVvdADyy7Xsk8IBpTnsJ8GtJFifZBTgJ+OTW6kjywKq6pKpeCXwDOLB/NJK0MBiiJWl++Ctg+FM6XgBMJPliki8Bz237PwDsmWQd8PvAV6Y6WVVtAl4KfAK4DPhCVZ07Qw2v2/JGRAZzpS/b7tFI0jyXH71iJ0mSJGlbeCdakiRJ6mSIliRJkjoZoiVJkqROhmhJkiSpkyFakiRJ6mSIliRJkjoZoiVJkqROhmhJkiSp0/8HI9hNGvlh+1MAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 864x432 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "visualize_importances(range(12),np.mean(cond_vals, axis=0),title=\"Average Neuron Importances\", axis_title=\"Neurons\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also look at the distribution of each neuron's attributions. Below we look at the distributions for neurons 7 and 9, and we can confirm that their attribution distributions are very close to 0, suggesting they are not learning substantial features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAEICAYAAACktLTqAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAWnUlEQVR4nO3dfbRkVX3m8e8TWlBkIm8XbLvRBtNR0dFE2xdMzBhJFF9GmFnoYIj2INrLiIlONBFkJczEaDCZFV+Wb+lBhURFCHEGRkmUQdFkomhjUEHE7iBC2wgXEUXNRDG/+aN2k+JS9956uy99+H7WuqvO2WefffbZ3fepU7tO3UpVIUnqlp9a6Q5IkqbPcJekDjLcJamDDHdJ6iDDXZI6yHCXpA4y3KURJXldkrOm2N73kxzRls9O8odTbPvdSX5vWu1pz2G4a1FJrk9yc5L795W9JMllK9itobW+7mgh+jdJHrRA3cuS/L8kdyT5XpIrkpyaZJ/ddarqjVX1kiGOe1mSRetV1X5Vdd3wZzTv8f5zkr+b0/bLqur1k7atPY/hrmGtAV651AdJsmbK7f074I3AscCBwNeBcxfZ7RVV9W+AtcCrgROAi5Nkyn2b6rlK/Qx3DetPgNck2X/QxiQPT3JJktuSXJvk+X3b7nYFO/cKM0klOSXJdmB7K3tyks8n+W57fPKc9l6f5P+2K+yPJzl4nn7/e+Avq+rqqvoR8Hrgl5I8dLETrqofVNVlwHOBo4Bnt+P/1yTvb8v3TfL+JN9Ocnvr66FJ3gA8BXh7e8Xw9gXOtZL8TN+hD25jeUeSTyV5SKu3odW960lh99gmeQTwbuCodrzb2/a7TfMkeWl7FXNbkov6X8W0tl+WZHuS7yR5x7Sf0LR8DHcNaxtwGfCauRvadM0lwAeBQ4AXAO9M8sgR2j8OeCJwZJIDgY8CbwMOAv4U+GiSg/rq/xpwUjve3oP6tbt77ad/HeBRw3asqm6gd/5PGbB5M/AA4LDW15cB/1RVpwN/S+9VwH5V9Yq+fe4613kOeSK9J6GDgSuBDwzRx2vasT/TjnePJ+EkTwP+CHg+vVcl3wA+NKfac4DHA49p9Z6x2LG1OhnuGsXvA7+ZZGZO+XOA66vqfVV1Z1V9Afgr4PgR2v6jqrqtqv6J3hXy9qr6i9beucBX6V2F7/a+qvpaq38+8HPztHsx8Pwkj05yv3YOBew7Qt8AdtGb1pnrx/RC/Weq6idVdUVVfW+RtvrPdZCPVtWnq+qfgdPpXY0fNmJ/BzkReG9VfaG1fVpre0NfnTOr6vb2hPZJ5h9XrXKGu4ZWVVcBHwFOnbPpIcAT27TE7W1K4ETggSM0f2Pf8oPoXVX2+wawrm/9W33LPwT2m6fPlwJn0Huy+QZwPXAHsHOEvtGOfduA8r8APgZ8KMmuJH+c5D6LtHXjsNur6vvtuPO+CTyCu41ra/vbjDGuWv0Md43qDOCl3D0QbgQ+VVX79/3sV1W/0bb/gLtfKQ8K/f4/T7qL3hNGvwcD3xynw1X1jqraWFWH0Av5NcBVw+7frpofR2+aZW7bP66q/1ZVRwJPpvcq5kW7N8/XpUUOeddVepL96L1i2EVvHGH+sVys3buNa5tOO4gxx1Wrm+GukVTVDuA84Lf6ij8C/GySFya5T/t5fHuTD3rzxv8xyb7tjcOTFznMxa29X0uyJsl/ojc//ZFR+9ve8HxUeh4MbAXeWlXfGWLffdvdNhcCn2v9mlvnl5P82yR7Ad+jN03zk7b5ZuCIUfsMPCvJLybZm97c++VVdWNVzdIL4l9PsleSFwP9bwzfDKxv+w3yQeCkJD/Xbu18Y2v7+jH6qFXOcNc4/gC46573qroDeDq9WwZ30Xtp/yZg973hbwZ+RC98zmGRNwir6tv0roBfTW/a4HeB51TVrWP09b70Qu379AL6M8BiH+p5e5I7Wn/fQu9q/5iq+pcBdR8IXEAv2K8BPgW8v217K3B8u/PkbSP0+YP0XiHdRu8Vw4l9214K/A69cXkk8Pd92z4BXA18K8k9xqpNUf1eO5+b6D0xnDBCv7QHiV/WIUnd45W7JHWQ4S5JHWS4S1IHGe6S1EGr4g8XHXzwwbVhw4aV7oYk7VGuuOKKW6tq7ifGgVUS7hs2bGDbtm0r3Q1J2qMkmftJ7rs4LSNJHWS4S1IHGe6S1EGGuyR1kOEuSR1kuEtSBxnuktRBhrskdZDhLkkdtCo+oSpJ9yYbTv3oXcvXn/nsJTmGV+6S1EGLhnuS9ya5Jck9vlA4yWuSVJKD23qSvC3JjiRfSvLYpei0JGlhw1y5nw0cM7ewfSP8rwI39BU/E9jYfrYA75q8i5KkUS0a7lX1aXpf1DvXm+l9cXH/l7AeC/x59XwW2D/J2qn0VJI0tLHm3JM8F/hmVX1xzqZ1wI196ztb2aA2tiTZlmTb7OzsON2QJM1j5HBPsi9wOvD7gzYPKKsBZVTV1qraVFWbZmYG/q15SdKYxrkV8qHA4cAXkwCsB76Q5An0rtQP66u7Htg1aSclSaMZ+cq9qr5cVYdU1Yaq2kAv0B9bVd8CLgJe1O6aeRLw3aq6abpdliQtZphbIc8FPgM8LMnOJCcvUP1i4DpgB/A/gJdPpZeSpJEsOi1TVS9YZPuGvuUCTpm8W5KkSfgJVUnqIMNdkjrIcJekDjLcJamDDHdJ6iDDXZI6yHCXpA4y3CWpgwx3Seogw12SOshwl6QOMtwlqYMMd0nqIMNdkjrIcJekDjLcJamDDHdJ6iDDXZI6yHCXpA4y3CWpgxYN9yTvTXJLkqv6yv4kyVeTfCnJ/0yyf9+205LsSHJtkmcsVcclSfMb5sr9bOCYOWWXAI+qqkcDXwNOA0hyJHAC8Mi2zzuT7DW13kqShrJouFfVp4Hb5pR9vKrubKufBda35WOBD1XVP1fV14EdwBOm2F9J0hCmMef+YuCv2/I64Ma+bTtb2T0k2ZJkW5Jts7OzU+iGJGm3icI9yenAncAHdhcNqFaD9q2qrVW1qao2zczMTNINSdIca8bdMclm4DnA0VW1O8B3Aof1VVsP7Bq/e5KkcYx15Z7kGOC1wHOr6od9my4CTkiyT5LDgY3A5ybvpiRpFIteuSc5F3gqcHCSncAZ9O6O2Qe4JAnAZ6vqZVV1dZLzga/Qm645pap+slSdlyQNtmi4V9ULBhS/Z4H6bwDeMEmnJEmT8ROqktRBhrskdZDhLkkdZLhLUgcZ7pLUQYa7JHWQ4S5JHWS4S1IHGe6S1EGGuyR1kOEuSR1kuEtSBxnuktRBhrskdZDhLkkdZLhLUgcZ7pLUQYa7JHWQ4S5JHWS4S1IHLRruSd6b5JYkV/WVHZjkkiTb2+MBrTxJ3pZkR5IvJXnsUnZekjTYMFfuZwPHzCk7Fbi0qjYCl7Z1gGcCG9vPFuBd0+mmJGkUi4Z7VX0auG1O8bHAOW35HOC4vvI/r57PAvsnWTutzkqShjPunPuhVXUTQHs8pJWvA27sq7ezld1Dki1JtiXZNjs7O2Y3JEmDTPsN1Qwoq0EVq2prVW2qqk0zMzNT7oYk3buNG+43755uaY+3tPKdwGF99dYDu8bvniRpHOOG+0XA5ra8Gbiwr/xF7a6ZJwHf3T19I0laPmsWq5DkXOCpwMFJdgJnAGcC5yc5GbgBeF6rfjHwLGAH8EPgpCXosyRpEYuGe1W9YJ5NRw+oW8Apk3ZKkjQZP6EqSR1kuEtSBxnuktRBhrskdZDhLkkdZLhLUgcZ7pLUQYa7JHWQ4S5JHWS4S1IHGe6S1EGGuyR1kOEuSR1kuEtSBxnuktRBhrskdZDhLkkdZLhLUgcZ7pLUQYa7JHXQROGe5L8kuTrJVUnOTXLfJIcnuTzJ9iTnJdl7Wp2VJA1n7HBPsg74LWBTVT0K2As4AXgT8Oaq2gh8Bzh5Gh2VJA1v0mmZNcD9kqwB9gVuAp4GXNC2nwMcN+ExJEkjGjvcq+qbwH8HbqAX6t8FrgBur6o7W7WdwLpB+yfZkmRbkm2zs7PjdkOSNMAk0zIHAMcChwMPAu4PPHNA1Rq0f1VtrapNVbVpZmZm3G5IkgaYZFrmV4CvV9VsVf0Y+DDwZGD/Nk0DsB7YNWEfJUkjmiTcbwCelGTfJAGOBr4CfBI4vtXZDFw4WRclSaOaZM79cnpvnH4B+HJrayvwWuC3k+wADgLeM4V+SpJGsGbxKvOrqjOAM+YUXwc8YZJ2JUmT8ROqktRBhrskdZDhLkkdZLhLUgcZ7pLUQYa7JHWQ4S5JHWS4S1IHGe6S1EGGuyR1kOEuSR1kuEtSBxnuktRBhrskdZDhLkkdZLhLUgcZ7pLUQYa7JHWQ4S5JHTRRuCfZP8kFSb6a5JokRyU5MMklSba3xwOm1VlJ0nAmvXJ/K/A3VfVw4DHANcCpwKVVtRG4tK1LkpbR2OGe5KeBXwLeA1BVP6qq24FjgXNatXOA4ybtpCRpNJNcuR8BzALvS/IPSc5Kcn/g0Kq6CaA9HjKFfkqSRjBJuK8BHgu8q6p+HvgBI0zBJNmSZFuSbbOzsxN0Q5I01yThvhPYWVWXt/UL6IX9zUnWArTHWwbtXFVbq2pTVW2amZmZoBuSpLnGDveq+hZwY5KHtaKjga8AFwGbW9lm4MKJeihJGtmaCff/TeADSfYGrgNOoveEcX6Sk4EbgOdNeAxJ0ogmCvequhLYNGDT0ZO0K0majJ9QlaQOMtwlqYMMd0nqIMNdkjrIcJekDjLcJamDDHdJ6iDDXZI6yHCXpA4y3CWpgwx3Seogw12SOshwl6QOMtwlqYMMd0nqIMNdkjrIcJekDjLcJamDDHdJ6iDDXZI6aOJwT7JXkn9I8pG2fniSy5NsT3Jekr0n76YkaRTTuHJ/JXBN3/qbgDdX1UbgO8DJUziGJGkEE4V7kvXAs4Gz2nqApwEXtCrnAMdNcgxJ0ugmvXJ/C/C7wL+09YOA26vqzra+E1g3aMckW5JsS7JtdnZ2wm5IkvqNHe5JngPcUlVX9BcPqFqD9q+qrVW1qao2zczMjNsNSdIAaybY9xeA5yZ5FnBf4KfpXcnvn2RNu3pfD+yavJuSpFGMfeVeVadV1fqq2gCcAHyiqk4EPgkc36ptBi6cuJeSpJEsxX3urwV+O8kOenPw71mCY0iSFjDJtMxdquoy4LK2fB3whGm0K0kaj59QlaQOMtwlqYMMd0nqIMNdkjrIcJekDjLcJamDDHdJ6iDDXZI6yHCXpA4y3CWpgwx3Seogw12SOmgqfzhMkrSwDad+dFmP55W7JHWQ4S5JHWS4S1IHGe6S1EGGuyR1kOEuSR3krZC6S/+tWtef+ewV7ImkSY195Z7ksCSfTHJNkquTvLKVH5jkkiTb2+MB0+uuJGkYk0zL3Am8uqoeATwJOCXJkcCpwKVVtRG4tK1LkpbR2OFeVTdV1Rfa8h3ANcA64FjgnFbtHOC4STspSRrNVN5QTbIB+HngcuDQqroJek8AwCHz7LMlybYk22ZnZ6fRDUlSM3G4J9kP+CvgVVX1vWH3q6qtVbWpqjbNzMxM2g1JUp+Jwj3JfegF+weq6sOt+OYka9v2tcAtk3VRkjSqSe6WCfAe4Jqq+tO+TRcBm9vyZuDC8bsnSRrHJPe5/wLwQuDLSa5sZa8DzgTOT3IycAPwvMm6KEka1djhXlV/B2SezUeP264kaXL++QFJ6iDDXZI6yHCXpA4y3CWpgwx3Seogw12SOshwl6QO8ss6JGmJ9H8BznLzyl2SOshwl6QOclpGAw3zfap+56q0ennlLkkdZLhLUgc5LaNFreQ7/pqc02dLbzWOseF+LzA3nPv/8xncUjc5LSNJHeSVu5bUtF6ursaXvUvt3njO0+LYGe73Snv6VIy/uNO3Gsd0mP+nS9HX1TgW4zDcNRUr9QvRlV/ElTDsk/yoYzxfu0vxXs9yXqgs9N7VauScuyR10JJduSc5BngrsBdwVlWduVTHurcZ9dOjq8WoV3SrcY5+mHNYzZbq/8Vq/P+227311d2ShHuSvYB3AL8K7AQ+n+SiqvrKtI+1p/7DTeul7mr8pRq1T9M6t2HqL1RnWk8yy/kkNsxxp2kpxmU5Tev3bqFtq+E8YemmZZ4A7Kiq66rqR8CHgGOX6FiSpDlSVdNvNDkeOKaqXtLWXwg8sape0VdnC7ClrT4MuHbqHbmng4Fbl+E4eyLHZmGOz/wcm4Ut5fg8pKpmBm1Yqjn3DCi727NIVW0Fti7R8QdKsq2qNi3nMfcUjs3CHJ/5OTYLW6nxWappmZ3AYX3r64FdS3QsSdIcSxXunwc2Jjk8yd7ACcBFS3QsSdIcSzItU1V3JnkF8DF6t0K+t6quXopjjWhZp4H2MI7Nwhyf+Tk2C1uR8VmSN1QlSSvLT6hKUgcZ7pLUQXtkuCc5MMklSba3xwPmqbe51dmeZHNf+eOSfDnJjiRvS5KF2k3ygCT/O8kXk1yd5KTlOdPxLPf4tG1PTXJlG59PLf1Zjm8lxqdtf3ySn7TPgaxKK/C7dWKSL7Wfv0/ymOU50+ElOSbJte2cTh2wfZ8k57XtlyfZ0LfttFZ+bZJnLNZmuwnl8jZO57UbUsZTVXvcD/DHwKlt+VTgTQPqHAhc1x4PaMsHtG2fA46idz/+XwPPXKhd4HV9yzPAbcDeKz0Oq2h89ge+Ajy4rR+y0mOwmsanre8FfAK4GDh+pcdgtYwN8OS+fZ8JXL7SYzDnXPcC/hE4Atgb+CJw5Jw6Lwfe3ZZPAM5ry0e2+vsAh7d29lqoTeB84IS2/G7gN8bu+0oP3pgDfi2wti2vBa4dUOcFwJ/1rf9ZK1sLfHVQvfnaBU4D3tn+wx4O7AB+aqXHYRWNz8uBP1zp816t49PWXwWcApzN6g73ZR+bvvoHAN9c6TGY06ejgI/1rZ8GnDanzseAo9ryGnqfRs3curvrzddm2+dWYM2gY4/6s0dOywCHVtVNAO3xkAF11gE39q3vbGXr2vLc8oXafTvwCHofxPoy8Mqq+pfpnMqSWO7x+VnggCSXJbkiyYumdiZLY1nHJ8k64D/QuxJb7Zb7/06/k+ld7a8m853rwDpVdSfwXeCgBfadr/wg4PbWxnzHGtqq/bKOJP8HeOCATacP28SAslqgfCHPAK4EngY8FLgkyd9W1feG7MvUrbLxWQM8DjgauB/wmSSfraqvDdmXqVtl4/MW4LVV9ZM2Bb2iVtnY7O7TL9ML918csg/LZZhzGnU8Bl1UTzR+g6zacK+qX5lvW5Kbk6ytqpuSrAVuGVBtJ/DUvvX1wGWtfP2c8t1/GmG+dk8Czqzea6UdSb4OPJze/OKKWGXjsxO4tap+APwgyaeBxwArFu6rbHw2AR9qwX4w8Kwkd1bV/xr9zCa3ysaGJI8GzqI3P//tMU5pKQ3zp1R219mZZA3wAHrvyy2076DyW4H9k6xpV+8T/dmWPXVa5iJg9zv0m4ELB9T5GPD0JAe0d+afTm/+6ibgjiRPau/kv6hv//navYHeVSlJDqX3Vyyvm+4pTdVyj8+FwFOSrEmyL/BE4Jppn9QULev4VNXhVbWhqjYAFwAvX6lgH8Kyjk2SBwMfBl64kq/0FjDMn1LpP7fjgU+0C8GLgBPa3TSHAxvpXRAObLPt88nWBsw//sNZ6TcsxnyT4yDgUmB7ezywlW+i961Pu+u9mN6bnzuAk/rKNwFX0XvH+u386yd152v3QcDH6c23XwX8+kqPwWoan7btd+jdMXMV8KqVHoPVNj59+57N6n5Ddbl/t84CvkNv2vNKYNtKj8GAMXkWvVeh/wic3sr+AHhuW74v8JdtLD4HHNG37+ltv2tpdw7N12YrP6K1saO1uc+4/fbPD0hSB+2p0zKSpAUY7pLUQYa7JHWQ4S5JHWS4S1IHGe6S1EGGuyR10P8H751v8GcKL2YAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXAAAAEICAYAAABGaK+TAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAScklEQVR4nO3de5CddX3H8fenhEsxKreACJSAUhXbETXe8NIpdCoCFdpBi1JNLcp4QbFqNchYrVYF24o6WhkKalRULDoDA04tg+Cl7UQDUgUjBgEBEyGICF6qoN/+cZ6Ek83Z7Nndc3b3F96vmZ0957n8nu/z22c/+zu/85wkVYUkqT2/M98FSJJmxgCXpEYZ4JLUKANckhplgEtSowxwSWqUAS4NkOTNSc4ZYXs/S3Jg9/hjSf5xhG2fleQto2pP7TDABUCSm5LcluRBfctemuSKeSxrKElO6AJy49cvklSSJ06y/RVJ/i/JPUnuTnJlkhVJdty4TVW9q6peOsSxr0gy5XZVtbiqbpjemQ083l8n+dqEtl9eVe+YbdtqjwGufouAU8Z9kCSLRtleVZ3XBeTiqloMvBK4AbhqK7udXFUPBvYGXg8cD3whSUZZ26jPVepngKvfPwFvSLLLoJVJHp3k0iR3JrkuyfP71m02Ep04UuxGxK9KshZY2y07NMk3kvy0+37ohPbekeS/upHyfybZY8jzWA58vIb4mHFV/byqrgCeCzwNOKo7/tuSfLJ7vFOSTyb5cZK7ulr3SvJO4JnAB7uR/we3cq6V5JF9h96j68t7knw5yf7ddku7bTcF/8a+TfIY4Czgad3x7urWbzYlk+RlSa7vfk4XJXn4hJ/Dy5OsTfKTJB8a9R8tzR0DXP1WA1cAb5i4optauRT4FLAn8ALgX5M8dhrtHws8BTg4yW7AJcAHgN2B9wKXJNm9b/sXAi/pjrfDoLoG1Lk/8Czg49Ooi6q6md75P3PA6uXAQ4H9ulpfDvyyqk4DvkpvNL+4qk7u22fTuU5yyBOAdwB7AFcD5w1R45ru2P/THW+LP7RJDgPeDTyf3quLHwCfmbDZ0cCTgMd12z17qmNrYTLANdHfA69OsmTC8qOBm6rqo1V1X1VdBXwOOG4abb+7qu6sql/SG+murapPdO19Gvgu8Gd923+0qr7Xbf9Z4JAhjvFi4KtVdeM06tpoHbDbgOX30gvuR1bVb6rqyqq6e4q2+s91kEuq6itV9SvgNHqj6v1mUPNEJwAfqaqrurZP7dpe2rfN6VV1V/dH63KG61ctQAa4NlNV1wAXAysmrNofeEo3hXBX9/L9BOBh02j+lr7HD6c3Ouz3A2Cfvuc/6nv8C2DxEMd4MbByGjX12we4c8DyTwBfBD6TZF2S9yTZfoq2bhl2fVX9rDvuwyfffGib9WvX9o+Zfb9qATLANchbgZex+S/9LcCXq2qXvq/FVfWKbv3PgZ37th8U7P1z0uvo/VHo93vAD2dadJKn0wuwC2aw737AE+lNiWymqu6tqn+oqoOBQ+m9GnnxxtWTNDnV/Pum0XaSxfRG/uvo9SNM3pdTtbtZv3ZTX7szi37VwmWAawtVdT1wPvCavsUXA7+f5EVJtu++ntS9sQa9edy/SLJz92bdiVMc5gtdey9MsijJX9KbL754FqUvBz5XVfcMu0NX7x8BFwJf7+qauM0fJ/nDJNsBd9ObUvlNt/o24MAZ1Hpkkmck2YHeXPiqqrqlqjbQC9u/SrJdkr8BHtG3323Avt1+g3wKeEmSQ7rbIt/VtX3TDGrUAmeAazJvBzbdE96F4p/Su91uHb2X4WcAG++dPhP4Nb2AWckUb8pV1Y/pjWRfT+8l/huBo6vqjpkUm2Qnem/IDTt98sEk93T1vo/efP4RVfXbAds+jN6o/m5gDfBl4JPduvcDx3V3dHxgGiV/it4rnTvpjfxP6Fv3MuDv6PXLY4H/7lv3JeBa4EdJtuirqroMeEt3Puvphf/x06hLDYn/oYMktckRuCQ1ygCXpEYZ4JLUKANckho1p//Qzh577FFLly6dy0NKUvOuvPLKO6pq4qej5zbAly5dyurVq+fykJLUvCQTP7UMOIUiSc0ywCWpUQa4JDXKAJekRhngktQoA1ySGmWAS1KjDHBJapQBLkmNmtNPYs7G0hWXbHp80+lHzWMlkrQwOAKXpEYZ4JLUKANckhplgEtSowxwSWqUAS5JjTLAJalRBrgkNcoAl6RGGeCS1CgDXJIaZYBLUqMMcElqlAEuSY0ywCWpUQa4JDVqqABP8rdJrk1yTZJPJ9kpyQFJViVZm+T8JDuMu1hJ0v2mDPAk+wCvAZZV1R8A2wHHA2cAZ1bVQcBPgBPHWagkaXPDTqEsAn43ySJgZ2A9cBhwQbd+JXDs6MuTJE1mygCvqh8C/wzcTC+4fwpcCdxVVfd1m90K7DOuIiVJWxpmCmVX4BjgAODhwIOA5wzYtCbZ/6Qkq5Os3rBhw2xqlST1GWYK5U+AG6tqQ1XdC3weOBTYpZtSAdgXWDdo56o6u6qWVdWyJUuWjKRoSdJwAX4z8NQkOycJcDjwHeBy4Lhum+XAheMpUZI0yDBz4KvovVl5FfDtbp+zgTcBr0tyPbA7cO4Y65QkTbBo6k2gqt4KvHXC4huAJ4+8IknSUPwkpiQ1ygCXpEYZ4JLUKANckhplgEtSowxwSWqUAS5JjTLAJalRBrgkNcoAl6RGGeCS1CgDXJIaZYBLUqMMcElqlAEuSY0ywCWpUQa4JDXKAJekRhngktQoA1ySGmWAS1KjDHBJapQBLkmNMsAlqVEGuCQ1atF8FzATS1dcsunxTacfNY+VSNL8cQQuSY0ywCWpUQa4JDXKAJekRhngktQoA1ySGmWAS1KjDHBJapQBLkmNMsAlqVEGuCQ1ygCXpEYNFeBJdklyQZLvJlmT5GlJdktyaZK13fddx12sJOl+w47A3w/8R1U9GngcsAZYAVxWVQcBl3XPJUlzZMoAT/IQ4FnAuQBV9euqugs4BljZbbYSOHZcRUqStjTMCPxAYAPw0STfTHJOkgcBe1XVeoDu+56Ddk5yUpLVSVZv2LBhZIVL0gPdMAG+CHgC8OGqejzwc6YxXVJVZ1fVsqpatmTJkhmWKUmaaJgAvxW4tapWdc8voBfotyXZG6D7fvt4SpQkDTJlgFfVj4BbkjyqW3Q48B3gImB5t2w5cOFYKpQkDTTs/4n5auC8JDsANwAvoRf+n01yInAz8LzxlChJGmSoAK+qq4FlA1YdPtpyJEnD8pOYktQoA1ySGmWAS1KjDHBJapQBLkmNMsAlqVEGuCQ1ygCXpEYZ4JLUKANckhplgEtSowxwSWqUAS5JjTLAJalRBrgkNcoAl6RGGeCS1CgDXJIaZYBLUqMMcElqlAEuSY0ywCWpUQa4JDXKAJekRhngktQoA1ySGmWAS1KjDHBJapQBLkmNMsAlqVEGuCQ1ygCXpEYZ4JLUKANckhplgEtSowxwSWqUAS5JjTLAJalRQwd4ku2SfDPJxd3zA5KsSrI2yflJdhhfmZKkiaYzAj8FWNP3/AzgzKo6CPgJcOIoC5Mkbd1QAZ5kX+Ao4JzueYDDgAu6TVYCx46jQEnSYMOOwN8HvBH4bfd8d+Cuqrqve34rsM+gHZOclGR1ktUbNmyYVbGSpPtNGeBJjgZur6or+xcP2LQG7V9VZ1fVsqpatmTJkhmWKUmaaNEQ2zwdeG6SI4GdgIfQG5HvkmRRNwrfF1g3vjIlSRNNOQKvqlOrat+qWgocD3ypqk4ALgeO6zZbDlw4tiolSVuYzX3gbwJel+R6enPi546mJEnSMIaZQtmkqq4Aruge3wA8efQlSZKG4ScxJalRBrgkNcoAl6RGGeCS1CgDXJIaZYBLUqMMcElqlAEuSY0ywCWpUQa4JDXKAJekRhngktQoA1ySGmWAS1KjDHBJapQBLkmNMsAlqVEGuCQ1ygCXpEYZ4JLUKANckhplgEtSowxwSWqUAS5JjTLAJalRBrgkNcoAl6RGGeCS1CgDXJIaZYBLUqMMcElqlAEuSY0ywCWpUQa4JDXKAJekRhngktQoA1ySGjVlgCfZL8nlSdYkuTbJKd3y3ZJcmmRt933X8ZcrSdpomBH4fcDrq+oxwFOBVyU5GFgBXFZVBwGXdc8lSXNkygCvqvVVdVX3+B5gDbAPcAywsttsJXDsuIqUJG1pWnPgSZYCjwdWAXtV1XrohTyw5yT7nJRkdZLVGzZsmF21kqRNhg7wJIuBzwGvraq7h92vqs6uqmVVtWzJkiUzqVGSNMBQAZ5ke3rhfV5Vfb5bfFuSvbv1ewO3j6dESdIgw9yFEuBcYE1Vvbdv1UXA8u7xcuDC0ZcnSZrMoiG2eTrwIuDbSa7ulr0ZOB34bJITgZuB542nREnSIFMGeFV9Dcgkqw8fbTmSHsiWrrhk0+ObTj9qHitpg5/ElKRGGeCS1CgDXJIaZYBLUqOGuQtlQfNND+mBxd/5+zkCl6RGGeCS1CgDXJIa1fwceD/nxqT29P/eanocgUtSowxwSWqUAS5JjTLAJalR29SbmJIWpolvVHqTwWg4ApekRhngktQoA1ySGmWAS1KjDHBJapQBLkmN8jZCSXPOf/9kNByBS1KjttkRuP8yoaRtnSNwSWqUAS5Jjdpmp1CkjZxO07bKEbgkNcoRuKSx8FbB8XMELkmNcgSukXPOedvnz3hhcAQuSY1yBD5PZjOCGWZucbI2HTnNnH03mP0yfxyBS1KjDHBJapRTKAtYSy9NZ3PLWEvnOQ7b0vmP8tbBUU0ztt6nW+MIXJIa9YAYgU/213hcf6Wn2+5CGy0M018LxWQ1LYR+hIX3s52Nhfjzn65t6ecBjsAlqVmpqpnvnBwBvB/YDjinqk7f2vbLli2r1atXz+hY8/XXf+Jf6YVQx1zWMJ8j8HEce7q3V47rnKf7SnA2r+omO+50bQsj8H7jHoGPcrSf5MqqWjZx+YxH4Em2Az4EPAc4GHhBkoNnXqIkaTpmM4XyZOD6qrqhqn4NfAY4ZjRlSZKmMuMplCTHAUdU1Uu75y8CnlJVJ0/Y7iTgpO7po4DrpnGYPYA7ZlTgtse+6LEfeuyH+z0Q+mL/qloyceFs7kLJgGVb/DWoqrOBs2d0gGT1oHmfByL7osd+6LEf7vdA7ovZTKHcCuzX93xfYN3sypEkDWs2Af4N4KAkByTZATgeuGg0ZUmSpjLjKZSqui/JycAX6d1G+JGqunZklfXMaOplG2Vf9NgPPfbD/R6wfTGr+8AlSfPHT2JKUqMMcElq1NgDPMkRSa5Lcn2SFQPW75jk/G79qiRL+9ad2i2/Lsmzp2qze0N1VZK1XZs7jPv8hjXH/fCxJDcmubr7OmTc5zesMfXDR5LcnuSaCW3tluTS7nq4NMmu4zy36Zrjvnhbkh/2XRNHjvPcpmPU/ZBkvySXJ1mT5Nokp/Rtv6CviWmrqrF90Xtz8/vAgcAOwP8CB0/Y5pXAWd3j44Hzu8cHd9vvCBzQtbPd1toEPgsc3z0+C3jFOM9vAffDx4Dj5vu856IfunXPAp4AXDOhrfcAK7rHK4Az5rsP5rEv3ga8Yb7Pey76AdgbeEK3zYOB7/X9bizYa2ImX+MegQ/zcftjgJXd4wuAw5OkW/6ZqvpVVd0IXN+1N7DNbp/Dujbo2jx2jOc2HXPWD3NwLrMxjn6gqr4C3DngeP1tLaTrAea+LxaqkfdDVa2vqqsAquoeYA2wz4C2Fto1MW3jDvB9gFv6nt/K/R25xTZVdR/wU2D3rew72fLdgbu6NiY71nyZy37Y6J1JvpXkzCQ7juIkRmAc/bA1e1XV+q6t9cCeM6589Oa6LwBO7q6JjyygqYOx9kM33fJ4YFW3aCFfE9M27gAf5uP2k20zquULwVz2A8CpwKOBJwG7AW8arsyxG0c/tGqu++LDwCOAQ4D1wL9MVeAcGVs/JFkMfA54bVXdPeMKF7BxB/gwH7fftE2SRcBD6b0EnGzfyZbfAezStTHZsebLXPYD3UvIqqpfAR+le3m9AIyjH7bmtiR7d23tDdw+48pHb077oqpuq6rfVNVvgX9jG78mkmxPL7zPq6rP922zkK+JaRt3gA/zcfuLgOXd4+OAL1XvHYaLgOO7d6APAA4Cvj5Zm90+l3dt0LV54RjPbTrmrB9g04VJN094LLDZHQnzaBz9sDX9bS2k6wHmuC82XhOdP2cbvia66/5cYE1VvXcrbS20a2L6xv0uKXAkvXeBvw+c1i17O/Dc7vFOwL/TewPi68CBffue1u13HfCcrbXZLT+wa+P6rs0d5/td4nnqhy8B36b3S/pJYPF8n/+Y++HT9KYF7qU3KjuxW747cBmwtvu+23yf/zz2xSe6a+Jb9EJs7/k+/3H1A/AMelMp3wKu7r6ObOGamO6XH6WXpEb5SUxJapQBLkmNMsAlqVEGuCQ1ygCXpEYZ4JLUKANckhr1/6jTAFCzrGJuAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.hist(cond_vals[:,9], 100);\n",
    "plt.title(\"Neuron 9 Distribution\")\n",
    "plt.figure()\n",
    "plt.hist(cond_vals[:,7], 100);\n",
    "plt.title(\"Neuron 7 Distribution\");\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we can look at the distributions of neurons 0 and 10, which appear to be learning strong features negatively correlated with survival."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXAAAAEICAYAAABGaK+TAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAVIElEQVR4nO3df7RlZX3f8fdHBkREwq8LGcFhME5VdFXS3KDR2iYCCQYirMQfGENGi51lTaxJtDqJmhhNG6iNJlk27Zol6sQYRYkWIqmVjJJoa9GBIEpQB8kAI+PMAIL4I0Hst3+cPcz1cO7cfe895977jO/XWnftn2fv731m7uc+9zln752qQpLUnoctdwGSpIUxwCWpUQa4JDXKAJekRhngktQoA1ySGmWAS7NI8t+TvGFMx1qT5JtJDuqWr07y0nEcuzve/0yyflzHUxsMcD0oyfYku5I8csa6lya5ehnL6i3J6Um+mOTbST6R5KT97Ls9yXeS3JfkniT/J8nLkjz4M1FVL6uqN/c47/YkZ+xvn6q6raoOr6rvze+7Gnm+Nyb5s6HjP7uqNi/22GqLAa5hq4BXTvokSVaN+XjHAh8C3gAcDWwFLp3jZT9XVY8CTgIuAl4LXDLOurraxvq9SnsZ4Br2FuDVSY4ctTHJE5JcleTuJF9K8vwZ275vWCDJi5N8asZyJfmVJNuAbd26pyf5bJJ7u+nTh4735iT/u+spf6wL6lF+Hrixqj5YVf8IvBF4SpInzPUNV9W9VXUF8AJgfZInd+d/d5Lf6+aPTfKRrrd+d5JPJnlYkvcAa4C/7IZIXpNkbfe9XpjkNuDjM9bNDPMfSfKZ7nu/PMnR3bl+MsmOoXbfnuSMJGcBvwW8oDvf54bbvqvr9UluTbI7yZ8m+aFu29461ie5LcmdSV43VxtpZTLANWwrcDXw6uEN3dDKVcCfA8cBLwT+JMmT5nH884CnAqd0gXUl8MfAMcBbgSuTHDNj/18EXtKd75BRdXWeBHxu70JVfQv4Sre+l6r6DLADeOaIza/qtk0BxzMI0aqqC4DbGPTmD6+q/zzjNf8aeCLwM7Oc8peBfwM8GniAQTvMVeNHgf8EXNqd7ykjdntx9/VTwGOBw4G3D+3zL4HHA6cDv53kiXOdWyuPAa5Rfht4RZKpofXnANur6l1V9UBVXQf8BfDceRz796vq7qr6DnA2sK2q3tMd733AF4Gfm7H/u6rqy93+HwBOneW4hwP3Dq27F3jUPGoDuIPBEMyw7wKrgZOq6rtV9cma+0ZCb6yqb3W1j/KeqvpC98vmDcDz977JuUgvAt5aVbdU1TeB3wTOH+r9/25VfaeqPsfgF9+oXwRa4QxwPURVfQH4CLBxaNNJwFO7YYR7ktzDICx+eB6Hv33G/KOBW4e23wqcMGP5azPmv80gqEf5JnDE0LojgPvmURvdue8esf4twM3Ax5LckmS4bUa5fR7bbwUOBmYbIpqP4Xa9lcF7G8fPWNe3XbWCGeCaze8A/5bvD9Pbgb+pqiNnfB1eVf+u2/4t4LAZ+48K9pm91jsY/FKYaQ3w1QXUeyMzepHdcM+PdOt7SfLjDL7fTw1vq6r7qupVVfVYBn8h/EaS0/dunuWQc/XQHzNjfg2DXv6dDLVj1yuf+dfQXMcdbtc1DIZods3xOjXGANdIVXUzg09x/PsZqz8C/LMkFyQ5uPv68Rnjp9cDP5/ksCSPAy6c4zR/1R3vF5OsSvIC4JTuPPP1YeDJSX4hyaEMhoFuqKovzvXCJEckOQd4P/BnVfX5Efuck+RxSQJ8A/he9wWDYHzsAmr+pSSnJDkMeBNwWfcxwy8DhyY5O8nBwOuBh8943S5g7cyPPA55H/DrSU5Ocjj7xswfWECNWsEMcO3Pm4AHPxNeVfcBPw2cz6CX9zXgYvaFy9uA+xkEzGbgvfs7eFXdxWBc/VXAXcBrgHOq6s75FlpVe4BfAP4j8HUGb5SeP8fL/jLJfQz+sngdgzdRXzLLvuuAv2YwVPNp4E+q6upu2+8Dr++GlWZ7k3WU9wDvZtCOh9L9sqyqe4GXA+9g8NfItxi8gbrXB7vpXUmuG3Hcd3bH/lvgH4B/BF4xj7rUiPhAB0lqkz1wSWqUAS5JjTLAJalRBrgkNWpJb7Jz7LHH1tq1a5fylJLUvGuvvfbOqhq+MnppA3zt2rVs3bp1KU8pSc1LMnzFMuAQiiQ1ywCXpEYZ4JLUqF4BnuTXk9yY5AtJ3pfk0O4+C9ck2Zbk0iSHTLpYSdI+cwZ4khMY3KNhuqqeDBzE4B4TFwNvq6p1DO49MdeNiyRJY9R3CGUV8IjuhvCHATuBZwGXdds3M3jSiiRpicwZ4FX1VeC/MHhs1E4GTzm5Frhnxu0pd/D9941+UJINSbYm2bpnz57xVC1J6jWEchRwLnAygyd9PBJ49ohdR97WsKo2VdV0VU1PTT3kc+iSpAXqM4RyBvAPVbWnqr4LfAh4OnDkjGfsncjg/tCSpCXS50rM24CndU8N+Q6Dp1hvBT7B4GG27wfWA5dPqkhJas3ajVc+OL/9orMnco4+Y+DXMHiz8jrg891rNgGvZfBcwJuBY4BLJlKhJGmkXvdCqarfYfCQ25luAU4be0WSpF68ElOSGmWAS1KjDHBJapQBLkmNMsAlqVEGuCQ1ygCXpEYZ4JLUKANckhplgEtSowxwSWqUAS5JjTLAJalRBrgkNcoAl6RGGeCS1CgDXJIa1eep9I9Pcv2Mr28k+bUkRye5Ksm2bnrUUhQsSRro80zML1XVqVV1KvBjwLeBDwMbgS1VtQ7Y0i1LkpbIfIdQTge+UlW3AucCm7v1m4HzxlmYJGn/5hvg5wPv6+aPr6qdAN30uFEvSLIhydYkW/fs2bPwSiVJ36d3gCc5BHgO8MH5nKCqNlXVdFVNT01Nzbc+SdIs5tMDfzZwXVXt6pZ3JVkN0E13j7s4SdLs5hPgL2Tf8AnAFcD6bn49cPm4ipIkza1XgCc5DDgT+NCM1RcBZybZ1m27aPzlSZJms6rPTlX1beCYoXV3MfhUiiRpGXglpiQ1ygCXpEYZ4JLUKANckhplgEtSowxwSWqUAS5JjTLAJalRBrgkNcoAl6RGGeCS1CgDXJIaZYBLUqMMcElqlAEuSY0ywCWpUQa4JDWq7yPVjkxyWZIvJrkpyU8kOTrJVUm2ddOjJl2sJGmfvj3wPwI+WlVPAJ4C3ARsBLZU1TpgS7csSVoicwZ4kiOAfwVcAlBV91fVPcC5wOZut83AeZMqUpL0UH164I8F9gDvSvJ3Sd6R5JHA8VW1E6CbHjfBOiVJQ/oE+CrgXwD/rap+FPgW8xguSbIhydYkW/fs2bPAMiVJw/oE+A5gR1Vd0y1fxiDQdyVZDdBNd496cVVtqqrpqpqempoaR82SJHoEeFV9Dbg9yeO7VacDfw9cAazv1q0HLp9IhZKkkVb13O8VwHuTHALcAryEQfh/IMmFwG3A8yZToiRplF4BXlXXA9MjNp0+3nIkSX15JaYkNcoAl6RGGeCS1CgDXJIaZYBLUqMMcElqlAEuSY0ywCWpUQa4JDXKAJekRhngktQoA1ySGmWAS1KjDHBJapQBLkmNMsAlqVEGuCQ1ygCXpEb1eqRaku3AfcD3gAeqajrJ0cClwFpgO/D8qvr6ZMqUJA2bTw/8p6rq1Kra+2zMjcCWqloHbOmWJUlLZDFDKOcCm7v5zcB5iy9HktRX3wAv4GNJrk2yoVt3fFXtBOimx416YZINSbYm2bpnz57FVyxJAnqOgQPPqKo7khwHXJXki31PUFWbgE0A09PTtYAaJUkj9OqBV9Ud3XQ38GHgNGBXktUA3XT3pIqUJD3UnAGe5JFJHrV3Hvhp4AvAFcD6brf1wOWTKlKS9FB9hlCOBz6cZO/+f15VH03yWeADSS4EbgOeN7kyJUnD5gzwqroFeMqI9XcBp0+iKKklazde+eD89ovOXsZK9IPGKzElqVEGuCQ1ygCXpEYZ4JLUKANckhplgEtSowxwSWqUAS5JjTLAJalRBrgkNcoAl6RGGeCS1CgDXJIaZYBLUqMMcElqlAEuSY0ywCWpUX2fSk+Sg4CtwFer6pwkJwPvB44GrgMuqKr7J1OmJK18M5/OtBTm0wN/JXDTjOWLgbdV1Trg68CF4yxMkrR/vQI8yYnA2cA7uuUAzwIu63bZDJw3iQIlSaP1HUL5Q+A1wKO65WOAe6rqgW55B3DCqBcm2QBsAFizZs3CK5UaMPwntA851iTN2QNPcg6wu6qunbl6xK416vVVtamqpqtqempqaoFlSpKG9emBPwN4TpKfBQ4FjmDQIz8yyaquF34icMfkypQkDZuzB15Vv1lVJ1bVWuB84ONV9SLgE8Bzu93WA5dPrEpJ0kMs5nPgrwV+I8nNDMbELxlPSZKkPnp/Dhygqq4Gru7mbwFOG39JkqQ+vBJTkhplgEtSowxwSWqUAS5JjTLAJalRBrgkNcoAl6RGGeCS1CgDXJIaZYBLUqMMcElqlAEuSY0ywCWpUQa4JDXKAJekRhngktQoA1ySGtXnqfSHJvlMks8luTHJ73brT05yTZJtSS5Ncsjky5Uk7dWnB/5PwLOq6inAqcBZSZ4GXAy8rarWAV8HLpxcmZKkYX2eSl9V9c1u8eDuq4BnAZd16zcD502kQknSSL3GwJMclOR6YDdwFfAV4J6qeqDbZQdwwiyv3ZBka5Kte/bsGUfNkiR6BnhVfa+qTgVOZPAk+ieO2m2W126qqumqmp6amlp4pZKk7zOvT6FU1T3A1cDTgCOTrOo2nQjcMd7SJEn70+dTKFNJjuzmHwGcAdwEfAJ4brfbeuDySRUpSXqoVXPvwmpgc5KDGAT+B6rqI0n+Hnh/kt8D/g64ZIJ1SpKGzBngVXUD8KMj1t/CYDxckrQMvBJTkhplgEtSowxwSWqUAS5JjTLAJalRBrgkNcoAl6RGGeCS1CgDXJIaZYBLUqMMcElqlAEuSY0ywCWpUQa4JDXKAJekRhngktQoA1ySGtXnmZiPSfKJJDcluTHJK7v1Rye5Ksm2bnrU5MuVJO3Vpwf+APCqqnoig6fR/0qSU4CNwJaqWgds6ZYlSUtkzgCvqp1VdV03fx+DJ9KfAJwLbO522wycN6kiJUkP1eep9A9KspbBA46vAY6vqp0wCPkkx83ymg3ABoA1a9YsplapaWs3Xvng/PaLzl7GSnSg6P0mZpLDgb8Afq2qvtH3dVW1qaqmq2p6ampqITVKkkboFeBJDmYQ3u+tqg91q3clWd1tXw3snkyJkqRR+nwKJcAlwE1V9dYZm64A1nfz64HLx1+eJGk2fcbAnwFcAHw+yfXdut8CLgI+kORC4DbgeZMpUZI0ypwBXlWfAjLL5tPHW44kqS+vxJSkRhngktQoA1ySGmWAS1KjDHBJatS8LqWXNFlebq/5sAcuSY0ywCWpUQa4JDXKAJekRvkmprQAM99sHMd+0kLYA5ekRhngktQoh1AkaRGWc5jMHrgkNcoAl6RGGeCS1Kg+z8R8Z5LdSb4wY93RSa5Ksq2bHjXZMiVJw/r0wN8NnDW0biOwparWAVu6ZUnSEpozwKvqb4G7h1afC2zu5jcD5425LknSHBY6Bn58Ve0E6KbHja8kSVIfE/8ceJINwAaANWvWTPp0UhO877fGYaE98F1JVgN0092z7VhVm6pquqqmp6amFng6SdKwhfbArwDWAxd108vHVpGk3uzJ/2Dr8zHC9wGfBh6fZEeSCxkE95lJtgFndsuSpCU0Zw+8ql44y6bTx1yLJGkevJmV1JP39tZK46X0ktQoA1ySGmWAS1KjDHBJapRvYkoN8PPe4zffNl2J/wb2wCWpUQa4JDXKIRRpyEr8U1kaxR64JDXKHrjE8l5lOdu5vfJz5Vop/zb2wCWpUQa4JDXKIRT9wFopfwZPgm/EjsdK/z9iD1ySGmWAS1KjHEKR9mMl/gk930+tjGs4ZZxtMYk6xnXMloac7IFLUqMW1QNPchbwR8BBwDuqamLPxmz1N6SWziR6ZgealfhzNImaflD+Lyy4B57kIOC/As8GTgFemOSUcRUmSdq/xQyhnAbcXFW3VNX9wPuBc8dTliRpLqmqhb0weS5wVlW9tFu+AHhqVf3q0H4bgA3d4uOBLy283BXlWODO5S5ihbAt9rEt9rEtBsbRDidV1dTwysWMgWfEuof8NqiqTcCmRZxnRUqytaqml7uOlcC22Me22Me2GJhkOyxmCGUH8JgZyycCdyyuHElSX4sJ8M8C65KcnOQQ4HzgivGUJUmay4KHUKrqgSS/CvwvBh8jfGdV3Ti2yla+A25YaBFsi31si31si4GJtcOC38SUJC0vr8SUpEYZ4JLUKAN8HpIcneSqJNu66VH72feIJF9N8valrHEp9GmHJCcluTbJ9UluTPKy5ah10nq2xalJPt21ww1JXrActU5a35+PJB9Nck+Sjyx1jZOU5KwkX0pyc5KNI7Y/PMml3fZrkqxd7DkN8PnZCGypqnXAlm55Nm8G/mZJqlp6fdphJ/D0qjoVeCqwMcmjl7DGpdKnLb4N/HJVPQk4C/jDJEcuYY1Lpe/Px1uAC5asqiXQ89YiFwJfr6rHAW8DLl7seQ3w+TkX2NzNbwbOG7VTkh8Djgc+tkR1LbU526Gq7q+qf+oWH86B+3+tT1t8uaq2dfN3ALuBh1xVdwDo9fNRVVuA+5aqqCXS59YiM9vnMuD0JKMuiOztQP2hmpTjq2onQDc9bniHJA8D/gD4D0tc21Kasx0AkjwmyQ3A7cDFXXgdaHq1xV5JTgMOAb6yBLUttXm1xQHmBAb/z/fa0a0buU9VPQDcCxyzmJP6QIchSf4a+OERm17X8xAvB/6qqm5f5C/XZTWGdqCqbgf+eTd08j+SXFZVu8ZV41IZR1t0x1kNvAdYX1X/bxy1LbVxtcUBqM+tRXrdfmQ+DPAhVXXGbNuS7Eqyuqp2dj+Mu0fs9hPAM5O8HDgcOCTJN6tqf+PlK84Y2mHmse5IciPwTAZ/OjZlHG2R5AjgSuD1VfV/J1TqxI3z/8UBps+tRfbusyPJKuCHgLsXc1KHUObnCmB9N78euHx4h6p6UVWtqaq1wKuBP20tvHuYsx2SnJjkEd38UcAzOHDuRDlTn7Y4BPgwg/8LH1zC2pbanG1xAOtza5GZ7fNc4OO12Cspq8qvnl8Mxqu2ANu66dHd+mkGTyQa3v/FwNuXu+7laAfgTOAG4HPddMNy172MbfFLwHeB62d8nbrctS9HW3TLnwT2AN9h0Cv9meWufUzf/88CX2bw/sbrunVvAp7TzR8KfBC4GfgM8NjFntNL6SWpUQ6hSFKjDHBJapQBLkmNMsAlqVEGuCQ1ygCXpEYZ4JLUqP8PC5m4kmrgwUUAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAEICAYAAABPgw/pAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAV+0lEQVR4nO3dfbBkdX3n8fdHRmANIiAD8uA4UEtUTAJYN0p0K0GwCAIRdoMGo+yYQCa6xjWllo6ajcbNRnA3IZtKspFVYXwIQjCGUWKU8GDWXSUMBpUHcUYywjgjMwqomARBv/tHnyHNpe9033u77537m/erqqvPw++c8+1f3/nM6V+f7k5VIUla+h632AVIksbDQJekRhjoktQIA12SGmGgS1IjDHRJaoSBLo0oyQNJjhzTvt6a5L3d9MoklWTZmPa9oqt1j3HsT0uHgb4bS7IpyT1Jfqxv2XlJrl/EskaSZM8kV3SPoZKcMG19klyQ5Nvd7d1JMsO+Tkjyoy4EH0iyOcnlSX66v11V7VNVdw6p64Qkm4fVX1W/V1XnjfBQh+r64IV9+76rq/WH49i/lg4DXcuA1036IOM6+5zms8ArgG8OWLcaOBM4Bvgp4HTg13eyry1VtQ/wROB44CvA/0ly0lgrZmJ9IRno4r8Db0yy36CVSZ6R5Ook9ya5I8lL+9Zdn+S8vvlXJvls33wleU2SDcCGbtnzktyY5Dvd/fOm7e+/Jvm/Sb6X5NNJDhxUV1X9oKr+sKo+Cww6E10F/H5Vba6qbwC/D7xyWGdUz+aq+m3gvcAF0x7Pv+2mT01yW1fnN5K8sXul80ng0L6z/UOTvKN7NfGhJN8FXtkt+9C0w/9qki1JtiZ5Q99xL0nyu33zj7wKSPJBYAXw8e54b5o+hNPVsK57Djcm+bW+fb2jezXyge6x3Jpkalg/addkoGs9cD3wxukruoC6Gvhz4CDgZcCfJnnWLPZ/JvBc4OgkBwBXAX8EPBn4A+CqJE/ua//LwK90x9tzUF0jehbwxb75L3bLZuMvgWf3D0n1eR/w61X1ROAngGur6vvAi+jO9rvblq79GcAVwH7Ah2c43guAo4CTgTX9wygzqapzgLuAX+iO9+4BzS4FNgOHAmcBvzftlceLgY90ta0D/njYcbVrMtAF8NvAa5Msn7b8dGBTVV1cVQ9X1ReAj9ILhVG9q6rurap/Bk4DNlTVB7v9XUpvaOMX+tpfXFVf7dpfDhw7x8e0D/CdvvnvAPvMNI4+gy1A6AXddA/R+09q36q6r+ubnflcVf1VVf2oe2yD/E5Vfb+qvgxcTO8/0HlJ8lTg3wFvrqp/qaqb6b3yOKev2Wer6q+7MfcP0hum0hJkoIuqugX4BLBm2qqnAc9Ncv+OG/By4Cmz2P3dfdOHAl+ftv7rwGF98/3j4f9EL5jn4gFg3775fYEHanbfRncYUMD9A9b9InAq8PUkn0nyM0P2dfeQ9dPbfJ1ef83XocC9VfW9afveWZ/v7Tj/0mSga4e3A7/Go/+h3w18pqr267vtU1Wv7tZ/H3hCX/tBQd8foFvo/SfRbwXwjfmVPtCtPPpM85hu2Wz8e+AL3VDKo1TVjVV1Br2hob+i92oCHv14H7XJCMd7at/0Cnr9BcP7eWf73gIckOSJ0/Y9iT7XIjPQBUBVbQQuA/5z3+JPAD+e5Jwkj+9uP53kmd36m4H/kOQJ3ZuF5w45zF93+/vlJMuS/BJwdHecWUuyV5K9u9k9k+zdN6TyAeD1SQ5LcijwBuCSEfaZbpu3A+cBbx3QZs8kL0/ypKp6CPgu//rG7D3Ak5M8aQ4P6b90ffkseu8jXNYtvxk4NckBSZ4C/Oa07e4BBl4fX1V3A/8PeFfXPz9F73maaRxfS5iBrn7vBB55A7B7mX4ycDa9M71v0rvqY6+uyYXAD+gFylqGhERVfZveuPwbgG8DbwJOr6pvzbHeO4B/pveq4lPd9I5XAO8BPg58GbiF3pux79nJvg5N8gC9oZobgZ8ETqiqT8/Q/hxgU3fVyqvoXT5JVX2F3puQd3bDVLMZNvkMsBG4Bvgffcf+IL03dTcBn+Zfg36HdwG/1R1v0JvILwNW0nsOPwa8vaqunkVdWiLiD1xIUhs8Q5ekRhjoktQIA12SGmGgS1IjFvTDAwceeGCtXLlyIQ8pSUveTTfd9K2qmv5J7sdY0EBfuXIl69evX8hDStKSl2T6J6wHcshFkhphoEtSIwx0SWqEgS5JjTDQJakRBrokNWKkQE+yX/ebiF9JcnuSn+m+yvPqJBu6+/0nXawkaWajnqH/T+BvquoZ9H4o4HZ6v25zTVUdRe/rPqf/2o0kaQENDfQk+wI/S+9HcXf82vr99H70dm3XbC29HwOWJC2SUT4peiSwHbg4yTHATcDrgIOraitAVW1NctCgjZOsBlYDrFixYixFazxWrrnqkelN55+2iJVIGodRhlyWAc8G/ldVHUfv9w1HHl6pqouqaqqqppYvH/pVBJKkORol0DcDm6vqhm7+CnoBf0+SQwC6+22TKVGSNIqhgV5V3wTuTvL0btFJwG3AOmBVt2wVcOVEKpQkjWTUb1t8LfDhJHsCd9L7RfLHAZcnORe4C3jJZEqUJI1ipECvqpuBqQGrThpvOZKkufKTopLUCANdkhphoEtSIwx0SWqEgS5JjTDQJakRBrokNcJAl6RGGOiS1AgDXZIaYaBLUiMMdElqhIEuSY0w0CWpEQa6JDXCQJekRhjoktQIA12SGmGgS1IjDHRJaoSBLkmNMNAlqREGuiQ1wkCXpEYY6JLUiGWjNEqyCfge8EPg4aqaSnIAcBmwEtgEvLSq7ptMmZKkYWZzhv6Cqjq2qqa6+TXANVV1FHBNNy9JWiTzGXI5A1jbTa8Fzpx/OZKkuRo10Av4dJKbkqzulh1cVVsBuvuDBm2YZHWS9UnWb9++ff4VS5IGGmkMHXh+VW1JchBwdZKvjHqAqroIuAhgamqq5lCjJGkEI52hV9WW7n4b8DHgOcA9SQ4B6O63TapISdJwQwM9yY8leeKOaeBk4BZgHbCqa7YKuHJSRUqShhtlyOVg4GNJdrT/86r6myQ3ApcnORe4C3jJ5MqU5m7lmqsemd50/mmLWIk0WUMDvaruBI4ZsPzbwEmTKEqSNHt+UlSSGmGgS1IjDHRJaoSBLkmNMNAlqREGuiQ1wkCXpEYY6JLUCANdkhphoEtSIwx0SWqEgS5JjTDQJakRBrokNcJAl6RGGOiS1AgDXZIaYaBLUiMMdElqhIEuSY0w0CWpEQa6JDXCQJekRhjoktQIA12SGmGgS1IjRg70JHsk+Yckn+jmj0hyQ5INSS5LsufkypQkDTObM/TXAbf3zV8AXFhVRwH3AeeOszBJ0uyMFOhJDgdOA97bzQc4Ebiia7IWOHMSBUqSRjPqGfofAm8CftTNPxm4v6oe7uY3A4cN2jDJ6iTrk6zfvn37vIqVJM1saKAnOR3YVlU39S8e0LQGbV9VF1XVVFVNLV++fI5lSpKGWTZCm+cDL05yKrA3sC+9M/b9kizrztIPB7ZMrkxJ0jBDz9Cr6i1VdXhVrQTOBq6tqpcD1wFndc1WAVdOrEpJ0lDzuQ79zcDrk2ykN6b+vvGUJEmai1GGXB5RVdcD13fTdwLPGX9JkqS58JOiktQIA12SGmGgS1IjDHRJasSs3hSVlrqVa656ZHrT+actYiXS+HmGLkmNMNAlqREGuiQ1wkCXpEYY6JLUCANdkhphoEtSIwx0SWqEgS5JjTDQJakRBrokNcJAl6RGGOiS1AgDXZIaYaBLUiMMdElqhIEuSY0w0CWpEQa6JDXCQJekRgwN9CR7J/n7JF9McmuS3+mWH5HkhiQbklyWZM/JlytJmskoZ+gPAidW1THAscApSY4HLgAurKqjgPuAcydXpiRpmKGBXj0PdLOP724FnAhc0S1fC5w5kQolSSMZaQw9yR5Jbga2AVcDXwPur6qHuyabgcNm2HZ1kvVJ1m/fvn0cNUuSBhgp0Kvqh1V1LHA48BzgmYOazbDtRVU1VVVTy5cvn3ulkqSdmtVVLlV1P3A9cDywX5Jl3arDgS3jLU2SNBujXOWyPMl+3fS/AV4I3A5cB5zVNVsFXDmpIiVJwy0b3oRDgLVJ9qD3H8DlVfWJJLcBH0nyu8A/AO+bYJ2SpCGGBnpVfQk4bsDyO+mNp0uSdgF+UlSSGjHKkIsasnLNVYtdgqQJ8QxdkhphoEtSIwx0SWqEgS5JjTDQJakRBrokNcJAl6RGGOiS1AgDXZIaYaBLUiMMdElqhIEuSY0w0CWpEQa6JDXCQJekRhjoktQIA12SGmGgS1IjDHRJaoSBLkmNMNAlqREGuiQ1wkCXpEYY6JLUiKGBnuSpSa5LcnuSW5O8rlt+QJKrk2zo7veffLmSpJmMcob+MPCGqnomcDzwmiRHA2uAa6rqKOCabl6StEiGBnpVba2qL3TT3wNuBw4DzgDWds3WAmdOqkhJ0nCzGkNPshI4DrgBOLiqtkIv9IGDZthmdZL1SdZv3759ftVKkmY0cqAn2Qf4KPCbVfXdUberqouqaqqqppYvXz6XGiVJIxgp0JM8nl6Yf7iq/rJbfE+SQ7r1hwDbJlOiJGkUo1zlEuB9wO1V9Qd9q9YBq7rpVcCV4y9PkjSqZSO0eT5wDvDlJDd3y94KnA9cnuRc4C7gJZMpUZI0iqGBXlWfBTLD6pPGW44kaa78pKgkNWKUIRctcSvXXLXYJUhaAJ6hS1IjDHRJaoSBLkmNMNAlqREGuiQ1wkCXpEYY6JLUCANdkhphoEtSI/ykqIBHf5p00/mnLWIlkubKM3RJaoSBLkmNcMhFO+VQjLR0eIYuSY0w0CWpEQa6JDXCMXQ9hj+IIS1NnqFLUiMMdElqhEMuS5CXEkoaxDN0SWqEgS5JjTDQJakRQ8fQk7wfOB3YVlU/0S07ALgMWAlsAl5aVfdNrkztChy7l3Zto5yhXwKcMm3ZGuCaqjoKuKablyQtoqGBXlV/B9w7bfEZwNpuei1w5pjrkiTN0lwvWzy4qrYCVNXWJAfN1DDJamA1wIoVK+Z4OO1qpn+a1CEYafFN/E3Rqrqoqqaqamr58uWTPpwk7bbmGuj3JDkEoLvfNr6SJElzMddAXwes6qZXAVeOpxxJ0lyNctnipcAJwIFJNgNvB84HLk9yLnAX8JJJFtmiSVwC6GWF0u5taKBX1ctmWHXSmGuRJM2DnxSVpEb4bYsaC4d7pMXnGbokNcJAl6RGGOiS1AjH0Je4mX7Q2R96lnY/nqFLUiMMdElqhEMuWjCTvrTRSye1u/MMXZIaYaBLUiMMdElqhGPoEzbbcV3HgSXNlWfoktQIA12SGrFbDLnMZ9hjuklfbjefNruKUfrboSVp/DxDl6RGGOiS1IjdYshlIcx2mEGSxs0zdElqhIEuSY0w0CWpEUtmDH1cl8LNNI49zkvnHCtfOPP5gY+F+FuQFpJn6JLUCANdkhqRqlqwg01NTdX69evntO0oL49nGnJxCGTp2NWeN4dfNFfj/DR0kpuqampYu3mdoSc5JckdSTYmWTOffUmS5mfOgZ5kD+BPgBcBRwMvS3L0uAqTJM3OfM7QnwNsrKo7q+oHwEeAM8ZTliRptuY8hp7kLOCUqjqvmz8HeG5V/ca0dquB1d3s04E75l7u2B0IfGuxixjBUqhzKdQI1jlOS6FGaKPOp1XV8mE7mM916Bmw7DH/O1TVRcBF8zjOxCRZP8obDYttKdS5FGoE6xynpVAj7F51zmfIZTPw1L75w4Et8ylGkjR38wn0G4GjkhyRZE/gbGDdeMqSJM3WnIdcqurhJL8BfArYA3h/Vd06tsoWxi45FDTAUqhzKdQI1jlOS6FG2I3qXNAPFkmSJseP/ktSIwx0SWpE84Ge5IAkVyfZ0N3vP6DNC5Lc3Hf7lyRndusuSfKPfeuOXaw6u3Y/7KtlXd/yI5Lc0G1/WfdG9YLXmOTYJJ9LcmuSLyX5pb51E+3LYV9FkWSvrm82dn21sm/dW7rldyT5+XHWNcsaX5/ktq7vrknytL51A5/7RarzlUm299VzXt+6Vd3fyIYkqxaxxgv76vtqkvv71i1kX74/ybYkt8ywPkn+qHscX0ry7L51s+vLqmr6BrwbWNNNrwEuGNL+AOBe4And/CXAWbtKncADMyy/HDi7m/4z4NWLUSPw48BR3fShwFZgv0n3Jb035r8GHAnsCXwROHpam/8E/Fk3fTZwWTd9dNd+L+CIbj97LFKNL+j723v1jhp39twvUp2vBP54wLYHAHd29/t30/svRo3T2r+W3oUbC9qX3bF+Fng2cMsM608FPknvsz3HAzfMtS+bP0On93UEa7vptcCZQ9qfBXyyqv5polU91mzrfESSACcCV8xl+1kYWmNVfbWqNnTTW4BtwNBPuI3BKF9F0V//FcBJXd+dAXykqh6sqn8ENnb7W/Aaq+q6vr+9z9P7fMdCm8/Xevw8cHVV3VtV9wFXA6fsAjW+DLh0AnUMVVV/R+8kcSZnAB+ons8D+yU5hDn05e4Q6AdX1VaA7v6gIe3P5rFP/H/rXgpdmGSvSRTJ6HXunWR9ks/vGBYCngzcX1UPd/ObgcMWsUYAkjyH3tnT1/oWT6ovDwPu7psf1AePtOn66jv0+m6UbReqxn7n0jtz22HQcz8Jo9b5i91zeUWSHR8y3OX6shu2OgK4tm/xQvXlKGZ6LLPuyyXzE3Q7k+RvgacMWPW2We7nEOAn6V1bv8NbgG/SC6aLgDcD71zEOldU1ZYkRwLXJvky8N0B7eZ0PeqY+/KDwKqq+lG3eGx9OeiQA5ZN74OZ2oz0NRZjMPJxkrwCmAJ+rm/xY577qvraoO0XoM6PA5dW1YNJXkXvlc+JI247DrM5ztnAFVX1w75lC9WXoxjb32UTgV5VL5xpXZJ7khxSVVu7kNm2k129FPhYVT3Ut++t3eSDSS4G3riYdXbDGFTVnUmuB44DPkrvZdqy7sxzzl/DMI4ak+wLXAX8VvcScse+x9aXA4zyVRQ72mxOsgx4Er2Xwgv1NRYjHSfJC+n9B/pzVfXgjuUzPPeTCKGhdVbVt/tm/zdwQd+2J0zb9vqxVzi75+xs4DX9CxawL0cx02OZdV/uDkMu64Ad7w6vAq7cSdvHjLN1wbVjnPpMYOA71WMwtM4k++8YpkhyIPB84LbqvYNyHb3x/xm3X6Aa9wQ+Rm9M8C+mrZtkX47yVRT99Z8FXNv13Trg7PSugjkCOAr4+zHWNnKNSY4D3gO8uKq29S0f+NxPoMZR6zykb/bFwO3d9KeAk7t69wdO5tGveBesxq7Op9N7Q/FzfcsWsi9HsQ74j93VLscD3+lOfmbflwv1Tu9i3eiNkV4DbOjuD+iWTwHv7Wu3EvgG8Lhp218LfJle+HwI2Gex6gSe19Xyxe7+3L7tj6QXQhuBvwD2WqQaXwE8BNzcdzt2IfqS3tUCX6V3pvW2btk76YUjwN5d32zs+urIvm3f1m13B/CiCf49Dqvxb4F7+vpu3bDnfpHqfBdwa1fPdcAz+rb91a6PNwK/slg1dvPvAM6ftt1C9+Wl9K72eojeWfe5wKuAV3XrQ+/Hgr7W1TM11770o/+S1IjdYchFknYLBrokNcJAl6RGGOiS1AgDXZIaYaBLUiMMdElqxP8HVDAID34gbm4AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.hist(cond_vals[:,0], 100);\n",
    "plt.title(\"Neuron 0 Distribution\")\n",
    "plt.figure()\n",
    "plt.hist(cond_vals[:,10], 100);\n",
    "plt.title(\"Neuron 10 Distribution\");"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have identified that some of the neurons are not learning important features, while others are. Can we now understand what each of these important neurons are looking at in the input? For instance, are they identifying different features in the input or similar ones?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To answer these questions, we can apply the third type of attributions available in Captum, **Neuron Attributions**. This allows us to understand what parts of the input contribute to activating a particular input neuron. For this example, we will apply Neuron Conductance, which divides the neuron's total conductance value into the contribution from each individual input feature."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To use Neuron Conductance, we create a NeuronConductance object, analogously to Conductance, passing in the model as well as the module (layer) whose output we would like to understand, in this case, net.sigmoid1, as before."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "neuron_cond = NeuronConductance(net, net.sigmoid1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now obtain the neuron conductance values for all the test examples by calling attribute on the NeuronConductance object. Neuron Conductance requires the neuron index in the target layer for which attributions are requested as well as the target index for networks with mutliple outputs, similar to layer conductance. As before, we provide target = 1, corresponding to survival, and compute neuron conductance for neurons 0 and 10, the significant neurons identified above. The neuron index can be provided either as a tuple or as just an integer if the layer output is 1-dimensional."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "neuron_cond_vals_10 = neuron_cond.attribute(test_input_tensor, neuron_index=10, target=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "neuron_cond_vals_0 = neuron_cond.attribute(test_input_tensor, neuron_index=0, target=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average Feature Importances for Neuron 0\n",
      "age :  -0.128\n",
      "sibsp :  -0.004\n",
      "parch :  0.003\n",
      "fare :  0.003\n",
      "female :  0.000\n",
      "male :  0.001\n",
      "embark_C :  0.006\n",
      "embark_Q :  -0.000\n",
      "embark_S :  -0.004\n",
      "class_1 :  0.002\n",
      "class_2 :  0.007\n",
      "class_3 :  -0.019\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAtEAAAGECAYAAADwYnzeAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAgAElEQVR4nO3deZxkVX3//9dbQERBFkFEQCcSXHDDOAFNXIis6teAu8bo8BMk6NcYf8Z8xaCAoAajfs1ijBIk4C64AWoEHMWFaGRAdpdxGQRBGTYVcMXP9497Woq2uqfPdPf0zPB6Ph716Lq3zr33c2rpetepU1WpKiRJkiTN3J0WugBJkiRpXWOIliRJkjoZoiVJkqROhmhJkiSpkyFakiRJ6mSIliRJkjoZoiVJ67wkr09ybZIfLXQtku4YDNHSHUySs5PckGTjha5lLrT+/CLJTSOnR89yn4uSVJIN56rOGRzzqCTvW1PHm06SA5N8eaHrmKkkOwJ/C+xSVfeao31WkouT3Glk3euTnDgX+59vSf4iyeVJbk7yiSRbLXRN0vrGEC3dgSRZBDwWKODP5+kYayx4jnhpVW06cvrKAtTwOxmsk/9fF+j2m637AtdV1TW9G66iv/cGnrPaVc1NDauzvwcD7wKeD2wL3AK8Yy6PIckQLd3RvAD4KnAisGRiZZJHJflRkg1G1j01yUXt/J2SHJbku0muS3LyxMjWyKjtQUl+AHyurT+l7fMnSb7Yntgn9n2PJKcn+WmSc9sI35dHLn9gkrOSXJ/kW0metTqdnW4/SZ6c5OuthiuSHDWy6Rfb3xsnRrYnjxRPHq1uI+JvSHIOQ2i5X5LNk7w7ydVJftj6uQEz0Pb9kiTLk/wsyTFJdkrylVbzyUnu3NrukeTKJH/fpjSsSPK8kX1tnuQ9SVa20cnXTIT8Nup8TpK3Jbke+DDwTuDRre83rur6GrkuliT5Qavh8JHLN2i1fbf15bw2eryq2+hJSS5r2/wwySvHXE97AWcB9271ntjW/3mSS5Pc2G6bB41ssyLJq9r9++ZpQuw/Aq+b6vL2uPnvdowLk+wx6Rh7jSz/7v4zzWNmVTW/MslFGR5TH05ylynqfh5welV9sapuAl4LPC3JZlO0l7QaDNHSHcsLgPe3075JtgWoqq8CNwNPGGn7F8AH2vmXAQcAj2cYnbsB+LdJ+3488CBg37b8X8DOwD2B89sxJ/xbO969GML8aKC/G0Mo+kDb9rnAOzISwmdiBvu5uV0fWwBPBl6c5IB22ePa3y06R7afDxwCbAZcDpwE/Ab4Q+ARwD7AwR3d2A94JPAo4P8AxzEEpB2Bh7Q+TbgXsDWwPcP1eVySB7TL/hXYHLgfw+30AuD/G9l2d+B7DNfTXwKHAl9pfd+itZnu+prwGOABwJ7AESMh8BWt1icBdwdeCNwyg9vo3cBfVdVmrb+fm3wFVdVngScCV7V6D0xyf+CDwMuBbYBPA6dPvOhontv6sUVV/WbyfpuPAT8FDpx8QZLtgU8Brwe2Al4JfDTJNlPsa5zfPWZmWPOzGO4TfwA8bFxdzYOBCycWquq7wK+A+3fUJmkVDNHSHUSSxzC87X1yVZ0HfJchKE/4IC2UtRGrJ7V1AH8FHF5VV1bVL4GjgGdMGqE7qqpurqqfA1TVCVX1s5H2D28johsATweOrKpbquoyhrA54X8BK6rqP6vqN1V1PvBR4BnTdO9f2ujdjUnOn8l+qursqrq4qn5bVRe1vj5+Rlfm1E6sqktbKNuKIdy9vF0v1wBvo296wJuq6qdVdSlwCXBmVX2vqn7C8CLlEZPav7aqfllVX2AIeM9q1/ezgVe322MF8FaGwD/hqqr613Y9/XxcITO8vl5XVT+vqgsZQtzD2/qDgddU1bdqcGFVXceqb+tfA7skuXtV3dAun4lnA5+qqrOq6tfAW4BNgD8ZafMvVXXFVP2d6DbDKO4R+f3PEPwl8Omq+nS7Ts4CljE8bmZq9DEz05qvqqrrgdOBXafY76bATyat+wnDiztJc8QQLd1xLGEIYde25Q8wMgLclp/WwsLTgPOr6vJ22X2Bj08EVeAbwK0M8y0nXDFxpr19f2x7+/6nwIp20dYMo2wbjrafdP6+wO4jofhGhtHX6T4w9rKq2qKd/mgm+0mye5LPtykOP2EYfd16mmPMxOR+bARcPXL8dzGMuM7Uj0fO/3zM8qYjyzdU1c0jy5czvGuwNXDntjx62fZT1D3WDK+v0W/GuGWkvh0ZXrRNtqrb+ukMofTyJF/IzD8wem9G+ltVv2XoY1ef27afBn7A8A7D5NqfOan2xwDbzbDGyTXMpOaprt/JbmIY8R91d+BnHbVJWoV18QMkkjol2YThreANcttXgG0MbJHk4W1k8LIklzOMno5O5YDhyfyFVXXOmH0vamdrZPVfAPsDezEE6M0ZpoAEWMkwxWEH4Nut/Y6TjvWFqtp7tTo78/18AHg78MSq+kWSf+K2UFhj2t8M3HVkeVyoH93uCuCXwNbTTBeYS1smudtIkL4Pw+j1tQwjuvcFLhu57Icj207u77j+T3d9rcoVwE6tnsnrp7yNqupcYP8kGwEvBU7m9veVqVwFPHRiIUnadtP1eTqvAT7E7z8m3ltVL5pim977y0xqnqlLue1dAJLcj+Hx/u0pt5DUzZFo6Y7hAIaR410Y3gLelWEu5pcY5rlO+ADD/OfHAaeMrH8n8IYk9wVIsk2S/ac53mYMAfI6hiDxxokLqupWhrmmRyW5a5IHTqrhk8D9kzw/yUbt9MejH7KaoVXtZzPg+hYId+P2U1tWAr9lmEM84QLgcUnuk2Rz4NXTHbyqrgbOBN6a5O4ZPpy5U5LZThmZzuuS3DnJYxmmSpzSru+TGW6/zdpt+Apguq/T+zGww6T5uNNdX6tyPHBMkp0zeFiSezDNbdT68bwkm7fpDT9luA/PxMnAk5Ps2QL43zLcH/+7o+bfqaqzgYu5/Ts37wOekmTf9s7LXTJ8wHOHdvkFwHNanxYz/XSkua75/a22x7Z550cDH6sqR6KlOWSIlu4YlgD/WVU/qKofTZwYRhafNzK3+YPAHsDnRqZ9APwzcBpwZpKfMXzDx+7THO89DG9N/5Bh9POrky5/KcPo9I+A97bj/hKgPdHvwzB3+KrW5k0MI2kzNoP9vAQ4uvXnCIYQM7HtLcAbgHPaW/WPanNePwxcBJzHEABX5QUMUykuYxiJ/wh9b/f3+FE7xlUMIerQqvpmu+yvGUZGvwd8meHF0gnT7OtzDKOZP0oycT+Y8vqagf/b2p/JEIbfDWwyg9vo+cCKNiXoUIZ5yKtUVd9qbf+VYST+KcBTqupXHTVP9hqGee4Tx7iC4d2Wv2d40XUF8Hfc9rz6WobR9xuA13H7Uex5rbnNoT+U4X5wDcMLoJf07kfS9FLV846WJM29JG8C7lVVS1bZWL8nw1erva+qdlhVW0nS3HAkWtIal+G7gR/W3trfDTgI+PhC1yVJ0kz5wUJJC2Ezhikc92Z4u/mtwKkLWpEkSR2cziFJkiR1cjqHJEmS1MkQLUmSJHVaJ+dEb7311rVo0aKFLkOSJEnrsfPOO+/aqtpm3GXrZIhetGgRy5YtW+gyJEmStB5rv+Q7ltM5JEmSpE6GaEmSJKmTIVqSJEnqZIiWJEmSOhmiJUmSpE6GaEmSJKmTIVqSJEnqZIiWJEmSOhmiJUmSpE6GaEmSJKmTIVqSJEnqZIiWJEmSOhmiJUmSpE4bLnQBkiRJ65JFh31qoUtYpRXHPnmhS1jvORItSZIkdXIkWpLuwBxRk6TV40i0JEmS1MkQLUmSJHUyREuSJEmdDNGSJElSJ0O0JEmS1MkQLUmSJHXyK+60XlgXvqYLZv5VXetCf/zaMUnSHZkj0ZIkSVKnORmJTrIf8M/ABsDxVXXspMs3Bt4DPBK4Dnh2Va1ol70aOAi4FXhZVZ0xFzVJ0nzwXQJp9fjY0fpm1iPRSTYA/g14IrAL8Nwku0xqdhBwQ1X9IfA24E1t212A5wAPBvYD3tH2J0mSJK215mI6x27Ad6rqe1X1K+BDwP6T2uwPnNTOfwTYM0na+g9V1S+r6vvAd9r+JEmSpLVWqmp2O0ieAexXVQe35ecDu1fVS0faXNLaXNmWvwvsDhwFfLWq3tfWvxv4r6r6yJjjHAIcAnCf+9znkZdffvms6l4d68JbUeCH1yTdMa0L/9PA/9Fa+3hfm1qS86pq8bjL5mJOdMasm5zMp2ozk22HlVXHAccBLF68eHbJX5K03jHQSVqT5mI6x5XAjiPLOwBXTdUmyYbA5sD1M9xWkiRJWqvMRYg+F9g5yR8kuTPDBwVPm9TmNGBJO/8M4HM1zCM5DXhOko2T/AGwM/C1OahJkiRJmjezns5RVb9J8lLgDIavuDuhqi5NcjSwrKpOA94NvDfJdxhGoJ/Ttr00ycnAZcBvgP9dVbfOtiZJkiRpPs3J90RX1aeBT09ad8TI+V8Az5xi2zcAb5iLOiRJkqQ1wV8slCRJkjoZoiVJkqROhmhJkiSp05zMiZYkSXPL772W1m6G6Dsw/0FLkiStHqdzSJIkSZ0M0ZIkSVInQ7QkSZLUyRAtSZIkdTJES5IkSZ0M0ZIkSVInQ7QkSZLUyRAtSZIkdTJES5IkSZ0M0ZIkSVInQ7QkSZLUyRAtSZIkdTJES5IkSZ0M0ZIkSVInQ7QkSZLUyRAtSZIkdTJES5IkSZ0M0ZIkSVInQ7QkSZLUyRAtSZIkdTJES5IkSZ0M0ZIkSVInQ7QkSZLUyRAtSZIkdTJES5IkSZ0M0ZIkSVInQ7QkSZLUyRAtSZIkdTJES5IkSZ0M0ZIkSVKnDWezcZKtgA8Di4AVwLOq6oYx7ZYAr2mLr6+qk5LcFTgF2Am4FTi9qg6bTT2SJEnqs+LYJy90Ceuk2Y5EHwYsraqdgaVt+XZa0D4S2B3YDTgyyZbt4rdU1QOBRwB/muSJs6xHkiRJmnezDdH7Aye18ycBB4xpsy9wVlVd30apzwL2q6pbqurzAFX1K+B8YIdZ1iNJkiTNu9mG6G2r6mqA9veeY9psD1wxsnxlW/c7SbYAnsIwmj1WkkOSLEuybOXKlbMsW5IkSVp9q5wTneSzwL3GXHT4DI+RMetqZP8bAh8E/qWqvjfVTqrqOOA4gMWLF9dU7SRJkqT5tsoQXVV7TXVZkh8n2a6qrk6yHXDNmGZXAnuMLO8AnD2yfBywvKr+aUYVS5IkSQtsttM5TgOWtPNLgFPHtDkD2CfJlu0Dhfu0dSR5PbA58PJZ1iFJkiStMbMN0ccCeydZDuzdlkmyOMnxAFV1PXAMcG47HV1V1yfZgWFKyC7A+UkuSHLwLOuRJEmS5t2svie6qq4D9hyzfhlw8MjyCcAJk9pcyfj50pIkSdJazV8slCRJkjoZoiVJkqROhmhJkiSpkyFakiRJ6mSIliRJkjoZoiVJkqROhmhJkiSpkyFakiRJ6mSIliRJkjoZoiVJkqROhmhJkiSpkyFakiRJ6mSIliRJkjoZoiVJkqROhmhJkiSpkyFakiRJ6mSIliRJkjoZoiVJkqROhmhJkiSpkyFakiRJ6mSIliRJkjoZoiVJkqROhmhJkiSpkyFakiRJ6mSIliRJkjoZoiVJkqROhmhJkiSpkyFakiRJ6mSIliRJkjoZoiVJkqROhmhJkiSpkyFakiRJ6mSIliRJkjoZoiVJkqROhmhJkiSp06xDdJKtkpyVZHn7u+UU7Za0NsuTLBlz+WlJLpltPZIkSdJ8m4uR6MOApVW1M7C0Ld9Okq2AI4Hdgd2AI0fDdpKnATfNQS2SJEnSvJuLEL0/cFI7fxJwwJg2+wJnVdX1VXUDcBawH0CSTYFXAK+fg1okSZKkeTcXIXrbqroaoP2955g22wNXjCxf2dYBHAO8FbhluoMkOSTJsiTLVq5cOfuqJUmSpNW04UwaJfkscK8xFx0+w+NkzLpKsivwh1X1/ydZNN0Oquo44DiAxYsX1wyPK0mSJM25GYXoqtprqsuS/DjJdlV1dZLtgGvGNLsS2GNkeQfgbODRwCOTrGi13DPJ2VW1B5IkSdJaai6mc5wGTHzbxhLg1DFtzgD2SbJl+0DhPsAZVfXvVXXvqloEPAb4tgFakiRJa7u5CNHHAnsnWQ7s3ZZJsjjJ8QBVdT3D3Odz2+notk6SJEla58xoOsd0quo6YM8x65cBB48snwCcMM1+VgAPmW09kiRJ0nzzFwslSZKkToZoSZIkqZMhWpIkSepkiJYkSZI6GaIlSZKkToZoSZIkqZMhWpIkSepkiJYkSZI6GaIlSZKkToZoSZIkqZMhWpIkSepkiJYkSZI6GaIlSZKkToZoSZIkqZMhWpIkSepkiJYkSZI6GaIlSZKkToZoSZIkqZMhWpIkSepkiJYkSZI6GaIlSZKkToZoSZIkqZMhWpIkSepkiJYkSZI6GaIlSZKkToZoSZIkqZMhWpIkSepkiJYkSZI6GaIlSZKkToZoSZIkqZMhWpIkSepkiJYkSZI6GaIlSZKkToZoSZIkqZMhWpIkSeo0qxCdZKskZyVZ3v5uOUW7Ja3N8iRLRtbfOclxSb6d5JtJnj6beiRJkqQ1YbYj0YcBS6tqZ2BpW76dJFsBRwK7A7sBR46E7cOBa6rq/sAuwBdmWY8kSZI072YbovcHTmrnTwIOGNNmX+Csqrq+qm4AzgL2a5e9EPgHgKr6bVVdO8t6JEmSpHk32xC9bVVdDdD+3nNMm+2BK0aWrwS2T7JFWz4myflJTkmy7VQHSnJIkmVJlq1cuXKWZUuSJEmrb5UhOslnk1wy5rT/DI+RMesK2BDYATinqv4I+Arwlql2UlXHVdXiqlq8zTbbzPDQkiRJ0tzbcFUNqmqvqS5L8uMk21XV1Um2A64Z0+xKYI+R5R2As4HrgFuAj7f1pwAHzaxsSZIkaeHMdjrHacDEt20sAU4d0+YMYJ8kW7YPFO4DnFFVBZzObQF7T+CyWdYjSZIkzbvZhuhjgb2TLAf2bsskWZzkeICquh44Bji3nY5u6wBeBRyV5CLg+cDfzrIeSZIkad6tcjrHdKrqOoYR5MnrlwEHjyyfAJwwpt3lwONmU4MkSZK0pvmLhZIkSVInQ7QkSZLUyRAtSZIkdTJES5IkSZ0M0ZIkSVInQ7QkSZLUyRAtSZIkdTJES5IkSZ0M0ZIkSVInQ7QkSZLUyRAtSZIkdTJES5IkSZ0M0ZIkSVInQ7QkSZLUyRAtSZIkdTJES5IkSZ0M0ZIkSVInQ7QkSZLUyRAtSZIkdTJES5IkSZ0M0ZIkSVInQ7QkSZLUyRAtSZIkdTJES5IkSZ0M0ZIkSVInQ7QkSZLUyRAtSZIkdTJES5IkSZ0M0ZIkSVInQ7QkSZLUyRAtSZIkdTJES5IkSZ0M0ZIkSVInQ7QkSZLUyRAtSZIkdZp1iE6yVZKzkixvf7ecot2S1mZ5kiUj65+b5OIkFyX5TJKtZ1uTJEmSNJ/mYiT6MGBpVe0MLG3Lt5NkK+BIYHdgN+DIJFsm2RD4Z+DPquphwEXAS+egJkmSJGnezEWI3h84qZ0/CThgTJt9gbOq6vqqugE4C9gPSDvdLUmAuwNXzUFNkiRJ0ryZixC9bVVdDdD+3nNMm+2BK0aWrwS2r6pfAy8GLmYIz7sA7x53kCSHJFmWZNnKlSvnoGxJkiRp9cwoRCf5bJJLxpz2n+FxMmZdJdmIIUQ/Arg3w3SOV4/bQVUdV1WLq2rxNttsM8PDSpIkSXNvw5k0qqq9prosyY+TbFdVVyfZDrhmTLMrgT1GlncAzgZ2bfv/btvXyYyZUy1JkiStTeZiOsdpwMS3bSwBTh3T5gxgn/Zhwi2Bfdq6HwK7JJkYWt4b+MYc1CRJkiTNmxmNRK/CscDJSQ4CfgA8EyDJYuDQqjq4qq5Pcgxwbtvm6Kq6vrV7HfDFJL8GLgcOnIOaJEmSpHkz6xBdVdcBe45Zvww4eGT5BOCEMe3eCbxztnVIkiRJa4q/WChJkiR1MkRLkiRJnQzRkiRJUidDtCRJktTJEC1JkiR1MkRLkiRJnQzRkiRJUidDtCRJktTJEC1JkiR1MkRLkiRJnQzRkiRJUidDtCRJktTJEC1JkiR1MkRLkiRJnQzRkiRJUidDtCRJktTJEC1JkiR1MkRLkiRJnQzRkiRJUidDtCRJktTJEC1JkiR1MkRLkiRJnQzRkiRJUidDtCRJktTJEC1JkiR1MkRLkiRJnQzRkiRJUidDtCRJktTJEC1JkiR1MkRLkiRJnQzRkiRJUidDtCRJktTJEC1JkiR1MkRLkiRJnQzRkiRJUqdZhegkWyU5K8ny9nfLKdp9JsmNST45af0fJPmftv2Hk9x5NvVIkiRJa8JsR6IPA5ZW1c7A0rY8zpuB549Z/ybgbW37G4CDZlmPJEmSNO9mG6L3B05q508CDhjXqKqWAj8bXZckwBOAj6xqe0mSJGltMtsQvW1VXQ3Q/t6zY9t7ADdW1W/a8pXA9lM1TnJIkmVJlq1cuXK1C5YkSZJma8NVNUjyWeBeYy46fJbHzph1NVXjqjoOOA5g8eLFU7aTJEmS5tsqQ3RV7TXVZUl+nGS7qro6yXbANR3HvhbYIsmGbTR6B+Cqju0lSZKkBTHb6RynAUva+SXAqTPdsKoK+DzwjNXZXpIkSVoosw3RxwJ7J1kO7N2WSbI4yfETjZJ8CTgF2DPJlUn2bRe9CnhFku8wzJF+9yzrkSRJkubdKqdzTKeqrgP2HLN+GXDwyPJjp9j+e8Bus6lBkiRJWtP8xUJJkiSpkyFakiRJ6mSIliRJkjoZoiVJkqROhmhJkiSpkyFakiRJ6mSIliRJkjoZoiVJkqROhmhJkiSpkyFakiRJ6mSIliRJkjoZoiVJkqROhmhJkiSpkyFakiRJ6mSIliRJkjoZoiVJkqROhmhJkiSpkyFakiRJ6mSIliRJkjoZoiVJkqROhmhJkiSpkyFakiRJ6mSIliRJkjoZoiVJkqROhmhJkiSpkyFakiRJ6mSIliRJkjoZoiVJkqROhmhJkiSpkyFakiRJ6mSIliRJkjoZoiVJkqROhmhJkiSpkyFakiRJ6mSIliRJkjrNKkQn2SrJWUmWt79bTtHuM0luTPLJSevfn+RbSS5JckKSjWZTjyRJkrQmzHYk+jBgaVXtDCxty+O8GXj+mPXvBx4IPBTYBDh4lvVIkiRJ8262IXp/4KR2/iTggHGNqmop8LMx6z9dDfA1YIdZ1iNJkiTNu9mG6G2r6mqA9veeq7OTNo3j+cBnpmlzSJJlSZatXLlytYqVJEmS5sKGq2qQ5LPAvcZcdPgc1vEO4ItV9aWpGlTVccBxAIsXL645PLYkSZLUZZUhuqr2muqyJD9Osl1VXZ1kO+Ca3gKSHAlsA/xV77aSJEnSQpjtdI7TgCXt/BLg1J6NkxwM7As8t6p+O8taJEmSpDVitiH6WGDvJMuBvdsySRYnOX6iUZIvAacAeya5Msm+7aJ3AtsCX0lyQZIjZlmPJEmSNO9WOZ1jOlV1HbDnmPXLGPm6uqp67BTbz+r4kiRJ0kLwFwslSZKkToZoSZIkqZMhWpIkSepkiJYkSZI6GaIlSZKkToZoSZIkqZMhWpIkSepkiJYkSZI6GaIlSZKkToZoSZIkqZMhWpIkSepkiJYkSZI6bbjQBaxLVhz75IUuQZIkSWsBR6IlSZKkToZoSZIkqZMhWpIkSepkiJYkSZI6GaIlSZKkToZoSZIkqZMhWpIkSepkiJYkSZI6GaIlSZKkToZoSZIkqZMhWpIkSepkiJYkSZI6GaIlSZKkToZoSZIkqVOqaqFr6JZkJXD5QtcxR7YGrl3oIubI+tQXsD9rs/WpL7B+9Wd96gvYn7XZ+tQXWL/6sz715b5Vtc24C9bJEL0+SbKsqhYvdB1zYX3qC9iftdn61BdYv/qzPvUF7M/abH3qC6xf/Vmf+jIdp3NIkiRJnQzRkiRJUidD9MI7bqELmEPrU1/A/qzN1qe+wPrVn/WpL2B/1mbrU19g/erP+tSXKTknWpIkSerkSLQkSZLUyRCtLkmOT7JLO3/TQtezkJIsSnLJQtcBkORlSb6R5P0LXctMzHe9SY5K8sr52Pd8S7JHkk8udB2SpOkZotWlqg6uqssWuo41KcmGC13DDLwEeFJVPW9VDdeS/sy4Xq09khyY5O2z3MeKJFvPsO29knwoyXeTXJbk00nuP5vjj+x7TfdlhySnJlme5HtJ3p5k49kcf9L+13R/Dk9yaZKLklyQZPdZHnveX/gmOSHJNWti8GO++5NkxySfb4MRlyb5m/k6VjvefPfnLkm+luTC1p/Xzdex5pIhep4l+USS89qd4pC27qAk305ydpL/mPjHl2SbJB9Ncm47/ekC1363JJ9qd+pLkjy71bx4pM1bk5yfZGmSbdq6l7UnvIuSfKitOyrJe5N8rj2JvGgN92VRkm8mOanV9ZEkd01yRLuuL0lyXJK09mcneWOSLwB/k2TbJB9v18WFSf6k7XqDdhtemuTMJJusyX61Wt8J3A84Lcmrkvx3kq+3vw9obQ5MckqS04Ez27q/a32/aE3+w5pU7+Htie3cVvP+I/V+IsnpSb6f5KVJXtHafDXJVq3di9q2F7bHzl3HHG+nJJ9pj8MvJXngGujjxP3t+Hbfen+SvZKc0+7/u7XT791Wk/Zzt3HXz7oqyQYdbQN8HDi7qnaqql2Avwe2na/6eqxGXz4GfKKqdgZ2BjYB/nGeyuvW2Z9HA/8L+KOqehiwF3DFfNU2h04E9lvoIubIb4C/raoHAY8C/nfau8TrqF8CT6iqhwO7AvsledQC17RqVeVpHk/AVu3vJsAlwPbACmArYCPgS8DbW5sPAI9p5+8DfGOBa3868B8jy5sDZwOL23IBz2vnjxjpx1XAxu38Fu3vUcCF7XrYmuEf7r3XYF8WtXr/tC2fALxy4vZp694LPKWdPxt4x8hlHwZe3s5v0K6LRQz/yHZt608G/nKBbqsV7Xq9O7BhW7cX8NF2/kDgypH74z4Mn54Ow4vpTwKPW4B63zhxnQFbAN8G7tbq/Q6wGbAN8BPg0NbubSO3xT1G9vl64K9H7m+vbOeXAju380VFupAAAAoBSURBVLsDn1tD97ffAA9t1+957T4XYH/gE9PcVnsAn2znx14/c1DfXwJfAy4A3tXu0zcBb2q1fhbYrT0Ovgf8+cj96FTgM8C3gCNH9vmJtu2lwCEj628Cjgb+B3jMyG2/SdvPi6ao8QnAF9eTvuw5uS/t9r8B2HQd7M/TgNNneR98AXARw/PCe7n9Y/ZFwLntso8Cd23rn8nwPHrhxPUJPHjk+rqI9lhfxWPzknl4zC9If0aOfyqw9/rQH+CuwPnA7nN9O8357b7QBazvJ24LjxcyBIHDgJNGLn8Zt4XPa9odbeL0Q2CzBaz9/sD3Gf55P7atO5vbQvSt3BYC7gdc0M5/BvgIw5PBpiPXw9Ej+34PcMAa7Msi4Acjy09geGJ5OsMTyMXt+j5spJ+PH2m/kvbCYNI+l48svwp4zQLdVisYnvx2ZBi9u6T16Zvt8gOB/xxp/5a2zcR97TvAQQtQ77JW60QdPwAe1OodfQH3A2D7dv6FwD+1849neCF6cbuvvnPk/vZKYFPg55MeV/P+4nTMfeM93PaC836tjqluqz24LUSPvX5mWduDgNOBjdryOxieMAt4Ylv3cYZ3LDYCHs5tj+0DgauBe3DbwMDE/4PJAwb3aMsFPGvSbb+IIQy+YJo6Xwa8bX3uC/B12ovwdaw/mzLcH7/danx8533wwQxBf+uJ+rh9SJvqxfHF3PZ/YGKA5l+57bF1Z2CTGTw25zREL2R/Rvr0A+Du63J/GF4wXkB70TiXt9F8ndaGuZHrrSR7MIwwPbqqbklyNsMd80FTbHKn1vbna6bC6VXVt5M8EngS8A9JzlzVJu3vk4HHAX8OvDbJgyddPrn9mjLu+O9geKK5IslRwF1GLr95Bvv85cj5WxmepBbSMcDnq+qpSRYxvBiYMNqfAP9QVe9ac6WNFeDpVfWt260c5leOXre/HVn+Lfzuf9eJDC/GLkxyIEMAHXUn4Maq2nVuy56RVdU/3W01Yez1M0t7Ao8Ezm2zlzZheAH/K4YXwDA8Gf6yqn6d5GKGJ+kJZ1XVdQBJPsYwgrkMeFmSp7Y2OzJMWbiO4XHx0Uk1nAr8Y1XN9oOl60pfwvj/d1kX+1NVN7XnhscCfwZ8OMlhVXXiVNtM8gTgI1V1bdvf9a2/Ex6S5PUM775sCpzR1p8DnJjkZIbpMQBfAQ5PsgPwsapaPsMa5tKC9SfJpgy34cur6qfrcn+q6lZg1yRbAB9P8pCqWis+vD8V50TPr82BG1qAfiDDvKW7Ao9PsmWGD3g9faT9mcBLJxaSLMQT/+8kuTdwS1W9j2Hk8o8mNbkT8Ix2/i+ALye5E7BjVX0e+D/c9iAD2D/DhwfuwRB2zp3nLkx2nzaXD+C5wJfb+WvbP6JnjN8MGKYEvBiGuYNJ7j5/Zc7K5gwj6jCMTE3lDOCFrd8k2T7JPee5tqnq+Os2Z5Qkj+jcfjPg6iQbAb/3IcX2pPL9JM9s+0+Sh8+y5rkyk9tqttfPOGF4N2zXdnpAVR0F/LracBAjob+qRl+0wJgXo5MGDB7OMMI68YL0F+3JcdQ5wBMn+jWFSxkC5frSl8WjK9r/kG0ZBlbWtf5QVbdW1dlVdSTD89bTp2s/yVQvKiacCLy0qh4KvG6i3qo6FHgNwwuBC5Lco6o+wDBg83PgjCRP6KhjrixIf9r/vY8C76+qj03VbjUs6O1TVTcyDCqs9fPXDdHz6zPAhkkuYhh1+irDk+YbGaYQfBa4jGGaBwxv+S3O8EGvy4BD13zJt/NQ4GtJLgAOZ3jbZtTNwIOTnMfwyvVohrdj3tdGSL7O8Bbmja3914BPMVwPx1TVVWugD6O+ASxpt8dWwL8D/8EwsvMJpg/1fwP8WevXeQxvd62N/pHhXYNzGG6LsarqTIY5+F9pffoIQyBd045heFv6ogyfmD+mc/vXMjyWzgK+OUWb5wEHJbmQIcysLR/Om8ltNdvrZ5ylwDMmXjQl2SrJfTu237ttswlwAEPoGjdgMJ0jGEZC3zFNm88BG2fkQ8hJ/jjJ49fBviwF7prkBa3ODYC3Mkzl+/mkdmt9f5I8IMnOI6t2BS7vqHMp8Kw2oELaB4VHjH1xnGSnqvqfqjoCuBbYMcn9gO9V1b8ApwEP66hjrqzx/rQXOe9mmJ72f9eD/mzTRqBp99+9mPp/+tpjoeaR3JFP3DZPeEOG+W9PXeia1kCfj6LNp1qg4y9iHj5M4snTungCns1tH/Q5jyFY3TRy+e0erxOXMYyYn8zwYvh3H14DNgb+q+3vFIZRpD1Gtx3Z1wqG+fAB/pNh6sBUdd67He+7DC+APsWkDyatQ33ZkSFELAduBN61rt42DO8Q/DfDINBFDG/db915H1zCbR9CO5Hbz7l9McNnHM5mmFN7Ylv/MYZBj0uAf251vrrdNy5gGLjaappjfpBh3vivGT5oPWefA1nT/WGYqlPt+p/4zMST1uH+PIxh4O2itv0Rc9WX+Tz5s98LIMlbGF5l3YVhCsff1Hp+Q2SYb3xTVb1lgY6/iOHDWg9ZiONL0oQMX5H5QeBpVXXeQtcjafUYoiVJkqROfjuHJGnBtHmXS8dctGe1b5pYV6xPfYF1sz/rYs3TsT9rN0eiJUmSpE5+O4ckSZLUyRAtSZIkdTJES9JaKMmtSS4YOS1ajX1skeQlc1+dJMk50ZK0FkpyU1VtuuqW0+5jEavx1Y5JNqjf/yU7SdIIR6IlaR3RfnL+zUnObb9s+ldt/aZJliY5P8nFSSZ+lfFYYKc2kv3mJHsk+eTI/t6e5MB2fkWSI5J8GXhmkp2SfCbJeUm+1H7tjiTPTHJJkguTfHHNXgOStPbwK+4kae20SZIL2vnvV9VTgYOAn1TVHyfZGDgnyZnAFQy/fPrTJFsDX01yGnAY8JCq2hUgyR6rOOYvquoxre1S4NCqWp5kd4afgX4Cw89C71tVP5z4mV5JuiMyREvS2unnE+F3xD7Aw5I8oy1vDuzM8BPGb0zyOOC3wPbAtqtxzA/DMLIN/AlwSpKJyzZuf88BTkxyMsPP/ErSHZIhWpLWHQH+uqrOuN3KYUrGNsAjq+rXSVYAdxmz/W+4/TS+yW1ubn/vBNw4JsRTVYe2keknAxck2XVd/JEESZot50RL0rrjDODFSTYCSHL/JHdjGJG+pgXoPwPu29r/DNhsZPvLgV2SbJxkc2DPcQepqp8C30/yzHacJHl4O79TVf1PVR0BXAvsOPfdlKS1nyPRkrTuOB5YBJyfYZ7FSuAA4P3A6UmWARcA3wSoquuSnJPkEuC/qurv2jSMi4DlwNenOdbzgH9P8hpgI+BDwIXAm5PszDAqvrStk6Q7HL/iTpIkSerkdA5JkiSpkyFakiRJ6mSIliRJkjoZoiVJkqROhmhJkiSpkyFakiRJ6mSIliRJkjoZoiVJkqRO/w+14Opc/9oYTQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 864x432 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "visualize_importances(feature_names, neuron_cond_vals_0.mean(dim=0).detach().numpy(), title=\"Average Feature Importances for Neuron 0\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From the data above, it appears that the primary input feature used by neuron 0 is age, with limited importance for all other features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average Feature Importances for Neuron 10\n",
      "age :  -0.012\n",
      "sibsp :  0.003\n",
      "parch :  0.000\n",
      "fare :  -0.015\n",
      "female :  0.055\n",
      "male :  -0.210\n",
      "embark_C :  0.016\n",
      "embark_Q :  -0.006\n",
      "embark_S :  -0.052\n",
      "class_1 :  0.022\n",
      "class_2 :  0.009\n",
      "class_3 :  -0.116\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAtEAAAGECAYAAADwYnzeAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAAgAElEQVR4nO3debhkVX3u8e8rDYgyyNAiMtjqxQETJdrBJNeBMDgmQSMOCdHmBiXGqyaPMRGjIo7BmMTcxOtVnMAZHMEhCrZilDjQaDM6gIiCoMyOSER/94+9jhTHM62uM/Q5fD/PU8/Zw9p7r7Wr6tRbq9auSlUhSZIkae5us9QVkCRJkpYbQ7QkSZLUyRAtSZIkdTJES5IkSZ0M0ZIkSVInQ7QkSZLUyRAtSVp2krw8ydVJvrfUdZF062SIlla4JKcnuS7J1ktdl/nQ2vOzJD8euf3umPtck6SSrJqves7hmMckecdiHW8mSQ5P8rmlrsdcJdkT+Btgn6q60zzts5Kcm+Q2I8tenuT4+dj/QkqyW5JTklze2rFm0vqtk7wlyQ+TfC/Jc5amptLKYoiWVrD2YvpgoIA/WqBjLFrwHPHMqtp25Pb5JajDr2SwLP+fLtH9N667ANdU1ZW9G87S3jsDT9rkWs1PHTbFL4GPA4+bZv0xwN4M5+33gb9L8oh5roN0q7Ms/+lLmrOnAF8AjgfWTSxM8jutR2qLkWWPTXJOm75NkqOSfDPJNUlOSrJTWzfRa3tEku8An2rL39v2+YMk/5nkPiP73jnJh1tP2Jmth+9zI+vvleS0JNcm+XqSJ2xKY2faT5JHJ/lKq8OlSY4Z2fQ/29/rJ3q2J/cUT+6tbj3ir0hyBvBT4G5Jdkjy5iRXJPlua+cWzEHb9zOSXJjkR0leluTuST7f6nxSkq1a2f2TXJbk79uQhkuSHDayrx2SvC3JVUm+neSFEyG/9TqfkeQ1Sa4FTgReD/xua/v1s52vkXOxLsl3Wh1eMLJ+i1a3b7a2nJWh93i2++hRSS5o23w3yXOnOE8HAacBd271Pb4t/6Mk5ye5vt039x7Z5pIkz2uP75/MEGL/EXjJdOvb8+a/2jHOTrL/pGMcNDL/q8fPDM+Z2er83CTnZHhOnZjktlPVq6q+X1WvA86cpl1PAV5WVddV1VeBNwKHT1NW0lxVlTdv3lboDbgIeAbwAODnwK4j674JHDwy/17gqDb91wzhew9ga+ANwLvbujUMPdtvA24PbNOW/zmwXSv/r8DGkX2/p91uB+wDXAp8rq27fZv/X8Aq4P7A1cB9pmnT6cBTp1g+436A/YHfZOg8uC/wfeAxk9q0amR/xwDvGJm/RZlWj+8A92nH2xL4UDtXtwfuCHwJ+Itp2jF5/wWcAmzf9nkjsB64G7ADcAGwbqQtNwH/0s73Q4GfAPds698GnNzujzXAN4Aj2rrD27bPavXepi373KT6zeV8vbFtf79W33u39X8LnAvcE0hbv/Mc7qMrgAe36R2B+09z7vYHLhuZv0dr/8Htfvg7hsf+Vm39JcBGYE/a43WKfRZDb+1ZtMcX8HLg+Da9O3AN8Kh2Tg5u86tHjnHQVPcvUzxn5ljnLzH0ju8EfBV4+izP91XtOGtGlu3Ylo0+9w8Fzl3q/0/evC33mz3R0gqV5EEMH9+eVFVnMYTmPx0p8m7gT1rZ7RjCwbvbur8AXlBVl1XVjQyB4NBJPXTHVNVPquoGgKp6S1X9aKT8/VqP6BYMHzO/uKp+WlUXACeM7OcPgEuq6q1VdVNVfRl4P8ML/XT+rfXeXZ/ky3PZT1WdXlXnVtUvq+qc1taHzulkTu/4qjq/qm5iCDqPBP66nZcrgdfQNzzgVVX1w6o6HzgPOLWqLq6qHwD/AfzWpPIvqqobq+ozwEeBJ7Tz/UTg+e3+uAT4Z+DJI9tdXlX/3s7TDVNVZI7n6yVVdUNVnQ2czRCWAZ4KvLCqvl6Ds6vqGma/r38O7JNk+xp6Tb/M3DwR+GhVnVZVPwf+iSGo/t5ImX+rqkuna+9Es4EXAUfn168h+DPgY1X1sXZOTgM2MDxv5mr0OTPXOl9eVdcCHwb27TjWhG3b3x+MLPsBwxssSWMwREsr1zqGEHZ1m38XI0M62vwft7Dwx8CXq+rbbd1dgA9OBFWGXrBfALuObH/pxET7+P7Y9vH9Dxl60QB2AVYz9JBdOtW27VgPHAnF1wOHATNdMPbsqrpDu91/LvtJ8sAkn25DHH4APL3VbxyT27ElcMXI8d/A0CM9V98fmb5hivltR+avq6qfjMx/m6HXchdgqzY/um73aeo9pTmer9FvxvjpSP32ZHjTNtls9/XjGELpt5N8JnO/YPTOjLS3qn7J0MauNrdtP8bwCcORU9T98ZPq/iBgtznWcXId5lLn6c5vjx+3v9uPLNse+NEm7EvSiOV4QYmkWSTZBngCsEVu/gqwrYE7JLlf6xm8IMm3GXpP/5QhVE+4FPjzqjpjin2vaZM1svhPgUOAgxgC9A7AdQwf5V/FMHxgD4ZhBTCErNFjfaaqDt6kxs59P+8CXgs8sqp+luRfuTkU1hTlf8Iw/GTCVKF+dLtLGYY07NJ6phfajkluPxKk92Lovb6aoUf3LgxDQCbWfXdk28ntnar9M52v2VwK3L3VZ/Lyae+jqjoTOCTJlsAzgZO45WNlOpczDD0Bhgs923YztXkmL2QYfjT5OfH2qnraNNv0Pl7mUuexVdV1Sa5g+JTgtLb4fsD583kc6dbInmhpZXoMQ8/xPgwfAe8L3Bv4LMNFRhPeBTwbeAjDmOgJrwdekeQuAElWJzlkhuNtxxAgr2EIEq+cWFFVvwA+AByT5HZJ7jWpDh8B7pHkyUm2bLffHr3Iao5m2892wLUtEO7HLYe2XMXwDQd3G1m2EXhIkr2S7AA8f6aDV9UVwKnAPyfZPsPFmXdPMu6QkZm8JMlWSR7MMFTive18n8Rw/23X7sPnADN9nd73gT3SLlxsZjpfs3kT8LIke2dw3yQ7M8N91NpxWJId2vCGHzI8hufiJODRSQ5sAfxvGB6P/9VR51+pqtMZxnSPfnLzDuAPkzy8ffJy2wwXeO7R1m8EntTatJaZhyPNe53bRYcTQ1C2nnQR4tuAFybZsT3/nsZwsbGkMRiipZVpHfDWqvpOVX1v4sbQs3jYyNjmdzNcpPWpkWEfAP+H4SK3U5P8iOEiwwfOcLy3MXw0/V2G3s8vTFr/TIbe6e8Bb2/HvRGgqn4EPIxh7PDlrcyruDkQzMkc9vMM4KWtPUczhJiJbX8KvAI4o31U/zttzOuJwDkMF5t9ZA7VeArDUIoLGHri30ffx/09vteOcTnwToaLzr7W1j2LoWf0YuBzDG+W3jLDvj7F0DP5vSQTj4Npz9cc/EsrfypDGH4zwwV9s91HTwYuaUOCns4wDnlWVfX1VvbfGXri/xD4w6r67446T/ZChnHuE8e4lOHTlr9neNN1KcMFlBOvoy9i6H2/DngJt+zFXow638DNQze+1uYnvJhheM23gc8Ar66qj2/icSQ1qer5hEuSxpfkVcCdqmrdrIX1azJ8tdo7qmqP2cpKkhaGPdGSFlyG7wa+b/tofz/gCOCDS10vSZI2lRcWSloM2zEM4bgzcCXDV66dvKQ1kiRpDA7nkCRJkjo5nEOSJEnqZIiWJEmSOi3LMdG77LJLrVmzZqmrIUmSpBXsrLPOurqqVk+1blmG6DVr1rBhw4alroYkSZJWsPbLvlNyOIckSZLUyRAtSZIkdTJES5IkSZ0M0ZIkSVInQ7QkSZLUyRAtSZIkdTJES5IkSZ0M0ZIkSVInQ7QkSZLUyRAtSZIkdTJES5IkSZ0M0ZIkSVInQ7QkSZLUadVSV0DSyrfmqI8udRVmdcmxj17qKkiSlhF7oiVJkqRO8xKikzwiydeTXJTkqCnWb53kxLb+i0nWtOVrktyQZGO7vX4+6iNJkiQtpLGHcyTZAvi/wMHAZcCZSU6pqgtGih0BXFdV/yPJk4BXAU9s675ZVfuOWw9JkiRpscxHT/R+wEVVdXFV/TfwHuCQSWUOAU5o0+8DDkySeTi2JEmStOjmI0TvDlw6Mn9ZWzZlmaq6CfgBsHNbd9ckX0nymSQPnu4gSY5MsiHJhquuumoeqi1JkiRtmvkI0VP1KNccy1wB7FVVvwU8B3hXku2nOkhVHVdVa6tq7erVq8eqsCRJkjSO+QjRlwF7jszvAVw+XZkkq4AdgGur6saqugagqs4CvgncYx7qJEmSJC2Y+QjRZwJ7J7lrkq2AJwGnTCpzCrCuTR8KfKqqKsnqdmEiSe4G7A1cPA91kiRJkhbM2N/OUVU3JXkm8AlgC+AtVXV+kpcCG6rqFODNwNuTXARcyxC0AR4CvDTJTcAvgKdX1bXj1kmSJElaSPPyi4VV9THgY5OWHT0y/TPg8VNs937g/fNRB0mSJGmx+IuFkiRJUidDtCRJktTJEC1JkiR1MkRLkiRJnQzRkiRJUidDtCRJktTJEC1JkiR1MkRLkiRJnQzRkiRJUidDtCRJktTJEC1JkiR1MkRLkiRJnQzRkiRJUidDtCRJktTJEC1JkiR1MkRLkiRJnQzRkiRJUidDtCRJktTJEC1JkiR1MkRLkiRJnQzRkiRJUidDtCRJktTJEC1JkiR1MkRLkiRJnQzRkiRJUqdVS10BSZK08q056qNLXYVZXXLso5e6ClpG7ImWJEmSOhmiJUmSpE6GaEmSJKmTIVqSJEnqZIiWJEmSOhmiJUmSpE5+xZ0k3Yr5tWOStGnsiZYkSZI6GaIlSZKkToZoSZIkqZMhWpIkSepkiJYkSZI6GaIlSZKkToZoSZIkqZPfEy1JktTB71cX2BMtSZIkdTNES5IkSZ0M0ZIkSVInQ7QkSZLUyRAtSZIkdfLbOW7FvLpYkiRp08xLT3SSRyT5epKLkhw1xfqtk5zY1n8xyZqRdc9vy7+e5OHzUR9JkiRpIY0dopNsAfxf4JHAPsCfJNlnUrEjgOuq6n8ArwFe1bbdB3gScB/gEcDr2v4kSZKkzdZ89ETvB1xUVRdX1X8D7wEOmVTmEOCENv0+4MAkacvfU1U3VtW3gIva/iRJkqTN1nyMid4duHRk/jLggdOVqaqbkvwA2Lkt/8KkbXef6iBJjgSOBNhrr73modr9lsMYYpj7OGLHG2++lsNjrefxs5Iea943m6/lcN/A3M/5cmjPrfWxtpLaAivvsbZY5qMnOlMsqzmWmcu2w8Kq46pqbVWtXb16dWcVJUmSpPkzHyH6MmDPkfk9gMunK5NkFbADcO0ct5UkSZI2K/MRos8E9k5y1yRbMVwoeMqkMqcA69r0ocCnqqra8ie1b++4K7A38KV5qJMkSZK0YMYeE93GOD8T+ASwBfCWqjo/yUuBDVV1CvBm4O1JLmLogX5S2/b8JCcBFwA3Af+7qn4xbp0kSZKkhTQvP7ZSVR8DPjZp2dEj0z8DHj/Ntq8AXjEf9ZAkSZIWgz/7LUmSJHXyZ78lSSvC5vgVWJJWLnuiJUmSpE6GaEmSJKmTIVqSJEnqZIiWJEmSOhmiJUmSpE6GaEmSJKmTX3HXwa9PkiRJEtgTLUmSJHUzREuSJEmdHM4hbYYcOiRJ0ubNnmhJkiSpkyFakiRJ6mSIliRJkjoZoiVJkqROhmhJkiSpkyFakiRJ6mSIliRJkjoZoiVJkqROhmhJkiSpkyFakiRJ6mSIliRJkjoZoiVJkqROhmhJkiSpkyFakiRJ6mSIliRJkjoZoiVJkqROhmhJkiSpkyFakiRJ6mSIliRJkjoZoiVJkqROhmhJkiSpkyFakiRJ6mSIliRJkjoZoiVJkqROhmhJkiSpkyFakiRJ6mSIliRJkjoZoiVJkqROhmhJkiSpkyFakiRJ6mSIliRJkjoZoiVJkqROhmhJkiSpkyFakiRJ6mSIliRJkjqNFaKT7JTktCQXtr87TlNuXStzYZJ1I8tPT/L1JBvb7Y7j1EeSJElaDOP2RB8FrK+qvYH1bf4WkuwEvBh4ILAf8OJJYfuwqtq33a4csz6SJEnSghs3RB8CnNCmTwAeM0WZhwOnVdW1VXUdcBrwiDGPK0mSJC2ZcUP0rlV1BUD7O9VwjN2BS0fmL2vLJry1DeV4UZJMd6AkRybZkGTDVVddNWa1JUmSpE23arYCST4J3GmKVS+Y4zGmCsbV/h5WVd9Nsh3wfuDJwNum2klVHQccB7B27dqaqowkSZK0GGYN0VV10HTrknw/yW5VdUWS3YCpxjRfBuw/Mr8HcHrb93fb3x8leRfDmOkpQ7QkSbcmlxz76KWugqQZjDuc4xRg4ts21gEnT1HmE8DDkuzYLih8GPCJJKuS7AKQZEvgD4DzxqyPJEmStODGDdHHAgcnuRA4uM2TZG2SNwFU1bXAy4Az2+2lbdnWDGH6HGAj8F3gjWPWR5IkSVpwsw7nmElVXQMcOMXyDcBTR+bfArxlUpmfAA8Y5/iSJEnSUvAXCyVJkqROhmhJkiSpkyFakiRJ6mSIliRJkjoZoiVJkqROhmhJkiSpkyFakiRJ6mSIliRJkjoZoiVJkqROhmhJkiSpkyFakiRJ6mSIliRJkjoZoiVJkqROhmhJkiSpkyFakiRJ6mSIliRJkjoZoiVJkqROhmhJkiSpkyFakiRJ6mSIliRJkjoZoiVJkqROhmhJkiSpkyFakiRJ6mSIliRJkjoZoiVJkqROhmhJkiSpkyFakiRJ6mSIliRJkjoZoiVJkqROhmhJkiSpkyFakiRJ6mSIliRJkjoZoiVJkqROhmhJkiSpkyFakiRJ6mSIliRJkjoZoiVJkqROhmhJkiSpkyFakiRJ6mSIliRJkjoZoiVJkqROhmhJkiSpkyFakiRJ6mSIliRJkjoZoiVJkqROhmhJkiSpkyFakiRJ6jRWiE6yU5LTklzY/u44TbmPJ7k+yUcmLb9rki+27U9MstU49ZEkSZIWw7g90UcB66tqb2B9m5/Kq4EnT7H8VcBr2vbXAUeMWR9JkiRpwY0bog8BTmjTJwCPmapQVa0HfjS6LEmAA4D3zba9JEmStDkZN0TvWlVXALS/d+zYdmfg+qq6qc1fBuw+XeEkRybZkGTDVVddtckVliRJksa1arYCST4J3GmKVS8Y89iZYllNV7iqjgOOA1i7du205SRJkqSFNmuIrqqDpluX5PtJdquqK5LsBlzZceyrgTskWdV6o/cALu/YXpIkSVoS4w7nOAVY16bXASfPdcOqKuDTwKGbsr0kSZK0VMYN0ccCBye5EDi4zZNkbZI3TRRK8lngvcCBSS5L8vC26nnAc5JcxDBG+s1j1keSJElacLMO55hJVV0DHDjF8g3AU0fmHzzN9hcD+41TB0mSJGmx+YuFkiRJUidDtCRJktTJEC1JkiR1MkRLkiRJnQzRkiRJUidDtCRJktTJEC1JkiR1MkRLkiRJnQzRkiRJUidDtCRJktTJEC1JkiR1MkRLkiRJnVYtdQUkSZK0dC459tFLXYVlyZ5oSZIkqZMhWpIkSepkiJYkSZI6GaIlSZKkToZoSZIkqZMhWpIkSepkiJYkSZI6GaIlSZKkToZoSZIkqZMhWpIkSepkiJYkSZI6GaIlSZKkToZoSZIkqZMhWpIkSepkiJYkSZI6GaIlSZKkToZoSZIkqZMhWpIkSepkiJYkSZI6GaIlSZKkToZoSZIkqZMhWpIkSepkiJYkSZI6GaIlSZKkToZoSZIkqZMhWpIkSepkiJYkSZI6GaIlSZKkToZoSZIkqZMhWpIkSepkiJYkSZI6GaIlSZKkToZoSZIkqZMhWpIkSeo0VohOslOS05Jc2P7uOE25jye5PslHJi0/Psm3kmxst33HqY8kSZK0GMbtiT4KWF9VewPr2/xUXg08eZp1f1tV+7bbxjHrI0mSJC24cUP0IcAJbfoE4DFTFaqq9cCPxjyWJEmStFkYN0TvWlVXALS/d9yEfbwiyTlJXpNk6+kKJTkyyYYkG6666qpNra8kSZI0tllDdJJPJjlvitsh83D85wP3An4b2Al43nQFq+q4qlpbVWtXr149D4eWJEmSNs2q2QpU1UHTrUvy/SS7VdUVSXYDruw5+EQvNnBjkrcCz+3ZXpIkSVoK4w7nOAVY16bXASf3bNyCN0nCMJ76vDHrI0mSJC24cUP0scDBSS4EDm7zJFmb5E0ThZJ8FngvcGCSy5I8vK16Z5JzgXOBXYCXj1kfSZIkacHNOpxjJlV1DXDgFMs3AE8dmX/wNNsfMM7xJUmSpKXgLxZKkiRJnQzRkiRJUidDtCRJktTJEC1JkiR1MkRLkiRJnQzRkiRJUidDtCRJktTJEC1JkiR1MkRLkiRJnQzRkiRJUidDtCRJktTJEC1JkiR1MkRLkiRJnQzRkiRJUidDtCRJktTJEC1JkiR1MkRLkiRJnQzRkiRJUidDtCRJktTJEC1JkiR1MkRLkiRJnQzRkiRJUidDtCRJktTJEC1JkiR1MkRLkiRJnQzRkiRJUidDtCRJktTJEC1JkiR1MkRLkiRJnQzRkiRJUidDtCRJktTJEC1JkiR1MkRLkiRJnQzRkiRJUidDtCRJktTJEC1JkiR1MkRLkiRJnQzRkiRJUidDtCRJktTJEC1JkiR1MkRLkiRJnQzRkiRJUidDtCRJktTJEC1JkiR1MkRLkiRJnQzRkiRJUqexQnSSnZKcluTC9nfHKcrsm+TzSc5Pck6SJ46su2uSL7btT0yy1Tj1kSRJkhbDuD3RRwHrq2pvYH2bn+ynwFOq6j7AI4B/TXKHtu5VwGva9tcBR4xZH0mSJGnBjRuiDwFOaNMnAI+ZXKCqvlFVF7bpy4ErgdVJAhwAvG+m7SVJkqTNzbghetequgKg/b3jTIWT7AdsBXwT2Bm4vqpuaqsvA3Yfsz6SJEnSgls1W4EknwTuNMWqF/QcKMluwNuBdVX1y9YTPVnNsP2RwJEAe+21V8+hJUmSpHk1a4iuqoOmW5fk+0l2q6orWki+cppy2wMfBV5YVV9oi68G7pBkVeuN3gO4fIZ6HAccB7B27dppw7YkSZK00MYdznEKsK5NrwNOnlygfePGB4G3VdV7J5ZXVQGfBg6daXtJkiRpczNuiD4WODjJhcDBbZ4ka5O8qZV5AvAQ4PAkG9tt37buecBzklzEMEb6zWPWR5IkSVpwsw7nmElVXQMcOMXyDcBT2/Q7gHdMs/3FwH7j1EGSJElabP5ioSRJktTJEC1JkiR1MkRLkiRJnQzRkiRJUidDtCRJktTJEC1JkiR1Gusr7iTp1uaSYx+91FWQJG0G7ImWJEmSOhmiJUmSpE6GaEmSJKmTIVqSJEnqZIiWJEmSOhmiJUmSpE6GaEmSJKmTIVqSJEnqZIiWJEmSOhmiJUmSpE6GaEmSJKmTIVqSJEnqZIiWJEmSOhmiJUmSpE6pqqWuQ7ckVwHfXup6zJNdgKuXuhLzZCW1BWzP5mwltQVWVntWUlvA9mzOVlJbYGW1ZyW15S5VtXqqFcsyRK8kSTZU1dqlrsd8WEltAduzOVtJbYGV1Z6V1BawPZuzldQWWFntWUltmYnDOSRJkqROhmhJkiSpkyF66R231BWYRyupLWB7NmcrqS2wstqzktoCtmdztpLaAiurPSupLdNyTLQkSZLUyZ5oSZIkqZMhWl2SvCnJPm36x0tdn6WUZE2S85a6HgBJnp3kq0neudR1mYuFrm+SY5I8dyH2vdCS7J/kI0tdD0nSzAzR6lJVT62qC5a6HospyaqlrsMcPAN4VFUdNlvBzaQ9c66vNh9JDk/y2jH3cUmSXeZY9k5J3pPkm0kuSPKxJPcY5/gj+17stuyR5OQkFya5OMlrk2w9zvEn7X+x2/OCJOcnOSfJxiQPHPPYC/7GN8lbkly5GJ0fC92eJHsm+XTrjDg/yV8t1LHa8Ra6PbdN8qUkZ7f2vGShjjWfDNELLMmHkpzVHhRHtmVHJPlGktOTvHHiH1+S1Unen+TMdvufS1z32yf5aHtQn5fkia3Oa0fK/HOSLydZn2R1W/bs9oJ3TpL3tGXHJHl7kk+1F5GnLXJb1iT5WpITWr3el+R2SY5u5/q8JMclSSt/epJXJvkM8FdJdk3ywXYuzk7ye23XW7T78PwkpybZZjHb1er6euBuwClJnpfkv5J8pf29ZytzeJL3JvkwcGpb9ret7ecs5j+sSfV9QXthO7PV+ZCR+n4oyYeTfCvJM5M8p5X5QpKdWrmntW3Pbs+d201xvLsn+Xh7Hn42yb0WoY0Tj7c3tcfWO5MclOSM9vjfr91+7b6atJ/bT3V+lqskW3SUDfBB4PSquntV7QP8PbDrQtWvxya05QPAh6pqb2BvYBvgHxeoet062/O7wB8A96+q+wIHAZcuVN3m0fHAI5a6EvPkJuBvqurewO8A/zvtU+Jl6kbggKq6H7Av8Igkv7PEdZpdVXlbwBuwU/u7DXAesDtwCbATsCXwWeC1rcy7gAe16b2Ary5x3R8HvHFkfgfgdGBtmy/gsDZ99Eg7Lge2btN3aH+PAc5u52EXhn+4d17Etqxp9f2fbf4twHMn7p+27O3AH7bp04HXjaw7EfjrNr1FOxdrGP6R7duWnwT82RLdV5e087o9sKotOwh4f5s+HLhs5PH4MIarp8PwZvojwEOWoL6vnDhnwB2AbwC3b/W9CNgOWA38AHh6K/eakfti55F9vhx41sjj7bltej2wd5t+IPCpRXq83QT8Zju/Z7XHXIBDgA/NcF/tD3ykTU95fuahfn8GfAnYCLyhPaZ/DLyq1fWTwH7teXAx8Ecjj6OTgY8DXwdePLLPD7VtzweOHFn+Y+ClwBeBB43c99u0/TxtmjoeAPznCmnLgZPb0u7/64Btl2F7/hj48JiPwacA5zC8LrydWz5nnwac2da9H7hdW/54htfRsyfOJ3CfkfN1Du25Pstz87wFeM4vSXtGjn8ycPBKaA9wO+DLwAPn+36a9/t9qSuw0m/cHB7PZggCRwEnjKx/NjeHzyvbA23i9l1guyWs+z2AbzH8835wW3Y6N4foX3BzCLgbsLFNfxx4H8OLwbYj5+GlI/t+G/CYRWzLGuA7I/MHMLywPI7hBeTcdr6PGmnnQ0VcMbMAAAjYSURBVEfKX0V7YzBpnxeOzD8PeOES3VeXMLz47cnQe3dea9PX2vrDgbeOlP+nts3EY+0i4IglqO+GVteJenwHuHer7+gbuO8Au7fpPwf+tU0/lOGN6Lntsfr6kcfbc4FtgRsmPa8W/M3pFI+Nt3HzG867tXpMd1/tz80hesrzM2bd7g18GNiyzb+O4QWzgEe2ZR9k+MRiS+B+3PzcPhy4AtiZmzsGJv4fTO4w2LnNF/CESff9GoYw+JQZ6vls4DUruS3AV2hvwpdZe7ZleDx+o9XxoZ2PwfswBP1dJurHLUPadG+Oz+Xm/wMTHTT/zs3Pra2Abebw3JzXEL2U7Rlp03eA7ZdzexjeMG6kvWmcz/tooW6bw9jIFSvJ/gw9TL9bVT9NcjrDA/Pe02xym1b2hsWp4cyq6htJHgA8CviHJKfOtkn7+2jgIcAfAS9Kcp9J6yeXXyxTHf91DC80lyY5BrjtyPqfzGGfN45M/4LhRWopvQz4dFU9NskahjcDE0bbE+AfquoNi1e1KQV4XFV9/RYLh/GVo+f2lyPzv4Rf/e86nuHN2NlJDmcIoKNuA1xfVfvOb7XnZLb6z3RfTZjy/IzpQOABwJlt9NI2DG/g/5vhDTAML4Y3VtXPk5zL8CI94bSqugYgyQcYejA3AM9O8thWZk+GIQvXMDwv3j+pDicD/1hV415YulzaEqb+f5fl2J6q+nF7bXgw8PvAiUmOqqrjp9tmkgOA91XV1W1/17b2TviNJC9n+PRlW+ATbfkZwPFJTmIYHgPweeAFSfYAPlBVF86xDvNpydqTZFuG+/Cvq+qHy7k9VfULYN8kdwA+mOQ3qmqzuHh/Oo6JXlg7ANe1AH0vhnFLtwMemmTHDBd4PW6k/KnAMydmkizFC/+vJLkz8NOqegdDz+X9JxW5DXBom/5T4HNJbgPsWVWfBv6Om59kAIdkuHhgZ4awc+YCN2GyvdpYPoA/AT7Xpq9u/4gOnXozYBgS8JcwjB1Msv3CVXMsOzD0qMPQMzWdTwB/3tpNkt2T3HGB6zZdPZ7VxoyS5Lc6t98OuCLJlsCvXaTYXlS+leTxbf9Jcr8x6zxf5nJfjXt+phKGT8P2bbd7VtUxwM+rdQcxEvqravRNC0zxZnRSh8H9GHpYJ96Q/qy9OI46A3jkRLumcT5DoFwpbVk7uqD9D9mVoWNlubWHqvpFVZ1eVS9meN163EzlJ5nuTcWE44FnVtVvAi+ZqG9VPR14IcMbgY1Jdq6qdzF02NwAfCLJAR31mC9L0p72f+/9wDur6gPTldsES3r/VNX1DJ0Km/34dUP0wvo4sCrJOQy9Tl9geNF8JcMQgk8CFzAM84DhI7+1GS70ugB4+uJX+RZ+E/hSko3ACxg+thn1E+A+Sc5ieOf6UoaPY97Reki+wvAR5vWt/JeAjzKch5dV1eWL0IZRXwXWtftjJ+D/AW9k6Nn5EDOH+r8Cfr+16yyGj7s2R//I8KnBGQz3xZSq6lSGMfifb216H0MgXWwvY/hY+pwMV8y/rHP7FzE8l04DvjZNmcOAI5KczRBmNpeL8+ZyX417fqayHjh04k1Tkp2S3KVj+4PbNtsAj2EIXVN1GMzkaIae0NfNUOZTwNYZuQg5yW8neegybMt64HZJntLquQXwzwxD+W6YVG6zb0+SeybZe2TRvsC3O+q5HnhC61Ah7ULhEVO+OU5y96r6YlUdDVwN7JnkbsDFVfVvwCnAfTvqMV8WvT3tTc6bGYan/csKaM/q1gNNe/wexPT/0zcfSzWO5NZ84+ZxwqsYxr89dqnrtAhtPoY2nmqJjr+GBbiYxJu35XgDnsjNF/qcxRCsfjyy/hbP14l1DD3mJzG8Gf7VxWvA1sB/tP29l6EXaf/RbUf2dQnDePgAb2UYOjBdPe/cjvdNhjdAH2XShUnLqC17MoSIC4HrgTcs1/uG4ROC/2LoBDqH4aP7XTofg+u4+SK047nlmNu/ZLjG4XSGMbXHt+UfYOj0OA/4P62ez2+PjY0MHVc7zXDMdzOMG/85w4XW83YdyGK3h2GoTrXzP3HNxKOWcXvuy9Dxdk7b/uj5astC3vzZ7yWQ5J8Y3mXdlmEIx1/VCr8jMow3/nFV/dMSHX8Nw8Vav7EUx5ekCRm+IvPdwB9X1VlLXR9Jm8YQLUmSJHXy2zkkSUumjbtcP8WqA6t908RysZLaAsuzPcuxzjOxPZs3e6IlSZKkTn47hyRJktTJEC1JkiR1MkRL0mYoyS+SbBy5rdmEfdwhyTPmv3aSJMdES9JmKMmPq2rb2UvOuI81bMJXOybZon79l+wkSSPsiZakZaL95Pyrk5zZftn0L9rybZOsT/LlJOcmmfhVxmOBu7ee7Fcn2T/JR0b299okh7fpS5IcneRzwOOT3D3Jx5OcleSz7dfuSPL4JOclOTvJfy7uGZCkzYdfcSdJm6dtkmxs09+qqscCRwA/qKrfTrI1cEaSU4FLGX759IdJdgG+kOQU4CjgN6pqX4Ak+89yzJ9V1YNa2fXA06vqwiQPZPgZ6AMYfhb64VX13Ymf6ZWkWyNDtCRtnm6YCL8jHgbcN8mhbX4HYG+GnzB+ZZKHAL8Edgd23YRjnghDzzbwe8B7k0ys27r9PQM4PslJDD/zK0m3SoZoSVo+Ajyrqj5xi4XDkIzVwAOq6udJLgFuO8X2N3HLYXyTy/yk/b0NcP0UIZ6qenrrmX40sDHJvsvxRxIkaVyOiZak5eMTwF8m2RIgyT2S3J6hR/rKFqB/H7hLK/8jYLuR7b8N7JNk6yQ7AAdOdZCq+iHwrSSPb8dJkvu16btX1Rer6mjgamDP+W+mJG3+7ImWpOXjTcAa4MsZxllcBTwGeCfw4SQbgI3A1wCq6pokZyQ5D/iPqvrbNgzjHOBC4CszHOsw4P8leSGwJfAe4Gzg1Un2ZugVX9+WSdKtjl9xJ0mSJHVyOIckSZLUyRAtSZIkdTJES5IkSZ0M0ZIkSVInQ7QkSZLUyRAtSZIkdTJES5IkSZ0M0ZIkSVKn/w/BK4o3Y/Bo5AAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 864x432 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "visualize_importances(feature_names, neuron_cond_vals_10.mean(dim=0).detach().numpy(), title=\"Average Feature Importances for Neuron 10\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From the visualization above, it is evident that neuron 10 primarily relies on the gender and class features, substantially different from the focus of neuron 0."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this demo, we have applied different attribution techniques in Captum including Integrated Gradients for feature attribution and Conductance for layer and neuron attribution in order to better understand the neural network predicting survival. Although larger networks are more difficult to analyze than this simple network, these basic building blocks for attribution can be utilized to improve model interpretability, breaking the traditional \"black-box\" characterization of neural networks and delving deeper into understanding how and why they make their decisions. "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}