ch15/Ch15.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "&copy; 2024 by Pearson Education, Inc. All Rights Reserved. The content in this notebook is based on the book [**Python for Programmers**](https://amzn.to/2VvdnxE)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<!-- CSS settings for this notbook -->\n",
       "<style>\n",
       "    h1 {color:#BB0000}\n",
       "    h2 {color:purple}\n",
       "    h3 {color:#0099ff}\n",
       "    hr {    \n",
       "        border: 0;\n",
       "        height: 3px;\n",
       "        background: #333;\n",
       "        background-image: linear-gradient(to right, #ccc, black, #ccc);\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%html\n",
    "<!-- CSS settings for this notbook -->\n",
    "<style>\n",
    "    h1 {color:#BB0000}\n",
    "    h2 {color:purple}\n",
    "    h3 {color:#0099ff}\n",
    "    hr {    \n",
    "        border: 0;\n",
    "        height: 3px;\n",
    "        background: #333;\n",
    "        background-image: linear-gradient(to right, #ccc, black, #ccc);\n",
    "    }\n",
    "</style>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# enable high-res images in notebook \n",
    "%config InlineBackend.figure_format = 'retina'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 15. Deep Learning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Objectives \n",
    "* What a **neural network** is and how it enables **deep learning**\n",
    "* Create **Keras neural networks**\n",
    "* Keras **layers**, **activation functions**, **loss functions** and **optimizers**\n",
    "* Use a Keras **convolutional neural network (CNN)** trained on the **MNIST dataset** to build a computer vision application that **recognizes handwritten digits** \n",
    "* Use a Keras **recurrent neural network (RNN)** trained on the **IMDb dataset** to create a sentiment analysis application that performs **binary classification** of **positive and negative movie reviews**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 15.1 Introduction\n",
    "* **Deep learning**&mdash;powerful subset of **machine learning** \n",
    "* Has produced impressive results in **computer vision** and many other areas \n",
    "* **Resource-intensive deep-learning solutions** are possible due to \n",
    "    * **big data**\n",
    "    * **significant processor power**\n",
    "    * **faster Internet speeds** \n",
    "    * advancements in **parallel computing hardware and software** "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Keras and TensorFlow\n",
    "* **Keras** offers a friendly interface to Google’s **TensorFlow**—the most widely used deep-learning library\n",
    "    * Also works with Microsoft’s **CNTK** and others\n",
    "* **François Chollet** of the **Google Mind team** developed **Keras** to make deep-learning capabilities **more accessible**\n",
    "    * His book [**_Deep Learning with Python_**](https://amzn.to/303gknb) is a must read\n",
    "* **Google has thousands of deep learning projects** internally &mdash; that number is growing quickly [\\[1\\]](http://theweek.com/speedreads/654463/google-more-than-1000-artificial-intelligence-projects-works), [\\[2\\]](https://www.zdnet.com/article/google-says-exponential-growth-of-ai-is-changing-nature-of-compute/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Models \n",
    "* **Deep learning models** connect multiple **layers**\n",
    "* Models **encapsulate sophisticated mathematical algorithms**\n",
    "    * You simply define, parameterize and manipulate objects\n",
    "* In general, **more data** leads to **better trained deep learning models**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Processing Power\n",
    "* **Deep learning** can require **significant processing power**\n",
    "* Training models on **big-data** can take **hours**, **days** or **more** \n",
    "* High-performance **GPUs** and **TPUs (Tensor Processing Units)** developed by **NVIDIA** and **Google** typically used to meet extraordinary processing demands of deep-learning applications\n",
    "* Our examples can be **trained in minutes to just less than an hour** on **conventional CPUs**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 15.1.1 Deep Learning Applications\n",
    "| <span class=\"width:50%\">&nbsp;</span> | &nbsp;\n",
    "| :--- | :--- |\n",
    "| Game playing | Computer vision: Object, pattern and facial recognition |\n",
    "| Self-driving cars | Robotics |\n",
    "| Improving customer experiences | Chatbots |\n",
    "| Diagnosing medical conditions | Google Search |\n",
    "| Facial recognition | Automated image captioning and video closed captioning |\n",
    "| Enhancing image resolution | Speech synthesis and recognition |\n",
    "| Language translation | Predicting election results |\n",
    "| Predicting earthquakes and weather | Google Sunroof to determine whether you can put solar panels on your roof |\n",
    "| <br>**_Generative applications_** | &nbsp; |\n",
    "| Generating original images | Processing existing images to look like a specified artist’s style\n",
    "| Adding color to black-and-white images and video | Creating music\n",
    "| Creating text (books, poetry) | Much more."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\"> "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 15.3 Custom Anaconda Environments\n",
    "* We use **TensorFlow's built-in version of Keras**\n",
    "* The version of TensorFlow we used requires **Python 3.6.x** \n",
    "    * Recently released TensorFlow 2.0 supports Python 3.7 \n",
    "* Easy to set up **custom environment** for Keras and TensorFlow\n",
    "    * Helps with **reproducibility** if code depends on specific Python or library versions\n",
    "    * Details in my [**Python Fundamentals LiveLessons videos**](https://learning.oreilly.com/videos/python-fundamentals/9780135917411/9780135917411-PFLL_Lesson15_06) and in [**Python for Programmers, Section 15.3**](https://learning.oreilly.com/library/view/Python+for+Programmers,+First+Edition/9780135231364/ch15.xhtml#ch15lev1sec3)    \n",
    "* Preconfigured **Docker**: [**`jupyter/tensorflow-notebook`**](https://hub.docker.com/r/jupyter/tensorflow-notebook/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Creating/Activating/Deactivating an Anaconda Environment\n",
    "```\n",
    "conda create -n tf_env python=3.6 anaconda tensorflow \n",
    "ipython jupyterlab scikit-learn matplotlib seaborn h5py \n",
    "pydot graphviz nodejs\n",
    "```\n",
    "\n",
    "* Computers with **Tensorflow-compatible NVIDIA GPUs**: [Replace `tensorflow` with **`tensorflow-gpu`** for better performance](https://www.tensorflow.org/install/gpu)\n",
    "* Activate the custom environment\n",
    "> ```\n",
    "> conda activate tf_env\n",
    "> ```\n",
    "* Deactivate the custom environment\n",
    ">```\n",
    "> conda deactivate\n",
    "> ``` \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 15.4 Neural Networks\n",
    "* Deep learning uses **artificial neural networks** to learn\n",
    "* Similar to how scientists believe our **brains** work"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Artificial Neural Network Diagram\n",
    "* The following diagram shows a three-**layer** artifical neural network\n",
    "* **Circles** represent **neurons**, **lines** between them simulate **synapses**&mdash;brain's connections between neurons\n",
    "* Output from one neuron becomes input to another\n",
    "* Diagram of a **fully connected network**\n",
    "    * Not all neural networks are fully connected\n",
    "    \n",
    "![Three-layer, fully connected neural network](./ch15images/neuralnet.png \"Three-layer, fully connected neural network\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### How Artificial Neurons Decide Whether to Activate Synapses (1 of 3)\n",
    "* During **training**, the network calculates **weights** for every **connection** between the **neurons in one layer** and **those in the next**\n",
    "* On a **neuron-by-neuron basis**, each of its **inputs** is **multiplied by** that **connection’s weight**\n",
    "* **Sum** of those weighted inputs is passed to the neuron’s **activation function**\n",
    "* **Activation function’s output** determines **which neurons to activate** based on the **inputs**—just like neurons in your brain respond to inputs from your senses"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### How Artificial Neurons Decide Whether to Activate Synapses (2 of 3)\n",
    "* Diagram of a **neuron** receiving three **inputs** (black dots) and producing an **output** (hollow circle) that would be passed to all or some of neurons in the next layer, depending on the types of the neural network’s layers\n",
    "\n",
    "![Neuron receiving three inputs (the black dots) and producing an output (the hollow circle) that would be passed to all or some of neurons in the next layer, depending on the types of the neural network’s layers](./ch15images/neuron.png \"Neuron receiving three inputs (the black dots) and producing an output (the hollow circle) that would be passed to all or some of neurons in the next layer, depending on the types of the neural network’s layers\")\n",
    "* **w1**, **w2** and **w3** are **weights**\n",
    "* In a **new model** that you train from scratch, these **values** are **initialized randomly** by the model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### How Artificial Neurons Decide Whether to Activate Synapses (3 of 3)\n",
    "* As the network **trains**, tries to **minimize error rate** between **network’s predicted labels** and **samples’ actual labels**\n",
    "* **Error rate** is known as the **loss**\n",
    "* **Calculation** that determines the **loss** is the **loss function**\n",
    "* **Backpropagation**&mdash;**During training**, the network determines the **amount that each neuron contributes to the loss**, then **adjusts the weights** throughout the layers in an effort to **minimize that loss**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 15.5 Tensors \n",
    "* Deep learning frameworks manipulate data in **tensors** &mdash; similar to **multidimensional arrays**\n",
    "    * Used to perform calculations that enable neural networks to learn\n",
    "* **Tensors** can **quickly become enormous** based on **number of dimensions** and **richness** of the data (e.g., images, audios and videos are richer than text)\n",
    "    * Manipulating them efficiently is crucial \n",
    "* For an **overview of 0D to 5D tensors** and what they might represent, see \n",
    "    * [**Python Fundamentals LiveLessons videos**](https://learning.oreilly.com/videos/python-fundamentals/9780135917411/9780135917411-PFLL_Lesson15_08) \n",
    "    * [**Python for Programmers, Section 15.7**](https://learning.oreilly.com/library/view/python-for-programmers/9780135231364/ch15.xhtml#ch15lev1sec5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 15.6 Convolutional Neural Networks for Vision; Multi-Classification with the MNIST Dataset (1 of 2)\n",
    "* **`MNIST` database of handwritten digits**\n",
    "    * “The MNIST Database.” MNIST Handwritten Digit Database, Yann LeCun, Corinna Cortes and Chris Burges. http://yann.lecun.com/exdb/mnist/.\n",
    "* Create a [**convolutional neural network**](https://en.wikipedia.org/wiki/Convolutional_neural_network) (also called a **convnet** or **CNN**)\n",
    "* Common in **computer-vision applications**\n",
    "    * Recognizing handwritten digits and characters\n",
    "    * Recognizing objects in images and video\n",
    "    * Self-driving cars\n",
    "* **Non-vision applications**\n",
    "    * natural-language processing \n",
    "    * recommender systems\n",
    "    * much more"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 15.6 Convolutional Neural Networks for Vision; Multi-Classification with the MNIST Dataset (2 of 2)\n",
    "* **60,000** labeled digit image samples for **training**, **10,000** for testing\n",
    "* **28-by-28 pixel images** (**784 features**), represented as **NumPy arrays**\n",
    "* **Grayscale pixel intensity** (shade) values **0-255** \n",
    "* **Convnet** will perform [**probabilistic classification**](https://en.wikipedia.org/wiki/Probabilistic_classification)\n",
    "\t* Model will output **10 probabilities** indicating likelihood a digit is **0-9**\n",
    "\t* **Highest probability** is the **predicted value**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Reproducibility in Keras and Deep Learning\n",
    "* **Reproducibility is difficult** because the libraries **heavily parallelize floating-point calculations** \n",
    "* Each time calculations execute, they may execute in a **different order**\n",
    "* Can produce **different results** in each execution\n",
    "* See the [**Keras FAQ on reproducibility**](https://keras.io/getting-started/faq/#how-can-i-obtain-reproducible-results-using-keras-during-development)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Components of a Keras Neural Network \n",
    "* **Network** (also called a **model**)\n",
    "    * Sequence of layers containing the neurons used to learn from the samples\n",
    "    * Each layer’s neurons receive inputs, process them (via an **activation function**) and produce outputs\n",
    "    * The more layers you **stack**, the **deeper** the network is, hence the term **deep learning**\n",
    "* **Loss function**\n",
    "    * Produces a measure of **how well the network predicts target values** \n",
    "    * **Lower loss values** indicate **better predictions**\n",
    "* **Optimizer**\n",
    "    * Attempts to **minimize the values produced by the loss function** to **tune the network** to make better predictions "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 15.6.1 Loading the MNIST Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "A module that was compiled using NumPy 1.x cannot be run in\n",
      "NumPy 2.0.2 as it may crash. To support both 1.x and 2.x\n",
      "versions of NumPy, modules must be compiled with NumPy 2.0.\n",
      "Some module may need to rebuild instead e.g. with 'pybind11>=2.12'.\n",
      "\n",
      "If you are a user of the module, the easiest solution will be to\n",
      "downgrade to 'numpy<2' or try to upgrade the affected module.\n",
      "We expect that some modules will need time to support NumPy 2.\n",
      "\n",
      "Traceback (most recent call last):  File \"/Users/pauldeitel/anaconda3/envs/pydsft/lib/python3.11/runpy.py\", line 198, in _run_module_as_main\n",
      "    return _run_code(code, main_globals, None,\n",
      "  File \"/Users/pauldeitel/anaconda3/envs/pydsft/lib/python3.11/runpy.py\", line 88, in _run_code\n",
      "    exec(code, run_globals)\n",
      "  File \"/Users/pauldeitel/anaconda3/envs/pydsft/lib/python3.11/site-packages/ipykernel_launcher.py\", line 18, in <module>\n",
      "    app.launch_new_instance()\n",
      "  File \"/Users/pauldeitel/anaconda3/envs/pydsft/lib/python3.11/site-packages/traitlets/config/application.py\", line 1075, in launch_instance\n",
      "    app.start()\n",
      "  File \"/Users/pauldeitel/anaconda3/envs/pydsft/lib/python3.11/site-packages/ipykernel/kernelapp.py\", line 739, in start\n",
      "    self.io_loop.start()\n",
      "  File \"/Users/pauldeitel/anaconda3/envs/pydsft/lib/python3.11/site-packages/tornado/platform/asyncio.py\", line 205, in start\n",
      "    self.asyncio_loop.run_forever()\n",
      "  File \"/Users/pauldeitel/anaconda3/envs/pydsft/lib/python3.11/asyncio/base_events.py\", line 604, in run_forever\n",
      "    self._run_once()\n",
      "  File \"/Users/pauldeitel/anaconda3/envs/pydsft/lib/python3.11/asyncio/base_events.py\", line 1909, in _run_once\n",
      "    handle._run()\n",
      "  File \"/Users/pauldeitel/anaconda3/envs/pydsft/lib/python3.11/asyncio/events.py\", line 80, in _run\n",
      "    self._context.run(self._callback, *self._args)\n",
      "  File \"/Users/pauldeitel/anaconda3/envs/pydsft/lib/python3.11/site-packages/ipykernel/kernelbase.py\", line 545, in dispatch_queue\n",
      "    await self.process_one()\n",
      "  File \"/Users/pauldeitel/anaconda3/envs/pydsft/lib/python3.11/site-packages/ipykernel/kernelbase.py\", line 534, in process_one\n",
      "    await dispatch(*args)\n",
      "  File \"/Users/pauldeitel/anaconda3/envs/pydsft/lib/python3.11/site-packages/ipykernel/kernelbase.py\", line 437, in dispatch_shell\n",
      "    await result\n",
      "  File \"/Users/pauldeitel/anaconda3/envs/pydsft/lib/python3.11/site-packages/ipykernel/ipkernel.py\", line 362, in execute_request\n",
      "    await super().execute_request(stream, ident, parent)\n",
      "  File \"/Users/pauldeitel/anaconda3/envs/pydsft/lib/python3.11/site-packages/ipykernel/kernelbase.py\", line 778, in execute_request\n",
      "    reply_content = await reply_content\n",
      "  File \"/Users/pauldeitel/anaconda3/envs/pydsft/lib/python3.11/site-packages/ipykernel/ipkernel.py\", line 449, in do_execute\n",
      "    res = shell.run_cell(\n",
      "  File \"/Users/pauldeitel/anaconda3/envs/pydsft/lib/python3.11/site-packages/ipykernel/zmqshell.py\", line 549, in run_cell\n",
      "    return super().run_cell(*args, **kwargs)\n",
      "  File \"/Users/pauldeitel/anaconda3/envs/pydsft/lib/python3.11/site-packages/IPython/core/interactiveshell.py\", line 3075, in run_cell\n",
      "    result = self._run_cell(\n",
      "  File \"/Users/pauldeitel/anaconda3/envs/pydsft/lib/python3.11/site-packages/IPython/core/interactiveshell.py\", line 3130, in _run_cell\n",
      "    result = runner(coro)\n",
      "  File \"/Users/pauldeitel/anaconda3/envs/pydsft/lib/python3.11/site-packages/IPython/core/async_helpers.py\", line 128, in _pseudo_sync_runner\n",
      "    coro.send(None)\n",
      "  File \"/Users/pauldeitel/anaconda3/envs/pydsft/lib/python3.11/site-packages/IPython/core/interactiveshell.py\", line 3334, in run_cell_async\n",
      "    has_raised = await self.run_ast_nodes(code_ast.body, cell_name,\n",
      "  File \"/Users/pauldeitel/anaconda3/envs/pydsft/lib/python3.11/site-packages/IPython/core/interactiveshell.py\", line 3517, in run_ast_nodes\n",
      "    if await self.run_code(code, result, async_=asy):\n",
      "  File \"/Users/pauldeitel/anaconda3/envs/pydsft/lib/python3.11/site-packages/IPython/core/interactiveshell.py\", line 3577, in run_code\n",
      "    exec(code_obj, self.user_global_ns, self.user_ns)\n",
      "  File \"/var/folders/84/stqwg9dd1gb8dy9dxp_4j82r0000gn/T/ipykernel_54942/3894215214.py\", line 1, in <module>\n",
      "    import tensorflow.keras.datasets as datasets #import mnist\n",
      "  File \"/Users/pauldeitel/anaconda3/envs/pydsft/lib/python3.11/site-packages/tensorflow/__init__.py\", line 49, in <module>\n",
      "    from tensorflow._api.v2 import __internal__\n",
      "  File \"/Users/pauldeitel/anaconda3/envs/pydsft/lib/python3.11/site-packages/tensorflow/_api/v2/__internal__/__init__.py\", line 8, in <module>\n",
      "    from tensorflow._api.v2.__internal__ import autograph\n",
      "  File \"/Users/pauldeitel/anaconda3/envs/pydsft/lib/python3.11/site-packages/tensorflow/_api/v2/__internal__/autograph/__init__.py\", line 8, in <module>\n",
      "    from tensorflow.python.autograph.core.ag_ctx import control_status_ctx # line: 34\n",
      "  File \"/Users/pauldeitel/anaconda3/envs/pydsft/lib/python3.11/site-packages/tensorflow/python/autograph/core/ag_ctx.py\", line 21, in <module>\n",
      "    from tensorflow.python.autograph.utils import ag_logging\n",
      "  File \"/Users/pauldeitel/anaconda3/envs/pydsft/lib/python3.11/site-packages/tensorflow/python/autograph/utils/__init__.py\", line 17, in <module>\n",
      "    from tensorflow.python.autograph.utils.context_managers import control_dependency_on_returns\n",
      "  File \"/Users/pauldeitel/anaconda3/envs/pydsft/lib/python3.11/site-packages/tensorflow/python/autograph/utils/context_managers.py\", line 19, in <module>\n",
      "    from tensorflow.python.framework import ops\n",
      "  File \"/Users/pauldeitel/anaconda3/envs/pydsft/lib/python3.11/site-packages/tensorflow/python/framework/ops.py\", line 62, in <module>\n",
      "    from tensorflow.python.framework import tensor as tensor_lib\n",
      "  File \"/Users/pauldeitel/anaconda3/envs/pydsft/lib/python3.11/site-packages/tensorflow/python/framework/tensor.py\", line 35, in <module>\n",
      "    from tensorflow.python.framework import tensor_util\n",
      "  File \"/Users/pauldeitel/anaconda3/envs/pydsft/lib/python3.11/site-packages/tensorflow/python/framework/tensor_util.py\", line 39, in <module>\n",
      "    from tensorflow.python.framework import fast_tensor_util\n"
     ]
    }
   ],
   "source": [
    "import tensorflow.keras.datasets as datasets #import mnist"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras.datasets import mnist"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* **`load_data` function** loads **training** and **testing sets**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "(X_train, y_train), (X_test, y_test) = datasets.mnist.load_data()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 15.6.2 Data Exploration\n",
    "* Check dimensions of the **training set images (`X_train`)**, **training set labels (`y_train`)**, **testing set images (`X_test`)** and **testing set labels (`y_test`)**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(60000, 28, 28)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(60000,)"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_train.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(10000, 28, 28)"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_test.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(10000,)"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_test.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Visualizing Digits—Display 24 MNIST Training Set Images (1 of 2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import seaborn as sns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Visualizing Digits—Display 24 MNIST Training Set Images (2 of 2)\n",
    "* Run cell several times to view different digits and see **why handwritten digit recognition is a challenge**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAABHEAAAMOCAYAAACZO5rbAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjAsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvlHJYcgAAAAlwSFlzAAAewgAAHsIBbtB1PgAAkedJREFUeJzs3Xl8jOf+//FPYolYYwlSJKgtKrQIRe37TikqVKjTUm1VT3toVWtpq62e7q2qtmJfaj9FqdprjaVCqaWWIoglBCFCfn/4db657ms6M4mZuXNnXs/Ho4/HvG/Xdc/nHLkyM5f7/oxfWlpamgAAAAAAACBL8ze7AAAAAAAAADjHJg4AAAAAAIAFsIkDAAAAAABgAWziAAAAAAAAWACbOAAAAAAAABbAJg4AAAAAAIAFsIkDAAAAAABgAWziAAAAAAAAWACbOAAAAAAAABbAJg4AAAAAAIAFsIkDAAAAAABgAWziAAAAAAAAWACbOAAAAAAAABbAJg4AAAAAAIAFsIkDAAAAAABgAWziAAAAAAAAWACbOAAAAAAAABbAJg4AAAAAAIAFsIkDAAAAAABgAWziAAAAAAAAWACbOJmwYcMGeeaZZyQ8PFyCgoIkMDBQypcvL126dJHZs2dLamqq2SUCHrNu3Trx8/PL9H8xMTFm/08A3CImJibDP/9vv/222WUDHnH8+HGZPHmy9OnTR2rUqCGFCxeWXLlySZEiRaR69ery7LPPyvr1680uE/AaPi8Aql27dsmIESOkdu3aEhISIgEBAfLAAw9IzZo1ZcCAATJ9+nQ5e/as2WVagl9aWlqa2UVYxeXLl+Wpp56SH3/80eG4WrVqyYwZM6RKlSpeqgzwnnXr1knTpk0zPX/FihXSpk0bN1YEmCMmJkb69++foTnjxo2TN954w0MVAd63e/duGTRokGzfvt2l8U2aNJGpU6dKaGiohysDzMHnBUB1/vx5efnll2XmzJlOxw4ZMkS++OILL1RlbTnNLsAqEhMTpX79+nLw4EHbsfLly0u9evUkMDBQjh49Kps2bZLbt2/Lzp07pUmTJrJ161YpW7aseUUDHlCqVCkZMmSIy+NXrVolhw8fFhGREiVKSIsWLTxVGmCaKlWqSPPmzZ2Oi4yM9EI1gPf88ccf2gZOpUqVpFq1alKsWDFJTEyUzZs3y6lTp0Tk3j8E1KtXTzZu3Cjly5c3o2TAY/i8AKhOnjwpTZo0kWPHjtmOlStXTmrWrClFixaV5ORkOXz4sOzZs0du3rxpYqXWwiaOiwYOHGj7hRwYGCjffPON9OnTRxlz/PhxiYqKks2bN8u5c+ekW7duEhsbK35+fmaUDHhExYoVXd4hv3PnjpQuXdqWo6KiJGdOfu0g+6lbty7/cgSfVqFCBRk4cKD06dNHSpUqpfzZ3bt3ZcqUKfLiiy/KjRs35MyZM7b3S7xHQnbC5wXg/1y5ckWaNm1q28CpWbOmfP7551K/fn1t7LVr12TZsmXCTUKuoSeOC3bv3i0LFiyw5ZiYGO0XsohI2bJl5aeffrL9y9KuXbtk1qxZXqsTyGpWrlyp3Nvar18/E6sBALhbSEiITJkyRQ4ePCjDhw/XNnBERPz9/eXpp5+WGTNm2I5t3bpVVq1a5c1SAY/i8wKgeuWVV+TPP/8UEZFGjRrJxo0b7W7giIjkz59fevbsKb169fJmiZbFJo4L5s2bZ3tco0YN6dGjxz+OLVCggIwcOdKWP/30U4/WBmRlU6dOtT1+5JFHpHr16iZWAwBwt8aNG0t0dLTkyJHD6diuXbtKnTp1bHnZsmWeLA3wKj4vAP9nz5498u2334rIvZ/3mTNnSt68eU2uKvtgE8cFW7dutT1u27at0/Ht2rWzPd6xY4ecPHnSI3UBWdmVK1dk6dKltsxVOACABg0a2B4fP37cvEIAN+PzAvB/vv76a9vjAQMGKO0VcP/YxHHBuXPnbI/DwsKcji9ZsqQEBATY8po1azxSF5CVzZ0719agLFeuXNK7d2+TKwIAmC193487d+6YWAngXnxeAO65c+eOzJ4925ajoqJMrCZ7osOoC9I3WHK16Vj6Ofv373d7TUBWl/5Wqnbt2klwcLCJ1QCelZiYKPPmzZPff/9drly5IoUKFZJSpUrJY489JuHh4WaXB2QZcXFxtsdlypQxsRLAvfi8ANyzb98+uXr1qoiI5MuXTx555BG5deuWxMTEyKxZs+TAgQNy9epVKVasmDz88MPSuXNn6devn+TOndvkyq2DTRwXBAcH2zrNu3Lp79mzZyUlJcWWDxw44KnSgCzpyJEjsnnzZlvmVipkd0uWLJElS5bY/bOIiAh54403HPZHAHzBX3/9pVxt0KJFCxOrAdyLzwvAPTt27LA9rly5shw9elS6d+8u+/btU8adPn1aTp8+LcuWLZPx48fL/PnzpWbNmt4u15K4ncoFtWvXtj1esWKF0/HLly9X8qVLl9xeE5CVpb8Kp2jRotK+fXsTqwHMFRcXJz179pT+/ftLamqq2eUAphk2bJjtFqrQ0FDp2LGjyRUB7sPnBeCev/76y/bY399fWrVqZdvAqVKlivTt21eio6OVDZtjx45Jo0aNZPfu3V6v14rYxHFBp06dbI9/++03pfu8UVJSkrzzzjvaMcBXpKWlKV8j27t3by6PRLZVrlw5GTFihKxevVrOnDkjt27dkmvXrsn+/fvlo48+ktDQUNvYmJgYeeGFF0ysFjDP1KlTla9fHj9+vNIPBLA6Pi8A9yQmJtoex8bGysmTJyUwMFDmzp0rBw4ckGnTpsmUKVNk586dsmbNGilWrJiIiFy/fl169uwpt2/fNqly6/BLS38zJv7RY489Jr/++quIiOTNm1cmT56sNWo9ceKE9O7dW7mNRETkwQcflCNHjnitVsBM69atk6ZNm9pybGys1KpVy8SKAM9ITEyUggULir//P/97SFJSkvTu3Vt+/PFH27ENGzZIw4YNvVEikCXExsZKw4YNbc3ue/bsKXPmzDG5KsD9+LwAiAwcOFC+++475di8efPkiSeesDt+y5Yt8thjj8ndu3dFRGTKlCkSHR3t6TItjStxXDRjxgzbLuGNGzckKipKKlSoIH379pVnnnlGmjdvLhUrVpTNmzeLv7+/8rWBBQoUMKtswOvS30r10EMPsYGDbCsoKMjhBo7Ivd//8+bNk0qVKtmOvf/++54uDcgyjh07Jh07drRt4ERERMikSZNMrgrwDD4vACJ58uRRcmRk5D9u4IiI1KtXTx5//HFbZpPfOTZxXFS2bFnZsmWL1KhRw3bs6NGjMmPGDJk8ebKsWbNGbt++Lfnz55dp06ZJZGSkbVxQUJAJFQPed+PGDeVyeXbRAZHAwEAZPny4La9du1Zu3bplYkWAd8THx0vLli3l7NmzIiJSvnx5WblypRQqVMjkygDP4PMCIJI/f34ld+3a1emc9GOMV6lBxyZOBlSoUEF27dolc+fOlSeeeELCwsIkMDBQChQoINWqVZMRI0bI/v37JSoqSulKX7p0afOKBrxo4cKFtnu6c+TIIVFRUSZXBGQNzZs3tz2+ceOGnDhxwsRqAM+7ePGitGzZUo4ePSoiIiEhIbJ69WoJCQkxuTLAs/i8AF9XtGhRJVetWtXpnPRjkpKS6BHlBF8xnkH+/v7So0cPp18Vm/6r1dLvsgPZWfpbqVq1asWbdeD/M66FixcvmlQJ4HlXr16VNm3ayP79+0Xk3hv6n3/+WcqVK2dyZYB38HkBvqxKlSpKNl6ZY49xTFJSErcYOsCVOB4QHx8vBw8etOX69eubWA3gHadOnZI1a9bYMrdSAf/n+vXrSs6XL59JlQCedf36dWnXrp3ExsaKiEjBggVl5cqV8tBDD5lcGZC18HkB2VW1atWU7MpVNcYx3HbrGJs4HjBz5kxbd+0qVapI7dq1Ta4I8LwZM2bYfu6DgoKUr9oEfN3u3buVzFVqyI5u3rwpnTp1Ur6dZ/ny5TS4B+zg8wKyq3Llykn58uVt+ffff3c6J/2YIkWK8I9dTrCJ42ZXrlyRCRMm2PKgQYNMrAbwnmnTptke9+zZU+tMD/iyKVOm2B6Hh4dLcHCwidUA7nf79m3p1q2b7YrMgIAAWbJkiTRo0MDkyoCsh88LyO7SNypetGiR0/HpxzRq1MgjNWUnbOK40Z07dyQ6OlrOnz8vIvcaNA0ePNjkqgDP2759uxw4cMCWuZUK2d21a9dcHrto0SKZOXOmLffp08cTJQGmuXPnjvTu3VuWL18uIiI5c+aUefPmSYsWLUyuDMh6+LwAXzB48GDJlSuXiIjExsbKDz/88I9jt2zZomzi8DnCOTZxXLRq1SoZPXq00kU+vQMHDkjLli1l8eLFInLvX6C+//57yZ07t/eKBEySvqFxpUqV5NFHHzWxGsDz5s+fL3Xr1pUZM2bI1atX7Y5JSkqScePGSY8ePSQtLU1ERMqUKSNDhw71ZqmAR6WlpcnAgQNl/vz5InKvoev06dO5pRY+ic8LwD0PPvigPPfcc7YcHR1tdyNn7dq10qlTJ9uthY8++iivHy7wS/v7nSUcmjNnjjz55JMicu++1YiICClSpIhcuXJFfv/9d9m7d69tbEBAgCxevFjatGljVrmA16SkpMgDDzxg+7add955R15//XWTqwI8KyYmRvr37y8iIrly5ZLw8HCpXLmyBAUFSWpqqpw8eVK2bNkiN27csM0pXLiwbNiwQWv4B1jZV199JUOGDLHlihUrSqtWrVyaW7RoURkzZoynSgO8js8LwP+5deuWtGzZUjZu3Gg7Fh4eLpGRkZIjRw7Zu3ev7Ny50/ZnISEhsm3bNilTpowZ5VoKXzGeCQcPHlS6yadXq1YtmTRpEk384DN+/PFH2waOv7+/9O3b1+SKAO+6ffu27N27V3lzbtSsWTP5/vvvJSwszIuVAZ739y0hfzt8+LAcPnzYpblhYWFs4iDb4vMCfF1AQID873//k8GDB8vs2bNF5N7VaOlbMPytbt268sMPP7CB4yI2cVzUoUMHWbRokfzyyy+ybds2iY+Pl4SEBAkMDJSQkBCpU6eOdO/eXdq2bSs5cuQwu1zAa9LfStWsWTN++cInPPnkk1KxYkXZsmWLbNmyRY4ePSoXL16Uixcvyt27dyUoKEgefPBBqVevnvTq1YtvHQEAH8DnBUBVqFAhmTVrlgwaNEimTZsmmzZtktOnT8udO3ekRIkS8uijj0qPHj2kS5cu4ufnZ3a5lsHtVAAAAAAAABZAY2MAAAAAAAALYBMHAAAAAADAAtjEAQAAAAAAsAA2cQAAAAAAACyATRwAAAAAAAALYBMHAAAAAADAAtjEAQAAAAAAsAA2cQAAAAAAACyATRwAAAAAAAALYBMHAAAAAADAAtjEAQAAAAAAsAA2cQAAAAAAACyATRwAAAAAAAALyOmuE928eVPi4uJERCQ4OFhy5nTbqZHFpKamSkJCgoiIRERESJ48eUyuKGthLfgO1oJjrAXfwVpwjLXgO1gLjrEWfAdrwTHWgu/wxFpw209LXFyc1KlTx12ng0Vs375dIiMjzS4jS2Et+CbWgo614JtYCzrWgm9iLehYC76JtaBjLfgmd60FbqcCAAAAAACwALddiRMcHGx7vH37dgkJCXHXqZHFxMfH23aO0/+94x7Wgu9gLTjGWvAdrAXHWAu+g7XgGGvBd7AWHGMt+A5PrAW3beKkv48vJCRESpcu7a5TIwvj/k0da8E3sRZ0rAXfxFrQsRZ8E2tBx1rwTawFHWvBN7lrLXA7FQAAAAAAgAWwiQMAAAAAAGABbOIAAAAAAABYAJs4AAAAAAAAFsAmDgAAAAAAgAWwiQMAAAAAAGABbOIAAAAAAABYAJs4AAAAAAAAFsAmDgAAAAAAgAWwiQMAAAAAAGABOc0uAAAAAAAA+Kb4+Hgljxs3Thvz3XffKTklJUXJn3/+uTbn+eefd0N1WQ9X4gAAAAAAAFgAmzgAAAAAAAAWwCYOAAAAAACABbCJAwAAAAAAYAE0NgYAwAv27dun5I0bN2pjkpOTlfzKK684Pe/w4cOVPH78+ExUBwAA4HmHDx/WjvXs2VPJe/bscXoePz8/JdtrhkxjYwAAAAAAAJiGTRwAAAAAAAALYBMHAAAAAADAAuiJAwCAmx05ckQ71rZtWyWfOXPG6XmM93vbM2fOHCXTEwcAAJglLS1NyX379lXy/v37tTnnzp1T8ocffqiNGTBggJKfffZZJRcvXjxDdVoZV+IAAAAAAABYAJs4AAAAAAAAFsAmDgAAAAAAgAXQE8fgr7/+0o6tXbtWyevWrVNySkqKNmfmzJlOn6tkyZJKPnv2rJKN9/mJiLz88stKrlSpktPnAexZuHChkrt162ZSJc4Z74EVEXnnnXeUbFxPgDeNGTNGybNnz9bGnD59Wsn2+t34+6v/tlKgQAElV69eXZtjfF0AMqtp06ZKNr7fcZe33npLyU2aNNHG2DsGZGXJyclK/u2335T8/fffa3M6deqk5Lp162pjgoODHT7vvn37tGO//vqrkl999VVtzPbt25VcpUoVh88DuGrRokVKnjVrlpLLli2rzTG+j/rXv/7l9Hl69+6t5KVLl7pYofVxJQ4AAAAAAIAFsIkDAAAAAABgAWziAAAAAAAAWACbOAAAAAAAABaQrRsb37lzRztmbPT13nvvKXnbtm3anEuXLt13LREREdoxYzMzY6PWSZMmaXPmzZunZHuNmPPly5eZEuFj1qxZo2Rjk9WAgABtTt68eZV8+fJlJaelpWlz7DVvzSh7zQD/85//KJnGxvAmYyPJqVOnKvnEiRNOzxEaGqodMzbxHjVqVCaqA5wbPXq0dsxTjYyNjA0sjVlE/1IJGh0jKzlw4IB27IknnlDy/v37nZ5n8uTJSm7RooU2pnXr1kqOjIxUctu2bbU5OXOqH/GSkpK0MfHx8UqmsTEyw15j7ZdeeknJpUqVUvIvv/yizSlXrpzT50pMTFSy8T1SVFSU03NkF1yJAwAAAAAAYAFs4gAAAAAAAFgAmzgAAAAAAAAWkK174kyfPl071r9//wyfJywsTMmNGjVScseOHbU5RYoUcThHROTcuXNKzp8/v5Jfe+01bY6xB8miRYu0MX369NGOAUZjx45VcqtWrZRsr8dM1apVlWzsq+OKHTt2aMeM/aCMihUrph0zrhfAU4y900REPv74YyVfuHDB6Xk6dOigZOMaFBF5+OGHM1YckEn2euKsX79eyd7qkWNP06ZNlWyv5xrgLT/88IOSX3zxRW3M2bNnM3xeY3+bihUramOM73e+/PJLJScnJzt9nhIlSmjHjO/pgMx49913tWOnT59W8sSJE5XsSv8be3bv3q3kxx57TMlDhw7N1HmtiCtxAAAAAAAALIBNHAAAAAAAAAtgEwcAAAAAAMAC2MQBAAAAAACwgGzV2PjDDz9U8vDhw53OKVCggJLtNVh97rnnlJwjR45MVOfchg0bnI7x91f33YoXL+6RWpD9GZtvd+rUKcPnyMycxMTEDM8ZMmSIdqxUqVIZPg/gitTUVCWfOnVKG+OskXHRokW1Y0uXLr2/wgAPW7t2rZKNjY0z0+jY2Cw5s+cBPOXu3bvasTlz5ij5P//5j5JdaWLct29fJffo0UMb07JlSyUHBARoY7799lslr1ixwulzG7Vo0UI7Zq/ZMeDMqlWrlPy///1PG9OvXz8lP/PMM255bmPDe2P2JVyJAwAAAAAAYAFs4gAAAAAAAFgAmzgAAAAAAAAWkK164ixYsEDJ9u5xDQ8PV/KyZcuUXK5cOfcX9g+M/Xdcucc1Z071r6xVq1ZurQlwN2MPnH//+98ZPsfo0aPdUwzggjNnzih54sSJGT7HuHHj3FUOYJomTZo4zK6w9/vblZ44mXkuIDMmTJigHRsxYkSGz5M/f36H56hatWqGzykiMn/+fCVfu3Ytw+f45JNPMvXcgNGzzz6r5OvXr2tjKlWq5K1yfBZX4gAAAAAAAFgAmzgAAAAAAAAWwCYOAAAAAACABVi2J86aNWu0Y9u3b1dyYGCgNubDDz9Usjd74BgVLVo0w3PMrBfIjG+++UbJFy5ccDrntdde81Q5gFOvvPLKfZ/DXu+nmjVrKjkyMvK+nwfI6saMGZOpeWvXrnVzJYB9xvcpmTV27FglZ6YHzi+//OLSMWeMry958+bN8DkAV4SGhmrHoqOjvV+Ij+FKHAAAAAAAAAtgEwcAAAAAAMAC2MQBAAAAAACwADZxAAAAAAAALMCyjY3PnDmjHbt7966SK1asqI1p166dx2pK7/Lly0oeN26cNmbOnDkZPu/w4cMzXROQFfj5+WnHHnzwQSWPGjXKW+UAmhIlSii5ePHi2pjz5887PEdycrJ2rHPnzkpu06aNNuajjz5SclBQkMPnAbKadevWmV0C4HGNGzfWjr3wwgv3fd7p06drx1JTUx3OKVKkiHZs2bJlSqaxMTIrLi5OyRcvXlSyvcbgJUuW9GhN4EocAAAAAAAAS2ATBwAAAAAAwALYxAEAAAAAALAAy/bEcUVoaKh2LCUlRcm5c+fO8HmvX7+u5Llz52pjpk6dquQNGzZk+HmeeOIJ7VjPnj0zfB7ATL/88ovTMRUqVFByYGCgp8oBnPr888+V3KdPH23Mnj17lGzs0/b2229rc86ePatk4+uEiMilS5eUvHjxYkelAqYz9sBp2rRphs/x1ltvuakawLkJEyYo+dixY07nGHuYffbZZ9qYnDnVj1WJiYlKLlSokDbnwoULSp49e7bTWoyM/dZERIKDgzN8HsCeTz75RMnXrl1T8siRI7U5xv5+VatWVXKVKlW0OR06dFBytWrVMlKmz+FKHAAAAAAAAAtgEwcAAAAAAMAC2MQBAAAAAACwAMv2xGnSpIl2rHz58kr+8ccftTHR0dFK7t69u5L37dunzYmNjXU4xpV7aV1Ru3ZtJU+fPl0bExAQ4JbnAjzhiy++0I6tXbtWyQULFtTGxMTEeKok4L7VrVvXpWPpjRkzRjvm5+fn9LlWr16t5G3btmXoeYH7YexnY+x3Y693jb2fdWeM7+FGjx6d4XMAmVW6dGklp6WlOZ1j7NU3c+ZMbYyxB+D58+eVbK9PTXx8vJKNvTtdwfqBuxj7+4k4f49u/DkXEalTp46Sw8LClGzv88KoUaOU/Oqrr2pjXn75ZSUXK1bMYW3ZGVfiAAAAAAAAWACbOAAAAAAAABbAJg4AAAAAAIAFsIkDAAAAAABgAZZtbGxsSiaiNy2eNGmSNmb27NkOsytq1Kih5KioKG1MmTJllPzee+85PW/79u2VTBNjWM28efO0Y6mpqUp+9tlntTElSpTwWE1AVlGxYkUlHz16VBuTnJys5P/9739KprExMsvYpNjYxNgVmWlibE/jxo3dch4gM3r27KnkHTt2aGM+/vhjJS9atMhhdsWhQ4cyPEdEf4/Ur18/JRcpUiRT5wWMDh8+rB1z1vh77Nix2rFhw4Y5nDN48GDt2DPPPKNke5+djx8/ruQPP/xQyQ888IDD581OuBIHAAAAAADAAtjEAQAAAAAAsAA2cQAAAAAAACzAsj1x7Bk1apSSjffWiYhMmzZNyWfOnFGyvT40vXr1UnJERISS/f31vbAuXbo4rFVEpGbNmkp+6aWXnM4BspLx48creePGjU7nNGjQwFPlAFma8R7x559/3qRKkN2NHj1aO+aufjbusH79eiXbq7dJkyYOM5BZxvftH3zwgTbG2IfG2ANn9+7d2pzOnTsref78+Up21lvkn/Tt21fJ77//fqbOAxhdv35dycbPvPYUKlRIyUOGDMnw85YsWVI7ZlxjHTp00MbMmTNHyXFxcUr+6aeftDmlSpXKcH1WwJU4AAAAAAAAFsAmDgAAAAAAgAWwiQMAAAAAAGABbOIAAAAAAABYQLZqbGxkbEomIvLqq6+6/Xk+++wz7djSpUudznviiSeUHBQU5K6SAK9Yvny50zHGZpTNmjXzUDUAACtYt26dwyzinkbMrjRDtvfczmS2QS2yppw59Y9Dw4cPd5jtSUxMVLKx+fGRI0ecnqNSpUrasZ49ezqdB2TGxIkTlXzu3LkMnyN37txuqSVHjhxK/u6777QxrVu3VvL+/fuVbO+LhXbs2HH/xWVBXIkDAAAAAABgAWziAAAAAAAAWACbOAAAAAAAABaQrXvieEpycrKSjfcT2hMeHq4di46OdldJgFds3rxZyQcPHnQ6x9j7KX/+/G6tCQCgckc/GVcZ+85kpseMp7irFld66wAnT55U8unTpzN8jpdeekk7Vrt27cyWBDi0YcMGJRcvXlwbY+zZeuLECSXv27dPm1OtWrX7ru2BBx7Qjv30009Kbtu2rZL37NmjzVmwYIGSu3Xrdt+1ZQVciQMAAAAAAGABbOIAAAAAAABYAJs4AAAAAAAAFkBPnEz417/+pWRX+oI8++yz2rGSJUu6rSbA3ZKSkrRjxvtIL168qOSHH35Ym9OrVy+31gW428qVK5Xs5+en5FatWrnleVzpnwZkJcZeMGvXrs3wOUaPHq0dW79+vZLN7KNj/N/41ltvOR0DGHsEiug/J7dv33Z6nrJlyyp58ODB91MWkCHG9yVXr17Vxvzyyy9KfvHFF5X85ptvanPGjx+v5MqVK2e2REWpUqWUPGjQICU///zz2pxJkyYpmZ44AAAAAAAA8Bo2cQAAAAAAACyATRwAAAAAAAALYBMHAAAAAADAAmhs7IKzZ88qefHixRk+R58+fdxUDeAZN2/eVHKPHj20MefOnXN4js8//1w7VqRIkfsrDPAwY2O8xMREJdv7nd+4cWMlG18nPvvsM23Ovn37lGxsoAy4i73mvGPGjHE6z9i42B0Nfe01Ngas7scff9SOudLI2GjatGnuKAfIFGOjYGMWEQkNDVXyp59+qmR775GMX3zy9NNPK9mV5sJHjhzRji1atEjJ//3vf5WcN29ebY69xsvZAVfiAAAAAAAAWACbOAAAAAAAABbAJg4AAAAAAIAF0BPH4NKlS9qx9957T8nXr193ep7XXntNyYULF76/wgA3u3r1qpK///57Jf/0009OzzFlyhQlP/bYY/dfGGAy49po3769NiZXrlxKTktLU3JSUpLT5ylevLh2rFOnTkp+4403nJ4HMHKlD429fjfu6IEDZEfnz59X8rfffpvhc1SoUEE7Vr169UzXBHhDvnz5lLx//34lP/PMM9qc+fPnK3njxo1Kjo6Odkttxs/XAwcO1MZk188mXIkDAAAAAABgAWziAAAAAAAAWACbOAAAAAAAABbAJg4AAAAAAIAF+Hxj499//13J9hpYHj9+PMPnXbRokZLtNS7r1atXhs8LuIuxkfHLL7+sZD8/P21Oz549HWbAit566y0lDxs2TMnGRsciIsnJyff9vE2bNtWOTZo06b7PC9jjSrNjAPZ98803Sk5ISMjwOUaMGKEdK1SoUKZrAsyQO3duJcfExGhjjF/ws3LlSiUfPHjQ6fMYP6OLiFStWlXJxs/SjRo1cnre7IIrcQAAAAAAACyATRwAAAAAAAALYBMHAAAAAADAAny+J87169eV7Er/m4IFCyq5bt262pgmTZoouWvXrhmuDXCX1atXa8eMfT+MPXDs/VxPnTpVycb7YgErio6OVnLx4sWV/MUXX2hzjPd3u+Krr75Sco8ePTJ8DgCA550+fVrJu3btuu9zPvXUU/d9DsAKKleu7DDj/nElDgAAAAAAgAWwiQMAAAAAAGABbOIAAAAAAABYgM/3xPnhhx8yPGfAgAFK/vjjj91VDuAW58+fV/LgwYOdzsmZU/11ULt2bW0MPXDgC9q1a+cwAwCyt4kTJyp50aJFTufkyZPH4TmM77MAILO4EgcAAAAAAMAC2MQBAAAAAACwADZxAAAAAAAALIBNHAAAAAAAAAvw+Q5bISEhSjY2JRMRGTFihJJHjhzp0ZqA+3X37l0lX7t2TRtTsmRJJY8aNUrJzz33nPsLAwAAyOKM75ECAwOVnJycrM0pW7askqOjo91dFgCICFfiAAAAAAAAWAKbOAAAAAAAABbAJg4AAAAAAIAF+HxPnGHDhjnMgBUZ7+WOj483qRIAAABref755x1mADATV+IAAAAAAABYAJs4AAAAAAAAFsAmDgAAAAAAgAWwiQMAAAAAAGABbOIAAAAAAABYAJs4AAAAAAAAFuC2rxhPTU21PebrjLO39H+/6f/ecQ9rwXewFhxjLfgO1oJjrAXfwVpwjLXgO1gLjrEWfIcn1oLbNnESEhJsj+vUqeOu0yKLS0hIkLJly5pdRpbCWvBNrAUda8E3sRZ0rAXfxFrQsRZ8E2tBx1rwTe5aC9xOBQAAAAAAYAF+aWlpae440c2bNyUuLk5ERIKDgyVnTrdd5IMsJjU11bZ7HBERIXny5DG5oqyFteA7WAuOsRZ8B2vBMdaC72AtOMZa8B2sBcdYC77DE2vBbZs4AAAAAAAA8BxupwIAAAAAALAANnEAAAAAAAAsgE0cAAAAAAAAC2ATBwAAAAAAwALYxAEAAAAAALAANnEAAAAAAAAsgE0cAAAAAAAAC2ATBwAAAAAAwALYxAEAAAAAALAANnEAAAAAAAAsgE0cAAAAAAAAC2ATBwAAAAAAwALYxAEAAAAAALAANnEAAAAAAAAsgE0cAAAAAAAAC2ATBwAAAAAAwALYxAEAAAAAALAANnEAAAAAAAAsgE0cAAAAAAAAC2ATBwAAAAAAwALYxAEAAAAAALAANnEAAAAAAAAsgE0cAAAAAAAAC2ATBwAAAAAAwALYxAEAAAAAALCAnGYXYCXHjx+Xn3/+WdavXy9xcXFy8uRJuXbtmhQoUEBKly4t9erVk969e0vjxo3NLhXwql27dsm8efNk9erVcvr0abl06ZIULVpUSpYsKQ8//LA0bdpUWrZsKSVLljS7VMBtLl26JLGxsbJjxw6JjY2VEydOyIULFyQhIUH8/PykcOHCUq1aNWnSpIk89dRTUqpUKbNLBryG1wX4Il4XAF1KSorMnTtXZs+eLfv375dz585J4cKFpVy5cvL4449LdHS0FCtWzOwyLcUvLS0tzewisrrdu3fLoEGDZPv27S6Nb9KkiUydOlVCQ0M9XBlgrvPnz8vLL78sM2fOdDp2yJAh8sUXX3ihKsA7OnToIMuWLXNpbO7cueW1116TN998U/z9uQgW2RevC/BlvC4AqoMHD0rv3r1l9+7d/zimePHiMmXKFGnXrp0XK7M2rsRxwR9//KFt4FSqVEmqVasmxYoVk8TERNm8ebOcOnVKRETWrVsn9erVk40bN0r58uXNKBnwuJMnT0qTJk3k2LFjtmPlypWTmjVrStGiRSU5OVkOHz4se/bskZs3b5pYKeB5JUqUkCpVqkhoaKjky5dPbty4IYcPH5YdO3ZIamqqpKSkyJgxY+T48eMSExNjdrmAR/C6APwfXhfg606dOiXNmzeXM2fOiIiIn5+fNGrUSCpUqCDnz5+X1atXS3Jyspw/f166dOkiK1askObNm5tctTWwiZMBFSpUkIEDB0qfPn20yx/v3r0rU6ZMkRdffFFu3LghZ86ckaioKNm8ebP4+fmZVDHgGVeuXJGmTZva3qjXrFlTPv/8c6lfv7429tq1a7Js2TLhoj9kN02aNJHOnTtLixYtpFy5cnbHnD17VoYOHSrz5s0TEZGpU6dKx44dpVu3bt4sFfA4XhcAXheA9KKiomwbOGFhYbJ06VKpXr267c8vXLggvXr1kl9++UVu374tPXr0kKNHj0pQUJBJFVsHt1O5YP369XLs2DHp27ev5MiRw+HYRYsWyeOPP27LP/30k7Ru3drTJQJe9a9//Uu+/fZbERFp1KiRrFixQvLmzWtyVUDWlJaWJs2aNZN169aJiEjLli1l1apV5hYFuBmvC4DreF1Adrd8+XJp3769iNy7dTA2NlYiIiK0cdevX5fq1avLn3/+KSIir732mrz77rterdWKuAHTBY0bN5bo6GinGzgiIl27dpU6derYsqv3xQJWsWfPHtsb9QIFCsjMmTN5ow444OfnJwMGDLDlXbt2mVgN4H68LgAZw+sCsrsvv/zS9rhfv352N3BERPLlyydjx4615UmTJklqaqrH67M6NnE8oEGDBrbHx48fN68QwAO+/vpr2+MBAwZI6dKlTawGsIbixYvbHiclJZlYCeB+vC4AGcfrArKra9euyS+//GLL/fv3dzi+e/fuUqBAARG59w1vGzZs8Gh92QGbOB6QvgfOnTt3TKwEcK87d+7I7NmzbTkqKsrEagDrOHDggO1xWFiYiZUA7sXrApA5vC4gu9q8ebPcunVLRO5daRMZGelwfEBAgDz66KO2vGbNGo/Wlx2wieMBcXFxtsdlypQxsRLAvfbt2ydXr14VkXu/lB955BG5deuWTJo0SRo3bizFixeXPHnySOnSpaVDhw4yefJkSUlJMblqwFxnzpyRDz/80JZpXonshNcFION4XUB2ln6DMiIiQnLmdP5dSjVr1rQ7H/bx7VRu9tdffym7hy1atDCxGsC9duzYYXtcuXJlOXr0qHTv3l327dunjDt9+rScPn1ali1bJuPHj5f58+crv5yB7C45OVmOHTsmK1askA8++EDOnz8vIiKVKlWSESNGmFwd4D68LgCu4XUBvuKPP/6wPXb1KrPQ0FDb44MHD7q9puyGTRw3GzZsmO0WqtDQUOnYsaPJFQHu89dff9ke+/v7S6tWreTkyZMiIlKlShWJjIyUHDlyyN69e21N+o4dOyaNGjWSjRs3yiOPPGJK3YCnbdq0SRo2bOhwTJs2bWTmzJlSqFAhL1UFeB6vC4B9vC7AV128eNH2uESJEi7NKVmypO3xpUuX3F5TdsMmjhtNnTpVFixYYMvjx4+XgIAAEysC3CsxMdH2ODY2VkREAgMDJSYmRnr06KGMXbt2rfTo0UMuXLgg169fl549e8r+/fslV65c3iwZMF1QUJB8+eWX0rt3b7NLAdyO1wUg43hdQHZ27do12+PAwECX5qQfl34+7KMnjpvExsbKoEGDbLlnz578Yka2c/36de3Y1KlTtTfqIiJNmzaVpUuXir//vV8zhw8flpkzZ3q8RsAMDzzwgAwZMkSGDBkizz33nPTt21fq1KkjOXPmlMTERImKipJmzZrJoUOHzC4VcCteFwD7eF2Ar7p586btce7cuV2ak/7Ch+TkZLfXlN1wJY4bHDt2TDp27Gj7gY2IiJBJkyaZXBXgfnny5FFyZGSkPPHEE/84vl69evL444/L/PnzRURkzpw5Eh0d7ckSAVOUL19evvjiC+34mTNnZOTIkRITEyNr166VRx99VNauXSs1atQwoUrA/XhdAOzjdQG+Kv3rgquN7P/+NisR16/e8WVciXOf4uPjpWXLlnL27FkRufcLe+XKldzbimwpf/78Su7atavTOenHbN682e01AVnZAw88IFOmTJEXX3xRREQuX74sTz75pK13GmB1vC4AGcPrArK79K8Lrl5Vk36c8XUFOjZx7sPFixelZcuWcvToURERCQkJkdWrV0tISIjJlQGeUbRoUSVXrVrV6Zz0Y5KSkiQpKcntdQFZ3fjx46VgwYIicu+rM1esWGFyRYB78LoAZA6vC8iu0r8unDt3zqU5f18QISJSpEgRt9eU3bCJk0lXr16VNm3ayP79+0Xk3g/rzz//LOXKlTO5MsBzqlSpomRXdsqNY3izDl+UN29eqV+/vi3/+uuvJlYDuA+vC0Dm8LqA7Kpy5cq2xydOnHBpzt/faiiiv65AxyZOJly/fl3atWtn+xaGggULysqVK+Whhx4yuTLAs6pVq6ZkV954G8dwqyF8VeHChW2P03/9JmBlvC4AmcfrArKj8PBw2+O4uDhJTU11OmfXrl1258M+NnEy6ObNm9KpUyfbbnnevHll+fLlUqtWLZMrAzyvXLlyUr58eVv+/fffnc5JP6ZIkSKSL18+j9QGZHXx8fG2x1wqjOyC1wUg83hdQHZUv35927dNXb9+3Xbhwz+5deuWbN261ZabNWvm0fqyAzZxMuD27dvSrVs3WbNmjYjc+yq0JUuWSIMGDUyuDPCe9A0pFy1a5HR8+jGNGjXySE1AVnfx4kXZsmWLLfOvTMhOeF0AMo7XBWRX+fPnl+bNm9tyTEyMw/ELFy60XaFZuHBhXhdcwCaOi+7cuSO9e/eW5cuXi4hIzpw5Zd68edKiRQuTKwO8a/DgwZIrVy4REYmNjZUffvjhH8du2bJFebPO18giu7h06ZLLY9PS0uT555+3fX1mQECAdOjQwVOlAV7H6wLA6wKQ3nPPPWd7PGXKFFsfWaMbN27Im2++acvPPvus5MyZ0+P1WR2bOC5IS0uTgQMHyvz580VExN/fX6ZPny6dOnUyuTLA+x588EHlF3N0dLTdN+xr166VTp06yd27d0VE5NFHH2XNINuYNm2aREZGyrRp0+Tq1av/OG7v3r3Stm1bmTNnju3Yq6++qn2jD2BlvC4AvC4A6bVv314aNmwoIiIpKSnSoUMHiYuLU8ZcvHhRunTpIkeOHBGRe7cUDh8+3Ou1WpFfWlpamtlFZHVfffWVDBkyxJYrVqworVq1cmlu0aJFZcyYMZ4qDTDFrVu3pGXLlrJx40bbsfDwcImMjJQcOXLI3r17ZefOnbY/CwkJkW3btkmZMmXMKBdwu08++USGDRsmIveuzKxSpYpUrlxZChcuLH5+fnLx4kXZu3ev7Y3J37p16yZz5szhX5mQ7fC6AF/H6wKgOnXqlNSpU8fW+8nf318aN24s5cuXl4SEBFm9erXcuHFDRO6tmZ9++km5DQv/jE0cF4wePTrTGzFhYWFy/Phx9xYEZAFXrlyRwYMHy+zZsx2Oq1u3rvzwww+8UUe2MnHiROXKA2cKFCggo0ePlqFDh0qOHDk8WBlgHl4X4Mt4XQB0Bw8elCeffFL27Nnzj2OCg4NlypQp0r59e+8VZnFs4riATRzgn23YsEGmTZsmmzZtktOnT8udO3ekRIkS8uijj0qPHj2kS5cu4ufnZ3aZgNsdOnRIVq9eLdu2bZP9+/fLyZMnJTExUUREChYsKCEhIfLwww9LixYtpFu3bpI/f35zCwa8hNcF+CpeFwBdSkqKzJkzR2bPni379++Xc+fOSVBQkJQvX166du0qAwYMkGLFipldpqWwiQMAAAAAAGABNDYGAAAAAACwADZxAAAAAAAALIBNHAAAAAAAAAtgEwcAAAAAAMAC2MQBAAAAAACwADZxAAAAAAAALIBNHAAAAAAAAAtgEwcAAAAAAMAC2MQBAAAAAACwADZxAAAAAAAALIBNHAAAAAAAAAtgEwcAAAAAAMAC2MQBAAAAAACwADZxAAAAAAAALCCnu0508+ZNiYuLExGR4OBgyZnTbadGFpOamioJCQkiIhIRESF58uQxuaKshbXgO1gLjrEWfAdrwTHWgu9gLTjGWvAdrAXHWAu+wxNrwW0/LXFxcVKnTh13nQ4WsX37domMjDS7jCyFteCbWAs61oJvYi3oWAu+ibWgYy34JtaCjrXgm9y1FridCgAAAAAAwALcdiVOcHCw7fH27dslJCTEXadGFhMfH2/bOU7/9457WAu+g7XgGGvBd7AWHGMt+A7WgmOsBd/BWnCMteA7PLEW3LaJk/4+vpCQECldurS7To0sjPs3dawF38Ra0LEWfBNrQcda8E2sBR1rwTexFnSsBd/krrXA7VQAAAAAAAAWwCYOAAAAAACABbCJAwAAAAAAYAFs4gAAAAAAAFgAmzgAAAAAAAAWQKtwwAd88cUX2rF33nlHyS+++KKSX3vtNY/WBAAAAADIGK7EAQAAAAAAsAA2cQAAAAAAACyATRwAAAAAAAALoCcO4APGjh2rHbtw4YKS33//fSVHRUVpc0JDQ91bGGABX331lXZsz549Sp48ebLT83Tu3FnJ9evX18a89NJLSs6dO7fzAoEs5O2331bym2++qY2JjIxU8ltvvaWNadeunXsLAwAgm+BKHAAAAAAAAAtgEwcAAAAAAMAC2MQBAAAAAACwADZxAAAAAAAALIDGxkA2tG3bNiUnJSU5nWNsIkkTY+Ce8ePHa8dOnz6tZD8/P6fnWbp0qcMsIrJx40YlDxs2TMnNmjVz+jyAmerUqaNke2vj5MmTSjb+3IvQ2BgAsgvj55Bvv/1WG/Pyyy8r2d5rR1hYmJJXrlyp5EqVKmW2RMvhShwAAAAAAAALYBMHAAAAAADAAtjEAQAAAAAAsAB64rjBpUuXtGPr169X8oIFC7QxM2fOVHLr1q2V/MMPP2hzChQokJkS4WP++OMPJd+6dcvpnNdee81T5QBw0bJly5Rs7BUyadIkbU7Pnj09WhPgbkuWLFGysY8OAMC6du7cqeTOnTsrOT4+Xptj7IHjSj+1GTNmKHns2LEZqtPKuBIHAAAAAADAAtjEAQAAAAAAsAA2cQAAAAAAACyAnjguuH37tpJXr16t5K1bt2pzMnNPnvG77idMmOCW8yJ7u3nzpnbsk08+cTqvY8eOSo6IiHBXSfdtx44dSh4/frw25vz580oODAzUxnz33XdKDg0NdUN18DU1a9bUjpUuXdrhnJSUFO3Y7t27M/zcV69eVfKgQYOczqFHDrzp8uXLSn7//fedzilbtqyHqgEyLikpSTs2f/58JR84cCDD501LS1PyhQsXtDFTp05Vcr9+/bQx7733npJLlCiR4VoATzH2vxERGTNmjJLt9cBxh8mTJyt58ODB2piQkBCPPLfZuBIHAAAAAADAAtjEAQAAAAAAsAA2cQAAAAAAACyATRwAAAAAAAAL8LnGxsYmY8ZGZR988IE2Jy4uTsm7du1yf2F2XL9+3SvPA2u7du2adsyVBqrDhw/3RDlOGRuFi+jNWr///nu3PFfjxo2VbFzL+fPnd8vzIHtbsmRJhuckJydrx/73v/8peeTIkdqY48ePK/nOnTtKvnLlijZn3LhxSqaxMbxp4cKFSl63bp2SH3roIW1O3rx5PVkSYHPo0CHt2Mcff6zkjRs3amMy08jYyPiZw8/PTxtjPDZt2jRtTGxsrJKN72UAbzI2Au/cubM2xlONjI2MX3LSvn17bcyAAQOUbK/5cY4cOdxbmBdwJQ4AAAAAAIAFsIkDAAAAAABgAWziAAAAAAAAWEC27oljr3fASy+9pOSYmBjvFJMJf/75p9klwAJu3rypHbN333VW8dVXX2nHpkyZomRj/X369NHmPPDAA0q+deuWNubrr79WcrNmzZS8aNEibU6pUqW0Y0BGBQYGasd69OjhMIuIFCpUSMnGe88BMxl7NImIrFy5UsnGPiDG910i9COD5+zcuVPJzZs318YYf68af2ZFstb7qN9//93sEgCbTp06KTkz/W/s9dFp1KiR03knT55U8qeffqrk3377TZvz4osvKtnfX7+G5bnnnnP63FkNV+IAAAAAAABYAJs4AAAAAAAAFsAmDgAAAAAAgAWwiQMAAAAAAGAB2bqx8fvvv68dy0wj44CAACWHh4cruWTJktqcVq1aKbldu3bamG+++UbJH330UYZrA55++mmzS3Do4sWLSv7vf//rdE6DBg2U/Mknn2hjihQp4vQ8Dz74oJKNzc1OnDihzaGxMbxl7dq12rGUlBQTKgFcY6/B6oIFC5RcvXp1JXft2tWjNQHpffzxx0o2szl8njx5lGyvmf0vv/yi5DNnzjg97759+5RcrVq1TFQH6D9vxs+mIiJjx45VsrERuL0m4MYvH3nmmWeU/Oabb2aozn9Svnx5JRvf54vo9T7//PPaGOPvieHDh7uhOs/iShwAAAAAAAALYBMHAAAAAADAAtjEAQAAAAAAsIBs3ROnbt262rGXX37Z4Zzg4GDtWKdOnZRctWrVDNdivB9PROTo0aMO5wwdOjTDz4Ps7/jx40retWuX0zmRkZEuHfMEYz+oU6dOaWMKFiyo5HfffVfJrvS/scd4D+6YMWOU/OOPP2pz6tevn6nngm9LSEhQ8qxZs7QxS5cuVfLGjRu1Mampqe4tDHAj4/she0JCQpRcuHBhT5UDaMqVK5fhOfny5dOOPffccw7nGPtjiujvd4y9Qux9xmjTpo2SXemJ89NPPymZnjjILOP7knHjxmlj7PW8Sc/e64KxN5q7euAYDR48WMn+/vr1KZMnT1by3r17tTHGnrlRUVFKLl26dCYr9ByuxAEAAAAAALAANnEAAAAAAAAsgE0cAAAAAAAAC8jWPXE6d+7s0jFvSE5O1o4tWbJEyUFBQUrOzH29yP6MPWUuXrzodM7rr7+uHcuVK5fbavrboUOHtGPbt29Xsr17a5999lklN2zY0C315M6d2+Fzd+jQwS3PA+u4ffu2duyHH37I8Hk+/fRTJZ84cULJ58+fz/A5XVGiRAnt2JdffumR5wKMjD/nIiJFixZV8vDhwzN83p07d2rH4uPjlczva7jC2NOjQoUK2piyZcsquXHjxp4syebIkSPasT/++EPJ9npoAp4ybdq0DM8ZMWKEkkePHq2NMb7/9pQcOXIo2V4vq65duyq5Zs2a2hjj55e5c+cq+d///ndmS/QYrsQBAAAAAACwADZxAAAAAAAALIBNHAAAAAAAAAtgEwcAAAAAAMACsnVj46zkhRdecDpm5MiRSg4LC/NUOchGXGmC99hjj3mhEpEBAwY4HWOvoXLHjh09UY7W9NleU1tkb3/99ZeS7TVHjYuL81Y5GWZsZDxr1ixtjLeacsL37Nixw+mY0qVLK7lJkyZO53z11VdKttcM2fj7unDhwkp+//33tTlPPfWU0+eGb+nXr59pz/3ZZ58p+fPPP9fGnDx5Usn2vvzBqF27dvdXGJABxjX07rvvmlRJ5oSEhCjZlabLX3zxhZJpbAwAAAAAAIBMYRMHAAAAAADAAtjEAQAAAAAAsAB64niIsQ/D6tWrtTGBgYFK7tGjh0drQvawcOFCJbty/7SnGHvOnD592umcp59+Wjvmjp49N27c0I517txZyVeuXLnv54G1pKSkKDkr979xhfHebsCdkpOTlfzWW28p2V4PNmOvgN27dyu5TZs22hzj7+v8+fNrY4y/r8+dO6fk6Ohobc727duVbOxrAHjSDz/8oOSXXnpJye56v1a1alW3nAe+Zc+ePdox4+dVe6pXr+6Basxj/GwgIvLll18qOTExUcnr16/X5pjdj5ArcQAAAAAAACyATRwAAAAAAAALYBMHAAAAAADAAuiJ4yHdunVT8smTJ7Uxn332mZJDQ0M9WhOyh3379pldgk1CQoKST5w44XTO448/7pFaDh48qB3bvHmzksuUKaPkKlWqeKQWZB05c6ovc6VLl9bGXL161WG2JyAgQMnBwcGZqE5n7Ctl7ANifN0QEfnvf/+rZGO/NcBV8+fPV/LKlSuVbK+nh7GnwogRI5RcpEgRbc6CBQuUbK8v2uLFi5U8ZswYJR8+fFibM3HiRCW/+eab2pjixYtrxwB3cOU9kDsMHTrUYRYRKV++vFdqgXV899132jHje44+ffpoY4y9nayuYcOG2jFj/zRjTzZjrzcReuIAAAAAAADABWziAAAAAAAAWACbOAAAAAAAABbAJg4AAAAAAIAF0NjYDWbMmKEd27Fjh5KfeuopbcwLL7zgsZqQff38889Kttdo0ltiY2MzPMfYXDizVq1apWRjM3F71q1bp2R7DTeRvYSFhSnZXpP5LVu2KNmVn2vjz3GXLl0yXpwdxtcK4+vL119/rc1p3bq1kjt37uyWWgCjwoULa8f+/PNPJd++fVvJy5cv1+aUK1fO6XMZ15Qxv/3229qct956y+kYe83BAXcYPHiwkv/zn/945Hk+//xzJfv76/8m//HHH3vkuWFdS5cu1Y4ZP0N06tTJW+WYZty4cdoxMz9LZRZX4gAAAAAAAFgAmzgAAAAAAAAWwCYOAAAAAACABdATJxOMfTUGDRqkjQkODlbyu+++68mSAFPs27fPK8/z0UcfaccmTJig5OvXr2tjGjVqpOTSpUu7tzBkC/Xq1XOYvenmzZsZnjNy5Egl0xMHnpI7d27tWO3atZVcqVIlJbvS/yYzjP2u7MmXL59Hnhuwx/jzdvfuXSUb+2WK6J8X5s+fr40x9vC4evWqkmfPnq3NMfbnMa5LZH9nzpxRckpKijambNmySq5Ro4YnS8oSrly54nRMQECAkkNCQjxVTqZxJQ4AAAAAAIAFsIkDAAAAAABgAWziAAAAAAAAWACbOAAAAAAAABZAY2MXGBtDde3aVcn2GqrOmjVLyYULF3b6PKmpqUrOmZO/HugaNmyo5I0bNzqds2DBAu3YM888c9+1GBvypaWlOZ2za9cup8fee+89Je/du9fpee01GJ84caLTeYAnXLt2TTsWExOjZGNDYhGRpKQkT5UEeMSzzz5ryvPu3LnT6ZiiRYt6oRLANZGRkU7HvPLKK9oxY/PZ559/XskJCQnanK+//lrJ9r4gAtnbsmXLlHzp0iVtzPnz55W8YcMGbUzFihXdW5iXGd9X3blzx+mcV199Vck9e/Z0a03uwJU4AAAAAAAAFsAmDgAAAAAAgAWwiQMAAAAAAGABNF0xmDp1qnbsv//9r5ITExOdnqdz585KzpUrlzbG2Fvn6tWrDp9XRKRq1apOnxvZ24gRI5S8adMmp3OGDh2qHTt8+LCSjT+PlSpV0ubkzp1byS1atFCyn5+fNsfYJycqKspxsXbYO290dLSSR48erY25cOGCkmfOnKnk7t27a3NKlSqV4fpgHbdv39aOLV26NMPn+eSTT5Rs7J1m757rkydPZvh5XDF8+HCPnBcw/v6217PJ2NOsZs2aHqll9+7dSp49e7Y2pnHjxkq2118EsBrjexVjvw57Dhw44KlyYBHG9/GBgYHaGOPv9Lffflsb8/TTT7u3MA9asmSJdsz4fi0+Pl4bY+w71adPH3eW5RFciQMAAAAAAGABbOIAAAAAAABYAJs4AAAAAAAAFpCte+KcPn1aO7Z161YlT5kyRcnLly/X5hjvCa9YsaKS8+XL57SWI0eOaMfmzZvncM66deu0YyNHjlTyG2+84fS5kb08+uijSq5fv742ZvPmzUq+deuWNsbYc+nDDz9Usr0+NCVLllRygQIFHBfrQcb+CPZ6+Njr35BeZGSkdoyeOFnX//73PyW/++67GT6HvV41sbGxma7J27p166Yd69ixowmVIDtq0KCBkgsVKqTkK1euaHN+//13JWemJ469XlXr169XsrEPmrHnmYhI27ZtM/zcgNWcOHFCyfberwFGxs+zIiIhISFKXrx4sZeqyRxj/8FvvvlGyWPHjtXmGP9321svPXv2VLLxs35WxJU4AAAAAAAAFsAmDgAAAAAAgAWwiQMAAAAAAGABbOIAAAAAAABYgGUbG9trzrRw4UIlDxs2TBvz119/OTxv7ty5tWOtW7dW8qxZs5ScP39+h+cUETl8+LB27MaNG0p+9dVXlWyvsbGxGW3p0qW1MX379lVyjhw5nNYH6yhSpIiSjc1eRUTeeustJX/99dfamNTU1Aw/99mzZx1me4xN0y5fvqyNsdd42Znjx48ruXjx4toYY5Pi3r17KzksLCzDzwvzGBuZbtu2zaRKPKdy5cpKfv3115Vsr4lxUFCQJ0uCDylfvrySjV+eYHyfIiIyYcIEJScnJyu5V69e2pydO3cq2d6XShjf7xgNHDhQOzZ06FCHcwArGjBggNklwIIaN26sZHvvFc6fP69ke5+Ta9So4da6/skPP/yg5L1792pjvv32WyUb63elyfeoUaO0Y8YvDrICrsQBAAAAAACwADZxAAAAAAAALIBNHAAAAAAAAAuwbE+cL7/8Ujv2wgsvZPg8oaGhSh49erQ2pn///hk+r1HFihWdjlm1apWSP/vsM22M8X5ve7UZ7y2fN2+eKyXCogoXLqwdM/7s9OnTRxvz+++/K/nXX391+lyrV69WcokSJZQ8YsQIbU69evWUfObMGW2MsYfCjz/+qOQOHTpoc4w9cIz9b0REAgMDtWOANxQsWFA71qNHDyXXr19fGxMVFaXkXLlyubcwIAP+/e9/KzkmJkYbs2/fPiUPGjRIyR999JE259ixY0pOSUnRxhQtWlTJQ4YMUfLw4cO1Ofb6GgJWYnw/JCKybNkyJRv7gtp7nWjRooV7C4PllS1bVjt26tQpJdvre7Z//34lN2rUyOlznThxQsn2PtMabd26Vcn2+tsY31sZ30fZ6xtorLdmzZraGCu+dnAlDgAAAAAAgAWwiQMAAAAAAGABbOIAAAAAAABYAJs4AAAAAAAAFmCZxsbffvutkt95551MnadcuXJKnjRpkpJbtmyZqfN6gr1GzeXLl1fy7t27tTHGZoBAnTp1nB6Ljo72Si3GZsj22Gv4CtSuXVvJxp+lc+fOea2Wl156SckPPPCAktu3b6/NCQ8P92RJgMdt3rxZO7ZgwQIljx07VsmHDh1yet4mTZpox4wNKocNG+ZChYC1JCUlKblfv37amAsXLijZ2PC1atWq2hxjU3Jg5syZ2rEnnnhCydu3b9fGvP7660o2Nta214A4M8qUKaNkf3/9WpO33npLyd767JIVcSUOAAAAAACABbCJAwAAAAAAYAFs4gAAAAAAAFiAZXriGO+xPnv2rNM5Xbp00Y69//77Sq5UqdJ91eVJ9u4x7NChg8MMANlVRESEkuPj402qBPBNBQoU0I4ZexL4co8CwBFj/xsRfb0sWbLE6XkCAgKUbK+HJmBUunRp7djChQuVPHfuXG3M0qVLlbxu3TqnzzV06FAlh4aGOp1j7DUIx7gSBwAAAAAAwALYxAEAAAAAALAANnEAAAAAAAAswDI9cYzat2+vHevWrZuS+/Tpo43JlSuXx2oCAAAAgOnTpyt50aJF2hhXeuAYGXuHDBgwIMPnAEREQkJClGyvLw29arImrsQBAAAAAACwADZxAAAAAAAALIBNHAAAAAAAAAtgEwcAAAAAAMACLNPY+OTJk2aXAAAAAMDH7du3Tzv29ttvK3nevHlK9vPzy9RzRUZGKvmNN97I1HkAZB9ciQMAAAAAAGABbOIAAAAAAABYAJs4AAAAAAAAFmCZnjgAAAAAYLZq1appx+bMmeMwA4C7cCUOAAAAAACABbCJAwAAAAAAYAFs4gAAAAAAAFgAmzgAAAAAAAAWwCYOAAAAAACABbCJAwAAAAAAYAFu+4rx1NRU2+P4+Hh3nRZZUPq/3/R/77iHteA7WAuOsRZ8B2vBMdaC72AtOMZa8B2sBcdYC77DE2vBbZs4CQkJtsd16tRx12mRxSUkJEjZsmXNLiNLYS34JtaCjrXgm1gLOtaCb2It6FgLvom1oGMt+CZ3rQVupwIAAAAAALAAv7S0tDR3nOjmzZsSFxcnIiLBwcGSM6fbLvJBFpOammrbPY6IiJA8efKYXFHWwlrwHawFx1gLvoO14BhrwXewFhxjLfgO1oJjrAXf4Ym14LZNHAAAAAAAAHgOt1MBAAAAAABYAJs4AAAAAAAAFsAmDgAAAAAAgAWwiQMAAAAAAGABbOIAAAAAAABYAJs4AAAAAAAAFsAmDgAAAAAAgAWwiQMAAAAAAGABbOIAAAAAAABYAJs4AAAAAAAAFsAmDgAAAAAAgAWwiQMAAAAAAGABbOIAAAAAAABYAJs4AAAAAAAAFsAmDgAAAAAAgAWwiQMAAAAAAGABbOIAAAAAAABYAJs4AAAAAAAAFsAmDgAAAAAAgAWwiQMAAAAAAGABbOIAAAAAAABYAJs4AAAAAAAAFsAmDgAAAAAAgAWwiQMAAAAAAGABbOIAAAAAAABYAJs4AAAAAAAAFsAmjpsMGzZM/Pz8bP+VLVvW7JIAj7h06ZKsWrVK3nnnHenatavUrFlTQkNDJTAwUPLmzSulSpWS1q1by/jx4+X06dNmlwt4xYYNG+SZZ56R8PBwCQoKksDAQClfvrx06dJFZs+eLampqWaXCHhMTEyM8h7Ilf/efvtts8sGPOL48eMyefJk6dOnj9SoUUMKFy4suXLlkiJFikj16tXl2WeflfXr15tdJmAqPjvfH7+0tLQ0s4uwuu3bt0u9evXk7t27tmNhYWFy/Phx84oCPKRDhw6ybNkyl8bmzp1bXnvtNXnzzTfF3589Y2Q/ly9flqeeekp+/PFHh+Nq1aolM2bMkCpVqnipMsB7YmJipH///hmaM27cOHnjjTc8VBHgfbt375ZBgwbJ9u3bXRrfpEkTmTp1qoSGhnq4MiBr4bPz/ctpdgFWd/v2bRk4cKDyQwj4ihIlSkiVKlUkNDRU8uXLJzdu3JDDhw/Ljh07JDU1VVJSUmTMmDFy/PhxiYmJMbtcwK0SExOlfv36cvDgQdux8uXLS7169SQwMFCOHj0qmzZtktu3b8vOnTulSZMmsnXrVv61CdlalSpVpHnz5k7HRUZGeqEawHv++OMPbQOnUqVKUq1aNSlWrJgkJibK5s2b5dSpUyIism7dOqlXr55s3LhRypcvb0bJgNfx2dk92MS5T++//77ExcWJiEjv3r1l1qxZJlcEeFaTJk2kc+fO0qJFCylXrpzdMWfPnpWhQ4fKvHnzRERk6tSp0rFjR+nWrZs3SwU8auDAgbYNnMDAQPnmm2+kT58+ypjjx49LVFSUbN68Wc6dOyfdunWT2NhY8fPzM6NkwOPq1q0rX3zxhdllAKapUKGCDBw4UPr06SOlSpVS/uzu3bsyZcoUefHFF+XGjRty5swZ22sErwvwBXx2dg/ub7gPBw8etN3THRUVJS1btjS5IsDzXnnlFfnXv/71jxs4IiIlS5aUOXPmSJMmTWzHJk2a5IXqAO/YvXu3LFiwwJZjYmK0DRwRkbJly8pPP/1k+1fWXbt28YYFALKhkJAQmTJlihw8eFCGDx+ubeCIiPj7+8vTTz8tM2bMsB3bunWrrFq1ypulAqbgs7P7sImTSWlpaTJw4EC5deuWFC5cWD766COzSwKyFD8/PxkwYIAt79q1y8RqAPf6+yozEZEaNWpIjx49/nFsgQIFZOTIkbb86aeferQ2AID3NW7cWKKjoyVHjhxOx3bt2lXq1Kljy672GgSsis/O7sUmTiZNnDhRfv31VxERmTBhghQvXtzkioCsJ/26SEpKMrESwL22bt1qe9y2bVun49u1a2d7vGPHDjl58qRH6gIAWEODBg1sj2noiuyOz87uxSZOJpw6dUpGjBghIiINGzZUrjYA8H8OHDhgexwWFmZiJYB7nTt3zvbYlZ/tkiVLSkBAgC2vWbPGI3UBAKwhfQ+cO3fumFgJ4Fl8dnY/GhtnwuDBgyUpKUly584tkyZNohEZYMeZM2fkww8/tGWaGiM7SUtLsz129TUg/Zz9+/e7vSYgK0hMTJR58+bJ77//LleuXJFChQpJqVKl5LHHHpPw8HCzywOyjL+bu4qIlClTxsRKAM/is7P7sYmTQXPmzJEff/xRRESGDx/OGxIgneTkZDl27JisWLFCPvjgAzl//ryI3PuKzb934IHsIDg42PbNVK5cBn/27FlJSUmx5fRXqQHZyZIlS2TJkiV2/ywiIkLeeOMNhz2kAF/w119/KVdktmjRwsRqAM/hs7NncDtVBly8eFGGDh0qIiIVK1ZUGlUCvmjTpk3i5+dn+y9v3rzy0EMPySuvvGLbwGnTpo1s2bJFChUqZHK1gPvUrl3b9njFihVOxy9fvlzJly5dcntNQFYXFxcnPXv2lP79+0tqaqrZ5QCmGTZsmO0WqtDQUOnYsaPJFQHux2dnz2ETJwOGDRtm+2A6adIkpb8BAFVQUJDMnDlTVqxYIUWKFDG7HMCtOnXqZHv822+/Kd9WZZSUlCTvvPOOdgzITsqVKycjRoyQ1atXy5kzZ+TWrVty7do12b9/v3z00UcSGhpqGxsTEyMvvPCCidUC5pk6daosWLDAlsePH89nCmRLfHb2HL+09Dfp4x+tWrVKWrduLSIi/fr1k5iYGG1MTEyM9O/fX0TuNbqk0zyyuz///NP2FYFpaWmSlJQkf/zxh+zatcv2r6xNmzaVr7/+WipVqmRmqYDbPfbYY7ZvWsibN69MnjxZevfurYw5ceKE9O7dWzZv3qwcf/DBB+XIkSNeqxXwpMTERClYsKD4+//zvw0mJSVJ7969bZfVi4hs2LBBGjZs6I0SgSwhNjZWGjZsKDdv3hQRkZ49e8qcOXNMrgpwPz47exabOC64fv26VKtWTY4fPy5FixaVgwcPSrFixbRx/CAC95w5c0ZGjhxp+4VduHBhWbt2rdSoUcPcwgA3On78uERGRsqFCxdsxx588EGpV6+eBAYGytGjR2Xjxo1y+/Zt8ff3lzZt2thuq3r44Ydl9+7dZpUOmCI5OVkefvhhOXTokIiItG/fXtnUAbKzY8eOSf369eXs2bMicq9H1MaNG7ndHNkOn509j9upXDBy5EjbD9V///tfuz+EAP7PAw88IFOmTJEXX3xRREQuX74sTz75JF+hiWylbNmysmXLFmVz8ujRozJjxgyZPHmyrFmzRm7fvi358+eXadOmSWRkpG1cUFCQCRUD5goMDJThw4fb8tq1a+XWrVsmVgR4R3x8vLRs2dK2gVO+fHlZuXIlGzjIlvjs7Hls4jixa9cu+fzzz0Xk3m0h/fr1M7kiwDrGjx8vBQsWFJF738bjSgNYwEoqVKggu3btkrlz58oTTzwhYWFhEhgYKAUKFJBq1arJiBEjZP/+/RIVFaX8C1Pp0qXNKxowUfPmzW2Pb9y4ISdOnDCxGsDzLl68KC1btpSjR4+KiEhISIisXr1aQkJCTK4McD8+O3sHXzHuxN69e+Xu3bsiInLy5El59NFH/3FsQkKC7XF8fLwydtSoUdK+fXvPFQpkQXnz5pX69evLTz/9JCIiv/76q3To0MHkqgD38vf3lx49ejj92uQdO3bYHqe/KgfwJcYPrhcvXjSpEsDzrl69Km3atJH9+/eLiEjRokXl559/lnLlyplcGeAZfHb2DjZxMuDo0aO2XXRnUlJSZNu2bbac/ocU8CWFCxe2PebNOnxVfHy8HDx40Jbr169vYjWAea5fv67kfPnymVQJ4FnXr1+Xdu3aSWxsrIiIFCxYUFauXCkPPfSQyZUB3sFnZ8/hdioAHhUfH297zFeNw1fNnDnT9i9TVapUkdq1a5tcEWAOY0NvbilBdnTz5k3p1KmT8g2Gy5cvl1q1aplcGYDsgE0cJ6KjoyUtLc2l/6ZMmWKbFxYWpvxZdHS0ef8jAJNcvHhRtmzZYsvh4eEmVgOY48qVKzJhwgRbHjRokInVAOZK/14pPDxcgoODTawGcL/bt29Lt27dZM2aNSIiEhAQIEuWLJEGDRqYXBngeXx29g42cQC47NKlSy6PTUtLk+eff972zSMBAQH0w4HPuXPnjkRHR8v58+dFRKRq1aoyePBgk6sC3OfatWsuj120aJHMnDnTlvv06eOJkgDT3LlzR3r37i3Lly8XEZGcOXPKvHnzpEWLFiZXBiA7YRMHgMv+/prkadOmydWrV/9x3N69e6Vt27YyZ84c27FXX31VihYt6o0yAa9YtWqVjB49WvnWqfQOHDggLVu2lMWLF4vIvY3M77//XnLnzu29IgEPmz9/vtStW1dmzJjxj68LSUlJMm7cOOnRo4ekpaWJiEiZMmVk6NCh3iwV8Ki0tDQZOHCgzJ8/X0TuNb2fPn26dOrUyeTKAGQ3NDYGkCGxsbHSr18/yZkzp1SpUkUqV64shQsXFj8/P7l48aLs3btXjhw5oszp1q2bvPXWWyZVDHjGpUuXZMyYMTJmzBipUqWKRERESJEiReTKlSvy+++/y969e21jAwICZPHixVK3bl0TKwY8Y/v27dK3b1/JlSuXhIeHS+XKlSUoKEhSU1Pl5MmTsmXLFrlx44ZtfOHChWX58uU0NUa2MnHiRImJibHlBx98UDZt2iSbNm1yOrdo0aIyZswYD1YHIDthEweAywICAmyPU1NTZd++fbJv375/HF+gQAEZPXq0DB06VHLkyOGNEgFTHDx4UPn2qfRq1aolkyZNoqElsr3bt2/L3r17lQ1Mo2bNmsn3338vYWFhXqwM8Ly/b5v92+HDh+Xw4cMuzQ0LC2MTB4DL2MQB4LLBgwdL8+bNZfXq1bJt2zbZv3+/nDx5UhITE0Xk3tdnhoSEyMMPPywtWrSQbt26Sf78+c0tGvCQDh06yKJFi+SXX36Rbdu2SXx8vCQkJEhgYKCEhIRInTp1pHv37tK2bVs2MZFtPfnkk1KxYkXZsmWLbNmyRY4ePSoXL16Uixcvyt27dyUoKEgefPBBqVevnvTq1YtvZgMA4D75pf19czIAAAAAAACyLBobAwAAAAAAWACbOAAAAAAAABbAJg4AAAAAAIAFsIkDAAAAAABgAWziAAAAAAAAWACbOAAAAAAAABbAJg4AAAAAAIAFsIkDAAAAAABgAWziAAAAAAAAWACbOAAAAAAAABbAJg4AAAAAAIAFsIkDAAAAAABgAWziAAAAAAAAWEBOd53o5s2bEhcXJyIiwcHBkjOn206NLCY1NVUSEhJERCQiIkLy5MljckVZC2vBd7AWHGMt+A7WgmOsBd/BWnCMteA7WAuOsRZ8hyfWgtt+WuLi4qROnTruOh0sYvv27RIZGWl2GVkKa8E3sRZ0rAXfxFrQsRZ8E2tBx1rwTawFHWvBN7lrLXA7FQAAAAAAgAW47Uqc4OBg2+Pt27dLSEiIu06NLCY+Pt62c5z+7x33sBZ8B2vBMdaC72AtOMZa8B2sBcdYC76DteAYa8F3eGItuG0TJ/19fCEhIVK6dGl3nRpZGPdv6lgLvom1oGMt+CbWgo614JtYCzrWgm9iLehYC77JXWuB26kAAAAAAAAsgE0cAAAAAAAAC2ATBwAAAAAAwALYxAEAAAAAALAANnEAAAAAAAAsgE0cAAAAAAAAC2ATBwAAAAAAwALYxAEAAAAAALAANnEAAAAAAAAsgE0cAAAAAAAAC2ATBwAAAAAAwALYxAEAAAAAALAANnEAAAAAAAAsgE0cAAAAAAAAC2ATBwAAAAAAwAJyml0AAAAAAFjF6dOntWMjRoxQ8g8//KDkGjVqaHPatWun5FdffVUbkzdv3syUCCAb40ocAAAAAAAAC2ATBwAAAAAAwALYxAEAAAAAALAAeuIAAOBmx44d046NHTtWybNmzdLGDBs2TMnvvfeeewsDAGTYwoULlfz0009rYxITE5VcrVo1JcfHx2tzRo8ereRt27ZpY3788Ucl+/vzb/DwDHt9m4w/b++//76SW7Vq5dGaYB+/BQAAAAAAACyATRwAAAAAAAALYBMHAAAAAADAAizTE2fRokVKTkhIcDonKChIO9ajRw93lQRkCVeuXNGOvf3220pOTk7Wxhjv1d63b1+Gn3vVqlVKPnLkiDYmLS1NyX5+fhl+Hnuee+45JT/44IPamKioKCUXL17cLc8NGE2ePFnJI0aM0MZcunTJ6Xk++OADJefOnVvJb731ljYnR44crpQIAMikIUOGKLlkyZLamI0bNyo5PDxcycaeOSIibdu2VfKKFSu0McbXBXuvL4A72HuPvmfPHiW/8cYbSvZUT5zLly9rxwoXLuyR57IirsQBAAAAAACwADZxAAAAAAAALIBNHAAAAAAAAAtgEwcAAAAAAMACsmxjY2Mj4759+yr5+vXrTs+RJ08e7VhKSoqS+/Tpk4nqvOPgwYPasVu3bmX4PKVLl1Zy0aJFM10Tsh5jsz0RkdmzZ2f4PO5oQOzKHHc1Nv7qq6+cnve7775T8k8//aRk49oAXBUTE6NkY6PtChUqaHMmTJig5IIFC2pjnn32WSWPGzfO4fOI2G+wCVid8f3aiRMnnM45evSokufNm5fh561fv752bODAgRk+D7KXmzdvKvmTTz7Rxhi/MMLI3vvvb7/9VsmPPPKINmbq1KlKfuWVV5ScM2eW/TgHi3n66ae1Y6+++qqSf/vtN48899ChQ5W8Zs0abczy5cuVXKZMGY/UYgVciQMAAAAAAGABbOIAAAAAAABYAJs4AAAAAAAAFpBlb6JMSEhQsis9cIyM96+KiJw9ezbTNXna6tWrlfzEE09oYxITEzN83qZNmyrZ3j2GsK5hw4ZpxypVqqTkM2fOaGOMvQPi4uKUbK+nR6tWrTJcn7HXTpcuXbQxDzzwgMNzTJkyRTs2YsQIp8/9xx9/KHn37t1KpicOXGHvZ+2DDz5QsvG+7A0bNmhzgoODnT6X8byXLl1ypUTAUoy/i1etWqWNWbdunZKNPc08xd7rDT1x8Nhjjyk5X758bjlv9erVldyzZ09tjLHP4c8//6zktm3buqUW4IUXXtCOGfv5nTp1Ssnz58/X5nTv3j3Dz23sGbVv3z5tzFtvvaXk77//PsPPk11wJQ4AAAAAAIAFsIkDAAAAAABgAWziAAAAAAAAWACbOAAAAAAAABaQZRsb+wJjk2JjY7LU1FS3PE9ycrJbzoOsqVatWi4dsxJjk8u5c+dm6jwRERFK7tixY6Zrgu+YMWOGkj/55BNtjLF5+I8//qhkV5oY2zN69Gglr127VsnFihXL1HkBbzl9+rSSjetJxLXG9JlRuHBhJVerVs3pnNq1ays5Mw38kf0ZG74aGx27i/G1xR7j6wKNjeFJAwYMUPKYMWOUPG7cOG1OZhobu/LFOz/88IOSaWwMAAAAAACALI1NHAAAAAAAAAtgEwcAAAAAAMAC6Iljog8++EDJ7uqBkydPHiW//PLLbjkv4CnPPfeckmfPnq3kq1evOj3HK6+84tIxwJlZs2Yp+datW9oYY3+lChUquOW527Vrp+Tw8HAl58zJyzaylvXr1yv5xRdfVPLevXszdd7q1asruVSpUkru1auXNqdmzZpKdqUnDuAKb/VKql+/vleeB3CVs158+/bt047Nnz9fya70yDl37lzGCvNxXIkDAAAAAABgAWziAAAAAAAAWACbOAAAAAAAABbgczfXL1u2TMnPP/+8ko39ZDzp8OHDHjlvjRo1lPzEE0945HmA69evK/nYsWPamHfffVfJc+bMcXrefPnyKblq1araGGPfkoiICKfnBez597//reQVK1YouUOHDtqc9957zyO1zJ07V8lPPfWUkt955x1tDr2f4C2fffaZduw///mPku31kHJm+PDhTs9bpEiRDJ8XAHB/IiMjlZw7d24lp6SkaHOSkpI8WhO4EgcAAAAAAMAS2MQBAAAAAACwADZxAAAAAAAALIBNHAAAAAAAAAvwucbGly9fVvLdu3c98jy///67kvft26eNOXXqlEeeG/CWHTt2KLl58+bamLS0NCX7+flpY2rXrq3kb775RsnGZt1AZtn7vWv8eStbtqySJ0yYoM3JkSPHfddy9OhR7dirr76qZGPDwNTU1Pt+XsBVxkbGxmbDIplrZGxkb10uXrxYyUWLFlVy586d7/t5AQCO1a1bV8ndunVT8uzZs7U5xi9/6NSpk5KNv89FRPLnz++0FuNnihs3big5b968Ts+RXXAlDgAAAAAAgAWwiQMAAAAAAGABbOIAAAAAAABYQJbtiTN9+nSPnLdDhw5KzpMnj5Lt9Rsw3qv922+/KfnLL7/U5hw7dkzJR44cyVCdripYsKB2zN4964Azf/zxh5LfeecdbcyqVauUbOzXERgYqM0x9vh48MEHtTFdu3ZVcr58+RwXC2RSv379tGPXrl1T8vvvv6/kKlWquOW5t27dquRWrVppY5KSkhyeIywszC21APacPn1ayZ9++qmS3dH/xp6ZM2c6PZYrVy4lv/3229oc3v8AgGfFxMQo2V5PnEOHDim5S5cuSt64caM2Z8iQIUru37+/Nub69etK/vDDD5X85ptvanOyK67EAQAAAAAAsAA2cQAAAAAAACyATRwAAAAAAAALyLI9cfr27avkTZs2ueW8xj4fxp4eycnJ2pwvvvjCLc/tCfb6Ozz++OMmVIKsLDExUTs2Y8YMJb/44otK9vPz0+YY+3EYe3oMGzZMm/PII4+4WibgccYeZ/YULVr0vp9n7ty52jHj69rt27ednic8PFzJnTp1ur/CAAdKliypZOP7CWP/ARGRwYMHKzkoKMgttSxfvlzJxn6Er732mjanYcOGSq5Xr55bagEAZN7+/fuVbO89Umb6x3733XdKpicOAAAAAAAAshQ2cQAAAAAAACyATRwAAAAAAAALYBMHAAAAAADAArJsY2NvmTBhgtklZEiePHmUbGzUDNhjbGIsIjJ06FCHcypUqKAdW716tZLLlClzf4UBWZCxwf2NGze0McYGfAsWLFDyBx98oM0xNjLOnTu3NsbYbN/YcDxfvnx2KgbcI0eOHEo2vkfy5num/v37K7lSpUpKvnv3rjZn48aNSqaxMazm0KFDZpcAuN3ly5eV3KtXL5MqyT64EgcAAAAAAMAC2MQBAAAAAACwADZxAAAAAAAALCDL9sRp3bq1kmvXrq3k2NhYb5aTZRUoUMDsEpAFrVmzRsnGvhr29OjRQ8lz5sxxa01AVtC3b1/t2KhRo5Rs7MUxevRobc6JEyccPk+uXLm0Yx9//LGSv/vuO23Mvn37lFykSBGHzwNkB1evXtWOLVmyxOGcgIAA7ViTJk3cVRLg0M6dO7VjixYtUrKxv83Zs2e1Oc8//7ySZ86c6fS5mzdv7kqJQJYVERGhHYuLizOhEuviShwAAAAAAAALYBMHAAAAAADAAtjEAQAAAAAAsAA2cQAAAAAAACwgyzY2DgsLU/K3336r5Icfftgjz9uhQwft2I8//qjkYcOGKTl//vzanI0bNyp53bp191+c2K8PMKpVq5aSQ0NDtTF//fWXkpctW6bkZs2aaXOio6MdjildunRGygS87vXXX9eO5ciRQ8kjR45Usr0mxsbXKGNj8BEjRmhzzpw5o+TXXntNGxMSEqLkzp07a2MAo7t37yr5t99+08ZMnz5dyS+99JI2xt5rhScYG3gb14+IyIEDB5ScJ08eJU+YMEGbU6dOHTdUB+g+/PBDJQ8fPlwbY3wPZGy0vXfvXm1Oz549nT53tWrVlNyyZUunc4CspG7dukr++eeftTHPPvuskmfPnu3RmqyOK3EAAAAAAAAsgE0cAAAAAAAAC2ATBwAAAAAAwAKybE8co/DwcCXb61EwaNAgJd+5c8fpeT/77DMllypVShtz6dIlh2OM/RRERMaNG6dkd/XEoT8CXFGoUCElr1ixQhvz+OOPK/mPP/5Q8vr167U5xmMFCxZUckBAgDbHuC79/Py0Mf3793d43qCgIG0OkBn+/vq/XRh707Rq1UrJxn4jIiIVK1ZUsvFn9NatW9ocYz+1mzdvamO+//57JdtbU4BRt27dlLx48WJtjLGnzNmzZ7Uxs2bNuu9aEhISlDx06FBtjCu9Doy9D409cFq0aJHx4gAXLViwQMnGfmqvvvqqNmfUqFFKzpcvn5JTUlK0OW3atFHy2rVrtTHG93T2XseArMzYQ7NAgQLaGON7L3riOMZvAQAAAAAAAAtgEwcAAAAAAMAC2MQBAAAAAACwAMv0xMmdO7eSQ0NDtTHLly/3yHPnz5/f4Z8nJiZqx+bNm3ffz2vsAyQi0rBhw/s+L3yPvZ+lAwcOKHn+/PlKXrJkiTbn119/VbK93lRGY8eOVbK9njjGMcWLF1eyvf4OdevWdfrcQGbUqlXrvs9x9OhR7djq1auVXKZMGW1MnTp17vu54Xvs/Y40qlevnpIz0/9m165d2rEPP/xQyXv27FGy8bVGRH8dML4GiIhER0cruXTp0i5WCdy/rVu3Kvn27dtKbt68uTbH2APH6NixY9qxLVu2OK1l27ZtSl65cqWSW7du7fQcgKcY+1iKiFy9elXJxtcfe4zrx17vJ2OPwsuXLys5NjZWm1O7dm2nz21FXIkDAAAAAABgAWziAAAAAAAAWACbOAAAAAAAABbAJg4AAAAAAIAFWKaxcVZ28+ZN7di+ffvu+7zG5q4iImFhYfd9XsCe7t27O8wiIgkJCQ7z3LlztTmHDx92Osbo/PnzSjY2yhShsTGyNmOjcHteeeUV7ViRIkU8UQ6yuXLlyinZXgPV06dPKzkzX5Rg73fxtWvXHM6pUKGCdmzatGlKdqXpJeBNq1atUrLxC1bs/VwbPw8YGxC/9NJLTufYa3j/119/KfmXX35RMo2N4U3GtbB27VptzPTp05Vcs2ZNp+d94oknlPzzzz9rYyZPnqzkpKQkJX/00UfanMw08bcCrsQBAAAAAACwADZxAAAAAAAALIBNHAAAAAAAAAugJ04WduXKFe3Yxo0blZyZe9qBzAoODnaYhw4dqs15/vnnlZyWlub0eSpVqqRk432yQFZz4cIFJX/77bdO50RFRXmqHPiYMWPGKPmpp57Sxhw6dMhhdpexY8cqeeDAgdqYkJAQjzw34C4vvviikp955hklly9fPsPn9PfX/+18yJAhSm7VqpU2pnPnzhl+LsBb7PW7caUHjjNVqlTJ8JwDBw7c9/NaBVfiAAAAAAAAWACbOAAAAAAAABbAJg4AAAAAAIAFsIkDAAAAAABgATQ2zsL27NmjHWvfvr2Sr1696qVqAJGbN28q+euvv3aYRfTmmX5+ftqY2rVrK/mbb75RcpEiRTJUJ+Btf/31l8MMeJKxSfa1a9e0MTt27HB6njJlyii5b9++Ga6lbNmySs6Zk7easJ6nn35ayblz51byiRMntDkbNmxQcr169ZTcuHFjbU6zZs2UnJSUpI0JCAhwXCwAn8OVOAAAAAAAABbAJg4AAAAAAIAFsIkDAAAAAABgAdyo7AZ58+bVjj366KNK3rp1a4bPa+9e9BdeeCHD5wFcUaJECSV36dJFG2Ps87Fy5Uqn5zX2QxgwYIA25p133lFy0aJFnZ4XyErs9TEw6tq1q5KDgoI8VA18jb+/+m9ygwcP1sbYOwbANZnpD5UZBQoU0I6FhIR45bkBWAdX4gAAAAAAAFgAmzgAAAAAAAAWwCYOAAAAAACABdATxw0KFiyoHVuyZImS16xZo4355JNPlNy6dWslv/7669qcgICATFQIOJeQkKDkb7/9NsPnqFOnjnZszJgxSm7VqlWGzwtkdQsXLnQ6pk+fPkrOkSOHp8oBAGQTx44dM7sEwOuee+457diIESOUHBgYqOSvv/7aozVlJVyJAwAAAAAAYAFs4gAAAAAAAFgAmzgAAAAAAAAWwCYOAAAAAACABdDY2EOKFy+u5F69emlj7B0DzGL8eZw7d642xti42Nh8217TYppxwxcYG9UbMwAAAFyTJ08e7VhKSooJlWRNXIkDAAAAAABgAWziAAAAAAAAWACbOAAAAAAAABZATxwAIiIya9YshxkAAAAAYC6uxAEAAAAAALAANnEAAAAAAAAsgE0cAAAAAAAAC2ATBwAAAAAAwALYxAEAAAAAALAANnEAAAAAAAAswG1fMZ6ammp7HB8f767TIgtK//eb/u8d97AWfAdrwTHWgu9gLTjGWvAdrAXHWAu+g7XgGGvBd3hiLbhtEychIcH2uE6dOu46LbK4hIQEKVu2rNllZCmsBd/EWtCxFnwTa0HHWvBNrAUda8E3sRZ0rAXf5K61wO1UAAAAAAAAFuCXlpaW5o4T3bx5U+Li4kREJDg4WHLmdNtFPshiUlNTbbvHERERkidPHpMrylpYC76DteAYa8F3sBYcYy34DtaCY6wF38FacIy14Ds8sRbctokDAAAAAAAAz+F2KgAAAAAAAAtgEwcAAAAAAMAC2MQBAAAAAACwADZxAAAAAAAALIBNHAAAAAAAAAtgEwcAAAAAAMAC2MQBAAAAAACwADZxAAAAAAAALIBNHAAAAAAAAAtgEwcAAAAAAMAC2MQBAAAAAACwADZxAAAAAAAALIBNHAAAAAAAAAtgEwcAAAAAAMAC2MQBAAAAAACwADZxAAAAAAAALIBNHAAAAAAAAAtgEwcAAAAAAMAC2MQBAAAAAACwADZxAAAAAAAALIBNHAAAAAAAAAtgEwcAAAAAAMAC2MQBAAAAAACwADZxAAAAAAAALIBNHAAAAAAAAAtgE8dF69atEz8/v0z/FxMTY/b/BMAtYmJiMvzz//bbb5tdNuARd+7ckb1798p3330ngwcPltq1a0vu3LltP/tNmjQxu0TAq7Zu3SpDhgyRmjVrSpEiRSRXrlxSsGBBqVixovTo0UNmzZolt27dMrtMwONSUlJk+vTp0q5dOwkLC5M8efJISEiI1K9fXz788EO5cOGC2SUCHsPnBc/KaXYBvqJkyZJmlwAAcKPFixdLVFSU3Lhxw+xSANNdvnxZBg4cKAsXLtT+LCkpSZKSkuTIkSPyww8/yJtvvinTpk2T+vXrm1Ap4HkHDx6U3r17y+7du5XjZ8+elbNnz8qWLVtkwoQJMmXKFGnXrp1JVQKwKjZxXFSqVCkZMmSIy+NXrVolhw8fFhGREiVKSIsWLTxVGmCaKlWqSPPmzZ2Oi4yM9EI1gHclJiaygQOISHJysrRs2VJ27txpOxYcHCyPPPKIlC5dWhISEmT//v3y559/iojI0aNHpWXLlrJmzRqpW7euWWUDHnHq1Clp3ry5nDlzRkRE/Pz8pFGjRlKhQgU5f/68rF69WpKTk+X8+fPSpUsXWbFihUvvpQCr4vOC+7GJ46KKFSvKF1984dLYO3fuSOnSpW05KipKcubk/2pkP3Xr1nV5XQDZVYkSJSQyMtL238qVK+XTTz81uyzAayZMmGDbwPH395exY8fKyy+/LIGBgbYxaWlpMnfuXBk0aJBcuXJFbty4Ic8884z89ttvZpUNeERUVJRtAycsLEyWLl0q1atXt/35hQsXpFevXvLLL7/I7du3pUePHnL06FEJCgoyqWLAs/i84H7sLHjAypUr5ezZs7bcr18/E6sBAHhCmzZt5MSJExIaGqoc37Ztm0kVAeaYMmWK7fGLL74oI0eO1Mb4+flJr169JFeuXNK9e3cREdm7d6/ExcVJRESE12oFPGn58uWyYcMGERHJnTu3/O9//9N+vosVKyZLliyR6tWry59//imXLl2SDz74QN59910zSgZgQTQ29oCpU6faHj/yyCPK7jsAIHsoWbKktoED+JqrV6/K8ePHbfnJJ590OL5Lly6SN29eWz506JCnSgO87ssvv7Q97tev3z9uUObLl0/Gjh1ry5MmTZLU1FSP1wcge2ATx82uXLkiS5cutWWuwgEAANnVtWvXlOzslpAcOXJIwYIFbfnu3bueKAvwumvXrskvv/xiy/3793c4vnv37lKgQAEREbl06ZLtCh4AcIZNHDebO3eu3Lx5U0REcuXKJb179za5IgAAAM8IDg6WPHny2PL+/fsdjj9//rycP3/elmvUqOGx2gBv2rx5s9y6dUtE7l1p46xJa0BAgDz66KO2vGbNGo/WByD7oCeOm6W/lapdu3YSHBxsYjWAZyUmJsq8efPk999/lytXrkihQoWkVKlS8thjj0l4eLjZ5QEAPCxXrlzStm1bWbRokYiIjBs3Tlq3bq3cMpXeiBEjbFffNG/eXCpVquS1WgFPOnDggO1xRESES19qUrNmTfn555+1+UB2wucF92MTx42OHDkimzdvtmVupUJ2t2TJElmyZIndP4uIiJA33nhDevTo4eWqAADe9O6778rPP/8s165dk927d0v16tVl1KhR0qBBA9tXjO/du1fee+892bRpk4iIhIeHKw2RAav7448/bI/DwsJcmpO+r9rBgwfdXhOQFfB5wf24ncqN0l+FU7RoUWnfvr2J1QDmiouLk549e0r//v1p1gcA2ViVKlVk06ZNUqZMGREROXr0qERHR0vFihUlMDBQQkNDpUOHDrJp0yYJCgqSIUOGyNatW23jgezg4sWLtsclSpRwaU7JkiVtjy9duuT2moCsjs8LmcMmjpukpaXJjBkzbLl3796SO3duEysCPKdcuXIyYsQIWb16tZw5c0Zu3bol165dk/3798tHH32k/MtSTEyMvPDCCyZWCwDwtBo1asihQ4fk888/l3z58v3juNatW0tUVJTS3BjIDtI3+Q4MDHRpTvpxxibhgNXxecFz/NLS0tLMLiI7WLdunTRt2tSWY2NjpVatWiZWBHhGYmKiFCxYUPz9/3kPOCkpSXr37i0//vij7diGDRukYcOG3igRMNXo0aNlzJgxIiLSuHFjWbdunbkFAV6QkJAg//nPf2TmzJly+/ZtKVmypDRo0ECKFi0qV65ckW3btilfRf7MM8/IV199JTly5DCvaMCNmjdvbmtOPGrUKOUrxP/JmjVrpHnz5iJy75vbuBIB2QWfFzyLK3HcJP2tVA899BAbOMi2goKCHP5CFhEpUKCAzJs3T2lY+f7773u6NACACQ4fPiyPPPKIxMTEiL+/v0ycOFFOnTol8+fPl0mTJsmcOXPkzz//lLlz50qhQoVEROSbb77hX12RraT/lraUlBSX5vz9bVYirl+9A1gBnxc8i00cN7hx44YsWLDAlqOjo80rBsgiAgMDZfjw4ba8du1a5c0KAMD6UlNT5fHHH5fTp0+LyL3NmUGDBmlX2Pj5+UmPHj1k4cKFtmMTJ06U7du3e7VewFPy589ve5ycnOzSnPTj0s8HfAWfFzKHTRw3WLhwoSQlJYnIvUsho6KiTK4IyBr+vkRY5N5m54kTJ0ysBgDgbgsWLJB9+/aJyL0Gx0899ZTD8c2aNZOWLVvaMt9QheyiaNGitsfnzp1zac7Zs2dtj4sUKeL2mgAr4PNCxrGJ4wbpb6Vq1aqVhISEmFgNkHUY10L6b24AAFjfTz/9ZHvcpEkTl+Y0a9bM9jg2NtbdJQGmqFy5su2xqx9CT548aXtcpUoVt9cEWAGfFzKOTZz7dOrUKVsTMxFupQLSu379upIdfWMJAMB6/r6NSkS9EsGR9OOuXLni9poAM4SHh9sex8XFudSkeNeuXXbnA76EzwsZxybOfZoxY4bcvXtXRO41cOrUqZPJFQFZx+7du5XMVWoAkL2kb8Z66dIll+ak/1fWoKAgd5cEmKJ+/foSEBAgIvc+lDq7yuzWrVuydetWW05/hRrgS/i8kHFs4tynadOm2R737NlT6UwP+Lr0vQ7Cw8MlODjYxGoAAO4WGhpqe7x27VqX5qS/grlChQpurwkwQ/78+ZXeHjExMQ7Hp++pWbhwYWnUqJEnywOyLD4vZBybOPdh+/btcuDAAVvmVipkd9euXXN57KJFi2TmzJm23KdPH0+UBAAwUYsWLWyPDx48KNOnT3c4fs2aNfLzzz/bcuvWrT1WG+Btzz33nO3xlClTZP/+/XbH3bhxQ958801bfvbZZyVnzpwerw/wBj4veB6bOPchfUPjSpUqyaOPPmpiNYDnzZ8/X+rWrSszZsyQq1ev2h2TlJQk48aNkx49ekhaWpqIiJQpU0aGDh3qzVIBAF7Qvn17paHrM888I19//bXcuXNHGZeWlibz5s2Txx9/3HasTJky0qtXL6/VCnha+/btpWHDhiIikpKSIh06dJC4uDhlzMWLF6VLly5y5MgREbn3rVTpv2IZsDo+L3ieX9rf/68hQ1JSUuSBBx6w3df9zjvvyOuvv25yVYBnxcTESP/+/UVEJFeuXBIeHi6VK1eWoKAgSU1NlZMnT8qWLVvkxo0btjmFCxeWDRs2SLVq1cwqG/CYdu3ayZkzZ5RjZ8+etX29bL58+ezeLrJ8+XJ54IEHvFIj4Gnbtm2TZs2aKb/7Q0JCpH79+lKsWDG5cuWKbN26VY4fP27784CAAPn5559tH3iB7OLUqVNSp04diY+PFxERf39/ady4sZQvX14SEhJk9erVtrWSM2dO+emnn5TbsACr4/OC57GJk0kLFy6Ubt26ici9X87Hjx+XMmXKmFwV4Fnpfym7olmzZvL9999LWFiYB6sCzFO2bFmXv0o2vWPHjknZsmXdXxBgku3bt0vfvn3l0KFDTseWK1dOpk+fLg0aNPBCZYD3HTx4UJ588knZs2fPP44JDg6WKVOmSPv27b1XGOAFfF7wPDZxMqlz586ydOlSEbl3P3j6+7uB7OrWrVsSGxsrW7ZskS1btsjRo0fl4sWLcvHiRbl7964EBQXJgw8+KPXq1ZNevXpJ7dq1zS4Z8Cg2cYD/k5qaKkuXLpXFixdLbGysnDlzRq5duyb58uWTEiVKSK1ataRTp07SvXt3yZUrl9nlAh6VkpIic+bMkdmzZ8v+/fvl3LlzEhQUJOXLl5euXbvKgAEDpFixYmaXCbgdnxc8j00cAAAAAAAAC6CxMQAAAAAAgAWwiQMAAAAAAGABbOIAAAAAAABYAJs4AAAAAAAAFsAmDgAAAAAAgAWwiQMAAAAAAGABbOIAAAAAAABYAJs4AAAAAAAAFsAmDgAAAAAAgAWwiQMAAAAAAGABbOIAAAAAAABYAJs4AAAAAAAAFsAmDgAAAAAAgAXkdNeJbt68KXFxcSIiEhwcLDlzuu3UyGJSU1MlISFBREQiIiIkT548JleUtbAWfAdrwTHWgu9gLTjGWvAdrAXHWAu+g7XgGGvBd3hiLbjtpyUuLk7q1KnjrtPBIrZv3y6RkZFml5GlsBZ8E2tBx1rwTawFHWvBN7EWdKwF38Ra0LEWfJO71gK3UwEAAAAAAFiA267ECQ4Otj3evn27hISEuOvUyGLi4+NtO8fp/95xD2vBd7AWHGMt+A7WgmOsBd/BWnCMteA7WAuOsRZ8hyfWgts2cdLfxxcSEiKlS5d216mRhXH/po614JtYCzrWgm9iLehYC76JtaBjLfgm1oKOteCb3LUWuJ0KAAAAAADAAtjEAQAAAAAAsAA2cQAAAAAAACyATRwAAAAAAAALYBMHAAAAAADAAtjEAQAAAAAAsAA2cQAAAAAAACyATRwAAAAAAAALYBMHAAAAAADAAtjEAQAAAAAAsICcZhcAwLecOHFCO3bhwgUlv/POO0pevHixNqdLly5KnjFjhjYmb968GS8QAAAAALIorsQBAAAAAACwADZxAAAAAAAALIBNHAAAAAAAAAtgEwcAAAAAAMACaGwMwK4DBw44Pfb4448rOSEhQZuzaNEiJY8aNUobY2xsnJaWpmQ/Pz9tjrHZ8cGDB7UxNWvW1I4BWdn58+eV/NRTTym5UaNG2pzXX3/dozXBdxl/NwcHB2tj6tSpo+QRI0Y4nVOrVi0lBwYGZrZEwDKOHTumHfvkk0+UPGXKFG3MG2+84fC8L730knYsd+7cGaoNgLVwJQ4AAAAAAIAFsIkDAAAAAABgAWziAAAAAAAAWAA9cQDYtWDBAu2YsZ+NsVeNsZeNq2OKFy+u5NDQUKf1FStWTMn0v0F28NVXXyl51apVDrMIPXHgPrdv31bytGnTlGyvP9nhw4eV3K1bNyXb+51fvnx5Jbdr104bM3LkSCWXLFnSTsVA1mXsDzVx4kRtzNWrV52eZ/jw4Q7//L333tOO7d69W8lhYWFOnwfwlkceeUQ7tmfPHq889+TJk7VjAwcO9MpzuxNX4gAAAAAAAFgAmzgAAAAAAAAWwCYOAAAAAACABdATxwXx8fFKTk5OVvL69eu1Ob/99pvT827evFnJtWvXVrKxNwLgSQsXLlTy+++/r40x9kOw1x/BqGrVqkpu2LChNuZf//qXkl3pb3PhwgWnYwCruXnzpsM/r1OnjpcqgS/atm2bkl955RUl9+jRQ5tjfK+yadMmJf/666/anK+//lrJX375pTbG2I9n69atSg4PD9fmAGZ67bXXlPzf//5XyUFBQdqcDh06OD3vnTt3lLxixQolX758WZvz0ksvKfnVV1/VxtSvX9/pcwPucPLkSSXb63/jymcKd/DW83gaV+IAAAAAAABYAJs4AAAAAAAAFsAmDgAAAAAAgAWwiQMAAAAAAGABPt/Y+Pjx40revn27Nub5559XsrGhalpamjYnM02TcuTIkeE5gLusWrVKydevX9fGhIaGKvn1119Xsr2mxZ5qPlmsWDGPnBfwlvPnz2vHJk2a5HBOZGSkp8oBZNasWQ7/vHfv3tqxIkWKKLlTp04Os4jIsGHDlDx9+nRtzNixY5VsbH78xRdfOKwVcKc///xTycbG2yIi7733npJbtGih5I8++kibExER4fS57969q+QxY8Yoedy4cdqcxYsXK9n4eUdEZM2aNUouXLiw01qAzFi0aJHTMXXr1lXy5MmTtTHGLwU6ffq0kmvUqKHNKVOmjJIrVKjgtBYr4EocAAAAAAAAC2ATBwAAAAAAwALYxAEAAAAAALCAbN0Tx3ivp4h+v2pcXJySz50759GaHDH2R7DXL6F48eLeKgc+xni/qr2+Th9//LGSu3bt6tGagOzM2ONDROTKlSsO5/Ts2dNT5QBy69Yth39++PBhtzxPyZIllfzqq69qYyZMmKBkZ7UB7rRv3z4ld+zYUcn2eswUKlRIycOHD1eyK/1v7PH3V//N3dgTJyUlRZvz4YcfKnnPnj3amAEDBijZ2OenQIECGSkTsDH+vv7ss8+UbPyZFhEZOXKkkqtVq6aNsXfMV3ElDgAAAAAAgAWwiQMAAAAAAGABbOIAAAAAAABYAJs4AAAAAAAAFpCtGxt/++232rHVq1dn+DxBQUFKrlu3rpIbNmyozTE2he3Tp4825osvvlCysYnfzJkztTnDhg1zWCvgqoSEBCUbG2k3btxYm0MjYyDzbt68qeQVK1ZoY9LS0pT81ltvKfmxxx5zf2HA/9e/f38lf//990p+5ZVXtDlt27ZVctWqVTP8vCtXrtSOJSUlOawNcJc///xTO+ZKI2OjpUuXKrlRo0b3VZerxo8frx1bvHixkg8ePOh0zIIFC5QcHR19v6XBRxl/px87dkzJZcuW1eZ06NDBkyVlO1yJAwAAAAAAYAFs4gAAAAAAAFgAmzgAAAAAAAAWkK164ly5ckXJW7Zs0cZUq1ZNyYULF1by4MGDtTlNmzZVcokSJTJboiJHjhwO/3zHjh1ueR7AnqeeekrJxj5Or7/+ujfLAbK9r776SsmxsbHaGONrVJs2bTxaE5BerVq1lFy+fHklG/saiIgsXLhQya70xLl8+bKSe/TooY0pUKCAkkuVKuX0vEBmGHsCijjvgdOsWTPtWIMGDdxV0n0bN26ckp944gmTKoEv+uOPP8wuIdvjShwAAAAAAAALYBMHAAAAAADAAtjEAQAAAAAAsIBs1RMnb968Sp4zZ4425qGHHlJy/vz5PVoTkBVcv35dO3bixAklG3shtGrVyiO17Ny5UztWrFgxJYeFhXnkuQEz/fvf/1aysQ+ViEj37t2V/Oijj3q0JiC9wMBAJQ8aNEjJw4cP1+ZMmDBByd26dVNyeHi4Nmfy5MlKTkpK0saMGjVKybwuwFNGjBiR4TnGfpkizntdetPjjz+u5Nq1a2tj7PVlAzLq7t272rHly5c7nJOYmKgdi4uLU3KlSpW0MQEBARkrLhvjShwAAAAAAAALYBMHAAAAAADAAtjEAQAAAAAAsAA2cQAAAAAAACwgWzU2zpUrl5Lr1q1rUiW6HTt2aMcmTpxoQiXwRQcPHtSO/fHHH0o2NgZ/9913tTnGBsRGxmaV9thrbFy8eHElP/LII0qePn16hmsBzPT9999nal7r1q3dXAmQecZm3Lt379bGGL9Ewvja0alTJ23OG2+8oeQGDRpoY4YMGeJynUBGfPDBB0resmWL0znGL0IZOnSoW2tyN39/f4cZcJcNGzZox9avX+9wzuXLl7VjNWrUUHLlypW1Mb/88ouSH3jgAVdKzJZY0QAAAAAAABbAJg4AAAAAAIAFsIkDAAAAAABgAdmqJ05W9vTTT2vHrly5omRjTxJ795EDmZGWlub02LVr15Rs7Flgb46fn5/T53FlzPnz55X8008/KblRo0banN9//107Bphl+/btSnaln8crr7yiHatdu7bbagLul7GPxn/+8x9tzOLFi5U8c+ZMJc+YMUObU7hwYSUvWbJEGxMYGOhqmUCGTJkyRckpKSlO57z88stKLlCggFtr8rRhw4Zpx5588kklHzp0yFvlwMKSk5OV3KdPH7ect1y5cko29u4UEWnbtq2Sly9fruRSpUq5pRYr4EocAAAAAAAAC2ATBwAAAAAAwALYxAEAAAAAALAAeuJ4SEJCgpKTkpKczomIiFByr1693FoTfJexL80/HXP2588++6zDOQ0aNNCOVa1a1Ul1OuP9tfbui3377beVbK+HD+ApJ06cUHJUVJSSb968qc0JDw9X8tChQ7UxOXPysoys6+GHH9aOvfDCC0r+4IMPnJ7nmWeeUXKRIkXuqy7gnxh77Inov7/tyZ07t5KN79GtJiQkxOmYyZMnK/ndd9/1VDmwsJ07dyr5zJkzTucYXztG/b/27jY066qPA/jfaDUNNdKFtKKCngyTorYe6AlCpGLRzIq0MikqYpUxLCGFVJZWmBRERRhBa6iVsfLFXgSZ1askJAc1a4UQRO1NEuUWi90v7u7uzv9cXdce/nM7uz6fV37Pzvlfv3BnbT8uf1u3LtqTn3/Z0tIS7dmxY0eQP/nkkyBX08/O3okDAAAAkABNHAAAAIAEaOIAAAAAJEATBwAAACABJigW4PDhw9Fac3NzkEsNUZsxY0aQN23aVGxh8JeLL744Wuvu7g5yfujqRPr444+D3NraGu3JD0VraGiI9ixevLjYwuAv+UHavb29QS41GHzr1q1Brq+vL74wOMpqa2uDnP/cHxoais58+eWXQf7jjz+iPfnBsjAapYauHjlypOK5/NfnpUuXFlbTRCj1s0rerFmzjkIlpO6WW24Jcqmv8SeccEKQ169fH+SmpqaKr/Pkk09Ga7t27QryCy+8EGSDjQEAAACYVDRxAAAAABKgiQMAAACQADNxCvDGG29Ea5999lnFc5s3bw7ytddeW1BFUNlkmoGTV1dXF+Tnn38+2tPe3h7kUjOlzMShCJ2dndFaR0dH2TONjY3Rms9HpqL+/v4Rn+nq6iqbsyzLbrrpplHXBP/z559/jupcTU1NwZUcXfn/7o0bN1Y8k581CKXk577u3r072jOaGTh5CxYsiNbMSvs/78QBAAAASIAmDgAAAEACNHEAAAAAEqCJAwAAAJAAg41H4f333w/yqlWroj3Tpk0Lcn19fbTnvvvuK7QumKqGhoYqru3du/dolUOV+f7776O1Up+T/2QoK1NRd3d3tPbSSy+VPbNo0aJo7cMPPwzy3XffHe357rvvgnzSSScNp0QIrFmzZlTnSv2yhJS0tLQEed++fRNUCVNNW1tbkEsNxJ49e/aYX+frr7+O1gYHB8f83KnCO3EAAAAAEqCJAwAAAJAATRwAAACABJiJMwyHDh0K8hNPPDHiZ7S3t0drtbW1o64JprK+vr4gt7a2Rnvyc6fWrl07rjVRvTo7O0d85uabby6+EJhgvb290drvv/8e5KampiBv2bIlOtPQ0BDkw4cPR3s++OCDIK9YsWLYdUI12bZtW7S2c+fOiudOPPHEIJ977rlFlcQUNmfOnKPyOs8++2y0NjAwcFReOwXeiQMAAACQAE0cAAAAgARo4gAAAAAkwEycnFK/f37lypVB7unpqficG264IchXXXXV2AqDf9HV1RWt5T//3nnnnWjPkiVLxq2mcn777bdo7b333gtyW1tbkEvduRkzZgR52bJlBVQHWXbHHXcEec+ePRXPNDc3B/n8888vsiSYtIaGhoJ88sknB/mss86KzsybNy/Iv/zyS7Snu7t77MXBKG3atCnIE/U9U5Zl2f79+4P86quvBvmVV16p+Iy5c+dGax0dHUG+/PLLR14cjEJ/f3+0lp+Dlv/ZIMuyrKamJsjLly8vtrCEeCcOAAAAQAI0cQAAAAASoIkDAAAAkABNHAAAAIAEGGyc89FHH0VrlYZaLly4MFrLDx075hj9MsbHihUrorVp06YFOT+gL8viIX1fffVVkOfPn19AdVm2a9euILe3t0d7Ojs7g5wflFlqSOzGjRuDfN555422RKrct99+G+Tt27cHOX+fsizLLrnkkiDnv+ZDtcjfj/wg/ddeey06k/++qdTw+vyQy6effjrI+QGXUMrDDz8cra1fv77iuQMHDgT5mWeeCfJll102tsL+kv9/R29vb7Tn0KFDQf7pp58qPrexsTHIGzZsiPYsWrRoOCVC4Up9z/TYY49VPPfII48EuaWlpbCaUqOzAAAAAJAATRwAAACABGjiAAAAACSg6mfiHDlyJMh33XXXiJ+xefPmaK2+vn7UNcFI/Pzzz9FafkbBvn37oj35OU35OTSl5oBU2pP/+Gj3LF68OMhvvvlmdGbu3LnRGoxGV1fXiM889dRTQa6rqyuoGpi85s2bF62dffbZQf7mm2+C/Nxzz0VnhjPD7ODBg0H+/PPPg3zFFVdUfAbce++90dq2bduC/MMPP0R7BgYGgrxmzZpiCxuD/Pc/pb4fevfdd4N86qmnjmtNpOnTTz8N8pVXXjniZ+TvSpZl2Y4dO4K8e/fuIL/99tvRmfzPAg0NDdGe4cyzqhbeiQMAAACQAE0cAAAAgARo4gAAAAAkQBMHAAAAIAFVN9h4cHAwyPmBZ6WGxObdf//9Qb7++uvHXhiM0tatW6O1/LDt4Qw/HunHS+0pNdx1/vz5ZXOWxUMuH3300YqvDUfL0qVLo7XrrrtuAiqBiXXppZdGa21tbUG+7bbbgvz4449HZ6ZPn15sYfAvTjvttGjtxRdfDPKGDRuiPfv37x+vksoq9b3XypUrg9zS0hLkiy66aFxrYmrYu3dvtNbU1BTk119/PdpzzjnnBDk/ZL7U8Pqenp6ytRx33HHR2urVq4Pc2toa7Zk9e3bZ51YT78QBAAAASIAmDgAAAEACNHEAAAAAElB1M3FefvnlIG/fvr3imfzsg3vuuafIkmBMVq1aNaw1oLSFCxcG+ZprrglyfuZHlmXZ8ccfP641QSpuvPHGIOdnKuTneWRZlvX391d87umnnx7kBQsWjKI6iDU3Nwe51Kynt956K8gDAwNBXrduXcXXufDCC4O8bNmyimdmzpwZrT344IMVz0ElBw8ejNZ+/fXXIN96663j8tpnnnlmkEvNu3nooYfG5bWnKu/EAQAAAEiAJg4AAABAAjRxAAAAABJQdTNxOjo6yn581qxZ0dqWLVuCnJ+fAEC6rr766iDv2bNnYgqBBE2fPj3I+bmB5ggy2Z1yyinR2urVq8ueWbt27XiVA+Pi9ttvj9Zqa2uDXGoGYE9PT5BramqCXGqWTX4GzvLly4M8Z86c8sVSkXfiAAAAACRAEwcAAAAgAZo4AAAAAAnQxAEAAABIQNUNNs4P2Pviiy+CvHPnzuiMQcYAAACkaObMmdHanXfeWTYzeXknDgAAAEACNHEAAAAAEqCJAwAAAJCAqpuJ88ADD5TNAAAAAJORd+IAAAAAJEATBwAAACABmjgAAAAACdDEAQAAAEiAJg4AAABAAjRxAAAAABJQ2K8YHxwc/PvPP/74Y1GPZRL659/vP//e+S93oXq4C+W5C9XDXSjPXage7kJ57kL1cBfKcxeqx3jchcKaOH19fX//ubGxsajHMsn19fVlZ5xxxkSXMam4C9XJXYi5C9XJXYi5C9XJXYi5C9XJXYi5C9WpqLvgn1MBAAAAJGDa0NDQUBEP6u/vzw4cOJBlWZbV1dVlxx5b2Jt8mGQGBwf/7h5fcMEFWW1t7QRXNLm4C9XDXSjPXage7kJ57kL1cBfKcxeqh7tQnrtQPcbjLhTWxAEAAABg/PjnVAAAAAAJ0MQBAAAASIAmDgAAAEACNHEAAAAAEqCJAwAAAJAATRwAAACABGjiAAAAACRAEwcAAAAgAZo4AAAAAAnQxAEAAABIgCYOAAAAQAI0cQAAAAASoIkDAAAAkABNHAAAAIAEaOIAAAAAJEATBwAAACABmjgAAAAACfgPGaTbw2YonFEAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 600x400 with 24 Axes>"
      ]
     },
     "metadata": {
      "image/png": {
       "height": 391,
       "width": 568
      }
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import numpy as np\n",
    "index = np.random.choice(np.arange(len(X_train)), 24, replace=False)  # 24 indices\n",
    "figure, axes = plt.subplots(nrows=4, ncols=6, figsize=(6, 4))\n",
    "\n",
    "for item in zip(axes.ravel(), X_train[index], y_train[index]):\n",
    "    axes, image, target = item\n",
    "    axes.imshow(image, cmap=plt.cm.gray_r)\n",
    "    axes.set_xticks([])  # remove x-axis tick marks\n",
    "    axes.set_yticks([])  # remove y-axis tick marks\n",
    "    axes.set_title(target)\n",
    "\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 15.6.3 Data Preparation\n",
    "* **Scikit-learn’s bundled datasets** were **preprocessed** into the **shapes its models required**\n",
    "* MNIST dataset **requires some preparation** for use in a Keras convnet"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Reshaping the Image Data (1 of 2)\n",
    "* **Keras convnets** require **NumPy array inputs** \n",
    "* Each **sample** must have the **shape**\n",
    "> `(`**width**`,` **height**`,` **channels**`)`\n",
    "* Each pixel has **one channel** (grayscale shade 0-255), so sample shapes will be \n",
    "> **`(28, 28, 1)`**\n",
    "* As the **neural network learns** from the images, it **creates many more channels**\n",
    "    * These channels will **represent more complex features**, like **edges**, **curves** and **lines**\n",
    "    * Enable network to **recognize digits** based on these features and how they’re **combined**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Reshaping the Image Data (1 of 2)\n",
    "* NumPy array method `reshape` receives a tuple representing the new shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "X_train = X_train.reshape((60000, 28, 28, 1)) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(60000, 28, 28, 1)"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "X_test = X_test.reshape((10000, 28, 28, 1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(10000, 28, 28, 1)"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_test.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Normalizing the Image Data \n",
    "* **Numeric feature values** may vary widely\n",
    "* Deep learning networks **perform better** on data that's **normalized** into\n",
    "    * the range **0.0-1.0**, or \n",
    "    * a range for which the data’s **mean is 0.0** and its **standard deviation is 1.0**\n",
    "        * S. Ioffe and Szegedy, C., “Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift.” https://arxiv.org/abs/1502.03167\n",
    "* Divide **each pixel** value by **255** to normalize into the range **0.0-1.0**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "X_train = X_train.astype('float32') / 255"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "X_test = X_test.astype('float32') / 255"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### One-Hot Encoding: Convert Labels to Categorical Data (1 of 2)\n",
    "* **Predictions** for each digit will be an **array of 10 probabilities** \n",
    "* To **evaluate model accuracy**, Keras **compares predictions to dataset's labels**\n",
    "    * Both must have the **same shape**\n",
    "    * MNIST labels are **individual integers 0-9**\n",
    "* Must **transform labels** into **categorical data arrays** matching the **prediction format**\n",
    "* Use [**one-hot encoding**](https://en.wikipedia.org/wiki/One-hot) to convert labels from integers into 10-element **arrays of 1.0s and 0.0s** \n",
    "    * **only one element is 1.0** and the **rest are 0.0s**\n",
    "* Categorical representation of a **7**\n",
    "> <pre>[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, <strong>1.0</strong>, 0.0, 0.0]</pre>\n",
    "* **`tensorflow.keras.utils`** function **`to_categorical`** performs **one-hot encoding**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### One-Hot Encoding: Convert Labels to Categorical Data (2 of 2)\n",
    "* Transform **`y_train`** and **`y_test`** into **two-dimensional arrays of categorical data**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from tensorflow.keras.utils import to_categorical"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "y_train = to_categorical(y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(60000, 10)"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_train.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0., 0., 0., 0., 0., 1., 0., 0., 0., 0.])"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_train[0]  # one sample’s categorical data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "y_test = to_categorical(y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(10000, 10)"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_test.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 15.6.4 Creating the Neural Network\n",
    "* Configure a **convolutional neural network**\n",
    "* **`Sequential` model** stacks layers to **execute sequentially**\n",
    "    * **output** of one layer becomes **input** to the next\n",
    "    * **Feed-forward network**\n",
    "    * Later, you’ll see that not all layers feed output to the next layer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from tensorflow.keras.models import Sequential "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "cnn = Sequential() "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Adding Layers to the Network\n",
    "* A typical **convnet** consists of \n",
    "\t* **input layer** that receives **training samples**\n",
    "\t* **hidden layers** that **learn** from training samples\n",
    "\t* **output layer** that **produces predictions**\n",
    "* Import layer classes for a basic **convnet**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from tensorflow.keras.layers import Conv2D, Dense, Flatten, MaxPooling2D "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Convolution (1 of 5)\n",
    "* We'll start with a **convolution layer**\n",
    "* Uses the **relationships between pixels in close proximity** to learn useful **features** (or patterns) in small areas of each sample\n",
    "* These **features** become **inputs** to **subsequent layers** "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Convolution (2 of 5)\n",
    "* Examine convolution on a 6-by-6 image\n",
    "* **3-by-3 shaded square** represents the **kernel**\n",
    "* **Convolution** performs calculations that **learn** from kernel's **9** features, then **outputs 1 new feature** \n",
    "![Convolution diagram in which the 3-by-3 shaded square represents the kernel in its initial position](./ch15images/convolution.png \"Convolution diagram in which the 3-by-3 shaded square represents the kernel in its initial position\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Convolution (3 of 5)\n",
    "* [**Kernels typically are 3-by-3**](https://www.quora.com/How-can-I-decide-the-kernel-size-output-maps-and-layers-of-CNN)\n",
    "    * We found convnets that used **5-by-5** and **7-by-7** \n",
    "    * Kernel-size is a **hyperparameter**\n",
    "* By looking at **features near one another**, the network begins to **recognize features** \n",
    "    * Like **edges**, **straight lines** and **curves**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Convolution (4 of 5)\n",
    "* **Complete pass** left-to-right and top-to-bottom is called a **filter**\n",
    "* For a **3-by-3 kernel**, the filter dimensions will be **two less than the input dimensions**\n",
    "    * For each 28-by-28 MNIST image, the filter will be 26-by-26 \n",
    "* **Number of filters** in the **convolutional layer** is commonly **32** or **64** for small images\n",
    "* Each filter produces different results\n",
    "* **Higher-resolution images** have **more features**, so they **require more filters**\n",
    "* [**Keras team’s pretrained convnets**](https://github.com/keras-team/keras-applications/tree/master/keras_applications) use 64, 128 or even 256 filters in their **first convolutional layers**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Convolution (5 of 5)\n",
    "* **Set of filters** produced by a **convolution layer** is called a **feature map**\n",
    "* Subsequent **convolution layers** combine features from previous feature maps to **recognize larger features** and so on\n",
    "\t* In **facial recognition**, **early layers** might recognize **lines**, **edges** and **curves**, and **subsequent layers** might **combine** those into **features** like **eyes**, **eyebrows**, **noses**, **ears** and **mouths**\n",
    "* After **learning a feature**, a network can **recognize that feature anywhere** in the **image**\n",
    "    * One reason **convnets** are popular for **object recognition** in images"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Adding a **`Conv2D`** Convolution Layer (1 of 2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/pauldeitel/anaconda3/envs/pydsft/lib/python3.11/site-packages/keras/src/layers/convolutional/base_conv.py:107: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead.\n",
      "  super().__init__(activity_regularizer=activity_regularizer, **kwargs)\n"
     ]
    }
   ],
   "source": [
    "cnn.add(Conv2D(filters=64, kernel_size=(3, 3), activation='relu', \n",
    "               input_shape=(28, 28, 1)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* **`filters=64`** — The number of **filters** in the resulting **feature map**.\n",
    "* **`kernel_size=(3, 3)`** — The **size of the kernel** used in each **filter**\n",
    "* **`activation='relu'`** — **Rectified Linear Unit activation function** is used to produce this layer’s output\n",
    "    * **Most widely used activation function** (Chollet, François. _Deep Learning with Python_. p. 72. Shelter Island, NY: Manning Publications, 2018)\n",
    "    * [**Good for performance** because it’s **easy to calculate**](https://towardsdatascience.com/exploring-activation-functions-for-neural-networks-73498da59b02) \n",
    "    * [Commonly recommended for **convolutional layers**](https://www.quora.com/How-should-I-choose-a-proper-activation-function-for-the-neural-network) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**NOTE:** Running in Colab, I got a `UserWarning` indicating that it's now preferred to create an `Input` object that specifies the input shape and add it as the first layer, rather than `input_shape` in the layer above."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Adding a **`Conv2D`** Convolution Layer (2 of 2)\n",
    "* **First layer** in the model, so we specify the shape of each sample with `input_shape=(28, 28,1)` \n",
    "\t* Creates an **input layer** to **load the samples** and pass them into the **`Conv2D` layer**, which is actually the **first hidden layer**\n",
    "* Each subsequent layer **infers `input_shape`** from previous layer’s **output shape**\n",
    "    * Makes it easy to **stack** layers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Dimensionality of the First Convolution Layer’s Output\n",
    "* Input samples are 28-by-28-by-1—that is, **784 features each**\n",
    "* Specified **64 filters** and a **3-by-3 kernel** for the layer, so the **feature map size is 26-by-26-by-64** for a total of **43,264 features** \n",
    "\t* **Significant increase in dimensionality** \n",
    "    * **Enormous** compared to numbers of features processed in our Machine Learning examples\n",
    "* As each layer adds features, feature map **dimensionality** grows significantly\n",
    "    * This is one of reason **deep learning** often requires **tremendous processing power**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Overfitting \n",
    "* Can occur when a **model is too complex** compared to what it is modeling\n",
    "* **Most extreme case**: Model **memorizes** its training data's features\n",
    "* **Overfitting** tends to occur in **deep learning** as the **dimensionality** becomes **too large** [\\[1\\]](https://cs231n.github.io/convolutional-networks/),[\\[2\\]](https://medium.com/@cxu24/why-dimensionality-reduction-is-important-dd60b5611543),[\\[3\\]](https://towardsdatascience.com/preventing-deep-neural-network-from-overfitting-953458db800a)\n",
    "* **Higher dimensionality** also increases (and sometimes explodes) **computation time**\n",
    "* For deep learning on **CPUs**, training could become **intolerably slow**\n",
    "* There are various techniques to **prevent overfitting** [\\[1\\]](https://towardsdatascience.com/deep-learning-3-more-on-cnns-handling-overfitting-2bd5d99abe5d), [\\[2\\]](https://www.kdnuggets.com/2015/04/preventing-overfitting-neural-networks.html) &mdash; we'll use **pooling**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Adding a Pooling Layer (1 of 3)\n",
    "* To **reduce overfitting** and **computation time**, a **convolution layer** is often followed by one or more layers that **reduce dimensionality** of **convolution layer’s output**\n",
    "* **Pooling compresses** (or **down-samples**) the results by **discarding features**\n",
    "    * Helps make the model **more general**\n",
    "* **Most common pooling technique** is called **max pooling**\n",
    "\t* Examines a 2-by-2 square of features and keeps only the maximum feature."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "### Adding a Pooling Layer (2 of 3)\n",
    "* 2-by-2 blue square in position 1 represents the initial pool of features to examine:\n",
    "\n",
    "![Max pooling diagram showing the 6-by-6 set of numeric values we wish to compress with the 2-by-2 blue square in position 1 representing the initial pool of features to examine, and the 3-by-3 square representing the results of max pooling](./ch15images/pooling.png \"Max pooling diagram showing the 6-by-6 set of numeric values we wish to compress with the 2-by-2 blue square in position 1 representing the initial pool of features to examine, and the 3-by-3 square representing the results of max pooling\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Adding a Pooling Layer (3 of 3)\n",
    "* Outputs **maximum feature** from each pool\n",
    "* **Pools do not overlap** \n",
    "* **Stride** for a 2-by-2 pool is **2**\n",
    "* Every group of four features is reduced to one, so 2-by-2 pooling **compresses** number of features by **75%**\n",
    "* Reduces previous layer’s output from **26-by-26-by-64** to **13-by-13-by-64**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "cnn.add(MaxPooling2D(pool_size=(2, 2), padding='same'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Adding Another Convolutional Layer and Pooling Layer\n",
    "* **Convnets** often have **many convolution and pooling layers**. \n",
    "* [Keras team’s convnets](https://github.com/keras-team/keras-applications/tree/master/keras_applications) tend to **double** the number of **filters** in subsequent **convolutional layers** to enable the models to learn more relationships between the features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "cnn.add(Conv2D(filters=128, kernel_size=(3, 3), activation='relu'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "cnn.add(MaxPooling2D(pool_size=(2, 2), padding='same'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* **Input** to the **second convolution layer** is the 13-by-13-by-64 **output of the first pooling layer**\n",
    "* **Output** of this **Conv2D layer** will be **11-by-11-by-128**\n",
    "* For **odd dimensions** like 11-by-11, **Keras pooling layers round down** by default (in this case to 10-by-10), so this pooling layer’s **output** will be **5-by-5-by-128**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Flattening the Results to One Dimension with a Keras **`Flatten`** Layer\n",
    "* Model's **final output** will be a **one-dimensional** array of 10 probabilities that classify the digits\n",
    "* To prepare for **one-dimensional final predictions**, need to **flatten** the previous layer’s output to **one dimension**\n",
    "* **`Flatten`** layer's output will be **1-by-3200** (5 &#215; 5 &#215; 128)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "cnn.add(Flatten())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Adding a Dense Layer to Reduce the Number of Features\n",
    "* Layers before the **`Flatten`** layer **learned digit features**\n",
    "* Now must **learn the relationships among those features** to **classify** which digit each image represents\n",
    "* Accomplished with **fully connected `Dense` layers**\n",
    "* The following **`Dense` layer** creates **128 neurons (`units`)** that **learn** from the 3200 outputs of the previous layer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "cnn.add(Dense(units=128, activation='relu'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Many **convnets** contain **at least one `Dense` layer** \n",
    "* **Convnets** geared to more complex image datasets with higher-resolution images like [**ImageNet**](http://www.image-net.org)—a dataset of over 14 million images—often have **several `Dense` layers**, commonly with **4096 neurons**\n",
    "* See the [Keras pretrained ImageNet convnets' code](https://github.com/keras-team/keras-applications/tree/master/keras_applications)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Adding Another Dense Layer to Produce the Final Output\n",
    "* Final **`Dense`** layer **classifies** inputs into **neurons** representing the classes **0-9**\n",
    "* The **`softmax` activation function** converts values of these 10 neurons into **classification probabilities**\n",
    "* **Neuron** with **highest probability** represents the **prediction** for a given digit image"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "cnn.add(Dense(units=10, activation='softmax'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Printing the Model’s Summary with the Model’s **`summary`** Method\n",
    "* Note layers' **output shapes** and **numbers of parameters**\n",
    "* **Parameters** are the **weights** that the network **learns** during training [\\[1\\]](https://hackernoon.com/everything-you-need-to-know-about-neural-networks-8988c3ee4491),[\\[2\\]](https://www.kdnuggets.com/2018/06/deep-learning-best-practices-weight-initialization.html) \n",
    "* **Relatively small network**, but needs to **learn nearly 500,000 parameters**! \n",
    "\t* This is for **tiny images** that are less than 1/4 the size of icons on smartphone home screens\n",
    "\t* Imagine how many features a network would have to learn to process high-resolution 4K video frames or the super-high-resolution images produced by today’s digital cameras \n",
    "* In the **`Output Shape`** column, **`None`** means the model does not know in advance how many training samples you’re going to provide"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\">Model: \"sequential\"</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1mModel: \"sequential\"\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓\n",
       "┃<span style=\"font-weight: bold\"> Layer (type)                    </span>┃<span style=\"font-weight: bold\"> Output Shape           </span>┃<span style=\"font-weight: bold\">       Param # </span>┃\n",
       "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩\n",
       "│ conv2d (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">Conv2D</span>)                 │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">26</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">26</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">64</span>)     │           <span style=\"color: #00af00; text-decoration-color: #00af00\">640</span> │\n",
       "├─────────────────────────────────┼────────────────────────┼───────────────┤\n",
       "│ max_pooling2d (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">MaxPooling2D</span>)    │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">13</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">13</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">64</span>)     │             <span style=\"color: #00af00; text-decoration-color: #00af00\">0</span> │\n",
       "├─────────────────────────────────┼────────────────────────┼───────────────┤\n",
       "│ conv2d_1 (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">Conv2D</span>)               │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">11</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">11</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">128</span>)    │        <span style=\"color: #00af00; text-decoration-color: #00af00\">73,856</span> │\n",
       "├─────────────────────────────────┼────────────────────────┼───────────────┤\n",
       "│ max_pooling2d_1 (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">MaxPooling2D</span>)  │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">6</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">6</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">128</span>)      │             <span style=\"color: #00af00; text-decoration-color: #00af00\">0</span> │\n",
       "├─────────────────────────────────┼────────────────────────┼───────────────┤\n",
       "│ flatten (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">Flatten</span>)               │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">4608</span>)           │             <span style=\"color: #00af00; text-decoration-color: #00af00\">0</span> │\n",
       "├─────────────────────────────────┼────────────────────────┼───────────────┤\n",
       "│ dense (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">Dense</span>)                   │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">128</span>)            │       <span style=\"color: #00af00; text-decoration-color: #00af00\">589,952</span> │\n",
       "├─────────────────────────────────┼────────────────────────┼───────────────┤\n",
       "│ dense_1 (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">Dense</span>)                 │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">10</span>)             │         <span style=\"color: #00af00; text-decoration-color: #00af00\">1,290</span> │\n",
       "└─────────────────────────────────┴────────────────────────┴───────────────┘\n",
       "</pre>\n"
      ],
      "text/plain": [
       "┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓\n",
       "┃\u001b[1m \u001b[0m\u001b[1mLayer (type)                   \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1mOutput Shape          \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1m      Param #\u001b[0m\u001b[1m \u001b[0m┃\n",
       "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩\n",
       "│ conv2d (\u001b[38;5;33mConv2D\u001b[0m)                 │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m26\u001b[0m, \u001b[38;5;34m26\u001b[0m, \u001b[38;5;34m64\u001b[0m)     │           \u001b[38;5;34m640\u001b[0m │\n",
       "├─────────────────────────────────┼────────────────────────┼───────────────┤\n",
       "│ max_pooling2d (\u001b[38;5;33mMaxPooling2D\u001b[0m)    │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m13\u001b[0m, \u001b[38;5;34m13\u001b[0m, \u001b[38;5;34m64\u001b[0m)     │             \u001b[38;5;34m0\u001b[0m │\n",
       "├─────────────────────────────────┼────────────────────────┼───────────────┤\n",
       "│ conv2d_1 (\u001b[38;5;33mConv2D\u001b[0m)               │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m11\u001b[0m, \u001b[38;5;34m11\u001b[0m, \u001b[38;5;34m128\u001b[0m)    │        \u001b[38;5;34m73,856\u001b[0m │\n",
       "├─────────────────────────────────┼────────────────────────┼───────────────┤\n",
       "│ max_pooling2d_1 (\u001b[38;5;33mMaxPooling2D\u001b[0m)  │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m6\u001b[0m, \u001b[38;5;34m6\u001b[0m, \u001b[38;5;34m128\u001b[0m)      │             \u001b[38;5;34m0\u001b[0m │\n",
       "├─────────────────────────────────┼────────────────────────┼───────────────┤\n",
       "│ flatten (\u001b[38;5;33mFlatten\u001b[0m)               │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m4608\u001b[0m)           │             \u001b[38;5;34m0\u001b[0m │\n",
       "├─────────────────────────────────┼────────────────────────┼───────────────┤\n",
       "│ dense (\u001b[38;5;33mDense\u001b[0m)                   │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m128\u001b[0m)            │       \u001b[38;5;34m589,952\u001b[0m │\n",
       "├─────────────────────────────────┼────────────────────────┼───────────────┤\n",
       "│ dense_1 (\u001b[38;5;33mDense\u001b[0m)                 │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m10\u001b[0m)             │         \u001b[38;5;34m1,290\u001b[0m │\n",
       "└─────────────────────────────────┴────────────────────────┴───────────────┘\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\"> Total params: </span><span style=\"color: #00af00; text-decoration-color: #00af00\">665,738</span> (2.54 MB)\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1m Total params: \u001b[0m\u001b[38;5;34m665,738\u001b[0m (2.54 MB)\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\"> Trainable params: </span><span style=\"color: #00af00; text-decoration-color: #00af00\">665,738</span> (2.54 MB)\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1m Trainable params: \u001b[0m\u001b[38;5;34m665,738\u001b[0m (2.54 MB)\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\"> Non-trainable params: </span><span style=\"color: #00af00; text-decoration-color: #00af00\">0</span> (0.00 B)\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1m Non-trainable params: \u001b[0m\u001b[38;5;34m0\u001b[0m (0.00 B)\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "cnn.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Visualizing a Model’s Structure with the **`plot_model` Function** from Module `tensorflow.keras.utils`\n",
    "* [See our discussion of `plot_model`](https://learning.oreilly.com/videos/python-fundamentals/9780135917411/9780135917411-PFLL_Lesson15_30)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Compiling the Model (1 of 2)\n",
    "* Complete the model by calling its **`compile` method**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "cnn.compile(optimizer='adam',\n",
    "            loss='categorical_crossentropy',\n",
    "            metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Compiling the Model (2 of 2)\n",
    "* `optimizer='adam'`—The **optimizer** this model uses to **adjust the weights** throughout the neural network **as it learns**\n",
    "\t* [**Keras optimizers**](https://keras.io/optimizers/)\n",
    "\t* `'adam'` performs well across a wide variety of models [\\[1\\]](https://medium.com/octavian-ai/which-optimizer-and-learning-rate-should-i-use-for-deep-learning-5acb418f9b2),[\\[2\\]](https://towardsdatascience.com/types-of-optimization-algorithms-used-in-neural-networks-and-ways-to-optimize-gradient-95ae5d39529f)\n",
    "* `loss='categorical_crossentropy'`—The **loss function** used by the optimizer in **multi-classification networks** (ours predicts 10 classes)\n",
    "\t* **Optimizer** attempts to **minimize the values returned by the loss function** \n",
    "\t* For **binary classification**, Keras provides **`'binary_crossentropy'`**, and for **regression**, **`'mean_squared_error'`**\n",
    "\t* [Other loss functions](https://keras.io/losses/)\n",
    "* `metrics=['accuracy']`—List of **metrics** the network will produce to help you **evaluate the model**\n",
    "\t* **Accuracy** commonly used in **classification models**\n",
    "\t* We’ll use it to check **percentage of correct predictions**\n",
    "\t* [Other metrics](https://keras.io/metrics/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 15.6.5 Training and Evaluating the Model (1 of 3)\n",
    "* **Train a Keras model** by calling its **`fit` method**\n",
    "```python\n",
    "cnn.fit(X_train, y_train, epochs=5, batch_size=64, validation_split=0.1)\n",
    "```\n",
    "* **`epochs=5`**&mdash;train neural networks iteratively over time\n",
    "    * Each **`epoch`** processes **every training dataset sample** once\n",
    "    * **Hyperparameter** that may need tuning\n",
    "* **`batch_size=64`**&mdash;**number of samples to process at a time**\n",
    "    * Most models specify a **power of 2 from 32 to 512**\n",
    "* [**`validation_split=0.1`**&mdash;model should reserve the **last** 10% of the training samples for validation](https://keras.io/getting-started/faq/#how-is-the-validation-split-computed) \n",
    "\t* After each **epoch**, model uses validation samples to **make predictions** and display the **validation loss and accuracy** \n",
    "    * Use **tune your layers** and the **`fit` method’s hyperparameters**, or possibly change the **layer composition** of your model\n",
    "    * Can specify **separate validation data** with **`validation_data` argument** "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 15.6.5 Training and Evaluating the Model (2 of 3)\n",
    "* Model took about 2.5 minutes to train on our CPU.\n",
    "* Older computer took 5+ minutes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/5\n",
      "\u001b[1m844/844\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m11s\u001b[0m 13ms/step - accuracy: 0.9008 - loss: 0.3213 - val_accuracy: 0.9835 - val_loss: 0.0565\n",
      "Epoch 2/5\n",
      "\u001b[1m844/844\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m12s\u001b[0m 14ms/step - accuracy: 0.9876 - loss: 0.0407 - val_accuracy: 0.9897 - val_loss: 0.0358\n",
      "Epoch 3/5\n",
      "\u001b[1m844/844\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m13s\u001b[0m 15ms/step - accuracy: 0.9914 - loss: 0.0257 - val_accuracy: 0.9913 - val_loss: 0.0329\n",
      "Epoch 4/5\n",
      "\u001b[1m844/844\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m13s\u001b[0m 16ms/step - accuracy: 0.9946 - loss: 0.0187 - val_accuracy: 0.9895 - val_loss: 0.0374\n",
      "Epoch 5/5\n",
      "\u001b[1m844/844\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m13s\u001b[0m 16ms/step - accuracy: 0.9953 - loss: 0.0150 - val_accuracy: 0.9930 - val_loss: 0.0286\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.src.callbacks.history.History at 0x30efbfad0>"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cnn.fit(X_train, y_train, epochs=5, batch_size=64, validation_split=0.1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## 15.6.5 Training and Evaluating the Model (3 of 3)\n",
    "* As training proceeds, **`fit`** shows the **progress** of each **epoch**, **how long** the epoch took to execute, and the **evaluation metrics** for that epoch\n",
    "* Impressive **training accuracy (`acc`**) and **validation accurracy (`acc`)**, given that **we have not yet tried to tune the hyperparameters** or **tweak the number and types of the layers** "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!--* In the following sample output, we highlighted the training accuracy (`acc`) and validation accuracy (`val_acc`) in bold: \n",
    "\n",
    "```\n",
    "Train on 54000 samples, validate on 6000 samples  \n",
    "Epoch 1/5  \n",
    "54000/54000 [==============================] - 68s 1ms/step - loss: 0.1407 - **acc: 0.9580** - val_loss: 0.0452 - **val_acc: 0.9867**  \n",
    "Epoch 2/5  \n",
    "54000/54000 [==============================] - 64s 1ms/step - loss: 0.0426 - **acc: 0.9867** - val_loss: 0.0409 - val_acc: **0.9878**  \n",
    "Epoch 3/5  \n",
    "54000/54000 [==============================] - 69s 1ms/step - loss: 0.0299 - **acc: 0.9902** - val_loss: 0.0325 - **val_acc: 0.9912**   \n",
    "Epoch 4/5  \n",
    "54000/54000 [==============================] - 70s 1ms/step - loss: 0.0197 - **acc: 0.9935** - val_loss: 0.0335 - **val_acc: 0.9903**  \n",
    "Epoch 5/5  \n",
    "54000/54000 [==============================] - 63s 1ms/step - loss: 0.0155 - **acc: 0.9948** - val_loss: 0.0297 - **val_acc: 0.9927**\n",
    "```-->\n",
    "\n",
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Evaluating the Model on Unseen Data with Model’s **`evaluate` Method**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1m313/313\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m1s\u001b[0m 4ms/step - accuracy: 0.9894 - loss: 0.0345\n"
     ]
    }
   ],
   "source": [
    "loss, accuracy = cnn.evaluate(X_test, y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.027324868366122246"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "loss"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9919999837875366"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "accuracy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Without tuning, our **convnet model** is **99+% accurate** for **unseen data samples**\n",
    "    * Can find models online that predict MNIST with even **higher accuracy**\n",
    "    * **Experiment** with different numbers of layers, types of layers and layer parameters and observe how those changes affect your results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Making Predictions with the Model’s **`predict` Method** "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "predictions = cnn.predict(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* The first digit should be a 7 (shown as `1.` at index 7)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "y_test[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Check the **probabilities** returned by **`predict`** for **first test sample**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "for index, probability in enumerate(predictions[0]):\n",
    "    print(f'{index}: {probability:.10%}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Our model believes this digit is a 7 with **nearly** 100% certainty\n",
    "* Not all predictions have this level of certainty"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Locating the Incorrect Predictions (1 of 2)\n",
    "* View some **incorrectly predicted images** to get a sense of digits **our model has trouble with**\n",
    "\t* If the model always mispredicts 8s, perhaps we need more 8s in our training data\n",
    "* To determine whether a prediction was correct, compare the index of the largest probability in `predictions[0]` to the index of the element containing **`1.0` in `y_test[0]`**\n",
    "\t* If **indices** are the same, **prediction was correct**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Locating the Incorrect Predictions (2 of 2)\n",
    "* **Reshape the samples** from the shape `(28, 28, 1)` that Keras required for learning back to `(28, 28)`, which **Matplotlib requires to display the images**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "images = X_test.reshape((10000, 28, 28))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "incorrect_predictions = []"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* In the following snippet, **`p`** is the **predicted value array**, and **`e`** is the **expected value array**\n",
    "* **NumPy’s `argmax` function** determines **index** of an array’s **highest valued element**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "for i, (p, e) in enumerate(zip(predictions, y_test)):\n",
    "    predicted, expected = np.argmax(p), np.argmax(e)\n",
    "\n",
    "    if predicted != expected:  # prediction was incorrect\n",
    "        incorrect_predictions.append(\n",
    "            (i, images[i], predicted, expected))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "len(incorrect_predictions)  # number of incorrect predictions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Visualizing Incorrect Predictions\n",
    "* **Display 24 of the incorrect images** labeled with each image’s index, predicted value (`p`) and expected value (`e`)\n",
    "* Before reading the **expected values**, look at each digit and write down what digit you think it is\n",
    "* This is an important part of **getting to know your data**\n",
    "<!--![24 incorrectly predicted digit images](./ch15images/incorrect24.png \"24 incorrectly predicted digit images\")-->\n",
    "\n",
    " "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "figure, axes = plt.subplots(nrows=4, ncols=6, figsize=(9, 6))\n",
    "\n",
    "for axes, item in zip(axes.ravel(), incorrect_predictions):\n",
    "    index, image, predicted, expected = item\n",
    "    axes.imshow(image, cmap=plt.cm.gray_r)\n",
    "    axes.set_xticks([])  # remove x-axis tick marks\n",
    "    axes.set_yticks([])  # remove y-axis tick marks\n",
    "    axes.set_title(f'index: {index}\\np: {predicted}; e: {expected}')\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Displaying the Probabilities for Several Incorrect Predictions\n",
    "* The following function displays the probabilities for the specified prediction array:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "def display_probabilities(prediction):\n",
    "    for index, probability in enumerate(prediction):\n",
    "        print(f'{index}: {probability:.10%}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "display_probabilities(predictions[583])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "display_probabilities(predictions[1014])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "display_probabilities(predictions[2035])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 15.6.6 Saving and Loading a Model (1 of 2)\n",
    "* Can **save state** of a model\n",
    "* Can **load it later** to \n",
    "    * Make more predictions\n",
    "    * Train more\n",
    "    * Train for new problems\n",
    "    * **Transfer learning** to a new model [\\[1\\]](https://towardsdatascience.com/transfer-learning-from-pre-trained-models-f2393f124751), [\\[2\\]](https://medium.com/nanonets/nanonets-how-to-use-deep-learning-when-you-have-limited-data-f68c0b512cab)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 15.6.6 Saving and Loading a Model (2 of 2)\n",
    "* Can store **model architecture** and **state**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cnn.save('mnist_cnn.keras')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Load a saved model "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras.models import load_model\n",
    "cnn = load_model('mnist_cnn.keras')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Can then invoke its methods\n",
    "    * Could call **`predict`** to make **additional predictions on new data**\n",
    "    * Could call **`fit`** to **train with additional data**\n",
    "* [Additional functions that enable you to **save and load various aspects of your models**](https://keras.io/getting-started/faq/#how-can-i-save-a-keras-model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 15.7 Visualizing Neural Network Training with TensorBoard\n",
    "* Visualization tools like Google's [**TensorBoard**](https://github.com/tensorflow/tensorboard/blob/master/README.md) ([\\[1\\]](https://www.tensorflow.org/guide/summaries_and_tensorboard)) can help you gain insights into what goes on under the hood in deep learning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![TensorBoard visualization of a 10-epoch run of our MNIST convnet](./ch15images/tensorboard.png \"TensorBoard visualization of a 10-epoch run of our MNIST convnet\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 15.8 ConvnetJS: Browser-Based Deep-Learning Training and Visualization \n",
    "* [**Karpathy’s ConvnetJS MNIST demo presents a scrollable dashboard** that updates dynamically as the model trains](https://cs.stanford.edu/people/karpathy/convnetjs/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 15.9 Recurrent Neural Networks for Sequences; Sentiment Analysis with the IMDb Dataset (1 of 4)\n",
    "**\\[NOTE: I cover this case study in detail only if we have time in this webinar (we typically don't). See my [12-video presentation of this case study on O'Reilly Online Learning](https://learning.oreilly.com/videos/python-fundamentals/9780135917411/9780135917411-PFLL_Lesson15_41) for a complete explanation.\\]**\n",
    "&nbsp;  \n",
    "&nbsp;  \n",
    "\n",
    "* **IMDb (the Internet Movie Database) movie reviews dataset** \n",
    "    * Maas, Andrew L.  and  Daly, Raymond E.  and  Pham, Peter T.  and  Huang, Dan  and  Ng, Andrew Y.  and  Potts, Christopher, \"Learning Word Vectors for Sentiment Analysis,\" _Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies_, June 2011. Portland, Oregon, USA. Association for Computational Linguistics, pp. 142–150. http://www.aclweb.org/anthology/P11-1015.\n",
    "* Perform **binary classification** to **predict** whether a review’s **sentiment** is **positive** or **negative**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 15.9 Recurrent Neural Networks for Sequences; Sentiment Analysis with the IMDb Dataset (2 of 4)\n",
    "* **Recurrent neural networks (RNNs)** process **sequences of data**\n",
    "    * time series\n",
    "    * text in sentences\n",
    "* **“Recurrent”** because the **neural network contains loops**\n",
    "    * **Output of a given layer** becomes the **input to that same layer** in the **next time step**\n",
    "* **Time step**\n",
    "    * **Next point in time** for a **time series**\n",
    "    * **Next word in a sequence of words** for a **text sequence**\n",
    "* **Loops in RNNs** help them **learn relationships** among data in the sequence"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 15.9 Recurrent Neural Networks for Sequences; Sentiment Analysis with the IMDb Dataset (3 of 4)\n",
    "* **“Good”** on its own has **positive sentiment**\n",
    "* **“Not good”** has **negative sentiment** \n",
    "    * **“not”** is **earlier** in the sequence \n",
    "* **RNNs** take into account the **relationships** among **earlier** and **later** data in a sequence\n",
    "* When determining text's meaning, there can be **many words to consider** and an **arbitrary number of words between them**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "# 15.9 Recurrent Neural Networks for Sequences; Sentiment Analysis with the IMDb Dataset (4 of 4)\n",
    "* **Long Short-Term Memory (LSTM)** layer makes the neural network **recurrent** \n",
    "* Optimized to handle **learning from sequences**\n",
    "* RNNs have been used for many tasks including:[\\[1\\]](https://www.analyticsindiamag.com/overview-of-recurrent-neural-networks-and-their-applications/),[\\[2\\]](https://en.wikipedia.org/wiki/Recurrent_neural_network#Applications),[\\[3\\]](http://karpathy.github.io/2015/05/21/rnn-effectiveness/)\n",
    "    * **predictive text input**—displaying possible next words as you type,\n",
    "    * **sentiment analysis**\n",
    "    * **responding to questions with predicted best answers** from a corpus\n",
    "    * **inter-language translation**\n",
    "    * **automated video closed captioning** &mdash; **speech recognition**\n",
    "    * **speech synthesis**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# More Info \n",
    "* See Lesson 15 in [**Python Fundamentals LiveLessons** here on O'Reilly Online Learning](https://learning.oreilly.com/videos/python-fundamentals/9780135917411)\n",
    "* See Chapter 15 in [**Python for Programmers** on O'Reilly Online Learning](https://learning.oreilly.com/library/view/python-for-programmers/9780135231364/)\n",
    "* See Chapter 16 in [**Intro Python for Computer Science and Data Science** on O'Reilly Online Learning](https://learning.oreilly.com/library/view/intro-to-python/9780135404799/)\n",
    "* Interested in a print book? Check out:\n",
    "\n",
    "| Python for Programmers<br>(640-page professional book) | Intro to Python for Computer<br>Science and Data Science<br>(880-page college textbook)\n",
    "| :------ | :------\n",
    "| <a href=\"https://amzn.to/2VvdnxE\"><img alt=\"Python for Programmers cover\" src=\"../images/PyFPCover.png\" width=\"150\" border=\"1\"/></a> | <a href=\"https://amzn.to/2LiDCmt\"><img alt=\"Intro to Python for Computer Science and Data Science: Learning to Program with AI, Big Data and the Cloud\" src=\"../images/IntroToPythonCover.png\" width=\"159\" border=\"1\"></a>\n",
    "\n",
    ">Please **do not** purchase both books&mdash;_Python for Programmers_ is a subset of _Intro to Python for Computer Science and Data Science_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "&copy; 1992-2024 by Pearson Education, Inc. All Rights Reserved. The content in this notebook is based on the book [**Python for Programmers**](https://amzn.to/2VvdnxE)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}