ch14/Ch14.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "&copy; 2024 by Pearson Education, Inc. All Rights Reserved. The content in this notebook is based on the book [**Python for Programmers**](https://amzn.to/2VvdnxE)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%%html\n",
    "<!-- CSS settings for this notbook -->\n",
    "<style>\n",
    "    h1 {color:#BB0000}\n",
    "    h2 {color:purple}\n",
    "    h3 {color:#0099ff}\n",
    "    hr {    \n",
    "        border: 0;\n",
    "        height: 3px;\n",
    "        background: #333;\n",
    "        background-image: linear-gradient(to right, #ccc, black, #ccc);\n",
    "    }\n",
    "</style>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# enable high-res images in notebook \n",
    "%config InlineBackend.figure_format = 'retina'\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 14. Machine Learning: Classification, Regression and Clustering"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 14.1 Introduction to Machine Learning\n",
    "* **Can we make computers learn?** \n",
    "* **machine learning**—one of the most exciting and promising subfields of **artificial intelligence**\n",
    "* **Rather than programming expertise into our applications, we program them to learn from data**\n",
    "* We build machine-learning **models** that make **remarkably accurate predictions**\n",
    "* The **“secret sauce”** is **data**&mdash;and **lots of it** \n",
    "* **Quickly solve problems** that novices and experienced programmers alike probably would not have attempted just a few years ago \n",
    "* Friendly, hands-on introduction to **simpler machine-learning techniques**\n",
    "* Popular machine-learning applications (there are thousands) \n",
    "    * **Chatbots, computer vision, fraud detection, handwriting recognition, language translation, medical diagnosis, predicting loan defaults, recommender systems, self-driving cars, sentiment analysis, spam filtering, stock price forecasting and voice recognition**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 14.1.2 Two Main Types of Machine Learning "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Types of machine learning diagram](./ch14images/TypesOfMachineLearning.png \"Types of machine learning diagram\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Big Data and Big Computer Processing Power\n",
    "* **Exploding, low-cost computing power, memory and secondary storage** enable us to think differently about solution approaches \n",
    "* **Before machine learning**: “**I’m drowning in data** and I don’t know what to do with it” \n",
    "* **With machine learning**: “**Flood me with big data** so I can use machine learning to extract insights and make valuable predictions from the data”"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 14.2 Case Study: Classification with k-Nearest Neighbors and the Digits Dataset, Part 1\n",
    "* To process mail efficiently and route each letter to the correct destination, postal service computers must be able to **scan handwritten names, addresses and zip codes** and **recognize the letters and digits**\n",
    "* **Scikit-learn** enables even novice programmers to make such machine-learning problems manageable\n",
    "\n",
    "### Supervised Machine Learning: Classification \n",
    "* Attempt to **predict the distinct class** (category) to which a **sample** belongs\n",
    "    * **Binary classification**&mdash;**two** classes (e.g., “dog” or “cat”)\n",
    "* [**Digits dataset**](http://scikit-learn.org/stable/datasets/index.html#optical-recognition-of-handwritten-digits-dataset) bundled with scikit-learn\n",
    "    * 8-by-8 pixel images representing 1797 hand-written digits (0 through 9) \n",
    "* Goal: **Predict** which digit an image represents\n",
    "    * **Multi-classification**&mdash;**10 possible digits** (the classes)\n",
    "* Train a classification model using **labeled data**—know in advance each digit’s class\n",
    "* We’ll use one of the simplest machine-learning classification algorithms, **k-nearest neighbors (k-NN)**, to **recognize handwritten digits** "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 14.2.1 k-Nearest Neighbors Algorithm (k-NN) \n",
    "* Predict a sample’s class by looking at the **_k_ training samples** **nearest in \"distance\"** to the **sample** \n",
    "* Filled dots represent four distinct classes—A (blue), B (green), C (red) and D (purple) \n",
    "* **Class with the most “votes” wins**\n",
    "    * **Odd _k_ value** **avoids ties** &mdash; there’s never an equal number of votes\n",
    "    \n",
    "<img src=\"./ch14images/nearest.png\" alt=\"Diagram for the discussion of the k-nearest neighbors algorithm\" width=300/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 14.2.2 Loading the Dataset with the **`load_digits` Function** \n",
    "* Returns a **`Bunch`** object containing **digit samples** and **metadata**\n",
    "* A **`Bunch`** is a dictionary with additional **dataset-specific attributes**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.datasets import load_digits"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "digits = load_digits()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Displaying Digits Dataset's Description\n",
    "* **Digits dataset** is a subset of the [**UCI (University of California Irvine) ML hand-written digits dataset**](http://archive.ics.uci.edu/ml/datasets/Optical+Recognition+of+Handwritten+Digits)\n",
    "    * Original dataset: **5620 samples**—3823 for **training** and 1797 for **testing** \n",
    "    * **Digits dataset**: Only the **1797 testing samples**\n",
    "* A Bunch’s **`DESCR` attribute** contains dataset's description \n",
    "    * Each sample has **`64` features** (**`Number of Attributes`**) that represent an **8-by-8 image** with **pixel values `0`–`16`** (**`Attribute Information`**)\n",
    "    * **No missing values** (**`Missing Attribute Values`**) \n",
    "* **64 features** may seem like a lot\n",
    "    * Datasets can have **hundreds**, **thousands** or even **millions of features**\n",
    "    * Processing datasets like these can require enormous computing capabilities"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(digits.DESCR)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Checking the Sample and Target Sizes (1 of 2)\n",
    "* `Bunch` object’s **`data`** and **`target`** attributes are **NumPy arrays**:\n",
    "    * **`data` array**: The **1797 samples** (digit images), each with **64 features** with values**&nbsp;0** (white) to **16** (black), representing **pixel intensities**\n",
    "    ![Pixel intensities in grayscale shades from white (0) to black (16)](./ch14images/grays.png \"Pixel intensities in grayscale shades from white (0) to black (16)\")\n",
    "\n",
    "    * **`target` array**: The **images’ labels**, (classes) indicating **which digit** each image represents"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "digits.target[:20]  # first twenty target values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Checking the Sample and Target Sizes (2 of 2)\n",
    "* Confirm number of **samples** and **features** (per sample) via `data` array’s **`shape`**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "digits.data.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Confirm that **number of target values matches number of samples** via `target` array’s `shape`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "digits.target.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### A Sample Digit Image \n",
    "* Images are **two-dimensional**—width and a height in pixels \n",
    "* Digits dataset's `Bunch` object has an **`images` attribute**\n",
    "    * Each element is an **8-by-8 array** representing a **digit image’s pixel intensities**\n",
    "* Scikit-learn stores the intensity values as **NumPy type `float64`**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "digits.images[13]  # show array for sample image at index 13"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Visualization of `digits.images[13]`\n",
    "\n",
    "    <img src=\"./ch14images/sampledigit3.png\" alt=\"Image of a handwritten digit 3\" width=\"200px\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Preparing the Data for Use with Scikit-Learn (1 of 2)\n",
    "* Scikit-learn estimators require samples to be stored in a **two-dimensional array of floating-point values** (or **list of lists** or **pandas `DataFrame`**): \n",
    "\t* Each **row** represents one **sample** \n",
    "\t* Each **column** in a given row represents one **feature** for that sample\n",
    "* Multi-dimensional data samples must be **flattened** into a **one-dimensional array** \n",
    "* For **categorical features** (e.g., **strings** like `'spam'` or `'not-spam'`), you’d have to **preprocess** those features into **numerical values**—known as **one-hot encoding** (discussed later in deep learning)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Preparing the Data for Use with Scikit-Learn (2 of 2)\n",
    "* **`load_digits`** returns the **preprocessed data** ready for machine learning \n",
    "* **8-by-8 array `digits.images[13]`** corresponds to **1-by-64 array `digits.data[13]`**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "digits.data[13]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 14.2.3 Visualizing the Data\n",
    "* **Data exploration**—familiarize yourself with the data\n",
    "* **Visualizing** helps you get a sense of your data\n",
    "\n",
    "### Creating the Diagram\n",
    "* **Color map `plt.cm.gray_r`** is for **grayscale** with **0 for white**\n",
    "* [**Matplotlib’s color map names**](https://matplotlib.org/examples/color/colormaps_reference.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "figure, axes = plt.subplots(nrows=4, ncols=6, figsize=(6, 4))\n",
    "\n",
    "for item in zip(axes.ravel(), digits.images, digits.target):\n",
    "    axes, image, target = item \n",
    "    axes.imshow(image, cmap=plt.cm.gray_r)\n",
    "    axes.set_xticks([])  # remove x-axis tick marks\n",
    "    axes.set_yticks([])  # remove y-axis tick marks\n",
    "    axes.set_title(target)\n",
    "plt.tight_layout()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 14.2.4 Splitting the Data for Training and Testing\n",
    "* Typically **train** a model with a **subset of a dataset**\n",
    "* **Save a portion for testing**, so you can evaluate a model’s performance using **unseen data**\n",
    "* Function **`train_test_split`** **shuffles** the data to **randomize** it, then **splits** the **samples** in the `data` array and the **target values** in the `target` array into **training** and **testing sets**\n",
    "    * Shuffling helps ensure that the **training and testing sets** have **similar characteristics**\n",
    "* Convention: \n",
    "    * **Uppercase `X`** represents **samples**\n",
    "    * **Lowercase `y`** represents **target values**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "    digits.data, digits.target, random_state=11)  # random_state for reproducibility"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Scikit-learn bundled classification datasets have **balanced classes**\n",
    "    * Samples are **divided evenly** among the classes\n",
    "    * **Unbalanced classes** could lead to incorrect results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training and Testing Set Sizes \n",
    "* **By default**, `train_test_split` reserves **75%** of the data for **training** and **25%** for **testing**\n",
    "    * Customizable: See how in my [**Python Fundamentals LiveLessons** videos](https://learning.oreilly.com/videos/python-fundamentals/9780135917411/9780135917411-PFLL_Lesson14_11) or in [**Python for Programmers**, Section 14.2.4](https://learning.oreilly.com/library/view/python-for-programmers/9780135231364/ch14.xhtml#ch14lev2sec8)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_test.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 14.2.5 Creating the Model \n",
    "* In **scikit-learn**, **models** are called **estimators** \n",
    "* **`KNeighborsClassifier`** estimator implements the **k-nearest neighbors algorithm**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.neighbors import KNeighborsClassifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "knn = KNeighborsClassifier()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 14.2.6 Training the Model with the `KNeighborsClassifier` Object’s **`fit` method** (1 of 2)\n",
    "* Load **sample training set (`X_train`)** and **target training set (`y_train`)** into the estimator"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "knn.fit(X=X_train, y=y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* **`n_neighbors`** corresponds to **_k_ in the k-nearest neighbors algorithm** \n",
    "* [`KNeighborsClassifier` default settings](http://scikit-learn.org/stable/modules/generated/sklearn.neighbors.KNeighborsClassifier.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 14.2.6 Training the Model with the `KNeighborsClassifier` Object’s **`fit` method** (2 of 2)\n",
    "* **`fit` normally loads data** into an **estimator** then performs complex calculations **behind the scenes** that **learn** from the data to train a model\n",
    "* **`KNeighborsClassifier`’s `fit` method** **just loads the data** \n",
    "    * **No initial learning process** \n",
    "    * The **estimator** is **lazy** &mdash; work is performed only when you use it to make predictions\n",
    "* **Lots of models** have **significant training phases** that can take minutes, hours, days or more \n",
    "    * High-performance **GPUs** and **TPUs** can significantly **reduce model training time**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 14.2.7 Predicting Digit Classes with the `KNeighborsClassifier`’s  **`predict` method** (1 of 2)\n",
    "* Returns an array containing the **predicted class of each test image**: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "predicted = knn.predict(X=X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "expected = y_test"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* **`predicted` digits** vs. **`expected` digits** for the first 20 test samples&mdash;see **index 18**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "predicted[:20]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "expected[:20]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 14.2.7 Predicting Digit Classes with the `KNeighborsClassifier`’s **`predict` method** (2 of 2)\n",
    "* Locate **all incorrect predictions** for the **entire test set**: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "wrong = [(p, e) for (p, e) in zip(predicted, expected) if p != e]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "wrong"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* **Incorrectly predicted only 10 of the 450 test samples**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 14.3 Case Study: Classification with k-Nearest Neighbors and the Digits Dataset, Part 2\n",
    "## 14.3.1 Metrics for Measuring Model Accuracy \n",
    "\n",
    "### Estimator Method `score`\n",
    "* Returns an **indication of how well the estimator performs** on **test data** \n",
    "* For **classification estimators**, returns the **prediction accuracy** for the test data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f'{knn.score(X_test, y_test):.2%}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* `kNeighborsClassifier` with default **_k_** of 5 achieved **97.78% prediction accuracy** using only the estimator’s **default parameters**\n",
    "* Can use **hyperparameter tuning** to try to determine the **optimal value for _k_**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Confusion Matrix (1 of 2)\n",
    "* Shows correct and incorrect predicted values (the **hits** and **misses**) for a given class "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import confusion_matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "confusion = confusion_matrix(y_true=expected, y_pred=predicted)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "confusion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Confusion Matrix (2 of 2)\n",
    "* **Correct predictions** shown on **principal diagonal** from top-left to bottom-right\n",
    "* **Nonzero values** not on **principal diagonal** indicate **incorrect predictions** \n",
    "* Each **row** represents **one distinct class** (0–9) \n",
    "* **Columns** specify how many **test samples** were classified into classes 0–9 \n",
    "* **Row 0** shows digit class **`0`**&mdash;**all 0s were predicted correctly**\n",
    ">`[45,  0,  0,  0,  0,  0,  0,  0,  0,  0]`\n",
    "* **Row 8** shows digit class **`8`**&mdash;**five 8s were predicted incorrectly**\n",
    ">`[ 0,  1,  1,  2,  0,  0,  0,  0, 39,  1]`\n",
    "\n",
    "    * **Correctly predicted 88.63%** (39 of 44) of `8`s\n",
    "    * 8s harder to recognize"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Visualizing the Confusion Matrix \n",
    "* A **heat map** displays **values** as **colors**\n",
    "* Convert the **confusion matrix** into a **`DataFrame`**, then graph it\n",
    "* **Principal diagonal** and **incorrect predictions** stand out nicely in **heat map**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "confusion_df = pd.DataFrame(confusion, index=range(10), columns=range(10))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "confusion_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import seaborn as sns\n",
    "import matplotlib"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "figure = plt.figure(figsize=(7, 6))\n",
    "axes = sns.heatmap(confusion_df, annot=True, \n",
    "                   cmap=matplotlib.colormaps['nipy_spectral_r']) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- ### Visualizing the Confusion Matrix (3 of 4)\n",
    "![Confusion matrix displayed as a heat map](./ch14images/confusion_nipy_spectral_r.png \"Confusion matrix displayed as a heat map\") -->"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 14.3.2 K-Fold Cross-Validation\n",
    "* Uses **all of your data** for **training and testing**\n",
    "* Gives a better sense of how well your model will make predictions\n",
    "* **Splits the dataset** into **_k_ equal-size folds** (unrelated to**&nbsp;k** in the k-nearest neighbors algorithm)\n",
    "* **Repeatedly trains** your model with **_k_ – 1 folds** and **test the model** with the **remaining fold**\n",
    "* Consider using **_k_ = 10** with **folds numbered 1 through 10**\n",
    "\t* **train** with **folds 1–9**, then **test** with **fold 10**\n",
    "\t* **train** with **folds 1–8 and 10**, then **test** with **fold 9**\n",
    "\t* **train** with **folds 1–7** and **9–10**, then **test** with **fold 8**\n",
    "    * ..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `KFold` Class\n",
    "* **`KFold`** class and function **`cross_val_score`** perform **k-fold cross validation** \n",
    "* **`n_splits=10`** specifies the **number of folds**\n",
    "* **`shuffle=True`** **randomizes** the data before **splitting it into folds** \n",
    "\t* Particularly **important** if the **samples** might be **ordered** or **grouped** (as in **Iris dataset** we'll see later)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import KFold"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "kfold = KFold(n_splits=10, random_state=11, shuffle=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Calling Function `cross_val_score` to Train and Test Your Model (1 of 2)\n",
    "* **`estimator=knn`** &mdash; **estimator** to validate\n",
    "* **`X=digits.data`** &mdash; **samples** to use for training and testing\n",
    "* **`y=digits.target`** &mdash; **target predictions** for the samples\n",
    "* **`cv=kfold`** &mdash; **cross-validation generator** that defines how to **split** the **samples** and **targets** for training and testing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import cross_val_score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "scores = cross_val_score(estimator=knn, X=digits.data, y=digits.target, cv=kfold)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Calling Function `cross_val_score` to Train and Test Your Model (2 of 2)\n",
    "* Lowest accuracy was **97.78%** &mdash; one was **100%**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "scores  # array of accuracy scores for each fold"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f'Mean accuracy: {scores.mean():.2%}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Mean accuracy even better than the **97.78% we achieved** when we **trained** the model with **75%** of the data and **tested** the model with **25%** earlier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 14.3.3 Running Multiple Models to Find the Best One (1 of 3)\n",
    "* **Difficult to know in advance** which machine learning model(s) will **perform best for a given dataset**\n",
    "    * Especially when they hide the details of how they operate\n",
    "* Even though the **`KNeighborsClassifier`** predicts digit images with a high degree of accuracy, it’s **possible** that other estimators are **even more accurate**\n",
    "* Let’s **compare** **`KNeighborsClassifier`**, **`SVC`** and **`GaussianNB`**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 14.3.3 Running Multiple Models to Find the Best One (2 of 3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.svm import SVC"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.naive_bayes import GaussianNB"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* **Create the estimators** "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "estimators = {\n",
    "    'KNeighborsClassifier': knn, \n",
    "    'SVC': SVC(),\n",
    "    'GaussianNB': GaussianNB()}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 14.3.3 Running Multiple Models to Find the Best One (3 of 3)\n",
    "* **Execute the models**: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for estimator_name, estimator_object in estimators.items():\n",
    "    kfold = KFold(n_splits=10, random_state=11, shuffle=True)\n",
    "    scores = cross_val_score(estimator=estimator_object, \n",
    "        X=digits.data, y=digits.target, cv=kfold)\n",
    "    print(f'{estimator_name:>20}: ' + \n",
    "          f'mean accuracy={scores.mean():.2%}; ' +\n",
    "          f'standard deviation={scores.std():.2%}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* **`KNeighborsClassifier`** and **`SVC`** estimators’ accuracies are identical so we might want to **perform hyperparameter tuning** on each to determine the best"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 14.3.4 Hyperparameter Tuning (1 of 3)\n",
    "* In real-world machine learning studies, you’ll want to **tune hyperparameters** to choose values that produce the **best possible predictions**\n",
    "* To **determine** the **best value** for **_k_** in the **kNN algorithm**, **try different values** and **compare performance**  \n",
    "* Scikit-learn also has **automated hyperparameter tuning** capabilities"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 14.3.4 Hyperparameter Tuning (2 of 3)\n",
    "* Create `KNeighborsClassifiers` with odd **k** values from 1 through 19\n",
    "* Perform **k-fold cross-validation** on each"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for k in range(1, 20, 2):  # k is an odd value 1-19; odds prevent ties\n",
    "    kfold = KFold(n_splits=10, random_state=11, shuffle=True)\n",
    "    knn = KNeighborsClassifier(n_neighbors=k)\n",
    "    scores = cross_val_score(estimator=knn, \n",
    "        X=digits.data, y=digits.target, cv=kfold)\n",
    "    print(f'k={k:<2}; mean accuracy={scores.mean():.2%}; ' +\n",
    "          f'standard deviation={scores.std():.2%}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 14.3.4 Hyperparameter Tuning (3 of 3)\n",
    "* **Machine learning** is not without its **costs**, especially in **big data** and **deep learning**\n",
    "* **Compute time grows with _k_**, because **k-NN** needs to perform **more calculations** to find the **nearest neighbors**\n",
    "* Can use function **`cross_validate`** to perform cross-validation **and** time the results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 14.4 Case Study: Time Series and Simple Linear Regression \n",
    "**Note:** I no longer cover this case study in this webinar due to lack of time. See the full presentation in my [**Python Fundamentals** videos (8 videos in this case study)](https://learning.oreilly.com/videos/python-fundamentals/9780135917411/9780135917411-PFLL_Lesson14_23)\n",
    "\n",
    "* **Simple linear regression** is the **simplest** regression algorithm\n",
    "* Given a collection of numeric values representing an **independent variable** and a **dependent variable**, simple linear regression **describes the relationship between these variables with a straight line**, known as the **regression line**\n",
    "* Using a **time series** of average New York City January high-temperature data for 1895 through 2018, we'll\n",
    "    * Perform **simple linear regression**\n",
    "    * Display a **scatter plot** with a **regression line** \n",
    "    * Use the **coefficient** and **intercept values** calculated by the estimator to **make predictions**\n",
    "* Temperature data stored in **`ave_hi_nyc_jan_1895-2018.csv`**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 14.5 Case Study: Multiple Linear Regression with the California Housing Dataset\n",
    "**Note:** I no longer cover this case study in this webinar due to lack of time. See the full presentation in my [**Python Fundamentals** videos (10 videos in this case study)](https://learning.oreilly.com/videos/python-fundamentals/9780135917411/9780135917411-PFLL_Lesson14_31)\n",
    "\n",
    "* [**California Housing dataset**](http://lib.stat.cmu.edu/datasets) bundled with scikit-learn \n",
    "* **Larger real-world dataset** \n",
    "    **20,640 samples**, each with **eight numerical features**\n",
    "\t* Pace, R. Kelley and Ronald Barry, Sparse Spatial Autoregressions, Statistics and Probability Letters, 33 (1997) 291-297. Submitted to the StatLib Datasets Archive by Kelley Pace (kpace@unix1.sncc.lsu.edu). [9/Nov/99]. \n",
    "* Perform **multiple linear regression** using **all eight numerical features** \n",
    "    * Make **more sophisticated housing price predictions** than if we were to use only a **single feature** or a **subset of the features**\n",
    "* **`LinearRegression`** estimator performs **multiple linear regression** by default"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 14.6 Case Study: Unsupervised Machine Learning, Part 1—Dimensionality Reduction (1 of 3)\n",
    "* **Unsupervised machine learning** and **visualization** can help you do this by **finding patterns and relationships among unlabeled samples**\n",
    "* Visualizing data with **two variables** is easy\n",
    "    * Plot data in **2D** with **one variable along each axis**\n",
    "    * Visualization libraries also can plot datasets with **three variables in 3D** \n",
    "* But how do you visualize data with **more than three dimensions**?\n",
    "    * **Digits dataset** samples each have **64 features (dimensions) and a target value** \n",
    "    * **Big data** samples can have **hundreds**, **thousands** or even **millions of features (dimensions)**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 14.6 Case Study: Unsupervised Machine Learning, Part 1—Dimensionality Reduction (2 of 3)\n",
    "* Must **reduce** the data to **two** or **three dimensions**\n",
    "* **Unsupervised machine learning** technique called **dimensionality reduction** \n",
    "    * There are also **supervised dimensionality-reduction** techniques\n",
    "* **Patterns in the data** might help you **choose the most appropriate machine learning algorithms** to use\n",
    "* See **clusters** of points? Might indicate **distinct classes** of information within the dataset\n",
    "\t* So a **classification algorithm** might be appropriate. \n",
    "\t* You’d still need to **determine the class** of the samples in each cluster\n",
    "\t* This might require **consulting with a domain expert** and **studying samples in a cluster** to see **what they have in common** "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 14.6 Case Study: Unsupervised Machine Learning, Part 1—Dimensionality Reduction (3 of 3)\n",
    "* **Dimensionality reduction** also serves other purposes\n",
    "    * **Training estimators on big data** with **significant numbers of dimensions** can take **hours, days, weeks or longer**. \n",
    "    * **Difficult for humans to think about highly dimensional data**\n",
    "    * Could eliminate or combine **closely correlated features** to **improve training performance** \n",
    "        * Might **reduce the accuracy** of the model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Loading the Digits Dataset\n",
    "* Let’s **ignore Digits dataset labels** and use **dimensionality reduction** to help visualize the data in two dimensions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from sklearn.datasets import load_digits"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "digits = load_digits()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Creating a `TSNE` Estimator for Dimensionality Reduction (1 of 2)\n",
    "* Uses an algorithm called **t-distributed Stochastic Neighbor Embedding (t-SNE)** to analyze a dataset’s features and reduce them to the specified number of dimensions \n",
    "\t* [Algorithm’s details](https://scikit-learn.org/stable/modules/manifold.html#t-sne) are **beyond scope**\n",
    "\t* We first tried the popular **`PCA`** (principal components analysis) estimator but did not like the results, so we switched to **`TSNE`**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Creating a `TSNE` Estimator for Dimensionality Reduction (2 of 2)\n",
    "* Create a `TSNE` object that **reduces a dataset’s features to two dimensions** \n",
    "* `random_state` for **reproducibility of the “render sequence”** when we display the digit clusters\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.manifold import TSNE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tsne = TSNE(n_components=2, learning_rate='auto', init='pca', random_state=11) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**New: `learning_rate='auto'` and `init='pca'` added because they will soon be new defaults for TSNE.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Transforming the Digits Dataset’s Features into Two Dimensions\n",
    "* **Lecture note: Takes about 15-20 seconds, so run code first**\n",
    "* Two steps\n",
    "\t* **Train the estimator** with the dataset\n",
    "\t* **Use the estimator** to **transform** the data into the **specified number of dimensions**\n",
    "* Can **perform separately** with `TSNE` methods **`fit`** and **`transform`**\n",
    "* Perform in **one statement** using **`fit_transform`**\n",
    "    * Returns array with **same number of rows** as `digits.data` and **two columns** "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "reduced_data = tsne.fit_transform(digits.data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "reduced_data.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Visualizing the Reduced Data (1 of 2)\n",
    "* Rather than Seaborn’s `scatterplot` function, use Matplotlib’s **`scatter` function**\n",
    "    * Returns collection of plotted items, which we’ll use in a second scatter plot"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "figure = plt.figure(figsize=(5, 5))\n",
    "dots = plt.scatter(reduced_data[:, 0], reduced_data[:, 1], c='black')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- ![Black and white Digits dataset clustering scatterplot after TSNE dimensionality reduction to two dimensions](./ch14images/digits_black.png \"Black and white Digits dataset clustering scatterplot after TSNE dimensionality reduction to two dimensions\") -->"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Visualizing the Reduced Data (2 of 2)\n",
    "* **Did not label axes** &mdash; they **do not correspond to specific features** of the original dataset\n",
    "* **New features** produced by **`TSNE`** could be quite different from **dataset’s original features**\n",
    "* Clear **clusters** of related data points\n",
    "* Appear to be **11 main clusters, rather than 10** \n",
    "* Some **\"loose\" data points**  \n",
    "    * Makes sense because, as you saw, **some digits were difficult to classify**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Visualizing the Reduced Data with Different Colors for Each Digit\n",
    "* **Don’t know** whether all the **items in each cluster** represent the **same digit** \n",
    "    * If not, then the clusters are not helpful \n",
    "* Use **`target`s** in **Digits dataset** to **color the dots** to see whether clusters indeed represent specific digits\n",
    "* **`c=digits.target`** &mdash; use `target` values determine dot colors\n",
    "* Last statement adds color bar key "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "figure = plt.figure(figsize=(6, 5))\n",
    "\n",
    "dots = plt.scatter(reduced_data[:, 0], reduced_data[:, 1],\n",
    "    c=digits.target, cmap=matplotlib.colormaps['nipy_spectral_r'])\n",
    " \n",
    "colorbar = plt.colorbar(dots)  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3D Plot\n",
    "* **Lecture Note: Run `digits3d.py` first**\n",
    "* Can use Matplotlib’s **`Axes3D`** for plotting in three-dimensional graphs\n",
    "* Run provided **`digits3d.py`** file from the command line\n",
    "    * Diagram in JupyterLab is not interactive without additional tools installed"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 14.7 Case Study: Unsupervised Machine Learning, Part 2—k-Means Clustering (1 of 2)\n",
    "* **Simplest** unsupervised machine learning algorithm \n",
    "* Analyze **unlabeled samples** and **attempt to place them in clusters**\n",
    "* **_k_** hyperparameter represents **number of clusters** to impose on the data\n",
    "* Organizes clusters using **distance calculations** similar to the **k-NN classification** "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 14.7 Case Study: Unsupervised Machine Learning, Part 2—k-Means Clustering (2 of 2)\n",
    "* Each **cluster** is grouped around a **centroid** (cluster’s **center point**)\n",
    "* Initially, the algorithm **chooses _k_ centroids at random** from **dataset’s samples**\n",
    "* **Remaining samples** placed in the cluster whose **centroid is the closest** \n",
    "* **Centroids are iteratively recalculated** and **samples re-assigned** to clusters until, for all clusters, **distances** from a given centroid to the samples in its cluster are **minimized**\n",
    "Results are:\n",
    "\t* **one-dimensional array of labels** indicating **cluster** to which **each sample belongs** \n",
    "\t* **two-dimensional array of clusters' centroids** "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "### Iris Dataset \n",
    "* **Iris dataset** &mdash; commonly analyzed with **classification and clustering**\n",
    "\t* Fisher, R.A., “The use of multiple measurements in taxonomic problems,” Annual Eugenics, 7, Part II, 179-188 (1936); also in “Contributions to Mathematical Statistics” (John Wiley, NY, 1950).\n",
    "* Dataset is **labeled** &mdash; we’ll **ignore labels** to demonstrate clustering\n",
    "    * Use labels later to determine **how well k-means algorithm clustered samples**\n",
    "* **\"Toy dataset\"** &mdash; has only **150 samples** and **four features**\n",
    "    * **50 samples** for each of **three _Iris_ flower species** (balanced classes)\n",
    "    * **Iris setosa**, **Iris versicolor** and **Iris virginica**\n",
    "    * Features: **sepal length**, **sepal width**, **petal length** and **petal width**, all measured in centimeters. \n",
    "    * **Sepals** are **larger outer parts** of each flower that protect smaller inside **petals** before buds bloom"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Iris setosa**: https://commons.wikimedia.org/wiki/File:Wild_iris_KEFJ_(9025144383).jpg.\n",
    "Credit: Courtesy of Nation Park services.\n",
    "\n",
    "<img src=\"./ch14images/Wild_iris_KEFJ_(9025144383).png\" alt=\"https://commons.wikimedia.org/wiki/File:Wild_iris_KEFJ_(9025144383).jpg. Credit: Courtesy of Nation Park services.\" width=300/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Iris versicolor**: https://commons.wikimedia.org/wiki/Iris_versicolor#/media/File:IrisVersicolor-FoxRoost-Newfoundland.jpg. \n",
    "Credit: Courtesy of Jefficus, https://commons.wikimedia.org/w/index.php?title=User:Jefficus&action=edit&redlink=1\n",
    "\n",
    "<img src=\"./ch14images/IrisVersicolor-FoxRoost-Newfoundland.png\" alt=\"Iris versicolor: https://commons.wikimedia.org/wiki/Iris_versicolor#/media/File:IrisVersicolor-FoxRoost-Newfoundland.jpg. Credit: Courtesy of Jefficus, https://commons.wikimedia.org/w/index.php?title=User:Jefficus&action=edit&redlink=1.\" width=300/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Iris virginica**: https://commons.wikimedia.org/wiki/File:IMG_7911-Iris_virginica.jpg. Credit: Christer T Johansson.\n",
    "\n",
    "<img src=\"./ch14images/IMG_7911-Iris_virginica.png\" alt=\"Iris virginica: https://commons.wikimedia.org/wiki/File:IMG_7911-Iris_virginica.jpg. Credit: Christer T Johansson.\" width=300/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 14.7.1 Loading the Iris Dataset\n",
    "* **Classifies samples** by **labeling** them with the integers **0, 1 and 2**, representing **Iris setosa**, **Iris versicolor** and **Iris virginica**, respectively "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.datasets import load_iris"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "iris = load_iris()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(iris.DESCR)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Checking the Numbers of Samples, Features and Targets "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "iris.data.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "iris.target.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Array `target_names` Contains Names for the `target` Array’s Numeric Labels\n",
    "* **`dtype='<U10'`** &mdash; elements are **strings with a max of 10 characters**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "iris.target_names"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Array `feature_names` Contains Names for Each Column in the `data` array:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "iris.feature_names"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 14.7.2 Exploring the Iris Dataset: Descriptive Statistics with a Pandas `DataFrame` \n",
    "### Create a `DataFrame` containing the `data` array’s contents\n",
    "* Use **`feature_names`** as the **column names**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "pd.set_option('display.precision', 2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "iris_df = pd.DataFrame(iris.data, columns=iris.feature_names)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "iris_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Add a column containing each sample’s species name\n",
    "* **List comprehension** uses each value in **`target` array** to look up the corresponding **species name** in **`target_names` array**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "iris_df['species'] = [iris.target_names[i] for i in iris.target]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Look at a few samples  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "iris_df.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###  Calling `describe` on the `'species'` column confirms that it contains three unique values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "iris_df.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "iris_df['species'].describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* We **know in advance** that there are **three classes** to which the samples belong\n",
    "    * This is **not** typically the case in **unsupervised machine learning**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 14.7.3 Visualizing the Dataset with a Seaborn pairplot (1 of 3)\n",
    "* To **learn more about your data**, **visualize** how the features relate to one another\n",
    "* Four features &mdash; cannot graph one against other three in a single graph\n",
    "* Can **plot pairs of features** against one another \n",
    "* **Seaborn function `pairplot`** creates a **grid of graphs**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import seaborn as sns\n",
    "sns.set_style('whitegrid')\n",
    "grid = sns.pairplot(data=iris_df, vars=iris_df.columns[0:4], hue='species')\n",
    "grid.fig.set_size_inches(9, 5.5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 14.7.4 Using a `KMeans` Estimator\n",
    "* Use k-means clustering via **`KMeans` estimator** to place each sample in the Iris dataset into a cluster\n",
    "\n",
    "### Creating the `KMeans` Estimator \n",
    "* [`KMeans` default arguments](https://scikit-learn.org/stable/modules/generated/sklearn.cluster.KMeans.html)\n",
    "* When you **train a `KMeans` estimator**, it calculates for each cluster a **centroid** representing the **cluster’s center data point** \n",
    "\t* Often, you’ll rely on **domain experts** to help **choose an appropriate _k_** (`n_clusters`). \n",
    "* Can also use **hyperparameter tuning** to estimate the appropriate **k**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.cluster import KMeans\n",
    "\n",
    "kmeans = KMeans(n_clusters=3, random_state=11, n_init='auto')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Fitting the Model Via the `KMeans` object’s `fit` Method\n",
    "* When the training completes, the `KMeans` object contains: \n",
    "\t* **`labels_` array** with values from **`0` to `n_clusters - 1`**, indicating the clusters to which the samples belong\n",
    "\t* **`cluster_centers_` array** in which **each row represents a centroid**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "kmeans.fit(iris.data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Comparing the Cluster Labels to the Iris Dataset’s Target Values (1 of 2)\n",
    "* **Iris dataset** is **labeled**, so we can look at **`target` array values** to get a sense of **how well the k-means algorithm clustered the samples** \n",
    "    * With **unlabeled data**, we’d depend on a **domain expert** to help **evaluate whether the predicted classes make sense**\n",
    "* First 50 samples are **Iris setosa**, next 50 are **Iris versicolor**, last 50 are **Iris virginica**\n",
    "    * **`target` array** represents these with values **0–2** \n",
    "* If **`KMeans` chose clusters perfectly**, then **each group of 50 elements in the estimator’s `labels_` array should have a distinct label**. \n",
    "    * **`KMeans` labels** are **not related** to dataset’s **`target` array** "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Comparing the Cluster Labels to the Iris Dataset’s Target Values (2 of 2)\n",
    "* First 50 samples should be **one cluster** "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(kmeans.labels_[0:50])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Next 50 samples should be a **second cluster** (two are not)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(kmeans.labels_[50:100])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Last 50 samples should be a **third cluster** (14 are not)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(kmeans.labels_[100:150])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "kmeans.cluster_centers_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Results confirm what we saw in **`pairplot` diagrams**\n",
    "    * **Iris setosa** is “in a class by itself” \n",
    "    * There is confusion between **Iris versicolor** and **Iris virginica**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 14.7.5 Dimensionality Reduction with Principal Component Analysis\n",
    "* Use **`PCA` estimator** to perform dimensionality reduction from **4 to 2 dimensions**\n",
    "\t* [Algorithm’s details](https://scikit-learn.org/stable/modules/decomposition.html#pca) beyond scope"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.decomposition import PCA"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pca = PCA(n_components=2, random_state=11)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Transforming the Iris Dataset’s Features into Two Dimensions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pca.fit(iris.data)  # trains estimator once"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "iris_pca = pca.transform(iris.data) # can be called many times to reduce data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* We'll call **`transform`** again to **reduce the cluster centroids from four dimensions to two** for plotting \n",
    "* **`transform`** returns an array with same number of rows as `iris.data`, but only two columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "iris_pca.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Visualizing the Reduced Data in a Scatter Plot\n",
    "* Place reduced data in a **`DataFrame`** and **add a species column** that we’ll use to **determine dot colors**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "iris_pca_df = pd.DataFrame(iris_pca, \n",
    "                           columns=['Component1', 'Component2'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "iris_pca_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "iris_pca_df['species'] = iris_df.species"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "iris_pca_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Scatterplot the Data with Seaborn\n",
    "* Each **centroid** in **`cluster_centers_`** array has **same number of features** (four) as dataset's samples\n",
    "* **To plot centroids**, we must **reduce their dimensions**\n",
    "* Think of a **centroid** as the **“average” sample in its cluster**\n",
    "\t* So each centroid should be **transformed** using **same `PCA` estimator** as **other samples**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "axes = sns.scatterplot(data=iris_pca_df, x='Component1', \n",
    "    y='Component2', hue='species', legend='brief') \n",
    "\n",
    "# reduce centroids to 2 dimensions\n",
    "iris_centers = pca.transform(kmeans.cluster_centers_)\n",
    "\n",
    "# plot centroids as larger black dots\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "dots = plt.scatter(iris_centers[:,0], iris_centers[:,1], s=100, c='k')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 14.7.6 Choosing the Best Clustering Estimator (1 of 4)\n",
    "* **Run multiple clustering algorithms** and see **how well they cluster Iris species** \n",
    "\t* We’re running `KMeans` here on the **small** Iris dataset\n",
    "    * If you experience **performance problems with `KMeans`** on larger datasets, consider **`MiniBatchKMeans`**\n",
    "    * Documentation indicates **`MiniBatchKMeans` is faster on large datasets** and the results are almost as good"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 14.7.6 Choosing the Best Clustering Estimator (2 of 4)\n",
    "* For the `DBSCAN` and `MeanShift` estimators, we do **not** specify number of clusters in advance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.cluster import DBSCAN, MeanShift,\\\n",
    "    SpectralClustering, AgglomerativeClustering"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "estimators = {\n",
    "    'KMeans': kmeans,\n",
    "    'DBSCAN': DBSCAN(),\n",
    "    'MeanShift': MeanShift(),\n",
    "    'SpectralClustering': SpectralClustering(n_clusters=3),\n",
    "    'AgglomerativeClustering': \n",
    "        AgglomerativeClustering(n_clusters=3)\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 14.7.6 Choosing the Best Clustering Estimator (3 of 4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for name, estimator in estimators.items():\n",
    "    estimator.fit(iris.data)\n",
    "    print(f'\\n{name}:')\n",
    "    for i in range(0, 101, 50):\n",
    "        labels, counts = np.unique(\n",
    "            estimator.labels_[i:i+50], return_counts=True)\n",
    "        print(f'{i}-{i+49}:')\n",
    "        for label, count in zip(labels, counts):\n",
    "            print(f'   label={label}, count={count}')          "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 14.7.6 Choosing the Best Clustering Estimator (4 of 4)\n",
    "* **`DBSCAN` correctly predicted three clusters** (labeled `-1`, `0` and `1`)\n",
    "    * Placed 84 of the 100 **Iris virginica** and **Iris versicolor** in the same cluster\n",
    "* **`MeanShift` predicted only two clusters** (labeled as `0` and `1`)\n",
    "    * Placed 99 of 100 **Iris virginica** and **Iris versicolor** samples in same cluster"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# More Info \n",
    "* See Lesson 14 in [**Python Fundamentals LiveLessons** here on O'Reilly Online Learning](https://learning.oreilly.com/videos/python-fundamentals/9780135917411)\n",
    "* See Chapter 14 in [**Python for Programmers** on O'Reilly Online Learning](https://learning.oreilly.com/library/view/python-for-programmers/9780135231364/)\n",
    "* See Chapter 15 in [**Intro Python for Computer Science and Data Science** on O'Reilly Online Learning](https://learning.oreilly.com/library/view/intro-to-python/9780135404799/)\n",
    "* Interested in a print book? Check out:\n",
    "\n",
    "| Python for Programmers<br>(640-page professional book) | Intro to Python for Computer<br>Science and Data Science<br>(880-page college textbook)\n",
    "| :------ | :------\n",
    "| <a href=\"https://amzn.to/2VvdnxE\"><img alt=\"Python for Programmers cover\" src=\"../images/PyFPCover.png\" width=\"150\" border=\"1\"/></a> | <a href=\"https://amzn.to/2LiDCmt\"><img alt=\"Intro to Python for Computer Science and Data Science: Learning to Program with AI, Big Data and the Cloud\" src=\"../images/IntroToPythonCover.png\" width=\"159\" border=\"1\"></a>\n",
    "\n",
    ">Please **do not** purchase both books&mdash;_Python for Programmers_ is a subset of _Intro to Python for Computer Science and Data Science_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "&copy; 1992-2024 by Pearson Education, Inc. All Rights Reserved. The content in this notebook is based on the book [**Python for Programmers**](https://amzn.to/2VvdnxE)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}