ch14/Ch14.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "&copy; 2019 by Pearson Education, Inc. All Rights Reserved. The content in this notebook is based on the book [**Python for Programmers**](https://amzn.to/2VvdnxE)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# enable high-res images in notebook \n",
    "%config InlineBackend.figure_format = 'retina'\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 14. Machine Learning: Classification, Regression and Clustering"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 14.1 Introduction to Machine Learning\n",
    "* **Can we make computers learn?** \n",
    "* **machine learning**—one of the most exciting and promising subfields of **artificial intelligence**\n",
    "* **Rather than programming expertise into our applications, we program them to learn from data**\n",
    "* We build machine-learning **models** that make **remarkably accurate predictions**\n",
    "* The **“secret sauce”** is **data**&mdash;and **lots of it** \n",
    "* **Quickly solve problems** that novices and experienced programmers alike probably would not have attempted just a few years ago \n",
    "* Friendly, hands-on introduction to **simpler machine-learning techniques**\n",
    "* Popular machine-learning applications (there are thousands) \n",
    "    * **Chatbots, computer vision, fraud detection, handwriting recognition, language translation, medical diagnosis, predicting loan defaults, recommender systems, self-driving cars, sentiment analysis, spam filtering, stock price forecasting and voice recognition**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 14.1.2 Two Main Types of Machine Learning "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Types of machine learning diagram](./ch14images/TypesOfMachineLearning.png \"Types of machine learning diagram\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Big Data and Big Computer Processing Power\n",
    "* **Exploding, low-cost computing power, memory and secondary storage** enable us to think differently about solution approaches \n",
    "* **Before machine learning**: “**I’m drowning in data** and I don’t know what to do with it” \n",
    "* **With machine learning**: “**Flood me with big data** so I can use machine-learning technology and powerful computing capabilities to extract insights and make valuable predictions from the data”"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 14.2 Case Study: Classification with k-Nearest Neighbors and the Digits Dataset, Part 1\n",
    "* To process mail efficiently and route each letter to the correct destination, postal service computers must be able to **scan handwritten names, addresses and zip codes** and **recognize the letters and digits**\n",
    "* **Scikit-learn** enables even novice programmers to make such machine-learning problems manageable"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Supervised Machine Learning: Classification \n",
    "* Attempt to **predict the distinct class** (category) to which a **sample** belongs\n",
    "    * **Binary classification**&mdash;**two** classes (e.g., “dog” or “cat”)\n",
    "* [**Digits dataset**](http://scikit-learn.org/stable/datasets/index.html#optical-recognition-of-handwritten-digits-dataset) bundled with scikit-learn\n",
    "    * 8-by-8 pixel images representing 1797 hand-written digits (0 through 9) \n",
    "* Goal: **Predict** which digit an image represents\n",
    "    * **Multi-classification**&mdash;**10 possible digits** (the classes)\n",
    "* Train a classification model using **labeled data**—know in advance each digit’s class\n",
    "* We’ll use one of the simplest machine-learning classification algorithms, **k-nearest neighbors (k-NN)**, to **recognize handwritten digits** "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 14.2.1 k-Nearest Neighbors Algorithm (k-NN) \n",
    "* Predict a sample’s class by looking at the **_k_ training samples** **nearest in \"distance\"** to the **sample** \n",
    "* Filled dots represent four distinct classes—A (blue), B (green), C (red) and D (purple) \n",
    "* **Class with the most “votes” wins**\n",
    "    * **Odd _k_ value** **avoids ties** &mdash; there’s never an equal number of votes\n",
    "    \n",
    "<img src=\"./ch14images/nearest.png\" alt=\"Diagram for the discussion of the k-nearest neighbors algorithm\" width=300/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 14.2.2 Loading the Dataset with the **`load_digits` Function** \n",
    "* Returns a **`Bunch`** object containing **digit samples** and **metadata**\n",
    "* A **`Bunch`** is a dictionary with additional **dataset-specific attributes**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.datasets import load_digits"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "digits = load_digits()  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Displaying Digits Dataset's Description\n",
    "* **Digits dataset** is a subset of the [**UCI (University of California Irvine) ML hand-written digits dataset**](http://archive.ics.uci.edu/ml/datasets/Optical+Recognition+of+Handwritten+Digits)\n",
    "    * Original dataset: **5620 samples**—3823 for **training** and 1797 for **testing** \n",
    "    * **Digits dataset**: Only the **1797 testing samples**\n",
    "* A Bunch’s **`DESCR` attribute** contains dataset's description \n",
    "    * Each sample has **`64` features** (**`Number of Attributes`**) that represent an **8-by-8 image** with **pixel values `0`–`16`** (**`Attribute Information`**)\n",
    "    * **No missing values** (**`Missing Attribute Values`**) \n",
    "* **64 features** may seem like a lot\n",
    "    * Datasets can have **hundreds**, **thousands** or even **millions of features**\n",
    "    * Processing datasets like these can require enormous computing capabilities"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(digits.DESCR)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Checking the Sample and Target Sizes (1 of 2)\n",
    "* `Bunch` object’s **`data`** and **`target`** attributes are **NumPy arrays**:\n",
    "    * **`data` array**: The **1797 samples** (digit images), each with **64 features** with values**&nbsp;0** (white) to **16** (black), representing **pixel intensities**\n",
    "    ![Pixel intensities in grayscale shades from white (0) to black (16)](./ch14images/grays.png \"Pixel intensities in grayscale shades from white (0) to black (16)\")\n",
    "\n",
    "    * **`target` array**: The **images’ labels**, (classes) indicating **which digit** each image represents"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "digits.target[::100]  # target values of every 100th sample"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Checking the Sample and Target Sizes (2 of 2)\n",
    "* Confirm number of **samples** and **features** (per sample) via `data` array’s **`shape`**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "digits.data.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Confirm that **number of target values matches number of samples** via `target` array’s `shape`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "digits.target.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### A Sample Digit Image \n",
    "* Images are **two-dimensional**—width and a height in pixels \n",
    "* Digits dataset's `Bunch` object has an **`images` attribute**\n",
    "    * Each element is an **8-by-8 array** representing a **digit image’s pixel intensities**\n",
    "* Scikit-learn stores the intensity values as **NumPy type `float64`**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "digits.images[13]  # show array for sample image at index 13"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Visualization of `digits.images[13]`\n",
    "\n",
    "    <img src=\"./ch14images/sampledigit3.png\" alt=\"Image of a handwritten digit 3\" width=\"200px\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Preparing the Data for Use with Scikit-Learn (1 of 2)\n",
    "* Scikit-learn estimators require samples to be stored in a **two-dimensional array of floating-point values** (or **list of lists** or **pandas `DataFrame`**): \n",
    "\t* Each **row** represents one **sample** \n",
    "\t* Each **column** in a given row represents one **feature** for that sample\n",
    "* Multi-dimensional data samples must be **flattened** into a **one-dimensional array** \n",
    "* For **categorical features** (e.g., **strings** like `'spam'` or `'not-spam'`), you’d have to **preprocess** those features into **numerical values**—known as **one-hot encoding** (discussed later in deep learning)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Preparing the Data for Use with Scikit-Learn (2 of 2)\n",
    "* **`load_digits`** returns the **preprocessed data** ready for machine learning \n",
    "* **8-by-8 array `digits.images[13]`** corresponds to **1-by-64 array `digits.data[13]`**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "digits.data[13]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 14.2.3 Visualizing the Data (1 of 2)\n",
    "* Always familiarize yourself with your data&mdash;called **data exploration** \n",
    "* Let's **visualize** the dataset’s first 24 images with **Matplotlib**\n",
    "* To see **how difficult a problem handwritten digit recognition is**, consider the **variations** among the images of the 3s in the first, third and fourth rows, and look at the images of the 2s in the first, third and fourth rows. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 14.2.3 Visualizing the Data (2 of 2)\n",
    "<img src=\"./ch14images/24digits.png\" alt=\"First 24 digit images in the digits dataset\" width=400/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Creating the Diagram\n",
    "* **Color map `plt.cm.gray_r`** is for **grayscale** with **0 for white**\n",
    "* [**Matplotlib’s color map names**](https://matplotlib.org/examples/color/colormaps_reference.html)&mdash;accessible via **`plt.cm` object** or a **string, like `'gray_r'`**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "figure, axes = plt.subplots(nrows=4, ncols=6, figsize=(6, 4))\n",
    "\n",
    "for item in zip(axes.ravel(), digits.images, digits.target):\n",
    "    axes, image, target = item \n",
    "    axes.imshow(image, cmap=plt.cm.gray_r)\n",
    "    axes.set_xticks([])  # remove x-axis tick marks\n",
    "    axes.set_yticks([])  # remove y-axis tick marks\n",
    "    axes.set_title(target)\n",
    "plt.tight_layout()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 14.2.4 Splitting the Data for Training and Testing (1 of 2)\n",
    "* Typically **train** a model with a **subset of a dataset**\n",
    "* **Save a portion for testing**, so you can evaluate a model’s performance using **unseen data**\n",
    "* Function **`train_test_split`** **shuffles** the data to **randomize** it, then **splits** the **samples** in the `data` array and the **target values** in the `target` array into **training** and **testing sets**\n",
    "    * Shuffling helps ensure that the **training and testing sets** have **similar characteristics**\n",
    "    * Returns a **tuple of four elements** in which the **first two** are the **samples** split into **training** and **testing sets**, and the **last two** are the **corresponding target values** split into **training** and **testing sets**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 14.2.4 Splitting the Data for Training and Testing (2 of 2)\n",
    "* Convention: \n",
    "    * **Uppercase `X`** represents **samples**\n",
    "    * **Lowercase `y`** represents **target values**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "    digits.data, digits.target, random_state=11)  # random_state for reproducibility"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Scikit-learn bundled classification datasets have **balanced classes**\n",
    "    * Samples are **divided evenly** among the classes\n",
    "    * **Unbalanced classes** could lead to incorrect results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training and Testing Set Sizes \n",
    "* **By default**, `train_test_split` reserves **75%** of the data for **training** and **25%** for **testing**\n",
    "    * See how to customize this in my [**Python Fundamentals LiveLessons** videos](https://learning.oreilly.com/videos/python-fundamentals/9780135917411/9780135917411-PFLL_Lesson14_11) or in [**Python for Programmers**, Section 14.2.4](https://learning.oreilly.com/library/view/python-for-programmers/9780135231364/ch14.xhtml#ch14lev2sec8)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_test.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 14.2.5 Creating the Model \n",
    "* In **scikit-learn**, **models** are called **estimators** \n",
    "* **`KNeighborsClassifier`** estimator implements the **k-nearest neighbors algorithm**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.neighbors import KNeighborsClassifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "knn = KNeighborsClassifier()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 14.2.6 Training the Model with the `KNeighborsClassifier` Object’s **`fit` method** (1 of 2)\n",
    "* Load **sample training set (`X_train`)** and **target training set (`y_train`)** into the estimator"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "knn.fit(X=X_train, y=y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* **`n_neighbors`** corresponds to **_k_ in the k-nearest neighbors algorithm** \n",
    "* [`KNeighborsClassifier` default settings](http://scikit-learn.org/stable/modules/generated/sklearn.neighbors.KNeighborsClassifier.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 14.2.6 Training the Model with the `KNeighborsClassifier` Object’s **`fit` method** (2 of 2)\n",
    "* **`fit` normally loads data** into an **estimator** then performs complex calculations **behind the scenes** that **learn** from the data to train a model\n",
    "* **`KNeighborsClassifier`’s `fit` method** **just loads the data** \n",
    "    * **No initial learning process** \n",
    "    * The **estimator** is **lazy** &mdash; work is performed only when you use it to make predictions\n",
    "* **Lots of models** have **significant training phases** that can take minutes, hours, days or more \n",
    "    * High-performance **GPUs** and **TPUs** can significantly **reduce model training time**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 14.2.7 Predicting Digit Classes with the `KNeighborsClassifier`’s  **`predict` method** (1 of 2)\n",
    "* Returns an array containing the **predicted class of each test image**: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "predicted = knn.predict(X=X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "expected = y_test"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* **`predicted` digits** vs. **`expected` digits** for the first 20 test samples&mdash;see **index 18**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "predicted[:20]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "expected[:20]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 14.2.7 Predicting Digit Classes with the `KNeighborsClassifier`’s **`predict` method** (2 of 2)\n",
    "* Locate **all incorrect predictions** for the **entire test set**: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "wrong = [(p, e) for (p, e) in zip(predicted, expected) if p != e]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "wrong"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* **Incorrectly predicted only 10 of the 450 test samples**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 14.3 Case Study: Classification with k-Nearest Neighbors and the Digits Dataset, Part 2\n",
    "## 14.3.1 Metrics for Measuring Model Accuracy \n",
    "\n",
    "### Estimator Method `score`\n",
    "* Returns an **indication of how well the estimator performs** on **test data** \n",
    "* For **classification estimators**, returns the **prediction accuracy** for the test data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f'{knn.score(X_test, y_test):.2%}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* `kNeighborsClassifier` with default **_k_** of 5 achieved **97.78% prediction accuracy** using only the estimator’s **default parameters**\n",
    "* Can use **hyperparameter tuning** to try to determine the **optimal value for _k_**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Confusion Matrix (1 of 2)\n",
    "* Shows correct and incorrect predicted values (the **hits** and **misses**) for a given class "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import confusion_matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "confusion = confusion_matrix(y_true=expected, y_pred=predicted)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "confusion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Confusion Matrix (2 of 2)\n",
    "* **Correct predictions** shown on **principal diagonal** from top-left to bottom-right\n",
    "* **Nonzero values** not on **principal diagonal** indicate **incorrect predictions** \n",
    "* Each **row** represents **one distinct class** (0–9) \n",
    "* **Columns** specify how many **test samples** were classified into classes 0–9 \n",
    "* **Row 0** shows digit class **`0`**&mdash;**all 0s were predicted correctly**\n",
    ">`[45,  0,  0,  0,  0,  0,  0,  0,  0,  0]`\n",
    "* **Row 8** shows digit class **`8`**&mdash;**five 8s were predicted incorrectly**\n",
    ">`[ 0,  1,  1,  2,  0,  0,  0,  0, 39,  1]`\n",
    "\n",
    "    * **Correctly predicted 88.63%** (39 of 44) of `8`s\n",
    "    * 8s harder to recognize"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Visualizing the Confusion Matrix \n",
    "* A **heat map** displays **values** as **colors**\n",
    "* Convert the **confusion matrix** into a **`DataFrame`**, then graph it\n",
    "* **Principal diagonal** and **incorrect predictions** stand out nicely in **heat map**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "confusion_df = pd.DataFrame(confusion, index=range(10), columns=range(10))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "confusion_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import seaborn as sns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "figure = plt.figure(figsize=(7, 6))\n",
    "axes = sns.heatmap(confusion_df, annot=True, \n",
    "                   cmap=plt.cm.nipy_spectral_r) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- ### Visualizing the Confusion Matrix (3 of 4)\n",
    "![Confusion matrix displayed as a heat map](./ch14images/confusion_nipy_spectral_r.png \"Confusion matrix displayed as a heat map\") -->"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 14.3.2 K-Fold Cross-Validation\n",
    "* Uses **all of your data** for **training and testing**\n",
    "* Gives a better sense of how well your model will make predictions\n",
    "* **Splits the dataset** into **_k_ equal-size folds** (unrelated to**&nbsp;k** in the k-nearest neighbors algorithm)\n",
    "* **Repeatedly trains** your model with **_k_ – 1 folds** and **test the model** with the **remaining fold**\n",
    "* Consider using **_k_ = 10** with **folds numbered 1 through 10**\n",
    "\t* **train** with **folds 1–9**, then **test** with **fold 10**\n",
    "\t* **train** with **folds 1–8 and 10**, then **test** with **fold 9**\n",
    "\t* **train** with **folds 1–7** and **9–10**, then **test** with **fold 8**\n",
    "    * ..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `KFold` Class\n",
    "* **`KFold`** class and function **`cross_val_score`** perform **k-fold cross validation** \n",
    "* **`n_splits=10`** specifies the **number of folds**\n",
    "* **`shuffle=True`** **randomizes** the data before **splitting it into folds** \n",
    "\t* Particularly **important** if the **samples** might be **ordered** or **grouped** (as in **Iris dataset** we'll see later)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import KFold"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "kfold = KFold(n_splits=10, random_state=11, shuffle=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Calling Function `cross_val_score` to Train and Test Your Model (1 of 2)\n",
    "* **`estimator=knn`** &mdash; **estimator** to validate\n",
    "* **`X=digits.data`** &mdash; **samples** to use for training and testing\n",
    "* **`y=digits.target`** &mdash; **target predictions** for the samples\n",
    "* **`cv=kfold`** &mdash; **cross-validation generator** that defines how to **split** the **samples** and **targets** for training and testing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import cross_val_score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "scores = cross_val_score(estimator=knn, X=digits.data, y=digits.target, cv=kfold)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Calling Function `cross_val_score` to Train and Test Your Model (2 of 2)\n",
    "* Lowest accuracy was **97.78%** &mdash; one was **100%**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "scores  # array of accuracy scores for each fold"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f'Mean accuracy: {scores.mean():.2%}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Mean accuracy even better than the **97.78% we achieved** when we **trained** the model with **75%** of the data and **tested** the model with **25%** earlier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 14.3.3 Running Multiple Models to Find the Best One (1 of 3)\n",
    "* **Difficult to know in advance** which machine learning model(s) will **perform best for a given dataset**\n",
    "    * Especially when they hide the details of how they operate\n",
    "* Even though the **`KNeighborsClassifier`** predicts digit images with a high degree of accuracy, it’s **possible** that other estimators are **even more accurate**\n",
    "* Let’s **compare** **`KNeighborsClassifier`**, **`SVC`** and **`GaussianNB`**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 14.3.3 Running Multiple Models to Find the Best One (2 of 3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.svm import SVC"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.naive_bayes import GaussianNB"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* **Create the estimators** \n",
    "* To avoid a scikit-learn warning, we supplied a keyword argument when creating the **`SVC`** estimator\n",
    "    * This argument’s value will become the default in scikit-learn version 0.22"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "estimators = {\n",
    "    'KNeighborsClassifier': knn, \n",
    "    'SVC': SVC(gamma='scale'),\n",
    "    'GaussianNB': GaussianNB()}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 14.3.3 Running Multiple Models to Find the Best One (3 of 3)\n",
    "* **Execute the models**: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for estimator_name, estimator_object in estimators.items():\n",
    "    kfold = KFold(n_splits=10, random_state=11, shuffle=True)\n",
    "    scores = cross_val_score(estimator=estimator_object, \n",
    "        X=digits.data, y=digits.target, cv=kfold)\n",
    "    print(f'{estimator_name:>20}: ' + \n",
    "          f'mean accuracy={scores.mean():.2%}; ' +\n",
    "          f'standard deviation={scores.std():.2%}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* **`KNeighborsClassifier`** and **`SVC`** estimators’ accuracies are identical so we might want to **perform hyperparameter tuning** on each to determine the best"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 14.3.4 Hyperparameter Tuning (1 of 3)\n",
    "* In real-world machine learning studies, you’ll want to **tune hyperparameters** to choose values that produce the **best possible predictions**\n",
    "* To **determine** the **best value** for **_k_** in the **kNN algorithm**, **try different values** and **compare performance**  \n",
    "* Scikit-learn also has **automated hyperparameter tuning** capabilities"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 14.3.4 Hyperparameter Tuning (2 of 3)\n",
    "* Create `KNeighborsClassifiers` with odd **k** values from 1 through 19\n",
    "* Perform **k-fold cross-validation** on each"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for k in range(1, 20, 2):  # k is an odd value 1-19; odds prevent ties\n",
    "    kfold = KFold(n_splits=10, random_state=11, shuffle=True)\n",
    "    knn = KNeighborsClassifier(n_neighbors=k)\n",
    "    scores = cross_val_score(estimator=knn, \n",
    "        X=digits.data, y=digits.target, cv=kfold)\n",
    "    print(f'k={k:<2}; mean accuracy={scores.mean():.2%}; ' +\n",
    "          f'standard deviation={scores.std():.2%}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 14.3.4 Hyperparameter Tuning (3 of 3)\n",
    "* **Machine learning** is not without its **costs**, especially in **big data** and **deep learning**\n",
    "* **Compute time grows with _k_**, because **k-NN** needs to perform **more calculations** to find the **nearest neighbors**\n",
    "* Can use function **`cross_validate`** to perform cross-validation **and** time the results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 14.4 Case Study: Time Series and Simple Linear Regression \n",
    "* **Simple linear regression** is the **simplest** regression algorithm\n",
    "* Given a collection of numeric values representing an **independent variable** and a **dependent variable**, simple linear regression **describes the relationship between these variables with a straight line**, known as the **regression line**\n",
    "* Using a **time series** of average New York City January high-temperature data for 1895 through 2018, we'll\n",
    "    * Perform **simple linear regression**\n",
    "    * Display a **scatter plot** with a **regression line** \n",
    "    * Use the **coefficient** and **intercept values** calculated by the estimator to **make predictions**\n",
    "* Temperature data stored in **`ave_hi_nyc_jan_1895-2018.csv`**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Loading the Average High Temperatures into a `DataFrame` \n",
    "* Load the data from `ave_hi_nyc_jan_1895-2018.csv`, rename the `'Value'` column to `'Temperature'`, remove `01` from the end of each date value and display a few data samples:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nyc = pd.read_csv('ave_hi_nyc_jan_1895-2018.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nyc.head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nyc.columns = ['Date', 'Temperature', 'Anomaly']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nyc.Date = nyc.Date.floordiv(100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nyc.head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Splitting the Data for Training and Testing (1 of 3)\n",
    "* We’ll use the **`LinearRegression`** estimator from **`sklearn.linear_model`** \n",
    "* By default, this estimator uses **all** the **numeric features** in a dataset to perform **multiple linear regression**  \n",
    "* For **simple linear regression** select **one** feature (the `Date` here) as the **independent variable**\n",
    "    * A column in `DataFrame` is a **one-dimensional** `Series` \n",
    "    * Scikit-learn estimators require training and testing data to be **two-dimensional** \n",
    "    * We'll transform **`Series` of _n_** elements, into two dimensions containing **_n_ rows** and **one column** "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Splitting the Data for Training and Testing (2 of 3)\n",
    "* `nyc.Date.values` returns NumPy array containing `Date` column’s values\n",
    "* **`reshape(-1, 1)`** tells `reshape` to **infer** the number of rows, based on the number of columns (`1`) and the number of elements (124) in the array\n",
    "    * Transformed array will have 124 rows and one column"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "    nyc.Date.values.reshape(-1, 1), nyc.Temperature.values, random_state=11)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Splitting the Data for Training and Testing (3 of 3)\n",
    "* Confirm the **75%–25% train-test split** "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_test.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training the Model (1 of 2)\n",
    "* [**LinearRegression default settings**](http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LinearRegression.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LinearRegression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "linear_regression = LinearRegression()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "linear_regression.fit(X=X_train, y=y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* To find the **best fitting regression line** for the data, the `LinearRegression` estimator **iteratively adjusts** the **slope** and **intercept** to **minimize** the **sum of the squares** of the data points’ **distances** from the line \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training the Model (2 of 2)\n",
    "* We'll soon use **slope** and **intercept** to make **predictions** with \n",
    "\n",
    "\\begin{equation}\n",
    "y = m x + b\n",
    "\\end{equation}\n",
    "\n",
    "* Slope is the estimator’s **`coeff_`** attribute (**m** in the equation) \n",
    "* Intercept is the estimator’s **`intercept_`** attribute (**b** in the equation)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "linear_regression.coef_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "linear_regression.intercept_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Testing the Model\n",
    "* Test the model using the data in **`X_test`** and check some of the **predictions**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "predicted = linear_regression.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "expected = y_test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for p, e in zip(predicted[::5], expected[::5]):  # check every 5th element\n",
    "    print(f'predicted: {p:.2f}, expected: {e:.2f}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Predicting Future Temperatures and Estimating Past Temperatures \n",
    "* Use the **coefficient** and **intercept** values to make **predictions** "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# lambda implements y = mx + b\n",
    "predict = (lambda x: linear_regression.coef_ * x + \n",
    "                     linear_regression.intercept_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "predict(2019)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "predict(1890)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Visualizing the Dataset with the Regression Line \n",
    "* Create a **scatter plot** with a regression line \n",
    "* **Cooler** temperatures shown in **darker colors**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import seaborn as sns\n",
    "\n",
    "axes = sns.scatterplot(data=nyc, x='Date', y='Temperature',\n",
    "    hue='Temperature', palette='winter', legend=False)  \n",
    "\n",
    "axes.set_ylim(10, 70)  # scale y-axis \n",
    "\n",
    "import numpy as np\n",
    "\n",
    "x = np.array([min(nyc.Date.values), max(nyc.Date.values)])\n",
    "\n",
    "y = predict(x)\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "line = plt.plot(x, y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Common Problems That Prevent Accurate Predictions: Overfitting/Underfitting\n",
    "* When creating a model, key goal is **making accurate predictions** for **data it has not yet seen** \n",
    "* **Underfitting** occurs when a **model is too simple to make predictions**, based on its training data\n",
    "    * You may use a **linear model**, such as **simple linear regression**, when problem really requires a **non-linear model**\n",
    "* **Overfitting** occurs when your **model is too complex**\n",
    "    * **Most extreme case** would be a **model that memorizes its training data**\n",
    "    * New data that **matches the training data** will produce **perfect predictions**, but the model will not know what to do with data it has never seen."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 14.5 Case Study: Multiple Linear Regression with the California Housing Dataset\n",
    "* [**California Housing dataset**](http://lib.stat.cmu.edu/datasets) bundled with scikit-learn \n",
    "* **Larger real-world dataset** \n",
    "    **20,640 samples**, each with **eight numerical features**\n",
    "\t* Pace, R. Kelley and Ronald Barry, Sparse Spatial Autoregressions, Statistics and Probability Letters, 33 (1997) 291-297. Submitted to the StatLib Datasets Archive by Kelley Pace (kpace@unix1.sncc.lsu.edu). [9/Nov/99]. \n",
    "* Perform **multiple linear regression** using **all eight numerical features** \n",
    "    * Make **more sophisticated housing price predictions** than if we were to use only a **single feature** or a **subset of the features**\n",
    "* **`LinearRegression`** estimator performs **multiple linear regression** by default"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 14.5.1 Loading the Dataset (1 of 3)\n",
    "* According to the California Housing Prices dataset’s description in scikit-learn\n",
    "> \"This dataset was **derived from the 1990 U.S. census**, using **one row per census block group**.  \n",
    ">  \n",
    "> \"A **block group** is the **smallest geographical unit** for which the U.S. Census Bureau publishes sample data (typically has a **population of 600 to 3,000 people**).\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 14.5.1 Loading the Dataset (2 of 3)\n",
    "* The dataset has **20,640 samples**—**one per block group**—with **eight features** each:\n",
    "\t* **median income**—in tens of thousands, so 8.37 would represent $83,700\n",
    "\t* **median house age**—in the dataset, the maximum value for this feature is 52\n",
    "\t* **average number of rooms** \n",
    "\t* **average number of bedrooms** \n",
    "\t* **block population**\n",
    "\t* **average house occupancy**\n",
    "\t* **house block latitude**\n",
    "\t* **house block longitude**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 14.5.1 Loading the Dataset (3 of 3)\n",
    "* **Target** &mdash; **median house value** in hundreds of thousands, so 3.55 would represent \\$355,000\n",
    "    * **Maximum** for this feature is**&nbsp;5** for **\\$500,000** \n",
    "* Reasonable to expect **more bedrooms**, **more rooms** or **higher income** would mean **higher house value**\n",
    "* **Combine all numeric features to make predictions**\n",
    "    * More likely to get **more accurate predictions** than with simple linear regression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Loading the Data with `sklearn.datasets` Function **`fetch_california_housing`** "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.datasets import fetch_california_housing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "california = fetch_california_housing()  # Bunch object "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(california.DESCR)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Getting to Know the Data\n",
    "* Confirm number of **samples/features**, number of **targets**, **feature names**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "california.data.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "california.target.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "california.feature_names"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 14.5.2 Exploring the Data with a Pandas `DataFrame` (1 of 4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.set_option('precision', 2)  # 2 digit precision for floats"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- # Used for command line outputs in IPython interactive mode\n",
    "#pd.set_option('max_columns', 9)  # display up to 9 columns in DataFrame outputs\n",
    "\n",
    "#pd.set_option('display.width', None)  # auto-detect the display width for wrapping\n",
    "-->"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 14.5.2 Exploring the Data with a Pandas `DataFrame` (2 of 4)\n",
    "* Second statement adds a **`DataFrame` column** for **median house values**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "california_df = pd.DataFrame(california.data, \n",
    "                             columns=california.feature_names)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "california_df['MedHouseValue'] = pd.Series(california.target)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 14.5.2 Exploring the Data with a Pandas `DataFrame` (3 of 4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "california_df.head()  # peek at first 5 rows"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 14.5.2 Exploring the Data with a Pandas `DataFrame` (4 of 4)\n",
    "* Calculate **`DataFrame`’s summary statistics** \n",
    "* Median income and house values are from 1990 and are **significantly higher today** \n",
    "* Output is **left-to-right scrollable in Jupyter** if it does not fit in your screen width"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "california_df.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 14.5.3 Visualizing the Features (1 of 3)\n",
    "* Helpful to **visualize** data by **plotting the target value** against **each** feature\n",
    "    Shows how **median home value** relates to **each feature**\n",
    "* To make our visualizations clearer, let’s use **`DataFrame` method **`sample`**** to **randomly select 10% of the 20,640 samples** for graphing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sample_df = california_df.sample(frac=0.1, random_state=17)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 14.5.3 Visualizing the Features (2 of 3)\n",
    "* Display **scatter plots** of several **features** \n",
    "* Each shows **feature** on **x-axis** and **median home value** on **y-axis**\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "sns.set_style('whitegrid')                                    \n",
    "\n",
    "for feature in ['MedInc', 'HouseAge', 'Latitude']:\n",
    "    plt.figure(figsize=(8, 4.5))  # 8\"-by-4.5\" Figure\n",
    "    sns.scatterplot(data=sample_df, x=feature, \n",
    "                    y='MedHouseValue', hue='MedHouseValue', \n",
    "                    palette='cool', legend=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 14.5.3 Visualizing the Features (3 of 3)\n",
    "* Some **interesting things** to notice in these graphs:\n",
    "\t* **Latitude and longitude graphs** each have **two areas** of especially significant density&mdash;**greater Los Angeles** and **greater San Francisco** areas where house prices tend to be higher\n",
    "    * Each graph shows a **horizontal line of dots at the y-axis value 5**, which represents the [**maximum median house value \\$500,000** listed in the 1990 census form](https://www.census.gov/prod/1/90dec/cph4/appdxe.pdf)\n",
    "    * **`HouseAge` graph** shows a **vertical line of dots** at the **x-axis value 52**\n",
    "        * **Highest home age** on the 1990 census form was **52**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- ![California Housing Dataset scatterplot of Median House Value vs. Median Income](./ch14images/medincome.png \"California Housing Dataset scatterplot of Median House Value vs. Median Income\")\n",
    " ![California Housing Dataset scatterplot of Median House Value vs. House Age](./ch14images/houseage.png \"California Housing Dataset scatterplot of Median House Value vs. House Age\")\n",
    " ![California Housing Dataset scatterplot of Median House Value vs. Average Rooms](./ch14images/averooms.png \"California Housing Dataset scatterplot of Median House Value vs. Average Rooms\")\n",
    " ![California Housing Dataset scatterplot of Median House Value vs. Average Bedrooms](./ch14images/avebedrooms.png \"California Housing Dataset scatterplot of Median House Value vs. Average Bedrooms\")\n",
    " ![California Housing Dataset scatterplot of Median House Value vs. Population](./ch14images/population.png \"California Housing Dataset scatterplot of Median House Value vs. Population\")\n",
    " ![California Housing Dataset scatterplot of Median House Value vs. Average Occupancy](./ch14images/aveoccupancy.png \"California Housing Dataset scatterplot of Median House Value vs. Average Occupancy\")\n",
    " ![California Housing Dataset scatterplot of Median House Value vs. Lattitude](./ch14images/lattitude.png \"California Housing Dataset scatterplot of Median House Value vs. Lattitude\")\n",
    " ![California Housing Dataset scatterplot of Median House Value vs. Longitude](./ch14images/longitude.png \"California Housing Dataset scatterplot of Median House Value vs. Longitude\")<hr style=\"height:2px; border:none; color:black; background-color:black;\"> -->"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 14.5.4 Splitting the Data for Training and Testing Using `train_test_split`  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "    california.data, california.target, random_state=11)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_test.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 14.5.5 Training the Model (1 of 3)\n",
    "* **`LinearRegression`** tries to use **all** features in a dataset’s `data` array\n",
    "    * **error** if any features are **categorical**  \n",
    "    * Categorical data must be preprocessed into numerical data or excluded\n",
    "* **Scikit-learn’s bundled datasets** are already in the **correct format** for training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LinearRegression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "linear_regression = LinearRegression()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "linear_regression.fit(X=X_train, y=y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 14.5.5 Training the Model (2 of 3)\n",
    "* **Separate coefficients** for each feature (stored in `coeff_`) and **one intercept** (stored in `intercept_`) \n",
    "    * **Positive coefficients** &mdash; median house value **increases** as feature value **increases** \n",
    "    * **Negative coefficients** &mdash; median house value **decreases** as feature value **increases**\n",
    "    * **HouseAge**, **AveOccup** and **Population** are **close to zero**, so these apparently have little to no affect on **median house value**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for i, name in enumerate(california.feature_names):\n",
    "    print(f'{name:>10}: {linear_regression.coef_[i]}')  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "linear_regression.intercept_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 14.5.5 Training the Model (3 of 3)\n",
    "* Can use coefficient values in following equation to **make predictions**:\n",
    "\n",
    "\\begin{equation}\n",
    "y = m_1 x_1 + m_2 x_2 + ... + m_n x_n + b\n",
    "\\end{equation}\n",
    "\n",
    "* <em>m</em><sub>1</sub>, <em>m</em><sub>2</sub>, …, <em>m</em><sub><em>n</em></sub> are the **feature coefficients**\n",
    "* <em>b</em> is the **intercept**\n",
    "* <em>x</em><sub>1</sub>, <em>x</em><sub>2</sub>, …, <em>x</em><sub><em>n</em></sub> are **feature values** (the **independent variables**)\n",
    "* <em>y</em> is the **predicted value** (the **dependent variable**)\n",
    "   "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 14.5.6 Testing the Model with the Estimator’s `predict `Method (1 of 2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "predicted = linear_regression.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "expected = y_test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "predicted[:5]  # first 5 predictions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "expected[:5]   # first five targets "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 14.5.6 Testing the Model with the Estimator’s `predict `Method (2 of 2)\n",
    "* In **classification**, **predictions** were **distinct classes** that **matched existing classes** in the dataset\n",
    "* In **regression**, it’s **tough to get exact predictions**, because you have **continuous outputs**\n",
    "    * Every possible value of <em>x</em><sub>1</sub>, <em>x</em><sub>2</sub> … <em>x</em><sub><em>n</em></sub> in the following calculation predicts a value\n",
    "\n",
    "\\begin{equation}\n",
    "y = m_1 x_1 + m_2 x_2 + ... + m_n x_n + b\n",
    "\\end{equation}\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 14.5.8 Regression Model Metrics\n",
    "* **Metrics for regression estimators** include **coefficient of determination** (**$R^{2}$ score**; 0.0-1.0)\n",
    "    * **1.0** &mdash; estimator **perfectly predicts** the **dependent variable’s value**, given independent variables' values\n",
    "    * **0.0** &mdash; **model cannot make predictions with any accuracy**, given independent variables’ values \n",
    "* Calculate with arrays representing the **expected** and **predicted results**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn import metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "metrics.r2_score(expected, predicted)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 14.5.9 Choosing the Best Model (1 of 2)\n",
    "* **Try several estimators** to determine whether any **produces better results** than `LinearRegression` \n",
    "* [Information about estimators used here](https://scikit-learn.org/stable/modules/linear_model.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.linear_model import ElasticNet, Lasso, Ridge"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "estimators = {\n",
    "    'LinearRegression': linear_regression,\n",
    "    'ElasticNet': ElasticNet(),\n",
    "    'Lasso': Lasso(),\n",
    "    'Ridge': Ridge()\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 14.5.9 Choosing the Best Model (2 of 2)\n",
    "* Run the estimators using **k-fold cross-validation** \n",
    "* **`cross_val_score` argument `scoring='r2'`** &mdash; report **$R^{2}$ scores** for **each fold**\n",
    "    * **1.0 is best**, so **`LinearRegression`** and **`Ridge`** appear to be **best models** for this dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import KFold, cross_val_score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for estimator_name, estimator_object in estimators.items():\n",
    "    kfold = KFold(n_splits=10, random_state=11, shuffle=True)\n",
    "    scores = cross_val_score(estimator=estimator_object, \n",
    "        X=california.data, y=california.target, cv=kfold,\n",
    "        scoring='r2')\n",
    "    print(f'{estimator_name:>16}: ' + \n",
    "          f'mean of r2 scores={scores.mean():.3f}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 14.6 Case Study: Unsupervised Machine Learning, Part 1—Dimensionality Reduction (1 of 3)\n",
    "* **Unsupervised machine learning** and **visualization** can help you do this by **finding patterns and relationships among unlabeled samples**\n",
    "* Visualizing data with **two variables** is easy\n",
    "    * Plot data in **2D** with **one variable along each axis**\n",
    "    * Visualization libraries also can plot datasets with **three variables in 3D** \n",
    "* But how do you visualize data with **more than three dimensions**?\n",
    "    * **Digits dataset** samples each have **64 features (dimensions) and a target value** \n",
    "    * **Big data** samples can have **hundreds**, **thousands** or even **millions of features (dimensions)**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 14.6 Case Study: Unsupervised Machine Learning, Part 1—Dimensionality Reduction (2 of 3)\n",
    "* Must **reduce** the data to **two** or **three dimensions**\n",
    "* **Unsupervised machine learning** technique called **dimensionality reduction** \n",
    "    * There are also **supervised dimensionality-reduction** techniques\n",
    "* **Patterns in the data** might help you **choose the most appropriate machine learning algorithms** to use\n",
    "* See **clusters** of points? Might indicate **distinct classes** of information within the dataset\n",
    "\t* So a **classification algorithm** might be appropriate. \n",
    "\t* You’d still need to **determine the class** of the samples in each cluster\n",
    "\t* This might require **consulting with a domain expert** and **studying samples in a cluster** to see **what they have in common** "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 14.6 Case Study: Unsupervised Machine Learning, Part 1—Dimensionality Reduction (3 of 3)\n",
    "* **Dimensionality reduction** also serves other purposes\n",
    "    * **Training estimators on big data** with **significant numbers of dimensions** can take **hours, days, weeks or longer**. \n",
    "    * **Difficult for humans to think about highly dimensional data**\n",
    "    * Could eliminate or combine **closely correlated features** to **improve training performance** \n",
    "        * Might **reduce the accuracy** of the model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Loading the Digits Dataset\n",
    "* Let’s **ignore Digits dataset labels** and use **dimensionality reduction** to help visualize the data in two dimensions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.datasets import load_digits"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "digits = load_digits()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Creating a `TSNE` Estimator for Dimensionality Reduction (1 of 2)\n",
    "* Uses an algorithm called **t-distributed Stochastic Neighbor Embedding (t-SNE)** to analyze a dataset’s features and reduce them to the specified number of dimensions \n",
    "\t* [Algorithm’s details](https://scikit-learn.org/stable/modules/manifold.html#t-sne) are **beyond scope**\n",
    "\t* We first tried the popular **`PCA`** (principal components analysis) estimator but did not like the results, so we switched to **`TSNE`**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Creating a `TSNE` Estimator for Dimensionality Reduction (2 of 2)\n",
    "* Create a `TSNE` object that **reduces a dataset’s features to two dimensions** \n",
    "* `random_state` for **reproducibility of the “render sequence”** when we display the digit clusters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.manifold import TSNE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tsne = TSNE(n_components=2, random_state=11)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Transforming the Digits Dataset’s Features into Two Dimensions\n",
    "* **Lecture note: Takes about 15-20 seconds, so run code first**\n",
    "* Two steps\n",
    "\t* **Train the estimator** with the dataset\n",
    "\t* **Use the estimator** to **transform** the data into the **specified number of dimensions**\n",
    "* Can **perform separately** with `TSNE` methods **`fit`** and **`transform`**\n",
    "* Perform in **one statement** using **`fit_transform`**\n",
    "    * Returns array with **same number of rows** as `digits.data` and **two columns** "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "reduced_data = tsne.fit_transform(digits.data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "reduced_data.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Visualizing the Reduced Data (1 of 2)\n",
    "* Rather than Seaborn’s `scatterplot` function, use Matplotlib’s **`scatter` function**\n",
    "    * Returns collection of plotted items, which we’ll use in a second scatter plot"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "figure = plt.figure(figsize=(5, 5))\n",
    "dots = plt.scatter(reduced_data[:, 0], reduced_data[:, 1], c='black')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<!-- ![Black and white Digits dataset clustering scatterplot after TSNE dimensionality reduction to two dimensions](./ch14images/digits_black.png \"Black and white Digits dataset clustering scatterplot after TSNE dimensionality reduction to two dimensions\") -->"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Visualizing the Reduced Data (2 of 2)\n",
    "* **Did not label axes** &mdash; they **do not correspond to specific features** of the original dataset\n",
    "* **New features** produced by **`TSNE`** could be quite different from **dataset’s original features**\n",
    "* Clear **clusters** of related data points\n",
    "* Appear to be **11 main clusters, rather than 10** \n",
    "* Some **\"loose\" data points**  \n",
    "    * Makes sense because, as you saw, **some digits were difficult to classify**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Visualizing the Reduced Data with Different Colors for Each Digit\n",
    "* **Don’t know** whether all the **items in each cluster** represent the **same digit** \n",
    "    * If not, then the clusters are not helpful \n",
    "* Use **`target`s** in **Digits dataset** to **color the dots** to see whether clusters indeed represent specific digits\n",
    "* **`c=digits.target`** &mdash; use `target` values determine dot colors\n",
    "* **`cmap=plt.cm.get_cmap('nipy_spectral_r', 10)`** &mdash; **color map** to use \n",
    "    * Specifically use **10 distinct colors** for the 10 digits \n",
    "* Last statement adds color bar key "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "figure = plt.figure(figsize=(6, 5))\n",
    "\n",
    "dots = plt.scatter(reduced_data[:, 0], reduced_data[:, 1],\n",
    "    c=digits.target, cmap=plt.cm.get_cmap('nipy_spectral_r', 10))\n",
    " \n",
    "colorbar = plt.colorbar(dots)  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3D Plot\n",
    "* **Lecture Note: Run `digits3d.py` first**\n",
    "* Can use Matplotlib’s **`Axes3D`** for plotting in three-dimensional graphs\n",
    "* Run provided **`digits3d.py`** file from the command line\n",
    "    * Diagram in JupyterLab is not interactive without additional tools installed"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 14.7 Case Study: Unsupervised Machine Learning, Part 2—k-Means Clustering (1 of 2)\n",
    "* **Simplest** unsupervised machine learning algorithm \n",
    "* Analyze **unlabeled samples** and **attempt to place them in clusters**\n",
    "* **_k_** hyperparameter represents **number of clusters** to impose on the data\n",
    "* Organizes clusters using **distance calculations** similar to the **k-NN classification** "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 14.7 Case Study: Unsupervised Machine Learning, Part 2—k-Means Clustering (2 of 2)\n",
    "* Each **cluster** is grouped around a **centroid** (cluster’s **center point**)\n",
    "* Initially, the algorithm **chooses _k_ centroids at random** from **dataset’s samples**\n",
    "* **Remaining samples** placed in the cluster whose **centroid is the closest** \n",
    "* **Centroids are iteratively recalculated** and **samples re-assigned** to clusters until, for all clusters, **distances** from a given centroid to the samples in its cluster are **minimized**\n",
    "Results are:\n",
    "\t* **one-dimensional array of labels** indicating **cluster** to which **each sample belongs** \n",
    "\t* **two-dimensional array of clusters' centroids** "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Iris Dataset \n",
    "* **Iris dataset** &mdash; commonly analyzed with **classification and clustering**\n",
    "\t* Fisher, R.A., “The use of multiple measurements in taxonomic problems,” Annual Eugenics, 7, Part II, 179-188 (1936); also in “Contributions to Mathematical Statistics” (John Wiley, NY, 1950).\n",
    "* Dataset is **labeled** &mdash; we’ll **ignore labels** to demonstrate clustering\n",
    "    * Use labels later to determine **how well k-means algorithm clustered samples**\n",
    "* **\"Toy dataset\"** &mdash; has only **150 samples** and **four features**\n",
    "    * **50 samples** for each of **three _Iris_ flower species** (balanced classes)\n",
    "    * **Iris setosa**, **Iris versicolor** and **Iris virginica**\n",
    "    * Features: **sepal length**, **sepal width**, **petal length** and **petal width**, all measured in centimeters. \n",
    "    * **Sepals** are **larger outer parts** of each flower that protect smaller inside **petals** before buds bloom"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Iris setosa**: https://commons.wikimedia.org/wiki/File:Wild_iris_KEFJ_(9025144383).jpg.\n",
    "Credit: Courtesy of Nation Park services.\n",
    "\n",
    "<img src=\"./ch14images/Wild_iris_KEFJ_(9025144383).png\" alt=\"https://commons.wikimedia.org/wiki/File:Wild_iris_KEFJ_(9025144383).jpg. Credit: Courtesy of Nation Park services.\" width=300/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Iris versicolor**: https://commons.wikimedia.org/wiki/Iris_versicolor#/media/File:IrisVersicolor-FoxRoost-Newfoundland.jpg. \n",
    "Credit: Courtesy of Jefficus, https://commons.wikimedia.org/w/index.php?title=User:Jefficus&action=edit&redlink=1\n",
    "\n",
    "<img src=\"./ch14images/IrisVersicolor-FoxRoost-Newfoundland.png\" alt=\"Iris versicolor: https://commons.wikimedia.org/wiki/Iris_versicolor#/media/File:IrisVersicolor-FoxRoost-Newfoundland.jpg. Credit: Courtesy of Jefficus, https://commons.wikimedia.org/w/index.php?title=User:Jefficus&action=edit&redlink=1.\" width=300/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Iris virginica**: https://commons.wikimedia.org/wiki/File:IMG_7911-Iris_virginica.jpg. Credit: Christer T Johansson.\n",
    "\n",
    "<img src=\"./ch14images/IMG_7911-Iris_virginica.png\" alt=\"Iris virginica: https://commons.wikimedia.org/wiki/File:IMG_7911-Iris_virginica.jpg. Credit: Christer T Johansson.\" width=300/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 14.7.1 Loading the Iris Dataset\n",
    "* **Classifies samples** by **labeling** them with the integers **0, 1 and 2**, representing **Iris setosa**, **Iris versicolor** and **Iris virginica**, respectively "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.datasets import load_iris"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "iris = load_iris()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(iris.DESCR)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Checking the Numbers of Samples, Features and Targets "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "iris.data.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "iris.target.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Array `target_names` Contains Names for the `target` Array’s Numeric Labels\n",
    "* **`dtype='<U10'`** &mdash; elements are **strings with a max of 10 characters**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "iris.target_names"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Array `feature_names` Contains Names for Each Column in the `data` array:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "iris.feature_names"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 14.7.2 Exploring the Iris Dataset: Descriptive Statistics with a Pandas `DataFrame` \n",
    "### Create a `DataFrame` containing the `data` array’s contents\n",
    "* Use **`feature_names`** as the **column names**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "pd.set_option('precision', 2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "iris_df = pd.DataFrame(iris.data, columns=iris.feature_names)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Add a column containing each sample’s species name\n",
    "* **List comprehension** uses each value in **`target` array** to look up the corresponding **species name** in **`target_names` array**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "iris_df['species'] = [iris.target_names[i] for i in iris.target]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Look at a few samples  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "iris_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###  Calling `describe` on the `'species'` column confirms that it contains three unique values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "iris_df['species'].describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* We **know in advance** that there are **three classes** to which the samples belong\n",
    "    * This is **not** typically the case in **unsupervised machine learning**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 14.7.3 Visualizing the Dataset with a Seaborn pairplot (1 of 3)\n",
    "* To **learn more about your data**, **visualize** how the features relate to one another\n",
    "* Four features &mdash; cannot graph one against other three in a single graph\n",
    "* Can **plot pairs of features** against one another \n",
    "* **Seaborn function `pairplot`** creates a **grid of graphs**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import seaborn as sns\n",
    "sns.set_style('whitegrid')\n",
    "grid = sns.pairplot(data=iris_df, vars=iris_df.columns[0:4], hue='species')\n",
    "grid.fig.set_size_inches(9, 5.5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 14.7.4 Using a `KMeans` Estimator\n",
    "* Use k-means clustering via **`KMeans` estimator** to place each sample in the Iris dataset into a cluster\n",
    "\n",
    "### Creating the `KMeans` Estimator \n",
    "* [`KMeans` default arguments](https://scikit-learn.org/stable/modules/generated/sklearn.cluster.KMeans.html)\n",
    "* When you **train a `KMeans` estimator**, it calculates for each cluster a **centroid** representing the **cluster’s center data point** \n",
    "\t* Often, you’ll rely on **domain experts** to help **choose an appropriate _k_** (`n_clusters`). \n",
    "* Can also use **hyperparameter tuning** to estimate the appropriate **k**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.cluster import KMeans\n",
    "\n",
    "kmeans = KMeans(n_clusters=3, random_state=11)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Fitting the Model Via the `KMeans` object’s `fit` Method\n",
    "* When the training completes, the `KMeans` object contains: \n",
    "\t* **`labels_` array** with values from **`0` to `n_clusters - 1`**, indicating the clusters to which the samples belong\n",
    "\t* **`cluster_centers_` array** in which **each row represents a centroid**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "kmeans.fit(iris.data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Comparing the Cluster Labels to the Iris Dataset’s Target Values (1 of 2)\n",
    "* **Iris dataset** is **labeled**, so we can look at **`target` array values** to get a sense of **how well the k-means algorithm clustered the samples** \n",
    "    * With **unlabeled data**, we’d depend on a **domain expert** to help **evaluate whether the predicted classes make sense**\n",
    "* First 50 samples are **Iris setosa**, next 50 are **Iris versicolor**, last 50 are **Iris virginica**\n",
    "    * **`target` array** represents these with values **0–2** \n",
    "* If **`KMeans` chose clusters perfectly**, then **each group of 50 elements in the estimator’s `labels_` array should have a distinct label**. \n",
    "    * **`KMeans` labels** are **not related** to dataset’s **`target` array** "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Comparing the Cluster Labels to the Iris Dataset’s Target Values (2 of 2)\n",
    "* First 50 samples should be **one cluster** "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(kmeans.labels_[0:50])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Next 50 samples should be a **second cluster** (two are not)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(kmeans.labels_[50:100])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Last 50 samples should be a **third cluster** (14 are not)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(kmeans.labels_[100:150])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Results confirm what we saw in **`pairplot` diagrams**\n",
    "    * **Iris setosa** is “in a class by itself” \n",
    "    * There is confusion between **Iris versicolor** and **Iris virginica**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 14.7.5 Dimensionality Reduction with Principal Component Analysis\n",
    "* Use **`PCA` estimator** to perform dimensionality reduction from **4 to 2 dimensions**\n",
    "\t* [Algorithm’s details](https://scikit-learn.org/stable/modules/decomposition.html#pca) beyond scope"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.decomposition import PCA"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pca = PCA(n_components=2, random_state=11)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Transforming the Iris Dataset’s Features into Two Dimensions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pca.fit(iris.data)  # trains estimator once"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "iris_pca = pca.transform(iris.data)  # can be called many times to reduce data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* We'll call **`transform`** again to **reduce the cluster centroids from four dimensions to two** for plotting \n",
    "* **`transform`** returns an array with same number of rows as `iris.data`, but only two columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "iris_pca.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Visualizing the Reduced Data in a Scatter Plot\n",
    "* Place reduced data in a **`DataFrame`** and **add a species column** that we’ll use to **determine dot colors**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "iris_pca_df = pd.DataFrame(iris_pca, \n",
    "                           columns=['Component1', 'Component2'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "iris_pca_df['species'] = iris_df.species"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Scatterplot the Data with Seaborn\n",
    "* Each **centroid** in **`cluster_centers_`** array has **same number of features** (four) as dataset's samples\n",
    "* **To plot centroids**, we must **reduce their dimensions**\n",
    "* Think of a **centroid** as the **“average” sample in its cluster**\n",
    "\t* So each centroid should be **transformed** using **same `PCA` estimator** as **other samples**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "axes = sns.scatterplot(data=iris_pca_df, x='Component1', \n",
    "    y='Component2', hue='species', legend='brief') \n",
    "\n",
    "# reduce centroids to 2 dimensions\n",
    "iris_centers = pca.transform(kmeans.cluster_centers_)\n",
    "\n",
    "# plot centroids as larger black dots\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "dots = plt.scatter(iris_centers[:,0], iris_centers[:,1], s=100, c='k')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:black; background-color:black;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 14.7.6 Choosing the Best Clustering Estimator (1 of 4)\n",
    "* **Run multiple clustering algorithms** and see **how well they cluster Iris species** \n",
    "\t* We’re running `KMeans` here on the **small** Iris dataset\n",
    "    * If you experience **performance problems with `KMeans`** on larger datasets, consider **`MiniBatchKMeans`**\n",
    "    * Documentation indicates **`MiniBatchKMeans` is faster on large datasets** and the results are almost as good"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 14.7.6 Choosing the Best Clustering Estimator (2 of 4)\n",
    "* For the `DBSCAN` and `MeanShift` estimators, we do **not** specify number of clusters in advance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.cluster import DBSCAN, MeanShift,\\\n",
    "    SpectralClustering, AgglomerativeClustering"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "estimators = {\n",
    "    'KMeans': kmeans,\n",
    "    'DBSCAN': DBSCAN(),\n",
    "    'MeanShift': MeanShift(),\n",
    "    'SpectralClustering': SpectralClustering(n_clusters=3),\n",
    "    'AgglomerativeClustering': \n",
    "        AgglomerativeClustering(n_clusters=3)\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 14.7.6 Choosing the Best Clustering Estimator (3 of 4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for name, estimator in estimators.items():\n",
    "    estimator.fit(iris.data)\n",
    "    print(f'\\n{name}:')\n",
    "    for i in range(0, 101, 50):\n",
    "        labels, counts = np.unique(\n",
    "            estimator.labels_[i:i+50], return_counts=True)\n",
    "        print(f'{i}-{i+50}:')\n",
    "        for label, count in zip(labels, counts):\n",
    "            print(f'   label={label}, count={count}')          "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 14.7.6 Choosing the Best Clustering Estimator (4 of 4)\n",
    "* **`DBSCAN` correctly predicted three clusters** (labeled `-1`, `0` and `1`)\n",
    "    * Placed 84 of the 100 **Iris virginica** and **Iris versicolor** in the same cluster\n",
    "* **`MeanShift` predicted only two clusters** (labeled as `0` and `1`)\n",
    "    * Placed 99 of 100 **Iris virginica** and **Iris versicolor** samples in same cluster"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# More Info \n",
    "* See **video** Lesson 14 in [**Python Fundamentals LiveLessons** here on Safari Online Learning](https://learning.oreilly.com/videos/python-fundamentals/9780135917411)\n",
    "* See **book** Chapter 14 in [**Python for Programmers** on Safari Online Learning](https://learning.oreilly.com/library/view/python-for-programmers/9780135231364/)\n",
    "* See Chapter 15 in **Intro Python for Computer Science and Data Science** on [VitalSource.com](https://www.vitalsource.com/products/intro-to-python-for-computer-science-and-data-paul-j-deitel-harvey-deitel-v9780135404812) or [RedShelf.com](https://redshelf.com/book/1157786/intro-to-python-for-computer-science-and-data-science-1157786-9780135404812-paul-j-deitel-harvey-deitel)\n",
    "* Interested in a print book? Check out:\n",
    "\n",
    "| Python for Programmers<br>(640-page professional book) | Intro to Python for Computer<br>Science and Data Science<br>(880-page college textbook)\n",
    "| :------ | :------\n",
    "| <a href=\"https://amzn.to/2VvdnxE\"><img alt=\"Python for Programmers cover\" src=\"../images/PyFPCover.png\" width=\"150\" border=\"1\"/></a> | <a href=\"https://amzn.to/2LiDCmt\"><img alt=\"Intro to Python for Computer Science and Data Science: Learning to Program with AI, Big Data and the Cloud\" src=\"../images/IntroToPythonCover.png\" width=\"159\" border=\"1\"></a>\n",
    "\n",
    ">Please **do not** purchase both books&mdash;our professional book **_Python for Programmers_** is a subset of our college textbook **_Intro to Python for Computer Science and Data Science_**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "&copy; 2019 by Pearson Education, Inc. All Rights Reserved. The content in this notebook is based on the book [**Python for Programmers**](https://amzn.to/2VvdnxE)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}