examples/Tutorial.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# An Introduction to SKLL"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "SKLL (Scikit-Learn Laboratory) provides a number of command-line utilities and an API that make it simpler to run scikit-learn machine learning experiments with pre-generated features.\n",
    "SKLL can be used in 2 ways:\n",
    "\n",
    "\n",
    "1. **Via the Command Line**\n",
    "2. **Via the API**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Using SKLL via the Command Line\n",
    "\n",
    "In this example, we will use the following four steps to understand how to use SKLL via the command line.\n",
    "\n",
    "- Get the data in a [SKLL compatible format](https://skll.readthedocs.io/en/latest/run_experiment.html#file-formats).\n",
    "- Create a SKLL [experiment configuration file](https://skll.readthedocs.io/en/latest/run_experiment.html#create-config) for cross-validation.\n",
    "- Run the machine learning experiment using the [*run_experiment*](https://skll.readthedocs.io/en/latest/run_experiment.html) command.\n",
    "- Examine experiment results using the various SKLL [utility scripts](https://skll.readthedocs.io/en/latest/utilities.html)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1a. Data Pre-processing\n",
    "\n",
    "We will use the [IRIS dataset](https://archive.ics.uci.edu/ml/datasets/Iris/) for this part. This dataset is for a simple 3-class classification task with 4 simple features. The SKLL utility script `make_iris_example_data.py` downloads the IRIS dataset using scikit-learn and pre-processes it to create the training and test splits that are contained within the `train` and `test`sub-directories under the `iris` directory. Each of the created sub-directories (`iris/train` and `iris/test`) contains the respective features in a [newline-delimited JSON](https://jsonlines.org) file format. SKLL supports this format natively and uses the `.jsonlines` extension for files in this format.\n",
    "\n",
    "Let's first take a look at our current directory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tutorial.ipynb               \u001b[31mmake_iris_example_data.py\u001b[m\u001b[m\n",
      "__init__.py                  \u001b[31mmake_titanic_example_data.py\u001b[m\u001b[m\n",
      "\u001b[34mboston\u001b[m\u001b[m                       \u001b[31mtest.csv\u001b[m\u001b[m\n",
      "\u001b[34miris\u001b[m\u001b[m                         \u001b[34mtitanic\u001b[m\u001b[m\n",
      "\u001b[31mmake_boston_example_data.py\u001b[m\u001b[m  \u001b[31mtrain.csv\u001b[m\u001b[m\n"
     ]
    }
   ],
   "source": [
    "!ls"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's run the script to download and pre-process the IRIS data in a format SKLL can use. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Retrieving iris data from servers...done\n",
      "Writing training and testing files...done\n"
     ]
    }
   ],
   "source": [
    "!python3 make_iris_example_data.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's look at the contents of the `iris` directory after that command."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iris/cross_val.cfg iris/evaluate.cfg\n",
      "\n",
      "iris/test:\n",
      "example_iris_features.jsonlines\n",
      "\n",
      "iris/train:\n",
      "example_iris_features.jsonlines\n"
     ]
    }
   ],
   "source": [
    "!ls iris/**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see that two `.jsonlines` files have been created -- one each under `iris/train` and `iris/test` respectively. Let's see what these files look like."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\"id\": \"EXAMPLE_96\", \"y\": \"versicolor\", \"x\": {\"f0\": 5.7, \"f1\": 2.9, \"f2\": 4.2, \"f3\": 1.3}}\n",
      "{\"id\": \"EXAMPLE_105\", \"y\": \"virginica\", \"x\": {\"f0\": 7.6, \"f1\": 3.0, \"f2\": 6.6, \"f3\": 2.1}}\n",
      "{\"id\": \"EXAMPLE_66\", \"y\": \"versicolor\", \"x\": {\"f0\": 5.6, \"f1\": 3.0, \"f2\": 4.5, \"f3\": 1.5}}\n",
      "{\"id\": \"EXAMPLE_0\", \"y\": \"setosa\", \"x\": {\"f0\": 5.1, \"f1\": 3.5, \"f2\": 1.4, \"f3\": 0.2}}\n",
      "{\"id\": \"EXAMPLE_122\", \"y\": \"virginica\", \"x\": {\"f0\": 7.7, \"f1\": 2.8, \"f2\": 6.7, \"f3\": 2.0}}\n"
     ]
    }
   ],
   "source": [
    "!head -5 iris/train/example_iris_features.jsonlines"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As [documented here](https://skll.readthedocs.io/en/latest/run_experiment.html#jsonlines-ndj-recommended), each line of such a file contains a JSON object with the following keys: `y` (the class label), `x` (a dictionary of feature values), and `id` (an optional instance ID). The `.jsonlines` format is the preferred file format for SKLL as it can represent sparse featuresets much more easily.\n",
    "\n",
    "SKLL accepts a variety of [file formats](https://skll.readthedocs.io/en/latest/run_experiment.html#feature-file-formats) and also provides the [*skll_convert*](https://skll.readthedocs.io/en/latest/utilities.html#skll-convert) utility to easily convert between them. For example, we could easily convert the `.jsonlines` file into `.csv` file like so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loading iris/train/example_iris_features.jsonlines...           done\n",
      "Writing iris/train/example_iris_features.csv...done           \n",
      "\n",
      "example_iris_features.csv       example_iris_features.jsonlines\n",
      "\n",
      "f0,f1,f2,f3,id,y\n",
      "5.7,2.9,4.2,1.3,EXAMPLE_96,versicolor\n",
      "7.6,3.0,6.6,2.1,EXAMPLE_105,virginica\n",
      "5.6,3.0,4.5,1.5,EXAMPLE_66,versicolor\n",
      "5.1,3.5,1.4,0.2,EXAMPLE_0,setosa\n"
     ]
    }
   ],
   "source": [
    "!skll_convert iris/train/example_iris_features.jsonlines iris/train/example_iris_features.csv \n",
    "print()\n",
    "!ls iris/train\n",
    "print()\n",
    "!head -5 iris/train/example_iris_features.csv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1b. Experiment configuration files"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The most common method of using SKLL is via its experiment configuration file which can then be run using the *run_experiment* command-line script. SKLL configuration files are basic configuration files that are similar in format to Windows INI files. \n",
    "\n",
    "There are 4 sections in a configuration file.\n",
    "\n",
    "\n",
    "1. [General](https://skll.readthedocs.io/en/latest/run_experiment.html#general)\n",
    "    - Defines the `experiment_name` and the `task` (both required)\n",
    "    - SKLL supports 5 types of machine learning [tasks](https://skll.readthedocs.io/en/latest/run_experiment.html#task).\n",
    "        1. `cross_validate`\n",
    "        2. `train`\n",
    "        3. `evaluate`\n",
    "        4. `predict`\n",
    "        5. `learning_curve`\n",
    "2. [Input](https://skll.readthedocs.io/en/latest/run_experiment.html#input)\n",
    "    - Defines the list of machine learners we want to run for this experiment via the `learners` option (required).\n",
    "    - Additionally, either the `train_directory` or `train_file` option must also be defined. \n",
    "    - Other fields may be required or optional depending on the task.\n",
    "3. [Tuning](https://skll.readthedocs.io/en/latest/run_experiment.html#tuning)\n",
    "    - Defines options related to tuning the model hyperparameters e.g., `grid_search`, `param_grids`, `objectives`, et cetera.\n",
    "    - Other fields may be required or optional depending on the task. \n",
    "4. [Output](https://skll.readthedocs.io/en/latest/run_experiment.html#output)\n",
    "    - Defines options related to the output produced by the machine learning task e.g.`probability`, `metrics`, `results`, et cetera.\n",
    "    - Other fields may be required or optional depending on the task. \n",
    "    \n",
    "Let's look at the SKLL configuration file for our IRIS example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[General]\n",
      "experiment_name = Iris_CV\n",
      "task = cross_validate\n",
      "\n",
      "[Input]\n",
      "# this could also be an absolute path instead (and must be if you're not\n",
      "# running things in local mode)\n",
      "train_directory = train\n",
      "featuresets = [[\"example_iris_features\"]]\n",
      "# there is only set of features to try with one feature file in it here.\n",
      "featureset_names = [\"example_iris\"]\n",
      "learners = [\"RandomForestClassifier\", \"SVC\", \"LogisticRegression\", \"MultinomialNB\"]\n",
      "suffix = .jsonlines\n",
      "\n",
      "[Tuning]\n",
      "grid_search = true\n",
      "objectives = ['f1_score_micro']\n",
      "\n",
      "[Output]\n",
      "# again, these can be absolute paths\n",
      "save_cv_models = true\n",
      "models = output\n",
      "results = output\n",
      "log = output\n",
      "predictions = output\n"
     ]
    }
   ],
   "source": [
    "!cat iris/cross_val.cfg"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, SKLL configuration files also support Python-style comments. In this particular example, we are going to do cross-validation on the IRIS training set with 4 different learners. The hyperparameters for each learner are tuned by searching a [default grid of values](https://skll.readthedocs.io/en/latest/run_experiment.html#param-grids-optional) defined in SKLL (the default grid can be overridden in the configuration file via the `param_grids` option) and picking the values that yield the largest value for the micro-averaged F1 score metric. The grid search happens inside each outer cross-validation loop using a 3-fold cross-validation on the fold training data. The configuration file also says that the results, the predictions, the logs, and the trained models for each fold will be saved in the `output` directory under the current directory."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1c. Running the experiment\n",
    "\n",
    "After creating the configuration file, we can now run the experiment using the `run_experiment` command. This scripts only takes a single required argument –– the path to the configuration file. However, it also [supports other optional arguments](https://skll.readthedocs.io/en/latest/run_experiment.html#using-run-experiment) for other use cases e.g. running an experiment on a DRMAA-compatible cluster rather than locally.\n",
    "\n",
    "Let's run our IRIS cross-validation experiment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2019-10-16 17:49:12,383 - experiment - WARNING - gridmap 0.10.1+ not available. Forcing local mode.  To run things on a DRMAA-compatible cluster, install gridmap>=0.10.1 via pip.\n",
      "2019-10-16 17:49:12,383 - Iris_CV_example_iris_RandomForestClassifier - INFO - Task: cross_validate\n",
      "2019-10-16 17:49:12,383 - Iris_CV_example_iris_RandomForestClassifier - INFO - Cross-validating (10 folds) on train, feature set ['example_iris_features'] ...\n",
      "Loading /Users/nmadnani/work/skll/examples/iris/train/example_iris_features.jsonlines...           done\n",
      "2019-10-16 17:49:12,386 - Iris_CV_example_iris_RandomForestClassifier - INFO - Cross-validating\n",
      "2019-10-16 17:49:12,387 - Iris_CV_example_iris_RandomForestClassifier - WARNING - Training data will be shuffled to randomize grid search folds. Shuffling may yield different results compared to scikit-learn.\n",
      "2019-10-16 17:50:05,631 - Iris_CV_example_iris_SVC - INFO - Task: cross_validate\n",
      "2019-10-16 17:50:05,631 - Iris_CV_example_iris_SVC - INFO - Cross-validating (10 folds) on train, feature set ['example_iris_features'] ...\n",
      "Loading /Users/nmadnani/work/skll/examples/iris/train/example_iris_features.jsonlines...           done\n",
      "2019-10-16 17:50:05,637 - Iris_CV_example_iris_SVC - INFO - Cross-validating\n",
      "2019-10-16 17:50:05,637 - Iris_CV_example_iris_SVC - WARNING - Training data will be shuffled to randomize grid search folds. Shuffling may yield different results compared to scikit-learn.\n",
      "2019-10-16 17:50:16,741 - Iris_CV_example_iris_LogisticRegression - INFO - Task: cross_validate\n",
      "2019-10-16 17:50:16,741 - Iris_CV_example_iris_LogisticRegression - INFO - Cross-validating (10 folds) on train, feature set ['example_iris_features'] ...\n",
      "Loading /Users/nmadnani/work/skll/examples/iris/train/example_iris_features.jsonlines...           done\n",
      "2019-10-16 17:50:16,748 - Iris_CV_example_iris_LogisticRegression - INFO - Cross-validating\n",
      "2019-10-16 17:50:16,749 - Iris_CV_example_iris_LogisticRegression - WARNING - Training data will be shuffled to randomize grid search folds. Shuffling may yield different results compared to scikit-learn.\n",
      "2019-10-16 17:50:27,323 - Iris_CV_example_iris_MultinomialNB - INFO - Task: cross_validate\n",
      "2019-10-16 17:50:27,323 - Iris_CV_example_iris_MultinomialNB - INFO - Cross-validating (10 folds) on train, feature set ['example_iris_features'] ...\n",
      "Loading /Users/nmadnani/work/skll/examples/iris/train/example_iris_features.jsonlines...           done\n",
      "2019-10-16 17:50:27,328 - Iris_CV_example_iris_MultinomialNB - INFO - Cross-validating\n",
      "2019-10-16 17:50:27,329 - Iris_CV_example_iris_MultinomialNB - WARNING - Training data will be shuffled to randomize grid search folds. Shuffling may yield different results compared to scikit-learn.\n"
     ]
    }
   ],
   "source": [
    "!run_experiment iris/cross_val.cfg"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The above log shows that SKLL reads in the training feature file for every learner. This is one of the many efficiency compromises that SKLL makes for the more convenient option of batched machine learning experiments using simple configuration files. However, this can be quite slow for experiments with very large feature files and multiple learners. For those more advanced cases, we recommend using the SKLL Python API directly (see next section below)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1d. Examining experiment output\n",
    "\n",
    "Let's examine the `iris` directory again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iris/cross_val.cfg iris/evaluate.cfg\n",
      "\n",
      "iris/output:\n",
      "Iris_CV.log\n",
      "Iris_CV_example_iris_LogisticRegression.log\n",
      "Iris_CV_example_iris_LogisticRegression.results\n",
      "Iris_CV_example_iris_LogisticRegression.results.json\n",
      "Iris_CV_example_iris_LogisticRegression_fold1.model\n",
      "Iris_CV_example_iris_LogisticRegression_fold10.model\n",
      "Iris_CV_example_iris_LogisticRegression_fold2.model\n",
      "Iris_CV_example_iris_LogisticRegression_fold3.model\n",
      "Iris_CV_example_iris_LogisticRegression_fold4.model\n",
      "Iris_CV_example_iris_LogisticRegression_fold5.model\n",
      "Iris_CV_example_iris_LogisticRegression_fold6.model\n",
      "Iris_CV_example_iris_LogisticRegression_fold7.model\n",
      "Iris_CV_example_iris_LogisticRegression_fold8.model\n",
      "Iris_CV_example_iris_LogisticRegression_fold9.model\n",
      "Iris_CV_example_iris_LogisticRegression_predictions.tsv\n",
      "Iris_CV_example_iris_MultinomialNB.log\n",
      "Iris_CV_example_iris_MultinomialNB.results\n",
      "Iris_CV_example_iris_MultinomialNB.results.json\n",
      "Iris_CV_example_iris_MultinomialNB_fold1.model\n",
      "Iris_CV_example_iris_MultinomialNB_fold10.model\n",
      "Iris_CV_example_iris_MultinomialNB_fold2.model\n",
      "Iris_CV_example_iris_MultinomialNB_fold3.model\n",
      "Iris_CV_example_iris_MultinomialNB_fold4.model\n",
      "Iris_CV_example_iris_MultinomialNB_fold5.model\n",
      "Iris_CV_example_iris_MultinomialNB_fold6.model\n",
      "Iris_CV_example_iris_MultinomialNB_fold7.model\n",
      "Iris_CV_example_iris_MultinomialNB_fold8.model\n",
      "Iris_CV_example_iris_MultinomialNB_fold9.model\n",
      "Iris_CV_example_iris_MultinomialNB_predictions.tsv\n",
      "Iris_CV_example_iris_RandomForestClassifier.log\n",
      "Iris_CV_example_iris_RandomForestClassifier.results\n",
      "Iris_CV_example_iris_RandomForestClassifier.results.json\n",
      "Iris_CV_example_iris_RandomForestClassifier_fold1.model\n",
      "Iris_CV_example_iris_RandomForestClassifier_fold10.model\n",
      "Iris_CV_example_iris_RandomForestClassifier_fold2.model\n",
      "Iris_CV_example_iris_RandomForestClassifier_fold3.model\n",
      "Iris_CV_example_iris_RandomForestClassifier_fold4.model\n",
      "Iris_CV_example_iris_RandomForestClassifier_fold5.model\n",
      "Iris_CV_example_iris_RandomForestClassifier_fold6.model\n",
      "Iris_CV_example_iris_RandomForestClassifier_fold7.model\n",
      "Iris_CV_example_iris_RandomForestClassifier_fold8.model\n",
      "Iris_CV_example_iris_RandomForestClassifier_fold9.model\n",
      "Iris_CV_example_iris_RandomForestClassifier_predictions.tsv\n",
      "Iris_CV_example_iris_SVC.log\n",
      "Iris_CV_example_iris_SVC.results\n",
      "Iris_CV_example_iris_SVC.results.json\n",
      "Iris_CV_example_iris_SVC_fold1.model\n",
      "Iris_CV_example_iris_SVC_fold10.model\n",
      "Iris_CV_example_iris_SVC_fold2.model\n",
      "Iris_CV_example_iris_SVC_fold3.model\n",
      "Iris_CV_example_iris_SVC_fold4.model\n",
      "Iris_CV_example_iris_SVC_fold5.model\n",
      "Iris_CV_example_iris_SVC_fold6.model\n",
      "Iris_CV_example_iris_SVC_fold7.model\n",
      "Iris_CV_example_iris_SVC_fold8.model\n",
      "Iris_CV_example_iris_SVC_fold9.model\n",
      "Iris_CV_example_iris_SVC_predictions.tsv\n",
      "Iris_CV_summary.tsv\n",
      "\n",
      "iris/test:\n",
      "example_iris_features.jsonlines\n",
      "\n",
      "iris/train:\n",
      "example_iris_features.csv       example_iris_features.jsonlines\n"
     ]
    }
   ],
   "source": [
    "!ls iris/**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see that the subdirectory `iris/output` has been created and that it contains several new files:\n",
    "- a plain text `.results` file that contains human-readable output of the results for each learner in our list\n",
    "- a `.results.json` file that contains the same results but in JSON format for programmatic use\n",
    "- a `.log` file containing log messages for that learner\n",
    "- the cross-validation predictions in `.tsv` format from each learner\n",
    "- 10 `.model` files for each learner - one per cross-validation fold\n",
    "- a `summary.tsv` file that contains the results for the entire experiment across all learners\n",
    "\n",
    "Let's first examine one of the `.results` files."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Experiment Name: Iris_CV\n",
      "SKLL Version: 1.5.3\n",
      "Training Set: train\n",
      "Training Set Size: 100\n",
      "Test Set: cv\n",
      "Test Set Size: n/a\n",
      "Shuffle: False\n",
      "Feature Set: [\"example_iris_features\"]\n",
      "Learner: LogisticRegression\n",
      "Task: cross_validate\n",
      "Number of Folds: 10\n",
      "Stratified Folds: True\n",
      "Feature Scaling: none\n",
      "Grid Search: True\n",
      "Grid Search Folds: 3\n",
      "Grid Objective Function: f1_score_micro\n",
      "Scikit-learn Version: 0.21.3\n",
      "Start Timestamp: 16 Oct 2019 17:50:16.741287\n",
      "End Timestamp: 16 Oct 2019 17:50:27.304682\n",
      "Total Time: 0:00:10.563395\n",
      "\n",
      "\n",
      "Fold: 1\n",
      "Model Parameters: {\"C\": 100.0, \"class_weight\": null, \"dual\": false, \"fit_intercept\": true, \"intercept_scaling\": 1, \"l1_ratio\": null, \"max_iter\": 1000, \"multi_class\": \"auto\", \"n_jobs\": null, \"penalty\": \"l2\", \"random_state\": 123456789, \"solver\": \"liblinear\", \"tol\": 0.0001, \"verbose\": 0, \"warm_start\": false}\n",
      "Grid Objective Score (Train) = 0.9547892720306513\n",
      "+------------+----------+--------------+-------------+-------------+----------+-------------+\n",
      "|            |   setosa |   versicolor |   virginica |   Precision |   Recall |   F-measure |\n",
      "+============+==========+==============+=============+=============+==========+=============+\n",
      "|     setosa |      [4] |            0 |           0 |       1.000 |    1.000 |       1.000 |\n",
      "+------------+----------+--------------+-------------+-------------+----------+-------------+\n",
      "| versicolor |        0 |          [4] |           0 |       1.000 |    1.000 |       1.000 |\n",
      "+------------+----------+--------------+-------------+-------------+----------+-------------+\n",
      "|  virginica |        0 |            0 |         [4] |       1.000 |    1.000 |       1.000 |\n",
      "+------------+----------+--------------+-------------+-------------+----------+-------------+\n",
      "(row = reference; column = predicted)\n",
      "Accuracy = 1.0\n",
      "Objective Function Score (Test) = 1.0\n",
      "\n",
      "Fold: 2\n",
      "Model Parameters: {\"C\": 10.0, \"class_weight\": null, \"dual\": false, \"fit_intercept\": true, \"intercept_scaling\": 1, \"l1_ratio\": null, \"max_iter\": 1000, \"multi_class\": \"auto\", \"n_jobs\": null, \"penalty\": \"l2\", \"random_state\": 123456789, \"solver\": \"liblinear\", \"tol\": 0.0001, \"verbose\": 0, \"warm_start\": false}\n",
      "Grid Objective Score (Train) = 0.978494623655914\n",
      "+------------+----------+--------------+-------------+-------------+----------+-------------+\n",
      "|            |   setosa |   versicolor |   virginica |   Precision |   Recall |   F-measure |\n",
      "+============+==========+==============+=============+=============+==========+=============+\n",
      "|     setosa |      [3] |            0 |           0 |       1.000 |    1.000 |       1.000 |\n",
      "+------------+----------+--------------+-------------+-------------+----------+-------------+\n",
      "| versicolor |        0 |          [4] |           0 |       1.000 |    1.000 |       1.000 |\n",
      "+------------+----------+--------------+-------------+-------------+----------+-------------+\n",
      "|  virginica |        0 |            0 |         [4] |       1.000 |    1.000 |       1.000 |\n",
      "+------------+----------+--------------+-------------+-------------+----------+-------------+\n",
      "(row = reference; column = predicted)\n",
      "Accuracy = 1.0\n",
      "Objective Function Score (Test) = 1.0\n",
      "\n",
      "Fold: 3\n",
      "Model Parameters: {\"C\": 10.0, \"class_weight\": null, \"dual\": false, \"fit_intercept\": true, \"intercept_scaling\": 1, \"l1_ratio\": null, \"max_iter\": 1000, \"multi_class\": \"auto\", \"n_jobs\": null, \"penalty\": \"l2\", \"random_state\": 123456789, \"solver\": \"liblinear\", \"tol\": 0.0001, \"verbose\": 0, \"warm_start\": false}\n",
      "Grid Objective Score (Train) = 0.9670003707823508\n",
      "+------------+----------+--------------+-------------+-------------+----------+-------------+\n",
      "|            |   setosa |   versicolor |   virginica |   Precision |   Recall |   F-measure |\n",
      "+============+==========+==============+=============+=============+==========+=============+\n",
      "|     setosa |      [3] |            0 |           0 |       1.000 |    1.000 |       1.000 |\n",
      "+------------+----------+--------------+-------------+-------------+----------+-------------+\n",
      "| versicolor |        0 |          [4] |           0 |       1.000 |    1.000 |       1.000 |\n",
      "+------------+----------+--------------+-------------+-------------+----------+-------------+\n",
      "|  virginica |        0 |            0 |         [4] |       1.000 |    1.000 |       1.000 |\n",
      "+------------+----------+--------------+-------------+-------------+----------+-------------+\n",
      "(row = reference; column = predicted)\n",
      "Accuracy = 1.0\n",
      "Objective Function Score (Test) = 1.0\n",
      "\n",
      "Fold: 4\n",
      "Model Parameters: {\"C\": 100.0, \"class_weight\": null, \"dual\": false, \"fit_intercept\": true, \"intercept_scaling\": 1, \"l1_ratio\": null, \"max_iter\": 1000, \"multi_class\": \"auto\", \"n_jobs\": null, \"penalty\": \"l2\", \"random_state\": 123456789, \"solver\": \"liblinear\", \"tol\": 0.0001, \"verbose\": 0, \"warm_start\": false}\n",
      "Grid Objective Score (Train) = 0.9770114942528737\n",
      "+------------+----------+--------------+-------------+-------------+----------+-------------+\n",
      "|            |   setosa |   versicolor |   virginica |   Precision |   Recall |   F-measure |\n",
      "+============+==========+==============+=============+=============+==========+=============+\n",
      "|     setosa |      [3] |            0 |           0 |       1.000 |    1.000 |       1.000 |\n",
      "+------------+----------+--------------+-------------+-------------+----------+-------------+\n",
      "| versicolor |        0 |          [2] |           2 |       1.000 |    0.500 |       0.667 |\n",
      "+------------+----------+--------------+-------------+-------------+----------+-------------+\n",
      "|  virginica |        0 |            0 |         [4] |       0.667 |    1.000 |       0.800 |\n",
      "+------------+----------+--------------+-------------+-------------+----------+-------------+\n",
      "(row = reference; column = predicted)\n",
      "Accuracy = 0.8181818181818182\n",
      "Objective Function Score (Test) = 0.8181818181818182\n",
      "\n",
      "Fold: 5\n",
      "Model Parameters: {\"C\": 1.0, \"class_weight\": null, \"dual\": false, \"fit_intercept\": true, \"intercept_scaling\": 1, \"l1_ratio\": null, \"max_iter\": 1000, \"multi_class\": \"auto\", \"n_jobs\": null, \"penalty\": \"l2\", \"random_state\": 123456789, \"solver\": \"liblinear\", \"tol\": 0.0001, \"verbose\": 0, \"warm_start\": false}\n",
      "Grid Objective Score (Train) = 0.9561781609195402\n",
      "+------------+----------+--------------+-------------+-------------+----------+-------------+\n",
      "|            |   setosa |   versicolor |   virginica |   Precision |   Recall |   F-measure |\n",
      "+============+==========+==============+=============+=============+==========+=============+\n",
      "|     setosa |      [3] |            0 |           0 |       1.000 |    1.000 |       1.000 |\n",
      "+------------+----------+--------------+-------------+-------------+----------+-------------+\n",
      "| versicolor |        0 |          [3] |           1 |       0.750 |    0.750 |       0.750 |\n",
      "+------------+----------+--------------+-------------+-------------+----------+-------------+\n",
      "|  virginica |        0 |            1 |         [2] |       0.667 |    0.667 |       0.667 |\n",
      "+------------+----------+--------------+-------------+-------------+----------+-------------+\n",
      "(row = reference; column = predicted)\n",
      "Accuracy = 0.8\n",
      "Objective Function Score (Test) = 0.8000000000000002\n",
      "\n",
      "Fold: 6\n",
      "Model Parameters: {\"C\": 100.0, \"class_weight\": null, \"dual\": false, \"fit_intercept\": true, \"intercept_scaling\": 1, \"l1_ratio\": null, \"max_iter\": 1000, \"multi_class\": \"auto\", \"n_jobs\": null, \"penalty\": \"l2\", \"random_state\": 123456789, \"solver\": \"liblinear\", \"tol\": 0.0001, \"verbose\": 0, \"warm_start\": false}\n",
      "Grid Objective Score (Train) = 0.9569444444444445\n",
      "+------------+----------+--------------+-------------+-------------+----------+-------------+\n",
      "|            |   setosa |   versicolor |   virginica |   Precision |   Recall |   F-measure |\n",
      "+============+==========+==============+=============+=============+==========+=============+\n",
      "|     setosa |      [3] |            0 |           0 |       1.000 |    1.000 |       1.000 |\n",
      "+------------+----------+--------------+-------------+-------------+----------+-------------+\n",
      "| versicolor |        0 |          [3] |           0 |       1.000 |    1.000 |       1.000 |\n",
      "+------------+----------+--------------+-------------+-------------+----------+-------------+\n",
      "|  virginica |        0 |            0 |         [3] |       1.000 |    1.000 |       1.000 |\n",
      "+------------+----------+--------------+-------------+-------------+----------+-------------+\n",
      "(row = reference; column = predicted)\n",
      "Accuracy = 1.0\n",
      "Objective Function Score (Test) = 1.0\n",
      "\n",
      "Fold: 7\n",
      "Model Parameters: {\"C\": 100.0, \"class_weight\": null, \"dual\": false, \"fit_intercept\": true, \"intercept_scaling\": 1, \"l1_ratio\": null, \"max_iter\": 1000, \"multi_class\": \"auto\", \"n_jobs\": null, \"penalty\": \"l2\", \"random_state\": 123456789, \"solver\": \"liblinear\", \"tol\": 0.0001, \"verbose\": 0, \"warm_start\": false}\n",
      "Grid Objective Score (Train) = 0.9558668582375479\n",
      "+------------+----------+--------------+-------------+-------------+----------+-------------+\n",
      "|            |   setosa |   versicolor |   virginica |   Precision |   Recall |   F-measure |\n",
      "+============+==========+==============+=============+=============+==========+=============+\n",
      "|     setosa |      [3] |            0 |           0 |       1.000 |    1.000 |       1.000 |\n",
      "+------------+----------+--------------+-------------+-------------+----------+-------------+\n",
      "| versicolor |        0 |          [3] |           0 |       1.000 |    1.000 |       1.000 |\n",
      "+------------+----------+--------------+-------------+-------------+----------+-------------+\n",
      "|  virginica |        0 |            0 |         [3] |       1.000 |    1.000 |       1.000 |\n",
      "+------------+----------+--------------+-------------+-------------+----------+-------------+\n",
      "(row = reference; column = predicted)\n",
      "Accuracy = 1.0\n",
      "Objective Function Score (Test) = 1.0\n",
      "\n",
      "Fold: 8\n",
      "Model Parameters: {\"C\": 10.0, \"class_weight\": null, \"dual\": false, \"fit_intercept\": true, \"intercept_scaling\": 1, \"l1_ratio\": null, \"max_iter\": 1000, \"multi_class\": \"auto\", \"n_jobs\": null, \"penalty\": \"l2\", \"random_state\": 123456789, \"solver\": \"liblinear\", \"tol\": 0.0001, \"verbose\": 0, \"warm_start\": false}\n",
      "Grid Objective Score (Train) = 0.9565613026819925\n",
      "+------------+----------+--------------+-------------+-------------+----------+-------------+\n",
      "|            |   setosa |   versicolor |   virginica |   Precision |   Recall |   F-measure |\n",
      "+============+==========+==============+=============+=============+==========+=============+\n",
      "|     setosa |      [3] |            0 |           0 |       1.000 |    1.000 |       1.000 |\n",
      "+------------+----------+--------------+-------------+-------------+----------+-------------+\n",
      "| versicolor |        0 |          [2] |           1 |       1.000 |    0.667 |       0.800 |\n",
      "+------------+----------+--------------+-------------+-------------+----------+-------------+\n",
      "|  virginica |        0 |            0 |         [3] |       0.750 |    1.000 |       0.857 |\n",
      "+------------+----------+--------------+-------------+-------------+----------+-------------+\n",
      "(row = reference; column = predicted)\n",
      "Accuracy = 0.8888888888888888\n",
      "Objective Function Score (Test) = 0.8888888888888888\n",
      "\n",
      "Fold: 9\n",
      "Model Parameters: {\"C\": 10.0, \"class_weight\": null, \"dual\": false, \"fit_intercept\": true, \"intercept_scaling\": 1, \"l1_ratio\": null, \"max_iter\": 1000, \"multi_class\": \"auto\", \"n_jobs\": null, \"penalty\": \"l2\", \"random_state\": 123456789, \"solver\": \"liblinear\", \"tol\": 0.0001, \"verbose\": 0, \"warm_start\": false}\n",
      "Grid Objective Score (Train) = 0.9565613026819925\n",
      "+------------+----------+--------------+-------------+-------------+----------+-------------+\n",
      "|            |   setosa |   versicolor |   virginica |   Precision |   Recall |   F-measure |\n",
      "+============+==========+==============+=============+=============+==========+=============+\n",
      "|     setosa |      [3] |            0 |           0 |       1.000 |    1.000 |       1.000 |\n",
      "+------------+----------+--------------+-------------+-------------+----------+-------------+\n",
      "| versicolor |        0 |          [3] |           0 |       1.000 |    1.000 |       1.000 |\n",
      "+------------+----------+--------------+-------------+-------------+----------+-------------+\n",
      "|  virginica |        0 |            0 |         [3] |       1.000 |    1.000 |       1.000 |\n",
      "+------------+----------+--------------+-------------+-------------+----------+-------------+\n",
      "(row = reference; column = predicted)\n",
      "Accuracy = 1.0\n",
      "Objective Function Score (Test) = 1.0\n",
      "\n",
      "Fold: 10\n",
      "Model Parameters: {\"C\": 1.0, \"class_weight\": null, \"dual\": false, \"fit_intercept\": true, \"intercept_scaling\": 1, \"l1_ratio\": null, \"max_iter\": 1000, \"multi_class\": \"auto\", \"n_jobs\": null, \"penalty\": \"l2\", \"random_state\": 123456789, \"solver\": \"liblinear\", \"tol\": 0.0001, \"verbose\": 0, \"warm_start\": false}\n",
      "Grid Objective Score (Train) = 0.9439894636015326\n",
      "+------------+----------+--------------+-------------+-------------+----------+-------------+\n",
      "|            |   setosa |   versicolor |   virginica |   Precision |   Recall |   F-measure |\n",
      "+============+==========+==============+=============+=============+==========+=============+\n",
      "|     setosa |      [3] |            0 |           0 |       1.000 |    1.000 |       1.000 |\n",
      "+------------+----------+--------------+-------------+-------------+----------+-------------+\n",
      "| versicolor |        0 |          [3] |           0 |       1.000 |    1.000 |       1.000 |\n",
      "+------------+----------+--------------+-------------+-------------+----------+-------------+\n",
      "|  virginica |        0 |            0 |         [3] |       1.000 |    1.000 |       1.000 |\n",
      "+------------+----------+--------------+-------------+-------------+----------+-------------+\n",
      "(row = reference; column = predicted)\n",
      "Accuracy = 1.0\n",
      "Objective Function Score (Test) = 1.0\n",
      "\n",
      "Fold: average\n",
      "Model Parameters: \n",
      "Grid Objective Score (Train) = \n",
      "+------------+-------------+----------+-------------+\n",
      "| Label      |   Precision |   Recall |   F-measure |\n",
      "|------------+-------------+----------+-------------|\n",
      "| setosa     |       1.000 |    1.000 |       1.000 |\n",
      "| versicolor |       0.975 |    0.892 |       0.922 |\n",
      "| virginica  |       0.908 |    0.967 |       0.932 |\n",
      "+------------+-------------+----------+-------------+\n",
      "Accuracy = 0.9507070707070706\n",
      "Objective Function Score (Test) = 0.9507070707070706\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!cat iris/output/Iris_CV_example_iris_LogisticRegression.results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see that it contains useful experiment metadata at the top and then it contains the results for each of the folds along with the final results that are the averages across the folds. Next let's look at the predictions file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "id\tprediction\n",
      "EXAMPLE_125\tvirginica\n",
      "EXAMPLE_130\tvirginica\n",
      "EXAMPLE_2\tsetosa\n",
      "EXAMPLE_123\tvirginica\n"
     ]
    }
   ],
   "source": [
    "!head -5 iris/output/Iris_CV_example_iris_LogisticRegression_predictions.tsv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This file simply contains the most likely label from the learner for each of the instances in the test split of each of the 10 cross-validation folds. Next let's look at the log file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2019-10-16 17:50:16,741 - INFO - Task: cross_validate\n",
      "2019-10-16 17:50:16,741 - INFO - Cross-validating (10 folds) on train, feature set ['example_iris_features'] ...\n",
      "2019-10-16 17:50:16,748 - INFO - Cross-validating\n",
      "2019-10-16 17:50:16,749 - WARNING - Training data will be shuffled to randomize grid search folds. Shuffling may yield different results compared to scikit-learn.\n"
     ]
    }
   ],
   "source": [
    "!cat iris/output/Iris_CV_example_iris_LogisticRegression.log"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The log file contains a record of all useful informational messages and warnings that might have been generated from both SKLL (and `scikit-learn`) during the experiment. For more detailed log messages, use the `--verbose` option for `run_experiment`. Next, let's examine the summary file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Index(['accuracy', 'additional_scores', 'cv_folds', 'end_timestamp',\n",
      "       'experiment_name', 'feature_scaling', 'featureset', 'featureset_name',\n",
      "       'fold', 'folds_file', 'grid_objective', 'grid_score', 'grid_search',\n",
      "       'grid_search_cv_results', 'grid_search_folds', 'learner_name',\n",
      "       'min_feature_count', 'model_params', 'pearson', 'save_cv_folds',\n",
      "       'save_cv_models', 'scikit_learn_version', 'score', 'shuffle',\n",
      "       'start_timestamp', 'stratified_folds', 'task', 'test_set_name',\n",
      "       'test_set_size', 'total_time', 'train_set_name', 'train_set_size',\n",
      "       'use_folds_file_for_grid_search', 'using_folds_file', 'version'],\n",
      "      dtype='object')\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "summary_df = pd.read_csv('iris/output/Iris_CV_summary.tsv', sep='\\t')\n",
    "print(summary_df.columns)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This file has many useful columns but let's just look at a small subset for now."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>learner_name</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>score</th>\n",
       "      <th>fold</th>\n",
       "      <th>featureset_name</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>RandomForestClassifier</td>\n",
       "      <td>0.916667</td>\n",
       "      <td>0.916667</td>\n",
       "      <td>1</td>\n",
       "      <td>example_iris</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>RandomForestClassifier</td>\n",
       "      <td>0.909091</td>\n",
       "      <td>0.909091</td>\n",
       "      <td>2</td>\n",
       "      <td>example_iris</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>RandomForestClassifier</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>3</td>\n",
       "      <td>example_iris</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>RandomForestClassifier</td>\n",
       "      <td>0.818182</td>\n",
       "      <td>0.818182</td>\n",
       "      <td>4</td>\n",
       "      <td>example_iris</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>RandomForestClassifier</td>\n",
       "      <td>0.800000</td>\n",
       "      <td>0.800000</td>\n",
       "      <td>5</td>\n",
       "      <td>example_iris</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>RandomForestClassifier</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>6</td>\n",
       "      <td>example_iris</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>RandomForestClassifier</td>\n",
       "      <td>0.888889</td>\n",
       "      <td>0.888889</td>\n",
       "      <td>7</td>\n",
       "      <td>example_iris</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>RandomForestClassifier</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>8</td>\n",
       "      <td>example_iris</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>RandomForestClassifier</td>\n",
       "      <td>0.777778</td>\n",
       "      <td>0.777778</td>\n",
       "      <td>9</td>\n",
       "      <td>example_iris</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>RandomForestClassifier</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>10</td>\n",
       "      <td>example_iris</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>RandomForestClassifier</td>\n",
       "      <td>0.911061</td>\n",
       "      <td>0.911061</td>\n",
       "      <td>average</td>\n",
       "      <td>example_iris</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>SVC</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1</td>\n",
       "      <td>example_iris</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>SVC</td>\n",
       "      <td>0.909091</td>\n",
       "      <td>0.909091</td>\n",
       "      <td>2</td>\n",
       "      <td>example_iris</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>SVC</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>3</td>\n",
       "      <td>example_iris</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>SVC</td>\n",
       "      <td>0.909091</td>\n",
       "      <td>0.909091</td>\n",
       "      <td>4</td>\n",
       "      <td>example_iris</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>15</td>\n",
       "      <td>SVC</td>\n",
       "      <td>0.800000</td>\n",
       "      <td>0.800000</td>\n",
       "      <td>5</td>\n",
       "      <td>example_iris</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>16</td>\n",
       "      <td>SVC</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>6</td>\n",
       "      <td>example_iris</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>17</td>\n",
       "      <td>SVC</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>7</td>\n",
       "      <td>example_iris</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>18</td>\n",
       "      <td>SVC</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>8</td>\n",
       "      <td>example_iris</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>19</td>\n",
       "      <td>SVC</td>\n",
       "      <td>0.888889</td>\n",
       "      <td>0.888889</td>\n",
       "      <td>9</td>\n",
       "      <td>example_iris</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>20</td>\n",
       "      <td>SVC</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>10</td>\n",
       "      <td>example_iris</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>21</td>\n",
       "      <td>SVC</td>\n",
       "      <td>0.950707</td>\n",
       "      <td>0.950707</td>\n",
       "      <td>average</td>\n",
       "      <td>example_iris</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              learner_name  accuracy     score     fold featureset_name\n",
       "0   RandomForestClassifier  0.916667  0.916667        1    example_iris\n",
       "1   RandomForestClassifier  0.909091  0.909091        2    example_iris\n",
       "2   RandomForestClassifier  1.000000  1.000000        3    example_iris\n",
       "3   RandomForestClassifier  0.818182  0.818182        4    example_iris\n",
       "4   RandomForestClassifier  0.800000  0.800000        5    example_iris\n",
       "5   RandomForestClassifier  1.000000  1.000000        6    example_iris\n",
       "6   RandomForestClassifier  0.888889  0.888889        7    example_iris\n",
       "7   RandomForestClassifier  1.000000  1.000000        8    example_iris\n",
       "8   RandomForestClassifier  0.777778  0.777778        9    example_iris\n",
       "9   RandomForestClassifier  1.000000  1.000000       10    example_iris\n",
       "10  RandomForestClassifier  0.911061  0.911061  average    example_iris\n",
       "11                     SVC  1.000000  1.000000        1    example_iris\n",
       "12                     SVC  0.909091  0.909091        2    example_iris\n",
       "13                     SVC  1.000000  1.000000        3    example_iris\n",
       "14                     SVC  0.909091  0.909091        4    example_iris\n",
       "15                     SVC  0.800000  0.800000        5    example_iris\n",
       "16                     SVC  1.000000  1.000000        6    example_iris\n",
       "17                     SVC  1.000000  1.000000        7    example_iris\n",
       "18                     SVC  1.000000  1.000000        8    example_iris\n",
       "19                     SVC  0.888889  0.888889        9    example_iris\n",
       "20                     SVC  1.000000  1.000000       10    example_iris\n",
       "21                     SVC  0.950707  0.950707  average    example_iris"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "summary_df[['learner_name', 'accuracy', 'score', 'fold', 'featureset_name']].head(22)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The summary file shows the micro-averaged F1 scores for each learner for each cross-validation fold and the averaged final results too."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As mentioned earlier, SKLL provides several [utility scripts](https://skll.readthedocs.io/en/latest/utilities.html) that can be run from the command line. We examined the `skll_convert` utility earlier. Next, let's examine the `print_model_weights` utility which can print out the model parameters (weights) for linear models on the command line which can be quite useful for debugging and interpretability."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "== intercept values ==\n",
      " 0.524944078798\tsetosa\n",
      " 8.268290425875\tversicolor\n",
      "-10.553640728997\tvirginica\n",
      "\n",
      "Number of nonzero features: 12\n",
      " 9.657326556465\tvirginica\tf3\n",
      "-5.211962709347\tvirginica\tf1\n",
      " 4.700791029369\tvirginica\tf2\n",
      "-4.674890949451\tsetosa\tf2\n",
      "-3.127417554908\tversicolor\tf1\n",
      " 2.863279474242\tsetosa\tf1\n",
      "-2.272367545674\tversicolor\tf3\n",
      "-2.247412344327\tvirginica\tf0\n",
      "-2.210812730402\tsetosa\tf3\n",
      " 1.087224918910\tversicolor\tf2\n",
      " 0.906657747409\tsetosa\tf0\n",
      "-0.191849837696\tversicolor\tf0\n"
     ]
    }
   ],
   "source": [
    "!print_model_weights iris/output/Iris_CV_example_iris_LogisticRegression_fold1.model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we have the `generate_predictions` script which can be used to generate predictions on new test data from an already existing model on disk."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loading iris/test/example_iris_features.jsonlines...           done\n",
      "id\tprediction\n",
      "EXAMPLE_73\tversicolor\n",
      "EXAMPLE_18\tsetosa\n",
      "EXAMPLE_118\tvirginica\n",
      "EXAMPLE_78\tversicolor\n",
      "EXAMPLE_76\tversicolor\n",
      "EXAMPLE_31\tsetosa\n",
      "EXAMPLE_64\tversicolor\n",
      "EXAMPLE_141\tvirginica\n",
      "EXAMPLE_68\tversicolor\n",
      "EXAMPLE_82\tversicolor\n",
      "EXAMPLE_110\tvirginica\n",
      "EXAMPLE_12\tsetosa\n",
      "EXAMPLE_36\tsetosa\n",
      "EXAMPLE_9\tsetosa\n",
      "EXAMPLE_19\tsetosa\n",
      "EXAMPLE_56\tversicolor\n",
      "EXAMPLE_104\tvirginica\n",
      "EXAMPLE_69\tversicolor\n",
      "EXAMPLE_55\tversicolor\n",
      "EXAMPLE_132\tvirginica\n",
      "EXAMPLE_29\tsetosa\n",
      "EXAMPLE_127\tvirginica\n",
      "EXAMPLE_26\tsetosa\n",
      "EXAMPLE_128\tvirginica\n",
      "EXAMPLE_131\tvirginica\n",
      "EXAMPLE_145\tvirginica\n",
      "EXAMPLE_108\tvirginica\n",
      "EXAMPLE_143\tvirginica\n",
      "EXAMPLE_45\tsetosa\n",
      "EXAMPLE_30\tsetosa\n",
      "EXAMPLE_22\tsetosa\n",
      "EXAMPLE_15\tsetosa\n",
      "EXAMPLE_65\tversicolor\n",
      "EXAMPLE_11\tsetosa\n",
      "EXAMPLE_42\tsetosa\n",
      "EXAMPLE_146\tvirginica\n",
      "EXAMPLE_51\tversicolor\n",
      "EXAMPLE_27\tsetosa\n",
      "EXAMPLE_4\tsetosa\n",
      "EXAMPLE_32\tsetosa\n",
      "EXAMPLE_142\tvirginica\n",
      "EXAMPLE_85\tversicolor\n",
      "EXAMPLE_86\tversicolor\n",
      "EXAMPLE_16\tsetosa\n",
      "EXAMPLE_10\tsetosa\n",
      "EXAMPLE_81\tversicolor\n",
      "EXAMPLE_133\tversicolor\n",
      "EXAMPLE_137\tvirginica\n",
      "EXAMPLE_75\tversicolor\n",
      "EXAMPLE_109\tvirginica\n"
     ]
    }
   ],
   "source": [
    "!generate_predictions iris/output/Iris_CV_example_iris_LogisticRegression_fold1.model iris/test/example_iris_features.jsonlines"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2. Using SKLL via the API\n",
    "\n",
    "While the command line tools are intended to be the primary method of using SKLL for normal users who want to run batched machine-learning experiments, the SKLL Python API can also be useful for building applications on top of SKLL and for other advanced use cases. \n",
    "\n",
    "The [*learner*](https://skll.readthedocs.io/en/latest/api/learner.html) and [*data*](https://skll.readthedocs.io/en/latest/api/data.html) modules can be used to broadly replicate the functionalities of command line tools. For example, the `skll.data.Reader` class can be used to programmatically read any SKLL-compatible feature file into a SKLL [*FeatureSet*](https://skll.readthedocs.io/en/latest/api/data.html#module-skll.data.featureset) object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'name': 'iris/test/example_iris_features.jsonlines', 'ids': array(['EXAMPLE_73', 'EXAMPLE_18', 'EXAMPLE_118', 'EXAMPLE_78',\n",
       "       'EXAMPLE_76', 'EXAMPLE_31', 'EXAMPLE_64', 'EXAMPLE_141',\n",
       "       'EXAMPLE_68', 'EXAMPLE_82', 'EXAMPLE_110', 'EXAMPLE_12',\n",
       "       'EXAMPLE_36', 'EXAMPLE_9', 'EXAMPLE_19', 'EXAMPLE_56',\n",
       "       'EXAMPLE_104', 'EXAMPLE_69', 'EXAMPLE_55', 'EXAMPLE_132',\n",
       "       'EXAMPLE_29', 'EXAMPLE_127', 'EXAMPLE_26', 'EXAMPLE_128',\n",
       "       'EXAMPLE_131', 'EXAMPLE_145', 'EXAMPLE_108', 'EXAMPLE_143',\n",
       "       'EXAMPLE_45', 'EXAMPLE_30', 'EXAMPLE_22', 'EXAMPLE_15',\n",
       "       'EXAMPLE_65', 'EXAMPLE_11', 'EXAMPLE_42', 'EXAMPLE_146',\n",
       "       'EXAMPLE_51', 'EXAMPLE_27', 'EXAMPLE_4', 'EXAMPLE_32',\n",
       "       'EXAMPLE_142', 'EXAMPLE_85', 'EXAMPLE_86', 'EXAMPLE_16',\n",
       "       'EXAMPLE_10', 'EXAMPLE_81', 'EXAMPLE_133', 'EXAMPLE_137',\n",
       "       'EXAMPLE_75', 'EXAMPLE_109'], dtype='<U11'), 'labels': array(['versicolor', 'setosa', 'virginica', 'versicolor', 'versicolor',\n",
       "       'setosa', 'versicolor', 'virginica', 'versicolor', 'versicolor',\n",
       "       'virginica', 'setosa', 'setosa', 'setosa', 'setosa', 'versicolor',\n",
       "       'virginica', 'versicolor', 'versicolor', 'virginica', 'setosa',\n",
       "       'virginica', 'setosa', 'virginica', 'virginica', 'virginica',\n",
       "       'virginica', 'virginica', 'setosa', 'setosa', 'setosa', 'setosa',\n",
       "       'versicolor', 'setosa', 'setosa', 'virginica', 'versicolor',\n",
       "       'setosa', 'setosa', 'setosa', 'virginica', 'versicolor',\n",
       "       'versicolor', 'setosa', 'setosa', 'versicolor', 'virginica',\n",
       "       'virginica', 'versicolor', 'virginica'], dtype='<U10'), 'features': <50x4 sparse matrix of type '<class 'numpy.float64'>'\n",
       "\twith 200 stored elements in Compressed Sparse Row format>, 'vectorizer': DictVectorizer(dtype=<class 'numpy.float64'>, separator='=', sort=True,\n",
       "               sparse=True)}"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from skll.data import Reader\n",
    "\n",
    "train_examples_reader = Reader.for_path('iris/train/example_iris_features.jsonlines')\n",
    "test_examples_reader = Reader.for_path('iris/test/example_iris_features.jsonlines')\n",
    "\n",
    "train_examples = train_examples_reader.read()\n",
    "test_examples = test_examples_reader.read()\n",
    "\n",
    "test_examples"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `Learner` class can be used to read in saved models from disk into memory and also to create new learners from scratch. First let's load in one of the models from our previous IRIS experiment and then use it to generate predictions for our test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['versicolor', 'setosa', 'virginica', 'versicolor', 'versicolor',\n",
       "       'setosa', 'versicolor', 'virginica', 'versicolor', 'versicolor',\n",
       "       'virginica', 'setosa', 'setosa', 'setosa', 'setosa', 'versicolor',\n",
       "       'virginica', 'versicolor', 'versicolor', 'virginica', 'setosa',\n",
       "       'virginica', 'setosa', 'virginica', 'virginica', 'virginica',\n",
       "       'virginica', 'virginica', 'setosa', 'setosa', 'setosa', 'setosa',\n",
       "       'versicolor', 'setosa', 'setosa', 'virginica', 'versicolor',\n",
       "       'setosa', 'setosa', 'setosa', 'virginica', 'versicolor',\n",
       "       'versicolor', 'setosa', 'setosa', 'versicolor', 'versicolor',\n",
       "       'virginica', 'versicolor', 'virginica'], dtype='<U10')"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from skll import Learner\n",
    "\n",
    "learner = Learner.from_file('iris/output/Iris_CV_example_iris_LogisticRegression_fold1.model')\n",
    "learner.predict(test_examples, class_labels=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, let's create a new learner, train and tune it on the training set via grid search using accuracy as our objective."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Training data will be shuffled to randomize grid search folds.  Shuffling may yield different results compared to scikit-learn.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0.9494949494949495"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# create a new Linear SVM learner with custom hyper-parameters that will be passed to scikit-learn\n",
    "new_learner = Learner('LinearSVC', model_kwargs = {'max_iter': 1000})\n",
    "best_objective_value, grid_search_results = learner.train(train_examples, grid_objective='accuracy')\n",
    "best_objective_value"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `Learner.train()` method also returns `grid_search_results` which is a dictionary containing useful intermediate results from the grid-search process e.g. the value of the objective for each of the parameter values in the grid and for each of the 3 cross-validation folds."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mean_fit_time</th>\n",
       "      <th>std_fit_time</th>\n",
       "      <th>mean_score_time</th>\n",
       "      <th>std_score_time</th>\n",
       "      <th>param_C</th>\n",
       "      <th>params</th>\n",
       "      <th>split0_test_score</th>\n",
       "      <th>split1_test_score</th>\n",
       "      <th>split2_test_score</th>\n",
       "      <th>mean_test_score</th>\n",
       "      <th>std_test_score</th>\n",
       "      <th>rank_test_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.002151</td>\n",
       "      <td>0.000145</td>\n",
       "      <td>0.000568</td>\n",
       "      <td>0.000100</td>\n",
       "      <td>0.01</td>\n",
       "      <td>{'C': 0.01}</td>\n",
       "      <td>0.657143</td>\n",
       "      <td>0.636364</td>\n",
       "      <td>0.65625</td>\n",
       "      <td>0.649919</td>\n",
       "      <td>0.009592</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.001767</td>\n",
       "      <td>0.000143</td>\n",
       "      <td>0.000412</td>\n",
       "      <td>0.000086</td>\n",
       "      <td>0.1</td>\n",
       "      <td>{'C': 0.1}</td>\n",
       "      <td>0.800000</td>\n",
       "      <td>0.757576</td>\n",
       "      <td>0.75000</td>\n",
       "      <td>0.769192</td>\n",
       "      <td>0.022003</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>0.001803</td>\n",
       "      <td>0.000123</td>\n",
       "      <td>0.000362</td>\n",
       "      <td>0.000007</td>\n",
       "      <td>1</td>\n",
       "      <td>{'C': 1.0}</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.818182</td>\n",
       "      <td>0.93750</td>\n",
       "      <td>0.918561</td>\n",
       "      <td>0.075425</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>0.001727</td>\n",
       "      <td>0.000055</td>\n",
       "      <td>0.000333</td>\n",
       "      <td>0.000005</td>\n",
       "      <td>10</td>\n",
       "      <td>{'C': 10.0}</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.818182</td>\n",
       "      <td>1.00000</td>\n",
       "      <td>0.939394</td>\n",
       "      <td>0.085710</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>0.001854</td>\n",
       "      <td>0.000118</td>\n",
       "      <td>0.000329</td>\n",
       "      <td>0.000008</td>\n",
       "      <td>100</td>\n",
       "      <td>{'C': 100.0}</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.848485</td>\n",
       "      <td>1.00000</td>\n",
       "      <td>0.949495</td>\n",
       "      <td>0.071425</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   mean_fit_time  std_fit_time  mean_score_time  std_score_time param_C  \\\n",
       "0       0.002151      0.000145         0.000568        0.000100    0.01   \n",
       "1       0.001767      0.000143         0.000412        0.000086     0.1   \n",
       "2       0.001803      0.000123         0.000362        0.000007       1   \n",
       "3       0.001727      0.000055         0.000333        0.000005      10   \n",
       "4       0.001854      0.000118         0.000329        0.000008     100   \n",
       "\n",
       "         params  split0_test_score  split1_test_score  split2_test_score  \\\n",
       "0   {'C': 0.01}           0.657143           0.636364            0.65625   \n",
       "1    {'C': 0.1}           0.800000           0.757576            0.75000   \n",
       "2    {'C': 1.0}           1.000000           0.818182            0.93750   \n",
       "3   {'C': 10.0}           1.000000           0.818182            1.00000   \n",
       "4  {'C': 100.0}           1.000000           0.848485            1.00000   \n",
       "\n",
       "   mean_test_score  std_test_score  rank_test_score  \n",
       "0         0.649919        0.009592                5  \n",
       "1         0.769192        0.022003                4  \n",
       "2         0.918561        0.075425                3  \n",
       "3         0.939394        0.085710                2  \n",
       "4         0.949495        0.071425                1  "
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.DataFrame(grid_search_results)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, let's evaluate this new trained learner on the test set using both accuracy as well as balanced accuracy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test Accuracy : 0.98\n",
      "Test Balanced Accuracy : 0.9791666666666666\n",
      "Test confusion matrix: [[19, 0, 0], [0, 15, 0], [0, 1, 15]]\n"
     ]
    }
   ],
   "source": [
    "(conf_matrix,\n",
    " accuracy,\n",
    " prf_dict,\n",
    " model_params,\n",
    " obj_score,\n",
    " metric_scores) = learner.evaluate(test_examples, grid_objective='accuracy', output_metrics=['balanced_accuracy'])\n",
    "\n",
    "print('Test Accuracy : {}'.format(obj_score))\n",
    "print('Test Balanced Accuracy : {}'.format(metric_scores['balanced_accuracy']))\n",
    "print('Test confusion matrix: {}'.format(conf_matrix))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This tutorial only covered some of the most basic functionality for SKLL. SKLL can do a whole lot more. Please read the [comprehensive documentation](https://skll.readthedocs.io) for more details. We also welcome any contributions you might want to make to SKLL. Please read the [contribution guidelines](https://skll.readthedocs.io/en/latest/contributing.html)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:sklldev]",
   "language": "python",
   "name": "conda-env-sklldev-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}