examples/time_series_datasets.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "allWIe5kwPcS"
   },
   "source": [
    "# Time Series Datasets\n",
    "\n",
    "This notebook shows how to create a time series dataset from some csv file in order to then share it on the [🤗 hub](https://huggingface.co/docs/datasets/index). We will use the GluonTS library to read the csv into the appropriate format. We start by installing the libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "id": "4XnNcdWbwrNo"
   },
   "outputs": [],
   "source": [
    "! pip install -q datasets gluonts orjson"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "dI1yo_vHw5CV"
   },
   "source": [
    "GluonTS comes with a pandas DataFrame based dataset so our strategy will be to read the csv file, and process it as a `PandasDataset`. We will then iterate over it and convert it to a 🤗 dataset with the appropriate schema for time series. So lets get started!\n",
    "\n",
    "## `PandasDataset`\n",
    "\n",
    "Suppose we are given multiple (10) time series stacked on top of each other in a dataframe with an `item_id` column that distinguishes different series:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 206
    },
    "id": "e9FaT_VpwuI2",
    "outputId": "8a10c908-41e1-4ca7-b420-01c0810c5c4b"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "  <div id=\"df-90cae087-2afb-413c-85e0-0f6b9a05e059\">\n",
       "    <div class=\"colab-df-container\">\n",
       "      <div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>target</th>\n",
       "      <th>item_id</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2021-01-01 00:00:00</th>\n",
       "      <td>-1.3378</td>\n",
       "      <td>A</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2021-01-01 01:00:00</th>\n",
       "      <td>-1.6111</td>\n",
       "      <td>A</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2021-01-01 02:00:00</th>\n",
       "      <td>-1.9259</td>\n",
       "      <td>A</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2021-01-01 03:00:00</th>\n",
       "      <td>-1.9184</td>\n",
       "      <td>A</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2021-01-01 04:00:00</th>\n",
       "      <td>-1.9168</td>\n",
       "      <td>A</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>\n",
       "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-90cae087-2afb-413c-85e0-0f6b9a05e059')\"\n",
       "              title=\"Convert this dataframe to an interactive table.\"\n",
       "              style=\"display:none;\">\n",
       "        \n",
       "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
       "       width=\"24px\">\n",
       "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
       "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
       "  </svg>\n",
       "      </button>\n",
       "      \n",
       "  <style>\n",
       "    .colab-df-container {\n",
       "      display:flex;\n",
       "      flex-wrap:wrap;\n",
       "      gap: 12px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert {\n",
       "      background-color: #E8F0FE;\n",
       "      border: none;\n",
       "      border-radius: 50%;\n",
       "      cursor: pointer;\n",
       "      display: none;\n",
       "      fill: #1967D2;\n",
       "      height: 32px;\n",
       "      padding: 0 0 0 0;\n",
       "      width: 32px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert:hover {\n",
       "      background-color: #E2EBFA;\n",
       "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "      fill: #174EA6;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert {\n",
       "      background-color: #3B4455;\n",
       "      fill: #D2E3FC;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert:hover {\n",
       "      background-color: #434B5C;\n",
       "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
       "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
       "      fill: #FFFFFF;\n",
       "    }\n",
       "  </style>\n",
       "\n",
       "      <script>\n",
       "        const buttonEl =\n",
       "          document.querySelector('#df-90cae087-2afb-413c-85e0-0f6b9a05e059 button.colab-df-convert');\n",
       "        buttonEl.style.display =\n",
       "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "\n",
       "        async function convertToInteractive(key) {\n",
       "          const element = document.querySelector('#df-90cae087-2afb-413c-85e0-0f6b9a05e059');\n",
       "          const dataTable =\n",
       "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
       "                                                     [key], {});\n",
       "          if (!dataTable) return;\n",
       "\n",
       "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
       "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
       "            + ' to learn more about interactive tables.';\n",
       "          element.innerHTML = '';\n",
       "          dataTable['output_type'] = 'display_data';\n",
       "          await google.colab.output.renderOutput(dataTable, element);\n",
       "          const docLink = document.createElement('div');\n",
       "          docLink.innerHTML = docLinkHtml;\n",
       "          element.appendChild(docLink);\n",
       "        }\n",
       "      </script>\n",
       "    </div>\n",
       "  </div>\n",
       "  "
      ],
      "text/plain": [
       "                     target item_id\n",
       "2021-01-01 00:00:00 -1.3378       A\n",
       "2021-01-01 01:00:00 -1.6111       A\n",
       "2021-01-01 02:00:00 -1.9259       A\n",
       "2021-01-01 03:00:00 -1.9184       A\n",
       "2021-01-01 04:00:00 -1.9168       A"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "url = (\n",
    "    \"https://gist.githubusercontent.com/rsnirwan/a8b424085c9f44ef2598da74ce43e7a3\"\n",
    "    \"/raw/b6fdef21fe1f654787fa0493846c546b7f9c4df2/ts_long.csv\"\n",
    ")\n",
    "df = pd.read_csv(url, index_col=0, parse_dates=True)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "s4WCjvBqxi3B"
   },
   "source": [
    "After converting it into a `pd.Dataframe` we can then convert it into GluonTS's `PandasDataset`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "id": "su1i8ZdDxf7c"
   },
   "outputs": [],
   "source": [
    "from gluonts.dataset.pandas import PandasDataset\n",
    "\n",
    "ds = PandasDataset.from_long_dataframe(df, target=\"target\", item_id=\"item_id\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "cYnHkLdex_n3"
   },
   "source": [
    "\n",
    "## 🤗 Datasets\n",
    "\n",
    "From here we have to map the pandas dataset's `start` field into a time stamp instead of a `pd.Period`. We do this by defining the following class:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "id": "gv5Ytpwrx4FQ"
   },
   "outputs": [],
   "source": [
    "class ProcessStartField():\n",
    "    ts_id = 0\n",
    "    \n",
    "    def __call__(self, data):\n",
    "        data[\"start\"] = data[\"start\"].to_timestamp()\n",
    "        data[\"feat_static_cat\"] = [self.ts_id]\n",
    "        self.ts_id += 1\n",
    "        \n",
    "        return data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "id": "DrhbT1QIyPMA"
   },
   "outputs": [],
   "source": [
    "from gluonts.itertools import Map\n",
    "\n",
    "process_start = ProcessStartField()\n",
    "\n",
    "list_ds = list(Map(process_start, ds))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Ug2kLNUPyeyJ"
   },
   "source": [
    "Next we need to define our schema features and create our dataset from this list via the `from_list` function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "id": "r1rQUtvGycaF"
   },
   "outputs": [],
   "source": [
    "from datasets import Dataset, Features, Value, Sequence\n",
    "\n",
    "features  = Features(\n",
    "    {    \n",
    "        \"start\": Value(\"timestamp[s]\"),\n",
    "        \"target\": Sequence(Value(\"float32\")),\n",
    "        \"feat_static_cat\": Sequence(Value(\"uint64\")),\n",
    "        # \"feat_static_real\":  Sequence(Value(\"float32\")),\n",
    "        # \"feat_dynamic_real\": Sequence(Sequence(Value(\"uint64\"))),\n",
    "        # \"feat_dynamic_cat\": Sequence(Sequence(Value(\"uint64\"))),\n",
    "        \"item_id\": Value(\"string\"),\n",
    "    }\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "id": "RrpP2oAxywC8"
   },
   "outputs": [],
   "source": [
    "dataset = Dataset.from_list(list_ds, features=features)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "GrbGlDtYzyIf"
   },
   "source": [
    "We can thus use this strategy to [share](https://huggingface.co/docs/datasets/share) the dataset to the hub."
   ]
  }
 ],
 "metadata": {
  "colab": {
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}