Testing_tutorial.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Testing_tutorial.ipynb",
      "provenance": [],
      "authorship_tag": "ABX9TyNYe/l5bi0/tqnpX8Ng2b5x",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/philsaurabh/Tutorials/blob/main/Testing_tutorial.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kIsVrkAYvhZ6"
      },
      "source": [
        "# Scipy\n",
        "SciPy stands for Scientific Python.\n",
        "\n",
        "It provides more utility functions for optimization, stats and signal processing.\n",
        "\n",
        "# statmodels\n",
        "\n",
        "statsmodels is a Python module that provides classes and functions for the estimation of many different statistical models, as well as for conducting statistical tests, and statistical data exploration. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "o2mzBgA8B-2B"
      },
      "source": [
        "# Importing Libraries"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "b2iRX35_B1ib"
      },
      "source": [
        "import pandas as pd\n",
        "import scipy.stats as st\n",
        "import statsmodels.api as sm\n",
        "from statsmodels.formula.api import ols"
      ],
      "execution_count": 11,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YPrXTd1lw0jc"
      },
      "source": [
        "# Cumulative Distribution Function"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "NQg7ANgMCNse",
        "outputId": "95e9a4d8-d5d6-42df-8e87-c87440fae901"
      },
      "source": [
        "st.norm.cdf(1.64)\n",
        "#st.norm.cdf([1.,0,-1.])# For more than one point"
      ],
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([0.84134475, 0.5       , 0.15865525])"
            ]
          },
          "metadata": {},
          "execution_count": 4
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MoIui_d6weCF"
      },
      "source": [
        "# Percent Point Function\n",
        "To find the median of a distribution, we can use the percent point function ppf, which is the inverse of the cdf"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "3acXc3IjCH2m",
        "outputId": "aa00b4f5-7631-4186-c82b-64b922c190ec"
      },
      "source": [
        "st.norm.ppf(.95)\n",
        "#st.norm.ppf([.95,0.05])"
      ],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "1.6448536269514722"
            ]
          },
          "metadata": {},
          "execution_count": 6
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Zl6Z3xrWCQbF",
        "outputId": "17781312-4c67-43f9-8021-91960b811140"
      },
      "source": [
        "a=st.norm.ppf(.975)\n",
        "b=st.norm.ppf(.025)\n",
        "two_sided=a-b\n",
        "two_sided"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "3.9199279690801085"
            ]
          },
          "metadata": {},
          "execution_count": 7
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bKFiBsep6pdX"
      },
      "source": [
        "# Critical value for z"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ucVHfI3KDW4o"
      },
      "source": [
        "```\n",
        "z-critical = stats.norm.ppf(1 - alpha) (use alpha = alpha/2 for two-sided)\n",
        "t-critical = stats.t.ppf(alpha/numOfTails, ddof)\n",
        "```"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "N3O6cT6KCeVS",
        "outputId": "1028ab17-5ce9-442a-be68-ea9fe5df19c4"
      },
      "source": [
        "# alpha to critical\n",
        "alpha = 0.05\n",
        "n_sided = 2 # 2-sided test\n",
        "z_crit = st.norm.ppf(1-alpha/n_sided)\n",
        "print('Z critical value: ',z_crit) "
      ],
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Z critical value:  1.959963984540054\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "gikJjP-RDFJ4",
        "outputId": "170eae35-c694-44ca-b91e-1ec51027c62d"
      },
      "source": [
        "# critical to alpha\n",
        "alpha = st.norm.sf(z_crit) * n_sided\n",
        "print('alpha: ',alpha) "
      ],
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "alpha:  0.05\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xzxTw_Yn6yYP"
      },
      "source": [
        "# T- stats\n",
        "for small samples (<50) we use t-statistics"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ttSC1fqRDLaJ",
        "outputId": "f53ee63a-30fa-4bcb-a6cd-f0d818d2d733"
      },
      "source": [
        "#n=999, p<0.05, 2-tail\n",
        "#(0.05,999)\n",
        "print(st.t.ppf(1-0.025, 999))\n",
        "\n",
        "#n=999, p<0.05%, Single tail\n",
        "print(st.t.ppf(1-0.05, 999)-st.t.ppf(1-0.025, 999))"
      ],
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "1.9623414611334487\n",
            "-0.3159611157059137\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "UxE-8DuWD7ks",
        "outputId": "ae61c228-7eef-4efa-8a3a-7c63f81e7ca4"
      },
      "source": [
        "# Another Example\n",
        "# n = 9, degree of freedom = 9-1 = 8\n",
        "# for 99% confidence interval, alpha = 1% = 0.01 and alpha/2 = 0.005\n",
        "ci = 99\n",
        "n = 999\n",
        "t = st.t.ppf(1- ((100-ci)/2/100), n-1) # 99% CI, t8,0.005\n",
        "print(t)"
      ],
      "execution_count": 16,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "2.580764586385345\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hjxkc9zD7nX_"
      },
      "source": [
        "# One-Way Anova"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Po3bju8yEGAT",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 203
        },
        "outputId": "d9ab3d55-dde7-42ae-997d-3db1ca14d3da"
      },
      "source": [
        "# To read data as dataframe\n",
        "df = pd.read_csv('https://raw.githubusercontent.com/philsaurabh/Tutorials/main/Advertising.csv')\n",
        "df.head()"
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Unnamed: 0</th>\n",
              "      <th>TV</th>\n",
              "      <th>radio</th>\n",
              "      <th>newspaper</th>\n",
              "      <th>sales</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1</td>\n",
              "      <td>230.1</td>\n",
              "      <td>37.8</td>\n",
              "      <td>69.2</td>\n",
              "      <td>22.1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>2</td>\n",
              "      <td>44.5</td>\n",
              "      <td>39.3</td>\n",
              "      <td>45.1</td>\n",
              "      <td>10.4</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>3</td>\n",
              "      <td>17.2</td>\n",
              "      <td>45.9</td>\n",
              "      <td>69.3</td>\n",
              "      <td>9.3</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>4</td>\n",
              "      <td>151.5</td>\n",
              "      <td>41.3</td>\n",
              "      <td>58.5</td>\n",
              "      <td>18.5</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>5</td>\n",
              "      <td>180.8</td>\n",
              "      <td>10.8</td>\n",
              "      <td>58.4</td>\n",
              "      <td>12.9</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   Unnamed: 0     TV  radio  newspaper  sales\n",
              "0           1  230.1   37.8       69.2   22.1\n",
              "1           2   44.5   39.3       45.1   10.4\n",
              "2           3   17.2   45.9       69.3    9.3\n",
              "3           4  151.5   41.3       58.5   18.5\n",
              "4           5  180.8   10.8       58.4   12.9"
            ]
          },
          "metadata": {},
          "execution_count": 2
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DrREu92xET3S"
      },
      "source": [
        "a= df['TV']\n",
        "b= df['radio']\n",
        "c= df['newspaper']\n",
        "F,p=st.f_oneway(a,b,c)"
      ],
      "execution_count": 12,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "URITdHo4FAik",
        "outputId": "f9d5db05-33c1-4a87-f9c9-9c8d3a4fe1b9"
      },
      "source": [
        "F"
      ],
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "358.8514595342597"
            ]
          },
          "metadata": {},
          "execution_count": 13
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "egOQdm5QFKtu",
        "outputId": "4de18965-1980-401c-e1cf-304f286594fc"
      },
      "source": [
        "p"
      ],
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "4.552931539744962e-103"
            ]
          },
          "metadata": {},
          "execution_count": 14
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "q8wTNoFZ70yu"
      },
      "source": [
        "# Two- Way ANOVA"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 110
        },
        "id": "rUTIWam2GNX5",
        "outputId": "b79a0b63-fd2f-43b8-bf01-c12bfb966e99"
      },
      "source": [
        "df_melt = pd.melt(df.reset_index(), id_vars=['index'], value_vars=['TV', 'radio', 'newspaper', 'sales'])\n",
        "# replace column names\n",
        "df_melt.columns = ['index', 'treatments', 'value']\n",
        "# Ordinary Least Squares (OLS) model\n",
        "model = ols('value ~ C(treatments)', data=df_melt).fit()\n",
        "anova_table = sm.stats.anova_lm(model, typ=2)\n",
        "anova_table"
      ],
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>sum_sq</th>\n",
              "      <th>df</th>\n",
              "      <th>F</th>\n",
              "      <th>PR(&gt;F)</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>C(treatments)</th>\n",
              "      <td>2.349842e+06</td>\n",
              "      <td>3.0</td>\n",
              "      <td>387.144309</td>\n",
              "      <td>5.150514e-155</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>Residual</th>\n",
              "      <td>1.610489e+06</td>\n",
              "      <td>796.0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                     sum_sq     df           F         PR(>F)\n",
              "C(treatments)  2.349842e+06    3.0  387.144309  5.150514e-155\n",
              "Residual       1.610489e+06  796.0         NaN            NaN"
            ]
          },
          "metadata": {},
          "execution_count": 15
        }
      ]
    }
  ]
}