ch12_Mastodon/ch12_Mastodon.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%%html\n",
    "<!-- CSS settings for this notbook -->\n",
    "<style>\n",
    "    h1 {color:#BB0000}\n",
    "    h2 {color:purple}\n",
    "    h3 {color:#0099ff}\n",
    "    hr {    \n",
    "        border: 0;\n",
    "        height: 3px;\n",
    "        background: #333;\n",
    "        background-image: linear-gradient(to right, #ccc, black, #ccc);\n",
    "    }\n",
    "</style>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "# Data Mining Mastodon"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "# Objectives\n",
    "* What is **Mastodon**?\n",
    "* Why we're presenting **Mastodon** rather than **Twitter**\n",
    "* **Data-mine Mastodon** with **Mastodon.py** library\n",
    "* Use various **Mastodon** API methods\n",
    "* **Get information** about a specific Mastodon account\n",
    "* **Look up trending hashtags** \n",
    "* **Search for toots** containing a specific hashtag\n",
    "* **Process streams of toots** as they’re happening\n",
    "* **Clean and preprocess toots** to prepare them for analysis\n",
    "* **Translate foreign language toots** into English \n",
    "* Tap into the **live streams of toots**\n",
    "<!--* Perform **sentiment analysis** on toots from the live stream-->\n",
    "* Create an **interactive map of Mastodon servers locations** from which Toots are received"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "# 12.1 Introduction \n",
    "* **Data mining** &mdash; searching large collections of data for **insights**\n",
    "* **Sentiment** in toots can help **make predictions**  \n",
    "    * **Stock prices**\n",
    "    * **Election results**\n",
    "    * Likely **revenues** for a **new movie** or, more generally, **product**\n",
    "    * **Success** of a company’s **marketing campaign**\n",
    "* Spot **comments on your company's products** \n",
    "* Spot **faults in competitors’ products** \n",
    "* Spot **trending topics**\n",
    "* **Connect to Mastodon** with easy-to-use **Web services**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## What Is Mastodon?\n",
    "* Free social network\n",
    "* Similar to Twitter, but decentralized and more privacy focused\n",
    "* No ads/profit model\n",
    "* Thousands of servers run by individuals and companies worldwide\n",
    "* **Federated** (known as the **Fediverse**)\n",
    "    * Independent servers distributed across the Internet\n",
    "    * Communication among the server nodes  \n",
    "    > https://en.wikipedia.org/wiki/Distributed_social_network\n",
    "    * Can communicate with accounts throughout the **Fediverse** \n",
    "* **Toots**\n",
    "    * Messages up to **500 characters**\n",
    "    * Some servers allow more\n",
    "* Anyone can generally choose to follow anyone else but depends on\n",
    "    * individual users' account settings \n",
    "    * specific server rules"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Mastodon Stats\n",
    "* https://mastodon-analytics.com/ "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Accessing Mastodon Data Programmatically \n",
    "* Anyone with an account can use the APIs\n",
    "* Access and manipulate accounts, servers, toots (statuses), timelines, trends, ...\n",
    "* Can **tap into the live stream** of toots for a given server or the **Fediverse** "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "# 12.2 Overview of the Mastodon APIs \n",
    "* **Web services** are methods that you call in the **cloud**\n",
    "* Each method has a **web service endpoint** represented by a **URL**\n",
    "* **Caution**: Internet connections can be lost, services can change and some services are not available in all countries, so **apps can be brittle**\n",
    "* Some **API categories** \n",
    "    * **Accounts API** — Access information about and manipulate **Mastodon user accounts**\n",
    "    * **Statuses API** — Access info about and post **status updates**, known as **toots**\n",
    "    * **Timelines API** — Toots and other \"events\" (follows, likes, ...) over time — since the inception of each Mastodon server\n",
    "    > Enables access to toots and other \"events\" from the **public fediverse**, **toots with specific hashtags**, **logged-in user's timeline** (including accounts the user follows) and **lists** for filtering a user's home timeline. Can use timelines to search for **past toots** containing specific hashtags and access **live toot streams**\n",
    "* **Mastodon API categories** under the **API METHODS** heading in the left column at\n",
    ">https://docs.joinmastodon.org/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "# 12.3 Creating a Mastodon Account "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Developer Accounts\n",
    "* **Mastodon does not have separate developer accounts**\n",
    "    * Anyone with a Mastodon account can be a developer\n",
    "    * Every server has its own rules — some servers allow anyone to join, some require approval"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Servers\n",
    "* Sign-up for main server: https://mastodon.social/auth/sign_up\n",
    "* Or, explore servers worldwide at\n",
    "    * https://joinmastodon.org/servers\n",
    "    * Many more servers than listed here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Mastodon Server Covenant\n",
    "* Servers listed at https://joinmastodon.org/servers adhere to the **Mastodon Server Covenant**\n",
    "    * https://joinmastodon.org/covenant\n",
    "    * **Active moderation** by the server administrators\n",
    "    * **Daily backups**\n",
    "    * **Multiple administrators** with emergency access to server infrastructure for fixing problems\n",
    "    * Administrator agrees to provide **at least 3 months notice** to users before **shutting down a server permanently** (if a server shuts down, the data is no longer available)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Mastodon.social — Original and Largest Overall Mastodon Server\n",
    "* Due to exodus from Twitter, **Deitel joined `mastodon.social`** \n",
    "    * Deitel requirement: Readers/viewers must be able to experiment with web services using free tiers\n",
    "    * New Twitter API free and initial paid tier have extremely limited capabilities\n",
    "* https://joinmastodon.org/servers enables you to filter servers based on\n",
    "    * region\n",
    "    * language\n",
    "    * topical focus of that server\n",
    "* Can join multiple servers\n",
    "* Can also **set up your own servers to create new Mastodon communities** with your own rules and restrictions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "# 12.4 What’s in a Mastodon API Response? \n",
    "* Mastodon API methods return **JSON (JavaScript Object Notation)** objects\n",
    "    * Like Twitter and most popular web services today\n",
    "* Text-based **data-interchange format** \n",
    "* Represents objects as **collections of name–value pairs** (like dictionaries)\n",
    "* Commonly used in web services\n",
    "* Human and computer readable"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## JSON\n",
    "* **JSON object format**:\n",
    "\n",
    "> ```\n",
    "> {propertyName1: value1, propertyName2: value2}\n",
    "> ```\n",
    "* **JSON array format (like Python list)**:\n",
    "\n",
    "> ```\n",
    "> [value1, value2, value3]\n",
    "> ```\n",
    "* **Mastodon.py handles the JSON for you** behind the scenes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Class `mastodon.AttribAccessDict` \n",
    "* Mastodon returns JSON as **`mastodon.AttribAccessDict` objects**\n",
    "* Python `dict` (dictionary) subclass\n",
    "* Access via\n",
    "    * traditional Python dictionary keys  \n",
    "    * attributes named to match the dictionary keys\n",
    "* **API ENTITIES** section of the Mastodon docs (https://docs.joinmastodon.org/) describes the 52 JSON objects you'll find in various Mastodon API responses"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Sample JSON for Trending Hashtags\n",
    "* A portion of the JSON response to a request for recent trending hashtags\n",
    "\n",
    "```json\n",
    "[{'name': 'caturday',\n",
    "  'url': 'https://mastodon.social/tags/caturday',\n",
    "  'history': [{'day': datetime.datetime(2023, 4, 22, 0, 0, tzinfo=datetime.timezone.utc),\n",
    "    'accounts': '719',\n",
    "    'uses': '828'},\n",
    "   {'day': datetime.datetime(2023, 4, 21, 0, 0, tzinfo=datetime.timezone.utc),\n",
    "    'accounts': '58',\n",
    "    'uses': '62'},\n",
    "   {'day': datetime.datetime(2023, 4, 20, 0, 0, tzinfo=datetime.timezone.utc),\n",
    "    'accounts': '25',\n",
    "    'uses': '32'},\n",
    "   {'day': datetime.datetime(2023, 4, 19, 0, 0, tzinfo=datetime.timezone.utc),\n",
    "    'accounts': '28',\n",
    "    'uses': '34'},\n",
    "   {'day': datetime.datetime(2023, 4, 18, 0, 0, tzinfo=datetime.timezone.utc),\n",
    "    'accounts': '19',\n",
    "    'uses': '22'},\n",
    "   {'day': datetime.datetime(2023, 4, 17, 0, 0, tzinfo=datetime.timezone.utc),\n",
    "    'accounts': '25',\n",
    "    'uses': '26'},\n",
    "   {'day': datetime.datetime(2023, 4, 16, 0, 0, tzinfo=datetime.timezone.utc),\n",
    "    'accounts': '225',\n",
    "    'uses': '254'}],\n",
    "  'following': False},\n",
    " {'name': 'ScreenshotSaturday',\n",
    "  'url': 'https://mastodon.social/tags/ScreenshotSaturday',\n",
    "  'history': [{'day': datetime.datetime(2023, 4, 22, 0, 0, tzinfo=datetime.timezone.utc),\n",
    "    'accounts': '56',\n",
    "    'uses': '59'},\n",
    "   {'day': datetime.datetime(2023, 4, 21, 0, 0, tzinfo=datetime.timezone.utc),\n",
    "    'accounts': '0',\n",
    "    'uses': '0'},\n",
    "   {'day': datetime.datetime(2023, 4, 20, 0, 0, tzinfo=datetime.timezone.utc),\n",
    "    'accounts': '0',\n",
    "    'uses': '0'},\n",
    "   {'day': datetime.datetime(2023, 4, 19, 0, 0, tzinfo=datetime.timezone.utc),\n",
    "    'accounts': '0',\n",
    "    'uses': '0'},\n",
    "   {'day': datetime.datetime(2023, 4, 18, 0, 0, tzinfo=datetime.timezone.utc),\n",
    "    'accounts': '3',\n",
    "    'uses': '3'},\n",
    "   {'day': datetime.datetime(2023, 4, 17, 0, 0, tzinfo=datetime.timezone.utc),\n",
    "    'accounts': '3',\n",
    "    'uses': '3'},\n",
    "   {'day': datetime.datetime(2023, 4, 16, 0, 0, tzinfo=datetime.timezone.utc),\n",
    "    'accounts': '24',\n",
    "    'uses': '26'}],\n",
    "  'following': False},\n",
    "  ...\n",
    " ]\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "# 12.5 Installing the Libraries Used in This Notebook\n",
    "\n",
    "## Installing Mastodon.py \n",
    "* https://github.com/halcy/Mastodon.py \n",
    "* Easy access to Mastodon APIs\n",
    "* Mastodon.py docs: https://mastodonpy.readthedocs.io/\n",
    "> `pip3 install Mastodon.py`\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## DeepL AI Translator \n",
    "* Mastodon's API supports translation, but not yet supported by Mastodon.py library\n",
    "* https://github.com/DeepLcom/deepl-python\n",
    "> `pip install --upgrade deepl`\n",
    "* DeepL requires an API key\n",
    "* Free one allows 500,000 characters/month\n",
    "* To get a key:\n",
    "> * Go to https://www.deepl.com/pro#developer\n",
    "> * Click **API**\n",
    "> * Click **Sign up for free**\n",
    "> * Under **DeepL API Free** click **Sign up for free**\n",
    "> * Specify an email/password and click **Continue**\n",
    "> * Fill in the form and provide a credit card — required to prevent “fraudulent multiple registrations”, then click **Continue**\n",
    "> * Read the terms and, if you agree, click **Sign up for free**\n",
    "> * Click the **Account Management** link on the thank you page\n",
    "> * Click the **Account** tab and scroll to **Authentication Key for DeepL API**\n",
    "> * Copy your key then open **`keys_mastodon.py`** and replace `'your key here'` with your DeepL key\n",
    ">> `deepL_key = 'your key here'`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Installing geopy \n",
    "* https://github.com/geopy/geopy\n",
    "* Convert locations, such as **Boston, MA**, into latitudes and longitudes, such as **42.3602534** and **-71.0582912**, for plotting on maps\n",
    "* We'll use the free **ArcGIS** service\n",
    ">`conda install -c conda-forge geopy`\n",
    "> * Windows users: **Run the Anaconda Prompt as an Administrator**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Folium Library and Leaflet.js JavaScript Mapping Library\n",
    "* https://github.com/python-visualization/folium\n",
    "* Creates interactive maps\n",
    "> `pip install folium`\n",
    "\n",
    "**Maps from OpenStreetMap.org**\n",
    "* Leaflet.js uses open-source maps from `OpenStreetMap.org`. \n",
    "* Copyrighted by the OpenStreetMap.org contributors\n",
    "* www.openstreetmap.org/copyright \n",
    "* www.opendatacommons.org/licenses/odbl"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 12.6 Preparing to Interact with Mastodon Programmatically"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Import Username and Password\n",
    "* Before executing this cell, ensure that your copy of `keys_mastodon.py` contains your Mastodon credentials\n",
    "* **Many Mastodon APIs do not require authentication**\n",
    "    * Some APIs optionally require authentication — determined by each server's administrator\n",
    "    * Some require authentication, such as those that enable administration of a mastodon server\n",
    "* See each method's documentation for **authentication requirements**\n",
    "    * Mastodon.py: https://mastodonpy.readthedocs.io/\n",
    "    * Main Mastodon docs: https://docs.joinmastodon.org/\n",
    "* **We will log in, so we are authenticated for calls that require authentication**, such as searching for accounts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import keys_mastodon"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Register a Mastodon App\n",
    "* Must be done **once per server** that you'll directly interact with via the API \n",
    "    * As you'll see, through one server, you can get access to the Fediverse data\n",
    "* For apps you are distributing (e.g., a mobile-phone app for interacting with Mastodon)\n",
    "    * must register **once for each device/server pair**\n",
    "    * for example, a mobil app might allow the user to manage accounts on multiple Mastodon servers\n",
    "* Arguments\n",
    "    * app name\n",
    "    * `api_base_url` — your specific Mastodon server\n",
    "    * `to_file` — file in which `create_app` saves app credentials to the specified file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from mastodon import Mastodon"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# create deiteltest app and save its credentials\n",
    "credentials = Mastodon.create_app(\n",
    "    'DeitelPythonDataScienceMastodonApp',\n",
    "    api_base_url='https://mastodon.social',\n",
    "    to_file='deiteltest_client_credentials.secret'\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:#AAA; background-color:#AAA;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "# 12.7 Creating a `Mastodon` object to Access Mastodon APIs\n",
    "* **`Mastodon` object** is your gateway to using the Mastodon APIs\n",
    "* Uses the info stored via the `to_file` parameter in preceding `create_app` call "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "<!-- mastodon = Mastodon(client_id='deiteltest_client_credentials.secret') -->"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create Mastodon Object for Authentication Purposes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mastodon = Mastodon(\n",
    "    client_id='deiteltest_client_credentials.secret',\n",
    "    api_base_url='https://mastodon.social'\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Generate authorization URL\n",
    "* Mastodon recently changed the authenticatin process\n",
    "* This creates a URL the user can use to log into Mastodon"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "auth_url = mastodon.auth_request_url(scopes=['read', 'write'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Have user log in\n",
    "* Open the URL in a browser and obtain the authorization code "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f\"Please visit this URL and authorize the app: {auth_url}\")\n",
    "code = input(\"Enter the authorization code: \")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Log in with the authorization code\n",
    "* Required for some API calls"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "access_token_str = mastodon.log_in(\n",
    "    code=code,\n",
    "    scopes=['read'], # can include read, write, follow, push\n",
    "    to_file='usercred.secret'\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true,
    "jupyter": {
     "source_hidden": true
    }
   },
   "source": [
    "<!-- ## Log into Mastodon\n",
    "* Log into the account\n",
    "* May not be required depending on the API methods you'll use\n",
    "\n",
    "access_token = mastodon.log_in(keys_mastodon.usr, keys_mastodon.pwd, \n",
    "    scopes=['read', 'write', 'follow'],\n",
    "    to_file='deiteltest_client_credentials.secret')\n",
    "\n",
    "mastodon = Mastodon(\n",
    "    client_id='deiteltest_client_credentials.secret',\n",
    "    access_token='usercred.secret',\n",
    "    api_base_url='https://mastodon.social'\n",
    ") -->"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## **Example:** Rate Limits\n",
    "* Typically, **300 calls per user** or **7500 calls per IP address** in **5 minutes**\n",
    "    * Can vary by server\n",
    "* Options **throw**, **wait** (default) and **pace**\n",
    "    * **throw**: `MastodonRateLimitError` when a request hits the rate limit — for apps that manage their own rate limiting\n",
    "    * **wait** (default): When rate limit hit, waits until rate limit resets (at end of five-minute interval), then tries again\n",
    "    * **pace**: Delays each request after the first, attempting to avoid hitting the rate limit; acts like **wait** mode if limit is hit\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Number of Calls Per 5 Minutes Allowed on This Server"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "mastodon.ratelimit_limit "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Number of Calls Remaining in Current Rate Interval Period"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "mastodon.ratelimit_remaining"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:#AAA; background-color:#AAA;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "# 12.8 **Example:** Getting a Mastodon Instance's (Server's) Info\n",
    "* **Instance** JSON description: https://docs.joinmastodon.org/entities/Instance/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get the Instance Info\n",
    "* Depending on server, might need to be logged in"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "instance = mastodon.instance() "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Print Some Instance Info\n",
    "* `title` ― server name\n",
    "* `uri` ― server address (access this in a web browser with `https:` followed by `uri`)\n",
    "* `stats.user_count` ― number of users on that server\n",
    "* `stats.status_count` ― cumulative number of toots posted to that server \n",
    "* `stats.domain_count` ― number of other known Mastodon servers in the fediverse"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "print(f'{\"server title\":>19}: {instance.title}')\n",
    "print(f'{\"uri\":>19}: {instance.uri}')\n",
    "print(f'{\"short_description\":>19}: {instance.short_description}')\n",
    "print(f'{\"stats.user_count\":>19}: {instance.stats.user_count:,}')\n",
    "print(f'{\"stats.status_count\":>19}: {instance.stats.status_count:,}')\n",
    "print(f'{\"stats.domain_count\":>19}: {instance.stats.domain_count:,}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:#AAA; background-color:#AAA;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "# 12.9 Searching for Mastodon Accounts By User Name\n",
    "* A mobile app might allows user to locate other accounts to follow\n",
    "* Can programmatically search for accounts containing a specified string"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## **Example:** Find Account Names Containing `'Mastodon'`\n",
    "* Returns list of **Account**s\n",
    "* **Account** JSON description: https://docs.joinmastodon.org/entities/Account/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "accounts = mastodon.account_search(q='Mastodon@mastodon.social') "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "len(accounts)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## **Example:** Basic Account Information for Top 3 Accounts with `mastodon` in the name\n",
    "* You can discover info about an account\n",
    "    * Might want to follow and account based on popularity (number of followers)\n",
    "    * Might want to follow some of the same accounts that a specific account follows\n",
    "* Each has many properties, including:\n",
    "    * `username` — user’s Mastodon handle \n",
    "    * `id` — account’s unique ID number\n",
    "    * `url` — URL used to access the account in a web browser\n",
    "    * `note` — account description (may contain HTML tags)\n",
    "    * `statuses_count` — number of toots posted by the account\n",
    "    * `followers_count` — account's number of followers\n",
    "    * `following_count` — number of other accounts this account follows\n",
    "* Sort by `followers_count` in descending order, then display top 3 accounts by followers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "sorted_accounts = sorted(accounts, key=lambda acct: acct.followers_count, reverse=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "print('username: ', sorted_accounts[0].username)\n",
    "print('id: ', sorted_accounts[0].id)\n",
    "print('url: ', sorted_accounts[0].url)\n",
    "print(f'statuses_count: {sorted_accounts[0].statuses_count:,}')\n",
    "print(f'followers_count: {sorted_accounts[0].followers_count:,}')\n",
    "print(f'following_count: {sorted_accounts[0].following_count:,}')\n",
    "print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Getting Your Own Account’s Information\n",
    "* Get via `Mastodon` object’s `me` method\n",
    "> `my_account = mastodon.me()`\n",
    "* Returns an **Account object** for the account you used to authenticate with Mastodon"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:#AAA; background-color:#AAA;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 12.10 Spotting Trending Hashtags: Mastodon Trends API\n",
    "* If a topic **“goes viral,”** thousands or even millions of people could be talking about it\n",
    "* Mastodon allows you to look up **trending hashtags**, **trending toots** and  **trending links** across the fediverse\n",
    "    * NOTE: **trending toots** and **trending links** are supposed to return lists of items, but each returns only one item at the moment \n",
    "    * **I filed an issue in the Mastodon.py GitHub repository** — the developers acknowledged the bug and are looking into it "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## **Example:** Getting a List of Trending Hashtags \n",
    "* `trending_tags` returns a list of trending hashtags in the fediverse\n",
    "* Returned as JSON **Tag** objects\n",
    "> https://docs.joinmastodon.org/entities/Tag/\n",
    "* Each contains\n",
    "    * `name`\n",
    "    * `url`\n",
    "    * `history` list of last 7 days' stats for the hashtag\n",
    "    * `following` ― whether the logged in account is following the trending tag"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "trends = mastodon.trending_tags(limit=20)  # 20 max, 10 default"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "trends[0] # sample dictionary for one hashtag"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## **Example:** Display Trending Hashtags in Descending Order By Toot Volume over the Last Seven Days"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "def tag_count(tag):\n",
    "    \"\"\"Counts number of times a hashtag was used in last 7 days\"\"\"\n",
    "    total_uses = 0\n",
    "    \n",
    "    for day in tag.history:\n",
    "        total_uses += int(day['uses'])  \n",
    "    \n",
    "    # add attribute to tag object specifying 7-day hashtag count\n",
    "    tag.seven_day_count = total_uses \n",
    "    return total_uses"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Sort the trends in **descending** order by toot volume:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "trends.sort(key=tag_count, reverse=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Display names, counts and URLs of the **top 20 trending topics**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "for tag in trends:\n",
    "    print(f'{tag.name}: {tag.seven_day_count}')\n",
    "    print(f'   {tag.url}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:#AAA; background-color:#AAA;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "# 12.11 Searching for Toots Containing Specific Hashtags; Getting More than One Page of Results "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "* Mastodon API methods often **return collections of objects** \n",
    "    * Docs describe these as \"Arrays\" \n",
    "    * Mastodon.py returns **lists of Python dictionaries**\n",
    "* For example, `timeline_xxx` functions can return:\n",
    "    * `timeline_hashtag` — toots **containing a specific hashtag**\n",
    "    * `timeline_public` — toots from the **fediverse's public timeline** \n",
    "    * `timeline_local` — toots posted on the **local server**\n",
    "    * `timeline_home` — toots in a user’s **home timeline** (includes accounts followed by the user)\n",
    "* Each Mastodon API call returns a maximum number of items per call\n",
    "    * known as a **page of results**\n",
    "    * default is often 10 or 20\n",
    "    * max is typically 40 for toots and 80 for accounts\n",
    "* **Mastodon.py can handle paging details** with utility functions (as you'll see momentarily)\n",
    "> https://mastodonpy.readthedocs.io/en/stable/12_utilities.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Functions `print_toot` and `print_toots` from `tootutilities.py`\n",
    "```python\n",
    "# DeepL Translator object used to autodetect source language and return English\n",
    "translator = deepl.Translator(keys_mastodon.deepL_key)\n",
    "\n",
    "def print_toot(toot):\n",
    "    \"\"\"Prints one toot's username and content (translated, if not English).\"\"\"\n",
    "    \n",
    "    # display the username of the sender \n",
    "    print(f'{toot.account.username}:')\n",
    "\n",
    "    # display toot text\n",
    "    if toot.language and toot.language.startswith('en'):\n",
    "        # render HTML version of toot in an HTML div indented 0.5 inches\n",
    "        text = profanity.censor(toot.content)\n",
    "        display(HTML(f'<div style=\"padding-left: 0.5in;\">{text}</div>'))\n",
    "\n",
    "    # if the language is not English, display original & translated text \n",
    "    else:            \n",
    "        # render HTML version of toot in an HTML div indented 0.5 inches\n",
    "        print(f'ORIGINAL: ')\n",
    "        display(HTML(f'<div style=\"padding-left: 0.5in;\">{toot.content}</div>'))\n",
    "\n",
    "        # translate content\n",
    "        result = translator.translate_text(toot.content, target_lang='en-us')\n",
    "\n",
    "        # render HTML version of toot in an HTML div indented 0.5 inches\n",
    "        print(f'TRANSLATED: ')\n",
    "        text = profanity.censor(result.text)\n",
    "        display(HTML(f'<div style=\"padding-left: 0.5in;\">{text}</div>'))\n",
    "        \n",
    "def print_toots(toots):\n",
    "    \"\"\"For each toot in toots, call print_toot to display \n",
    "       the username of the sender and toot text.\"\"\"\n",
    "    for toot in toots:\n",
    "        print_toot(toot)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## **Example:** Paging Through Results\n",
    "* We purposely grab only 2 toots per page of results to show the mechanics of paging"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "total_toots = 10 # how many toots to get\n",
    "pages = 5 # of pages of toots to process\n",
    "toots_per_page = total_toots // pages # 2 per page for demo purposes; can do 40/page\n",
    "hashtag = 'football'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get First Page of Toots Containing a Given Hashtag \n",
    "* `timeline_hashtag` searches public timeline of past toots containing hashtag you specify"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "result = mastodon.timeline_hashtag(hashtag, limit=toots_per_page)\n",
    "saved_toots = result # saved_toots will eventually contain all the toots"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get Remaining Pages of Toots Containing a Hashtag \n",
    "* Mastodon.py utility function **`fetch_next`** gets the next page of results\n",
    "* **Argument is the previous page of results**, which includes the info need by Mastodon.py to get the next page of result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "for toots in range(pages - 1): # for each remaining page\n",
    "    # save previous page of results\n",
    "    previous_result = result \n",
    "    \n",
    "    # use Mastodon.py utility function fetch_next to get next page of results\n",
    "    result = mastodon.fetch_next(previous_result) \n",
    "\n",
    "    # if there are results add them to saved_toots; otherwise, terminate loop\n",
    "    if result:\n",
    "        saved_toots += result\n",
    "    else:\n",
    "        break # no more results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "len(saved_toots) # total toots acquired"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import tootutilities \n",
    "tootutilities.print_toots(saved_toots)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr style=\"height:2px; border:none; color:#AAA; background-color:#AAA;\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "# 12.12 Cleaning/Preprocessing Toots for Analysis\n",
    "* **Data cleaning** is one of data scientists' most common and important tasks \n",
    "* Depending on the text analyses you wish to perform, you may need to normalize text so NLP tools can \"understand\" it\n",
    "    * abbreviations, slang, incorrectly spelled words, inconsistent formatting, etc. can limit NLP tools' ability to understand and analyze text \n",
    "* Some NLP tasks (Lesson 11) for normalizing social media posts\n",
    "    * Converting all text to the same case\n",
    "    * Removing `#` from hashtags, `@`-mentions, duplicates, hashtags\n",
    "    * Removing excess whitespace, punctuation, **stop words**, URLs\n",
    "    * **Stemming** and **lemmatization**\n",
    "    * **Tokenization**\n",
    "    * Removing formatting, like HTML, which NLP tools might not understand"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### **BeautifulSoup4** Library\n",
    "* https://www.crummy.com/software/BeautifulSoup/\n",
    "> `pip install beautifulsoup4`\n",
    "* Most popular library for **parsing HTML and extracting content** from it\n",
    "* Commonly used to **data-mine content in web pages**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "### **tweet-preprocessor** Library \n",
    "* https://github.com/s/preprocessor\n",
    "* Library designed to clean tweets, but useful for posts in general\n",
    "* `pip install tweet-preprocessor`\n",
    "* Can automatically remove any combination of:\n",
    "\n",
    "| Option | Option constant |\n",
    "| :--- | :--- |\n",
    "| **`OPT.MENTION`** | @-Mentions (e.g., `@nasa`) |\n",
    "| **`OPT.EMOJI`** | Emoji |\n",
    "| **`OPT.HASHTAG`** | Hashtag (e.g., `#mars`) |\n",
    "| **`OPT.NUMBER`** | Number |\n",
    "| **`OPT.RESERVED`** | Twitter reserved Words (`RT` and `FAV`) |\n",
    "| **`OPT.SMILEY`** | Smiley |\n",
    "| **`OPT.URL`** | URL |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## **Example:** Cleaning a Toot Containing HTML and a URL"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "toot_text = '<p style=\"padding-left: 3em\">A sample fake toot with a URL https://nasa.gov</p>'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* **BeautifulSoup** library can be used to **parse HTML and extract content** from it"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from bs4 import BeautifulSoup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "soup = BeautifulSoup(toot_text, 'html.parser') "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "plain_text = soup.get_text() # remove all HTML/CSS tags and commands"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "plain_text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* The **tweet-preprocessor** library’s module name is **`preprocessor`**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import preprocessor as p"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "p.set_options(p.OPT.URL)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "p.clean(plain_text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "# 12.13 Mastodon Streaming API\n",
    "* Your app can receive various Mastodon streams as they occur in real-time\n",
    "    * `stream_hashtag` — toots containing specified hashtag (home timeline and notifications)\n",
    "    * `stream_user` — events related to the logged in user account\n",
    "    * `stream_public` — public fediverse event stream\n",
    "    * `stream_local` — local server event stream\n",
    "    * `stream_list` — events for the specified user, but resticted to accounts from a list "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Creating a Subclass of `StreamListener` \n",
    "* Mastodon **pushes** data to your listener\n",
    "* Streaming rate varies \n",
    "* Create a **subclass of Mastodon.py’s `StreamListener` class** to process the stream\n",
    "* Mastodon.py calls `StreamListener` methods as it receives events\n",
    "    * `on_update(self, status)` is called when when a toot arrives from the stream\n",
    "    * `StreamListener` defines other **`on_`** methods for other \"events\"\n",
    "    > https://mastodonpy.readthedocs.io/en/stable/10_streaming.html#streamlistener\n",
    "    * Override only the methods your app needs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Class `TootListener` (Located in tootlistener.py)\n",
    "\n",
    "```python\n",
    "# tootlistener.py\n",
    "\"\"\"StreamListener subclass that processes tweets as they arrive.\"\"\"\n",
    "from mastodon import StreamListener\n",
    "import tootutilities \n",
    "\n",
    "# Global variable toot_listener_stream will be set to the stream handle so  \n",
    "# we can close the stream after a specified number of toots are received\n",
    "toot_listener_stream = None \n",
    "\n",
    "class TootListener(StreamListener):\n",
    "    \"\"\"Handles incoming toot stream.\"\"\"\n",
    "\n",
    "    def __init__(self, limit=10):\n",
    "        \"\"\"Create instance variables for tracking number of tweets.\"\"\"\n",
    "        self.toot_count = 0\n",
    "        self.TOOT_LIMIT = limit\n",
    "\n",
    "    def on_update(self, status):\n",
    "        \"\"\"Called when your listener receives a toot (status).\"\"\"\n",
    "        tootutilities.print_toot(status)\n",
    "        print()\n",
    "        self.toot_count += 1 # track number of toots received\n",
    "            \n",
    "        # if TOOT_LIMIT is reached, close the stream\n",
    "        if self.toot_count == self.TOOT_LIMIT:\n",
    "            toot_listener_stream.close()\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## **Example:** Streaming the Mastodon Federated Timeline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "### Creating a `TootListener` \n",
    "* `StreamListener` subclass `TootListener` manages the connection to the Mastodon stream and receives and processes the toots"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import tootlistener "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "toot_listener = tootlistener.TootListener(limit=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "### Streaming All Public Events\n",
    "* Events that are not toots are ignored by our `StreamListener` subclass\n",
    "    * notifications (e.g., someone followed you or reblogged your post)\n",
    "    * a toot deleted\n",
    "    * someone direct messaged you\n",
    "    * a toot was edited\n",
    "    * the streaming connection terminated\n",
    "* `stream_public` starts the live stream\n",
    "    * `toot_listener` receives each event—for toots, displays toot text\n",
    "    * `run_async=True` ensures that `stream_public` **returns a stream handle** we can use to **close the stream**\n",
    "* **Asynchronous vs. Synchronous Streams**\n",
    "    * `run_async=True` (asynchronous) runs the stream in a separate thread and returns a stream handle for managing the stream\n",
    "    * `run_async=False` (synchronous) runs the stream forever unless an unanticipated failure occurs, such as an unhandled exception"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# store the stream handle \n",
    "tootlistener.toot_listener_stream = mastodon.stream_public(\n",
    "    toot_listener, run_async=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "# 12.14 **Example:** Sentiment Analysis \n",
    "* Political researchers might use during elections to understand how people feel about specific politicians and issues, and **how they're likely to vote**\n",
    "* Companies might use to see **what people are saying about their products and competitors’ products**\n",
    "* Class `SentimentListener` (in `sentimentlistener.py`) checks sentiment on toots "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true,
    "tags": []
   },
   "source": [
    "## Class `SentimentListener`\n",
    "```python\n",
    "# sentimentlisener.py\n",
    "\"\"\"Tallies the number of positive, neutral and negative toots.\"\"\"\n",
    "import keys_mastodon\n",
    "from better_profanity import profanity \n",
    "from mastodon import StreamListener\n",
    "from bs4 import BeautifulSoup\n",
    "import deepl\n",
    "import preprocessor as p \n",
    "from textblob import TextBlob\n",
    "\n",
    "# load censored words list\n",
    "profanity.load_censor_words()\n",
    "\n",
    "# stream will be set to the stream handle so we can close \n",
    "# the stream after a specified number of toots are received\n",
    "sentiment_listener_stream = None\n",
    "\n",
    "# translator to autodetect source language and return English\n",
    "translator = deepl.Translator(keys_mastodon.deepL_key)\n",
    "\n",
    "class SentimentListener(StreamListener):\n",
    "    \"\"\"Handles incoming Tweet stream.\"\"\"\n",
    "\n",
    "    def __init__(self, sentiment_dict, limit=10):\n",
    "        \"\"\"Configure the SentimentListener.\"\"\"\n",
    "        self.sentiment_dict = sentiment_dict\n",
    "        self.toot_count = 0\n",
    "        self.TOOT_LIMIT = limit\n",
    "        \n",
    "        # tweet-preprocessor remove @ mentions, emojis, hashtags, URLs\n",
    "        p.set_options(p.OPT.MENTION, p.OPT.EMOJI, p.OPT.HASHTAG, p.OPT.URL)\n",
    "\n",
    "    def on_update(self, status):\n",
    "        \"\"\"Called when Mastodon pushes a new toot to you.\"\"\"\n",
    "\n",
    "        # if the toot is not a retoot\n",
    "        if status.reblogs_count == 0:                 \n",
    "            # remove all HTML/CSS tags and commands\n",
    "            plain_text = BeautifulSoup(\n",
    "                status.content, 'html.parser').get_text()\n",
    "\n",
    "            # clean the toot\n",
    "            text = p.clean(plain_text) \n",
    "\n",
    "            # possibly translate plain_text\n",
    "            if status.language and not status.language.startswith('en'):\n",
    "                try:\n",
    "                    result = translator.translate_text(\n",
    "                        plain_text, target_lang='en-us')\n",
    "                    text = profanity.censor(result.text) # save translated text\n",
    "                except:\n",
    "                    text = 'toot was empty'   \n",
    "\n",
    "            # update self.sentiment_dict with the polarity\n",
    "            blob = TextBlob(text)\n",
    "            if blob.sentiment.polarity > 0.1:\n",
    "                sentiment = '+'\n",
    "                self.sentiment_dict['positive'] += 1 \n",
    "            elif blob.sentiment.polarity < -0.1:\n",
    "                sentiment = '-'\n",
    "                self.sentiment_dict['negative'] += 1 \n",
    "            else:\n",
    "                sentiment = ' '\n",
    "                self.sentiment_dict['neutral'] += 1 \n",
    "\n",
    "            # display the toot\n",
    "            print(f'{sentiment} {status.account.username}: {text}\\n')\n",
    "\n",
    "            self.toot_count += 1 # track number of toots processed\n",
    "\n",
    "        # if TOOT_LIMIT reached, terminate streaming \n",
    "        if self.toot_count == self.TOOT_LIMIT:\n",
    "            sentiment_listener_stream.close()\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Main Application"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "### Specify number of toots to tally"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "limit = 10"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "### Set up Dictionary to Track Toot Sentiment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "sentiment_dict = {'positive': 0, 'neutral': 0, 'negative': 0}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "### Create `StreamListener` Subclass Object"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import sentimentlistener \n",
    "sentiment_listener = sentimentlistener.SentimentListener(sentiment_dict, limit)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "### Start Stream and Store Its Handle"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "sentimentlistener.sentiment_listener_stream = mastodon.stream_public(\n",
    "    sentiment_listener, run_async=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "#### Display summary of results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "print(f'Toot sentiment:')\n",
    "print('Positive:', sentiment_dict['positive'])\n",
    "print(' Neutral:', sentiment_dict['neutral'])\n",
    "print('Negative:', sentiment_dict['negative'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "# 12.15 **Example:** Geocoding and Mapping\n",
    "* Collect streaming toots\n",
    "* Look up sever locations and plot toots at those loctions on an interactive map\n",
    "* **Mastodon is privacy focused**\n",
    "    * Only server admins have access to any location data\n",
    "* Even in Twitter, geo location is off by default, though many accounts specify home location \n",
    "    * Sometimes invalid or fictitious \n",
    "* Map markers will show the sender's `location` and toot text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "### **geopy** library\n",
    "* https://github.com/geopy/geopy\n",
    "* Installed in Section 5\n",
    "* **Geocoding** — translate locations into **latitude** and **longitude**\n",
    "* **geopy** supports dozens of **geocoding web services**, many with **free or lite tiers**\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### **folium library** and Leaflet.js JavaScript Mapping Library\n",
    "* https://github.com/python-visualization/folium\n",
    "* Setup in Section 5\n",
    "* For maps — uses **Leaflet.js JavaScript mapping library** to display maps in a web page \n",
    "* Folium can save HTML files that you can view in your web browser or add to a website"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Getting and Mapping the Toots\n",
    "* We’ll use utility functions from our **`tootutilities.py`** file and class **`LocationListener`** in **`locationlistener.py`**\n",
    "* Each is included after the example"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Collections Required By LocationListener\n",
    "* a list (`toots`) to store the data from the toots we collect \n",
    "* a dictionary (`counts`) to track the total number of toots we collect and the number that have location data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "toots = [] \n",
    "counts = {'total_toots': 0, 'locations': 0}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "### Creating the LocationListener \n",
    "* Collect 50 toots \n",
    "* `LocationListener` will use utility function `get_toot_content` (located in `tootutilities.py`; discussed after this example) to place in a dictionary the `username`, toot `text` and Mastodon server `location` from each toot"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import locationlistener \n",
    "\n",
    "location_listener = locationlistener.LocationListener(\n",
    "    counts_dict=counts, toots_list=toots, limit=50)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Start Stream and Store Its Handle\n",
    "* We display the toot count so far and usernames to show progress"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "locationlistener.location_listener_stream = mastodon.stream_public(\n",
    "    location_listener, run_async=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "<!--\n",
    "\n",
    "### Displaying the Location Statistics\n",
    "* of the toots we processed, check the percentage of servers for which we were able to find locations (should be 100%)\n",
    "\n",
    "counts['total_toots']\n",
    "\n",
    "counts['locations']\n",
    "\n",
    "print(f'{counts[\"locations\"] / counts[\"total_toots\"]:.1%}')\n",
    "-->"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Geocoding the Locations\n",
    "* Use `get_geocodes` utility function (from `tootutilities.py`; discussed after this example) to geocode the location of each toot stored in the list of toots"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from tootutilities import get_geocodes\n",
    "\n",
    "bad_locations = get_geocodes(toots)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bad_locations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "<!--\n",
    "\n",
    "* For each toot with a valid location, the `get_geocodes` function adds the new keys `'latitude'` and `'longitude'` to that toot’s dictionary in the `toots` list — these will be used to plot map markers on our interactive map\n",
    "\n",
    "### Displaying the Bad Location Statistics\n",
    "* If geopy is unable to geo-encode a specific location `bad_locations` will be greater than 0\n",
    "\n",
    "bad_locations \n",
    "\n",
    "print(f'{bad_locations / counts[\"locations\"]:.1%}')\n",
    "-->"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Cleaning the Data\n",
    "* Before we plot the toot locations on a map, let’s use a pandas `DataFrame` to clean the data\n",
    "* When you create a `DataFrame` from the `toots` list, it may contain `NaN` for `'latitude'` and `'longitude'` if geopy was unable to geoencode a specific location \n",
    "* `NaN` cannot be plotted on a map, so remove any rows containing `NaN` by calling the `DataFrame`’s `dropna` method"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "toots[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "df = pd.DataFrame(toots)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "df = df.dropna() # if there are rows with missing data drop them"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "source_hidden": true
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Creating a Map with Folium\n",
    "Create a folium Map on which we’ll plot the toot locations\n",
    "\n",
    "Note: We used **Stamen map tiles** in our book: \n",
    "* Stamen recently lost their funding, which they used to maintain their own servers. \n",
    "* The `folium` maintainers have decided not to support Stamen map tiles directly moving forward, but you can specify a custom map tile link with `folium`.\n",
    "* Stamen has begun a new partnership with stadiamaps.com, which requires an API key. \n",
    "* See https://stadiamaps.com/stamen/ for details on setting up a free account and getting an API key. \n",
    "* In `keys_mastodon.py`, provide your API key in the `stadia_key` variable. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import folium"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "#usmap = folium.Map(location=[39.8283, -98.5795], \n",
    "#    tiles='Stamen Terrain', zoom_start=4, detect_retina=True) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "base_tile_url = 'https://tiles.stadiamaps.com/tiles/stamen_terrain/{z}/{x}/{y}@2x.png'\n",
    "tile_url = f'{base_tile_url}?api_key=\"{keys_mastodon.stadia_key}\")'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "usmap = folium.Map(location=[39.8283, -98.5795], \n",
    "    tiles=tile_url,\n",
    "    attr='Map tiles by Stamen Design, under CC BY 4.0. Data by OpenStreetMap, under ODbL.',\n",
    "    zoom_start=4, detect_retina=True)  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* `location` keyword argument specifies a sequence containing latitude and longitude coordinates for the **map’s center point** \n",
    "    * The values in this snippet are the **geographic center of the continental United States**\n",
    "    * In many places worldwide, the term `'football'` describes the sport we call soccer in the U.S., so some of the toots we plot may be outside the U.S\n",
    "    * You can zoom using the **+** and **–** buttons at the map’s top-left, or you can dragging the map with the mouse (that is, pan) to see anywhere in the world\n",
    "*  `zoom_start` keyword argument specifies the map’s initial zoom level, lower values show more of the world\n",
    "* `detect_retina` keyword argument enables folium to detect high-resolution screens to use higher-resolution maps from `OpenStreetMap.org`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Creating Popup Markers for the Toot Locations\n",
    "* Create `folium` `Popup` objects containing each toot’s text and add them to the `Map`\n",
    "* `DataFrame` method `itertuples` creates a named tuple from each row containing properties corresponding to each `DataFrame` column"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "for t in df.itertuples():\n",
    "    text = ''.join(['<p>' + t.username + '</p>', t.text if t.text else ''])\n",
    "    popup = folium.Popup(text)\n",
    "    marker = folium.Marker((t.latitude, t.longitude), \n",
    "                           popup=popup)\n",
    "    marker.add_to(usmap)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Creates a string (`text`) containing the user’s `username` and toot `text` \n",
    "* Creates a `folium` `Popup` to display the `text`\n",
    "* Creates a `folium` `Marker`\n",
    "    * tuple to specify the `Marker`’s latitude and longitude\n",
    "    * `popup` keyword argument associates the toot’s `Popup` object with the new `Marker`\n",
    "* Calls the `Marker`’s `add_to` method to specify the `Map` that will display the `Marker`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Saving the Map\n",
    "* Call the `Map`’s `save` method to store the map in an HTML file, which you can then double-click to open in your web browser"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "usmap.save('toot_map.html')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "usmap # displays the map in the notebook"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Class `LocationListener`\n",
    "\n",
    "```python\n",
    "# locationlistener.py\n",
    "\"\"\"Receives toots and stores a list of dictionaries containing \n",
    "each toot's username/content, server domain and server location.\"\"\"\n",
    "import keys_mastodon\n",
    "import tootutilities \n",
    "from mastodon import StreamListener\n",
    "from bs4 import BeautifulSoup\n",
    "import deepl\n",
    "import preprocessor as p \n",
    "from textblob import TextBlob\n",
    "\n",
    "# stream will be set to the stream handle so we can close \n",
    "# the stream after a specified number of toots are received\n",
    "location_listener_stream = None\n",
    "\n",
    "class LocationListener(StreamListener):\n",
    "    \"\"\"Handles incoming Toot stream to get location data.\"\"\"\n",
    "\n",
    "    def __init__(self, counts_dict, toots_list, limit=10):\n",
    "        \"\"\"Configure the LocationListener.\"\"\"\n",
    "        self.toots_list = toots_list\n",
    "        self.counts_dict = counts_dict\n",
    "        self.TOOT_LIMIT = limit\n",
    "\n",
    "    def on_update(self, status):\n",
    "        \"\"\"Called when Mastodon pushes a new toot to you.\"\"\"\n",
    "\n",
    "        # get toot's username, text and location\n",
    "        toot_data = tootutilities.get_toot_content(status)  \n",
    "        self.counts_dict['total_toots'] += 1 # it's an original toot\n",
    "\n",
    "        # ignore toots with no server location--can't plot on a map\n",
    "        if not toot_data.get('location'):  \n",
    "            return\n",
    "\n",
    "        self.counts_dict['locations'] += 1 \n",
    "        self.toots_list.append(toot_data) # store the toot\n",
    "        \n",
    "        print(f\"{self.counts_dict['locations']:2}: {toot_data['username']}\")\n",
    "\n",
    "        # if TOOT_LIMIT is reached, terminate streaming\n",
    "        if self.counts_dict['locations'] == self.TOOT_LIMIT:\n",
    "            location_listener_stream.close()\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Utility Functions in `tootutilities.py` "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Utility Function `get_toot_content`  \n",
    "* Receives a toot and creates a **dictionary** containing the **toot’s `username`, `text` and server `location`**\n",
    "\n",
    "```python\n",
    "def get_toot_content(toot):\n",
    "    \"\"\"Return dictionary with username of toot sender, toot content\n",
    "       translated to English (if necessary) and the sender account's \n",
    "       Mastodon server domain.\"\"\"\n",
    "\n",
    "    fields = {}\n",
    "    fields['username'] = toot.account.username\n",
    "\n",
    "    # possibly translate plain_text\n",
    "    if toot.language and not toot.language.startswith('en'):\n",
    "        try:\n",
    "            result = translator.translate_text(\n",
    "                toot.content, target_lang='en-us')\n",
    "            \n",
    "            # save translated text\n",
    "            fields['text'] = profanity.censor(result.text)\n",
    "        except:\n",
    "            fields['text'] = None\n",
    "    else:\n",
    "        fields['text'] = profanity.censor(toot.content)\n",
    "        \n",
    "    # look up server location\n",
    "    fields['location'] = get_domain_location_from_url(toot.url)\n",
    "\n",
    "    return fields\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Utility Function `get_domain_location_from_url`  \n",
    "* Receives the URL of a toot, extracts the domain name, looks up the domain's IP address, then uses the free tier of the ipgeolocation.io API to lookup the geographic location of that IP address\n",
    "*  Get a key at for ipgeolocation.io web service at: https://ipgeolocation.io/signup.html\n",
    "    * store it in `ipgeolocation_key` within in c`keys_mastodon.py`\n",
    "\n",
    "```python\n",
    "def get_domain_location_from_url(url):\n",
    "    \"\"\"Parse the domain from url, then look up the IP address \n",
    "       and get its city.\n",
    "       \n",
    "       Get a key at for ipgeolocation.io web service at: \n",
    "       https://ipgeolocation.io/signup.html\n",
    "    \"\"\"\n",
    "    \n",
    "    # get domain from url argument using Python \n",
    "    # urllib.parse module's urlparse function\n",
    "    domain = urlparse(url).netloc\n",
    "\n",
    "    # get IP address for hostname using Python \n",
    "    # socket module's gethostbyname function\n",
    "    ip_address = socket.gethostbyname(domain)\n",
    "    \n",
    "    # URL to access ipgeolocation.io web services\n",
    "    ipgeolocation_url = ('https://api.ipgeolocation.io/ipgeo?apiKey=' + \n",
    "       keys_mastodon.ipgeolocation_key + f'&ip={ip_address}')\n",
    "\n",
    "    # use Python requests module to invoke web service\n",
    "    response = requests.get(ipgeolocation_url)\n",
    "\n",
    "    # if successful get the location\n",
    "    if response.status_code == 200:\n",
    "        data = response.json() # convert JSON to dictionary\n",
    "        location = (f\"{data['city']}, {data['state_prov']}, \" +\n",
    "            f\"{data['country_code2']}\")\n",
    "        return location\n",
    "    else:\n",
    "        print('Error getting location\\n',\n",
    "              f'{response.status_code}: {response.text}')\n",
    "        return None\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `get_geocodes` Utility Function \n",
    "* Receives a list of dictionaries containing toots and **geocodes their server locations**\n",
    "* If geocoding is successful for a toot, adds the **latitude** and **longitude** to the toot’s **dictionary in `toot_list`**\n",
    "\n",
    "```python\n",
    "def get_geocodes(toot_list):\n",
    "    \"\"\"Get the latitude and longitude for each tweet's location.\n",
    "    Returns the number of tweets with invalid location data.\"\"\"\n",
    "    \n",
    "    print('Getting coordinates for mastodon server locations...')\n",
    "    geo = ArcGIS()  # geocoder\n",
    "    bad_locations = 0  \n",
    "\n",
    "    for toot in toot_list:\n",
    "        processed = False\n",
    "        delay = .1  # used if OpenMapQuest times out to delay next call\n",
    "        while not processed:\n",
    "            try:  # get coordinates for tweet['location']\n",
    "                geo_location = geo.geocode(toot['location'])\n",
    "                processed = True\n",
    "            except:  # timed out, so wait before trying again\n",
    "                print('Service timed out. Waiting.')\n",
    "                time.sleep(delay)\n",
    "                delay += .1\n",
    "\n",
    "        if geo_location:  \n",
    "            toot['latitude'] = geo_location.latitude\n",
    "            toot['longitude'] = geo_location.longitude\n",
    "        else:  \n",
    "            bad_locations += 1  # toot['location'] was invalid\n",
    "    \n",
    "    print('Done geocoding')\n",
    "    return bad_locations\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "<!--\n",
    "# More Info \n",
    "* See Lesson 12 in [**Python Fundamentals LiveLessons** here on O'Reilly Online Learning](https://learning.oreilly.com/videos/python-fundamentals/9780135917411)\n",
    "* See Chapter 12 in [**Python for Programmers** on O'Reilly Online Learning](https://learning.oreilly.com/library/view/python-for-programmers/9780135231364/)\n",
    "* See Chapter 13 in [**Intro Python for Computer Science and Data Science** on O'Reilly Online Learning](https://learning.oreilly.com/library/view/intro-to-python/9780135404799/)\n",
    "* Interested in a print book? Check out:\n",
    "\n",
    "| Python for Programmers<br>(640-page professional book) | Intro to Python for Computer<br>Science and Data Science<br>(880-page college textbook)\n",
    "| :------ | :------\n",
    "| <a href=\"https://amzn.to/2VvdnxE\"><img alt=\"Python for Programmers cover\" src=\"../images/PyFPCover.png\" width=\"150\" border=\"1\"/></a> | <a href=\"https://amzn.to/2LiDCmt\"><img alt=\"Intro to Python for Computer Science and Data Science: Learning to Program with AI, Big Data and the Cloud\" src=\"../images/IntroToPythonCover.png\" width=\"159\" border=\"1\"></a>\n",
    "\n",
    ">Please **do not** purchase both books&mdash;_Python for Programmers_ is a subset of _Intro to Python for Computer Science and Data Science_\n",
    "-->"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "------\n",
    "&copy;1992&ndash;2024 by Pearson Education, Inc. All Rights Reserved. This content is based on Chapter 12 of the book [**Intro to Python for Computer Science and Data Science: Learning to Program with AI, Big Data and the Cloud**](https://amzn.to/2VvdnxE).\n",
    "\n",
    "DISCLAIMER: The authors and publisher of this book have used their \n",
    "best efforts in preparing the book. These efforts include the \n",
    "development, research, and testing of the theories and programs \n",
    "to determine their effectiveness. The authors and publisher make \n",
    "no warranty of any kind, expressed or implied, with regard to these \n",
    "programs or to the documentation contained in these books. The authors \n",
    "and publisher shall not be liable in any event for incidental or \n",
    "consequential damages in connection with, or arising out of, the \n",
    "furnishing, performance, or use of these programs.                  "
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}