days/73-75-selenium/python-selenium.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using Python Selenium to Automate Tasks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### First day: Selenium by Example"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Selenium is a great tool to write functional/acceptance tests and automation scripts that require interaction with a webpage. \n",
    "\n",
    "In this lesson we get Selenium running and look at two use cases. Then we have you code 1 or 2 scripts using Selenium."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To follow along you need have Selenium installed and a webdriver:\n",
    "\n",
    "1. `pip install selenium` (if you installed the requirements.txt in my setup video in the appendix you should already have it)\n",
    "2. I used _PhantomJS_ before but now I got this error: `Selenium support for PhantomJS has been deprecated, please use headless versions of Chrome or Firefox instead`. So I downloaded the [ChromeDriver](https://sites.google.com/a/chromium.org/chromedriver/home) and put it in my `PATH` (`$HOME/bin`)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is the _Hello Selenium world_ example from [the docs](http://selenium-python.readthedocs.io/getting-started.html). Notice how easy it is to interact with forms:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from selenium import webdriver\n",
    "from selenium.webdriver.common.keys import Keys\n",
    "\n",
    "driver = webdriver.Chrome()  # replaced Firefox by Chrome\n",
    "driver.get(\"http://www.python.org\")\n",
    "assert \"Python\" in driver.title\n",
    "elem = driver.find_element_by_name(\"q\")\n",
    "elem.clear()\n",
    "elem.send_keys(\"pycon\")\n",
    "elem.send_keys(Keys.RETURN)\n",
    "assert \"No results found.\" not in driver.page_source\n",
    "driver.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Example 1. Packt ebook manager"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Packt gives away [a free ebook each day](https://www.packtpub.com/packt/offers/free-learning). I've been grabbing quite a few of them (back in our 100 Days we [wrote a notification script](https://github.com/pybites/100DaysOfCode/blob/master/076/packt_notification.py)). \n",
    "\n",
    "In this section I will make a simple Selenium script to search my collection for a title and provide me the download link. Of course my books are behind login, so I use Selenium to login first. This script/idea also came out of our own 100 days of code, see [day 66](https://github.com/pybites/100DaysOfCode/blob/master/066/packt.py)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How would this work? Quite simply:\n",
    "\n",
    "1. Go to the Packt login URL and login:\n",
    "    ![logged out](images/packt1.png)\n",
    "2. You get to your Account page:\n",
    "    ![logged in](images/packt2.png)\n",
    "3. Go the _My eBooks_ links and parse the html:\n",
    "    ![parse ebook html](images/packt3.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's get coding!\n",
    "\n",
    "First of all as per [The twelve-factor app](https://12factor.net/config) I store config details in the environment, __never__ in the script. \n",
    "\n",
    "`os.environ.get` lets you retrieve environment variables defaulting to `None` if not found. The check for `None` I consider a more Pythonic pattern and to increase the readability I added my own exception: `NoLogin`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "user = os.environ.get('PACKT_USER')\n",
    "pw = os.environ.get('PACKT_PW')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As shown in the example before we create a `driver` object and go to the `login` URL. We find the `edit-name` and `edit-pass` form elements and send the user and password (stored in the `user` and `pw` variables respectively). Appending the `Keys.RETURN` submits the form (html forms can be submitted by hitting Enter on an input field, not so on a textarea box, there it would enter a newline)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "login = 'https://www.packtpub.com/login'\n",
    "\n",
    "driver = webdriver.Chrome()\n",
    "driver.get(login)\n",
    "\n",
    "driver.find_element_by_id('edit-name').send_keys(user)\n",
    "driver.find_element_by_id('edit-pass').send_keys(pw + Keys.RETURN)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that at this point a Chrome browser window opened in the background. It will close when we close the driver later:\n",
    "\n",
    "![selenium web browser](images/packt4.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Als note that here is a natural delay between steps because we are using a notebook. In a script though steps run one after the other at lightning speed. \n",
    "\n",
    "Also in this case there is no pagination, so my 100+ books take some time to load. So if you use this as a script you might want to add: `driver.implicitly_wait(3)`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's get to the actual content, storing it in `elements`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "driver.find_element_by_link_text('My eBooks').click()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next I use a _dictionary comprehension_ to store the book id (`nid`) as keys and the book titles as values. You could sync this to a local file or sqlite DB if you want to cache these results. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'10068': 'Learning Ext JS 4 [eBook]',\n",
       " '10264': 'Implementing Splunk: Big Data Reporting and Development for Operational Intelligence [eBook]',\n",
       " '10744': 'Nagios Core Administration Cookbook [eBook]',\n",
       " '10763': 'Continuous Delivery and DevOps: A Quickstart guide [eBook]',\n",
       " '11441': 'Learning SciPy for Numerical and Scientific Computing [eBook]',\n",
       " '11703': 'Building Machine Learning Systems with Python [eBook]',\n",
       " '11723': 'Learning jQuery - Fourth Edition [eBook]',\n",
       " '11913': 'Mastering Web Application Development with AngularJS [eBook]',\n",
       " '12001': 'OpenCV Computer Vision with Python [eBook]',\n",
       " '12050': \"Magento : Beginner's Guide - Second Edition [eBook]\",\n",
       " '12318': 'Python Geospatial Development - Second Edition [eBook]',\n",
       " '12364': 'Object-Oriented JavaScript - Second Edition [eBook]',\n",
       " '12730': 'Learning Vaadin 7: Second Edition [eBook]',\n",
       " '12883': '3D Printing Blueprints [eBook]',\n",
       " '13253': 'Boost C++ Application Development Cookbook [eBook]',\n",
       " '13532': 'Python Data Visualization Cookbook [eBook]',\n",
       " '13804': 'Mastering ElasticSearch [eBook]',\n",
       " '13964': 'Mastering Apache Cassandra [eBook]',\n",
       " '14030': 'Network Analysis using Wireshark Cookbook [eBook]',\n",
       " '14165': 'Node Security [eBook]',\n",
       " '14254': 'Kali Linux Cookbook [eBook]',\n",
       " '14519': 'PySide GUI Application Development [eBook]',\n",
       " '14770': 'Raspberry Pi Cookbook for Python Programmers [eBook]',\n",
       " '15070': 'Kali Linux Social Engineering [eBook]',\n",
       " '15280': 'Learning Python Design Patterns [eBook]',\n",
       " '15363': 'Unity Multiplayer Games [eBook]',\n",
       " '15550': 'Blender 3D Printing Essentials [eBook]',\n",
       " '15597': 'Advanced Express Web Application Development [eBook]',\n",
       " '15642': 'Kali Linux – Assuring Security by Penetration Testing [eBook]',\n",
       " '16230': 'Applied Architecture Patterns on the Microsoft Platform (Second Edition) [eBook]',\n",
       " '16271': 'Linux Mint Essentials [eBook]',\n",
       " '16279': 'MongoDB Cookbook [eBook]',\n",
       " '16352': 'Backbone.js Patterns and Best Practices [eBook]',\n",
       " '16616': 'Social Media Mining with R [eBook]',\n",
       " '16625': 'Mastering Python Regular Expressions [eBook]',\n",
       " '16637': 'IPython Interactive Computing and Visualization Cookbook [eBook]',\n",
       " '16735': 'Mastering D3.js [eBook]',\n",
       " '16872': 'Node.js Blueprints [eBook]',\n",
       " '16904': 'Learning Web Development with Bootstrap and AngularJS [eBook]',\n",
       " '17007': 'Getting Started with LLVM Core Libraries [eBook]',\n",
       " '17011': 'Mastering Object-oriented Python [eBook]',\n",
       " '17102': 'Unity 4 Game Development Hotshot [eBook]',\n",
       " '17276': 'Mastering AngularJS Directives [eBook]',\n",
       " '17371': 'Penetration Testing with the Bash shell [eBook]',\n",
       " '17384': 'Natural Language Processing with Java and LingPipe Cookbook [eBook]',\n",
       " '17399': 'Machine Learning with Spark [eBook]',\n",
       " '17421': 'OpenCV for Secret Agents [eBook]',\n",
       " '17436': 'Learning Raspberry Pi [eBook]',\n",
       " '17477': 'MEAN Web Development [eBook]',\n",
       " '17490': 'Python 3 Text Processing with NLTK 3 Cookbook [eBook]',\n",
       " '17514': 'HBase Design Patterns [eBook]',\n",
       " '17645': 'Practical Data Science Cookbook [eBook]',\n",
       " '17682': 'Building Business Websites with Squarespace 7 [eBook]',\n",
       " '17722': 'Creative Greenfoot [eBook]',\n",
       " '17805': 'Mastering Machine Learning with scikit-learn [eBook]',\n",
       " '17814': 'AngularJS UI Development [eBook]',\n",
       " '17954': 'Python Machine Learning [eBook]',\n",
       " '17961': 'Web Development with Django Cookbook [eBook]',\n",
       " '17973': 'Python for Secret Agents [eBook]',\n",
       " '18044': 'Testing and Securing Android Studio Applications [eBook]',\n",
       " '18052': 'Mastering Python Design Patterns [eBook]',\n",
       " '18071': 'HBase Essentials [eBook]',\n",
       " '18127': 'Node.js Design Patterns [eBook]',\n",
       " '18678': 'Mockito Essentials [eBook]',\n",
       " '18879': 'Wireshark Essentials [eBook]',\n",
       " '18895': 'R Graphs Cookbook Second Edition [eBook]',\n",
       " '18942': 'scikit-learn Cookbook [eBook]',\n",
       " '18951': 'Learning JavaScript Data Structures and Algorithms [eBook]',\n",
       " '19168': 'Git Best Practices Guide [eBook]',\n",
       " '19294': 'Learning R for Geospatial Analysis  [eBook]',\n",
       " '19313': 'StartupPro: How to set up and grow a tech business [eBook]',\n",
       " '19387': 'Learning NGUI for Unity [eBook]',\n",
       " '19444': 'AngularJS Web Application Development Cookbook [eBook]',\n",
       " '19462': 'Go Programming Blueprints [eBook]',\n",
       " '19490': 'Learning Java by Building Android Games [eBook]',\n",
       " '19564': 'Mastering R for Quantitative Finance [eBook]',\n",
       " '19569': 'Raspberry Pi for Secret Agents - Second Edition [eBook]',\n",
       " '19599': 'Learning iOS 8 for Enterprise [eBook]',\n",
       " '19799': 'Learning C++ by Creating Games with UE4 [eBook]',\n",
       " '19853': 'Learning pandas [eBook]',\n",
       " '19884': 'Learning Docker [eBook]',\n",
       " '19932': \"Kali Linux Wireless Penetration Testing: Beginner's Guide [eBook]\",\n",
       " '1995': 'Drupal 6 Search Engine Optimization [eBook]',\n",
       " '20002': 'Learning Scrapy [eBook]',\n",
       " '20005': 'Arduino Development Cookbook [eBook]',\n",
       " '20047': 'Docker Cookbook [eBook]',\n",
       " '20117': 'Learning Robotics Using Python [eBook]',\n",
       " '20125': 'Functional Python Programming [eBook]',\n",
       " '20217': 'R Data Visualization Cookbook [eBook]',\n",
       " '20264': 'Python 3 Object-oriented Programming - Second Edition [eBook]',\n",
       " '20300': 'RESTful Web API Design with Node.js [eBook]',\n",
       " '20377': 'Mastering Embedded Linux Programming [eBook]',\n",
       " '20389': 'WordPress 4.x Complete [eBook]',\n",
       " '20615': 'Mastering Python for Finance [eBook]',\n",
       " '20820': 'Learning Python Network Programming [eBook]',\n",
       " '20825': 'Thinking in JavaScript [eBook]',\n",
       " '20826': 'Thinking in CSS [eBook]',\n",
       " '20892': 'Mastering Windows PowerShell Scripting [eBook]',\n",
       " '20893': 'Django By Example [eBook]',\n",
       " '20927': 'Learning .NET High-performance Programming [eBook]',\n",
       " '20987': 'Practical Machine Learning [eBook]',\n",
       " '2101': 'ASP.NET Data Presentation Controls Essentials [eBook]',\n",
       " '21051': 'Learning Reactive Programming with Java 8 [eBook]',\n",
       " '21104': 'Mastering matplotlib [eBook]',\n",
       " '21105': 'DevOps Automation Cookbook [eBook]',\n",
       " '21201': 'Learning Data Mining with Python [eBook]',\n",
       " '21307': 'Rust Essentials [eBook]',\n",
       " '21310': 'Clean Data [eBook]',\n",
       " '21392': 'Learning Flask Framework [eBook]',\n",
       " '21460': 'Unity 5.x Cookbook [eBook]',\n",
       " '21472': 'RESTful Java Web Services - Second Edition [eBook]',\n",
       " '21483': 'Blueprints Visual Scripting for Unreal Engine [eBook]',\n",
       " '21558': 'React.js Essentials [eBook]',\n",
       " '21583': 'Android Game Programming by Example [eBook]',\n",
       " '21612': 'Test-Driven Java Development [eBook]',\n",
       " '2166': 'MediaWiki Administrators’ Tutorial Guide [ebook]',\n",
       " '21834': 'SFML Game Development By Example [eBook]',\n",
       " '21931': 'Learning Object-Oriented Programming [eBook]',\n",
       " '2196': 'JasperReports 3.5 for Java Developers [eBook]',\n",
       " '21976': 'Mastering Git [eBook]',\n",
       " '21985': 'Python Game Programming By Example [eBook]',\n",
       " '21989': 'Machine Learning with R - Second Edition [eBook]',\n",
       " '22006': 'Responsive Web Design with HTML5 and CSS3 - Second Edition [eBook]',\n",
       " '22025': 'Getting Started with PowerShell [eBook]',\n",
       " '22031': 'Learning Python [eBook]',\n",
       " '22041': 'Learning PostgreSQL [eBook]',\n",
       " '22148': 'Python Parallel Programming Cookbook [eBook]',\n",
       " '2219': 'Expert Python Programming [eBook]',\n",
       " '22261': 'Learning AWS [eBook]',\n",
       " '22273': 'Mastering Responsive Web Design [eBook]',\n",
       " '22314': 'Learning ECMAScript 6 [eBook]',\n",
       " '22318': 'Building a Recommendation System with R [eBook]',\n",
       " '22369': 'Learning Penetration Testing with Python [eBook]',\n",
       " '22410': 'Cross-platform UI Development with Xamarin.Forms [eBook]',\n",
       " '2248': 'Object-Oriented Programming with PHP5 [eBook]',\n",
       " '22511': 'Unity AI Game Programming - Second Edition [eBook]',\n",
       " '2262': 'Service Oriented Architecture with Java [eBook]',\n",
       " '22653': 'Learning Bayesian Models with R [eBook]',\n",
       " '22681': 'Unity 5 Game Optimization [eBook]',\n",
       " '22791': 'D3.js By Example [eBook]',\n",
       " '22827': 'Learning TypeScript [eBook]',\n",
       " '22857': 'Python GUI Programming Cookbook [eBook]',\n",
       " '22889': 'Blender 3D by Example [eBook]',\n",
       " '22902': 'Procedural Content Generation for C++ Game Development [eBook]',\n",
       " '23013': 'GitHub Essentials [eBook]',\n",
       " '23045': 'Mastering Social Media Mining with R [eBook]',\n",
       " '23073': \"Magento 2 Developer's Guide [eBook]\",\n",
       " '23132': 'Neural Network Programming with Java [eBook]',\n",
       " '23153': 'Practical DevOps [eBook]',\n",
       " '23352': 'OpenCV By Example [eBook]',\n",
       " '23375': 'Web Development with Django Cookbook - Second Edition [eBook]',\n",
       " '23391': 'Learning Predictive Analytics with Python [eBook]',\n",
       " '23395': 'Python Unlocked [eBook]',\n",
       " '23411': 'Mastering Python [eBook]',\n",
       " '23427': 'Mastering JavaScript [eBook]',\n",
       " '23446': 'Raspberry Pi By Example [eBook]',\n",
       " '23460': 'Python Geospatial Development - Third Edition [eBook]',\n",
       " '23463': 'Bitcoin Essentials [eBook]',\n",
       " '23519': 'Bayesian Analysis with Python [eBook]',\n",
       " '23562': 'Mastering C# Concurrency [eBook]',\n",
       " '23679': 'AWS Administration – The Definitive Guide [eBook]',\n",
       " '23687': 'Mastering React [eBook]',\n",
       " '23694': 'Unity 5.x Shaders and Effects Cookbook [eBook]',\n",
       " '23780': 'Bootstrap Site Blueprints Volume II [eBook]',\n",
       " '23802': 'MongoDB Cookbook - Second Edition [eBook]',\n",
       " '23827': 'Learning Linux Binary Analysis [eBook]',\n",
       " '23846': 'Mastering Docker [eBook]',\n",
       " '23851': 'Android Programming for Beginners [eBook]',\n",
       " '23922': 'Building a Game with Unity and Blender [eBook]',\n",
       " '23924': 'Tkinter GUI Application Development Blueprints [eBook]',\n",
       " '23971': 'Learning Python Design Patterns - Second Edition [eBook]',\n",
       " '24015': 'Java Deep Learning Essentials [eBook]',\n",
       " '24068': 'Learning Python for Forensics [eBook]',\n",
       " '24109': 'Mastering Linux Shell Scripting [eBook]',\n",
       " '24161': 'Data Analysis with R [eBook]',\n",
       " '24179': 'Learning C# by Developing Games with Unity 5.x - Second Edition [eBook]',\n",
       " '24198': 'Flask By Example [eBook]',\n",
       " '24221': 'Python Machine Learning Blueprints: Intuitive data projects you can relate to [eBook]',\n",
       " '24258': 'Switching to Angular 2 [eBook]',\n",
       " '24294': 'Learning Angular 2 [eBook]',\n",
       " '24307': 'Advanced Penetration Testing for Highly-Secured Environments - Second Edition [eBook]',\n",
       " '24349': 'Principles of Data Science [eBook]',\n",
       " '24462': 'Learning PHP 7 [eBook]',\n",
       " '24475': 'Python Projects for Kids [eBook]',\n",
       " '24642': 'Multithreading with C# Cookbook - Second Edition [eBook]',\n",
       " '24661': 'Mastering Social Media Mining with Python [eBook]',\n",
       " '24680': 'Getting Started with Raspberry Pi Zero [eBook]',\n",
       " '24757': 'ReactJS Blueprints [eBook]',\n",
       " '24789': 'TypeScript Design Patterns [eBook]',\n",
       " '24807': 'R Deep Learning Essentials [eBook]',\n",
       " '24826': 'Machine Learning for the Web [eBook]',\n",
       " '24835': 'Modular Programming with Python [eBook]',\n",
       " '24907': 'C# 6 and .NET Core 1.0: Modern Cross-Platform Development [eBook]',\n",
       " '24949': 'Practical Linux Security Cookbook [eBook]',\n",
       " '25015': 'Node.js Web Development - Third Edition [eBook]',\n",
       " '25031': 'Practical Digital Forensics [eBook]',\n",
       " '25052': 'Mastering Wireshark [eBook]',\n",
       " '25098': 'Node.js Design Patterns - Second Edition [eBook]',\n",
       " '25143': 'Modern Python Cookbook [eBook]',\n",
       " '25232': 'Kubernetes Cookbook [eBook]',\n",
       " '25257': 'Expert Python Programming - Second Edition [eBook]',\n",
       " '25293': 'Professional CSS3 [eBook]',\n",
       " '25298': 'Practical Internet of Things Security [eBook]',\n",
       " '25319': 'MEAN Web Development - Second Edition [eBook]',\n",
       " '25366': 'Beginning C++ Game Programming [eBook]',\n",
       " '25608': 'Learning Python Application Development [eBook]',\n",
       " '25854': 'What You Need to Know about Docker [eBook]',\n",
       " '25976': 'Raspberry Pi for Secret Agents - Third Edition [eBook]',\n",
       " '26062': 'Learning Web Development with React and Bootstrap [eBook]',\n",
       " '26197': 'Automate it! - Recipes to upskill your business [eBook]',\n",
       " '26256': 'Artificial Intelligence with Python [eBook]',\n",
       " '26263': ' Skill Up 2016: Developer Skills Report [eBook]',\n",
       " '26288': 'What you need to know about Angular 2 [eBook]',\n",
       " '26325': 'Practical Data Analysis - Second Edition [eBook]',\n",
       " '26564': 'Learning PySpark [eBook]',\n",
       " '26678': 'PostgreSQL Development Essentials [eBook]',\n",
       " '26759': 'What You Need to Know about Python [eBook]',\n",
       " '26781': 'Building RESTful Python Web Services [eBook]',\n",
       " '27102': 'React Native Cookbook [eBook]',\n",
       " '27146': 'Python Data Science Essentials - Second Edition [eBook]',\n",
       " '27455': 'Scientific Computing with Python 3 [eBook]',\n",
       " '27683': 'What You Need to Know about Machine Learning [eBook]',\n",
       " '30182': 'Skill Up 2017 [eBook]',\n",
       " '4190': 'Matplotlib for Python Developers [eBook]',\n",
       " '4569': \"Python Testing: Beginner's Guide [eBook]\",\n",
       " '5188': 'Agile Web Application Development with Yii1.1 and PHP5 [eBook]',\n",
       " '5286': 'MySQL for Python [eBook]',\n",
       " '5317': 'Python 3 Object Oriented Programming [eBook]',\n",
       " '5568': 'PHP 5 Social Networking [eBook]',\n",
       " '5736': 'Java EE 6 with GlassFish 3 Application Server[eBook]',\n",
       " '5905': \"JavaScript Testing Beginner's Guide  [eBook]\",\n",
       " '6066': 'NHibernate 3.0 Cookbook [eBook]',\n",
       " '6088': 'PostgreSQL 9 Admin Cookbook [eBook]',\n",
       " '6195': 'Drupal 7 Module Development [eBook]',\n",
       " '6477': 'Statistical Analysis with R [eBook]',\n",
       " '6539': 'Using CiviCRM [eBook]',\n",
       " '6582': 'Blender 2.5 Materials and Textures Cookbook [eBook]',\n",
       " '6687': 'Linux Shell Scripting Cookbook [eBook]',\n",
       " '6972': \"Android User Interface Development: Beginner's Guide [eBook]\",\n",
       " '7193': \"LaTex Beginner's Guide [eBook]\",\n",
       " '7203': \"Python 3 Web Development Beginner's Guide [eBook]\",\n",
       " '7333': 'Java EE 6 Development with NetBeans 7 [eBook]',\n",
       " '7690': 'OpenCV 2 Computer Vision Application Programming Cookbook [eBook]',\n",
       " '8294': 'Learning jQuery, Third Edition [eBook]',\n",
       " '8386': 'Moodle 2.0 E-Learning Course Development [eBook]',\n",
       " '8689': 'Microsoft SharePoint 2010 and Windows PowerShell 2.0: Expert Cookbook [eBook]',\n",
       " '8717': 'PHP Ajax Cookbook [eBook]',\n",
       " '9017': \"Google SketchUp for Game Design: Beginner's Guide [eBook]\",\n",
       " '9368': \"PHP Application Development with NetBeans: Beginner's Guide [eBook]\",\n",
       " '9837': 'The Game Jam Survival Guide [eBook]',\n",
       " None: ''}"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "elements = driver.find_elements_by_class_name(\"product-line\")\n",
    "books = {e.get_attribute('nid'): e.get_attribute('title') for e in elements}\n",
    "books"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First time around I made the mistake to close `driver` after `elements`, but `get_attribute` I used in `books` still needs the session. So only now should you close it. This also closes the Chrome app:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "driver.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's write a function to receive a regex search term and book format and print the books that match the books in my collection:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "import re\n",
    "\n",
    "DOWNLOAD_URL = 'https://www.packtpub.com/ebook_download/{nid}/{ebook_format}'\n",
    "BOOK_FORMATS = 'pdf epub mobi'\n",
    "\n",
    "def get_books(grep, ebook_format):\n",
    "    \"\"\"Receives a grep regex and book format (epub, pdf, mobi)\n",
    "       and prints the titles + urls of matching ebooks\"\"\"\n",
    "    grep = grep.lower()\n",
    "    ebook_format = ebook_format.lower()\n",
    "    if ebook_format not in BOOK_FORMATS.split():\n",
    "        raise ValueError(f'Not a valid book format (valid are: {BOOK_FORMATS})')\n",
    "        \n",
    "    for nid, title in books.items():\n",
    "        if re.search(grep, title.lower()):\n",
    "            url = DOWNLOAD_URL.format(nid=nid, ebook_format=ebook_format)\n",
    "            print(title, url)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Python Data Science Essentials - Second Edition [eBook] https://www.packtpub.com/ebook_download/27146/mobi\n",
      "Python Machine Learning Blueprints: Intuitive data projects you can relate to [eBook] https://www.packtpub.com/ebook_download/24221/mobi\n",
      "Python Data Visualization Cookbook [eBook] https://www.packtpub.com/ebook_download/13532/mobi\n"
     ]
    }
   ],
   "source": [
    "get_books('python.*data', 'mobi')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Machine Learning for the Web [eBook] https://www.packtpub.com/ebook_download/24826/pdf\n",
      "Python Machine Learning Blueprints: Intuitive data projects you can relate to [eBook] https://www.packtpub.com/ebook_download/24221/pdf\n",
      "What You Need to Know about Machine Learning [eBook] https://www.packtpub.com/ebook_download/27683/pdf\n",
      "Mastering Machine Learning with scikit-learn [eBook] https://www.packtpub.com/ebook_download/17805/pdf\n",
      "Practical Machine Learning [eBook] https://www.packtpub.com/ebook_download/20987/pdf\n",
      "Machine Learning with R - Second Edition [eBook] https://www.packtpub.com/ebook_download/21989/pdf\n",
      "Machine Learning with Spark [eBook] https://www.packtpub.com/ebook_download/17399/pdf\n",
      "Python Machine Learning [eBook] https://www.packtpub.com/ebook_download/17954/pdf\n",
      "Building Machine Learning Systems with Python [eBook] https://www.packtpub.com/ebook_download/11703/pdf\n"
     ]
    }
   ],
   "source": [
    "get_books('Machine.*Learning', 'PDF')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Example 2. autocreate a PyBites banner"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Some time ago [I made a banner generator with Pillow and Flask](https://pybit.es/pillow-banner-flask.html). It is hosted [here](http://pybites-banners.herokuapp.com). \n",
    "\n",
    "Although this is nice what if I want to make banners automatically? Let's try to do so using Selenium:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's break the task down into various steps:\n",
    "\n",
    "1. Although the site can be used without login, authenticated users have their banners stored, so go straight to login URL:\n",
    "    ![go to site](images/banner1.png)\n",
    "\n",
    "2. And login: \n",
    "    ![login](images/banner2.png)\n",
    "\n",
    "3. We need to locate the form elements and provide the proper data, then click the submit button:\n",
    "    ![provide data](images/banner3.png)\n",
    "\n",
    "4. We need to download the output image it generates:\n",
    "    ![get banner](images/banner4.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After previous exercise this should be quite straight-forward:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "user = os.environ.get('PB_BANNER_USER')\n",
    "pw = os.environ.get('PB_BANNER_PW')\n",
    "\n",
    "class NoLogin(Exception):\n",
    "    pass\n",
    "\n",
    "if user is None or pw is None:\n",
    "    raise NoLogin('Set PB_BANNER_USER and PB_BANNER_PW in your env')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "login = 'https://pybites-banners.herokuapp.com/login'\n",
    "\n",
    "driver = webdriver.Chrome()\n",
    "driver.get(login)\n",
    "\n",
    "driver.find_element_by_id('username').send_keys(user)\n",
    "driver.find_element_by_id('password').send_keys(pw + Keys.RETURN)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "from datetime import datetime\n",
    "\n",
    "def get_title():\n",
    "    \"\"\"Creates a title to store banner as, e.g. newsYYYYWW\n",
    "       (YYYY = year, WW = week number)\"\"\"\n",
    "    now = datetime.now()\n",
    "    year = now.year\n",
    "    week = str(now.isocalendar()[1]).zfill(2)\n",
    "    return f'news{year}{week}'\n",
    "\n",
    "title = get_title()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "now = datetime.now()\n",
    "year = now.year\n",
    "week = str(now.isocalendar()[1]).zfill(2)\n",
    "news_option = 'pybites-news'\n",
    "\n",
    "bg_image = 'http://www.allwhitebackground.com/images/2/2210.jpg'\n",
    "banner_text = f'from pybites import News -> Twitter Digest {year} Week {week}'\n",
    "\n",
    "driver.find_element_by_id('name').send_keys(title)\n",
    "driver.find_element_by_xpath(f'//select[@name=\"image_url1\"]/option[text()=\"{news_option}\"]').click()\n",
    "driver.find_element_by_id('text').send_keys(banner_text)\n",
    "driver.find_element_by_id('image_url2').send_keys(bg_image + Keys.RETURN)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And the result:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![resulting banner](images/banner5.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "driver.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Second + third day: practice time!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now it's your turn. The goal is to have you get your hands dirty using Python Selenium."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Testing with Selenium"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I deliberately left testing out, because we have a nice code challenge for you to practice:\n",
    "\n",
    "- First checkout the docs: [Using Selenium to write tests](http://selenium-python.readthedocs.io/getting-started.html#using-selenium-to-write-tests)\n",
    "- Then head over to [Code Challenge 32 - Test a Simple Django App With Selenium](https://codechalleng.es/challenges/32/) and try to automate testing of [PyBites first ever Django App](http://pyplanet.herokuapp.com)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Scratch your own itch"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Although the testing option is our favorite, we want you to be free.\n",
    "\n",
    "In this notebook I gave you two examples of automated tasks. Maybe you want to try them yourself, build them out?\n",
    "\n",
    "Or what about trying to use Selenium on your favorite website or service? e.g. login to Facebook, Twitter, Reddit and read/post content ...\n",
    "\n",
    "There are many possibilities. Again, having you code up a script using what you just learned will make all the difference in what you get out of this lesson. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Have fun and remember: _Keep calm and code in Python!_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Time to share what you've accomplished!\n",
    "\n",
    "Be sure to share your last couple of days work on Twitter or Facebook. Use the hashtag **#100DaysOfCode**.\n",
    "\n",
    "Here are [some examples](https://twitter.com/search?q=%23100DaysOfCode) to inspire you. Consider including [@talkpython](https://twitter.com/talkpython) and [@pybites](https://twitter.com/pybites) in your tweets.\n",
    "\n",
    "*See a mistake in these instructions? Please [submit a new issue](https://github.com/talkpython/100daysofcode-with-python-course/issues) or fix it and [submit a PR](https://github.com/talkpython/100daysofcode-with-python-course/pulls).*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "venv",
   "language": "python",
   "name": "venv"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}