|
| 1 | +# Python3 Requests Module Examples |
| 2 | +This file contains examples for different actions using the Requests III module. All examples are Python3. |
| 3 | + |
| 4 | +## https://3.python-requests.org/user/quickstart |
| 5 | + |
| 6 | +### Download Images |
| 7 | +```python3 |
| 8 | +r = requests.get('https://imgs.xkcd.com/comics/python.png') |
| 9 | + |
| 10 | +with open('img.jpg', 'wb') as f: |
| 11 | + for chunk in r.iter_content(chunk_size=128): |
| 12 | + f.write(chunk) |
| 13 | +``` |
| 14 | + |
| 15 | +### Save Responses in JSON Dictionary |
| 16 | +```python3 |
| 17 | +payload = {'username': 'corey', 'password': 'testing'} |
| 18 | +r = requests.post('https://httpbin.org/post', data=payload) |
| 19 | + |
| 20 | +r_dict = r.json() #Saves JSON response as a dictionary |
| 21 | + |
| 22 | +print(r_dict['form']) #Retrieve particular value |
| 23 | +``` |
| 24 | + |
| 25 | +### Timeout in Seconds - Should be used for production code |
| 26 | +```python3 |
| 27 | +requests.get('https://github.com/', timeout=0.001) |
| 28 | +``` |
| 29 | + |
| 30 | +### Errors and Exceptions |
| 31 | + |
| 32 | +In the event of a network problem (e.g. DNS failure, refused connection, etc), Requests will raise a **```ConnectionError```** exception. |
| 33 | +**```Response.raise_for_status()```** will raise an **```HTTPError```** if the HTTP request returned an unsuccessful status code. |
| 34 | +If a request times out, a **```Timeout```** exception is raised. |
| 35 | +If a request exceeds the configured number of maximum redirections, a **```TooManyRedirects```** exception is raised. |
| 36 | + |
| 37 | +### Web Scraper with Request and BeautifulSoup4 |
| 38 | +```python3 |
| 39 | +from bs4 import BeautifulSoup |
| 40 | +import requests |
| 41 | +import csv |
| 42 | + |
| 43 | +source = requests.get('http://coreyms.com').text |
| 44 | + |
| 45 | +soup = BeautifulSoup(source, 'lxml') |
| 46 | + |
| 47 | +csv_file = open('cms_scrape.csv', 'w') |
| 48 | + |
| 49 | +csv_writer = csv.writer(csv_file) |
| 50 | +csv_writer.writerow(['headline', 'summary', 'video_link']) |
| 51 | + |
| 52 | +for article in soup.find_all('article'): |
| 53 | + headline = article.h2.a.text |
| 54 | + print(headline) |
| 55 | + |
| 56 | + summary = article.find('div', class_='entry-content').p.text |
| 57 | + print(summary) |
| 58 | + |
| 59 | + try: |
| 60 | + vid_src = article.find('iframe', class_='youtube-player')['src'] |
| 61 | + |
| 62 | + vid_id = vid_src.split('/')[4] |
| 63 | + vid_id = vid_id.split('?')[0] |
| 64 | + |
| 65 | + yt_link = f'https://youtube.com/watch?v={vid_id}' |
| 66 | + except Exception as e: |
| 67 | + yt_link = None |
| 68 | + |
| 69 | + print(yt_link) |
| 70 | + |
| 71 | + print() |
| 72 | + |
| 73 | + csv_writer.writerow([headline, summary, yt_link]) |
| 74 | + |
| 75 | +csv_file.close() |
| 76 | +``` |
0 commit comments