Skip to content

crawl_google_results.py update, modularize, documentation and doctest #4847

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 24 commits into from
Closed
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
be1c08a
Added large changes to web_programming/crawl_google_results.py with d…
Oct 1, 2021
dec909c
Added large changes to web_programming/crawl_google_results.py with d…
Oct 1, 2021
6b31ad7
Update web_programming/crawl_google_results.py
appledora Oct 1, 2021
4c4dc95
Update web_programming/crawl_google_results.py
appledora Oct 1, 2021
e1cc719
fixed typo in test
Oct 2, 2021
8c1f319
undo the changes to requirements.txt
Oct 2, 2021
af468bf
Fixed doctest error
Oct 2, 2021
6b774c2
Fixed doctest error, removed print() from method, created new script …
Oct 2, 2021
5c9a337
Fixed doctest error, removed print() from method, created new script …
Oct 2, 2021
2c8e276
Fixed doctest error, removed print() from method, created new script …
Oct 2, 2021
eb9c3f2
Revert changes to crawl_google_results.py
cclauss Oct 2, 2021
331ebcc
Update get_google_search_results.py
cclauss Oct 2, 2021
9cd8d6d
Update get_google_search_results.py
cclauss Oct 2, 2021
ac1b85f
Changed entire algorithm as suggested, added documentations
Oct 3, 2021
8cd3e8a
Changed entire algorithm as suggested, added documentations
Oct 3, 2021
f0c71e0
Changed entire algorithm as suggested, added documentations
Oct 3, 2021
8a4b8d2
Updated requirements.txt
Oct 3, 2021
f2a773e
Updated doctests
Oct 3, 2021
59d76c4
Update requirements.txt
cclauss Oct 4, 2021
1d4ba22
Apply suggestions from code review
cclauss Oct 4, 2021
6da76d2
removed session related libraries because problem with mypy
Oct 5, 2021
ed77a9f
Update web_programming/get_google_search_results.py
appledora Oct 18, 2021
783b91a
Update web_programming/get_google_search_results.py
appledora Oct 19, 2021
7c9e276
Update get_google_search_results.py
appledora Oct 21, 2021
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
fixed typo in test
  • Loading branch information
appledora committed Oct 2, 2021
commit e1cc719a559070294d1a1df64c25947b812c3b56
32 changes: 32 additions & 0 deletions requirements.txt
Original file line number Diff line number Diff line change
@@ -1,18 +1,50 @@
autopep8
beautifulsoup4
brotlipy
certifi
cffi
chardet
conda
conda-package-handling
cryptography
cycler
fake_useragent
idna
keras
kiwisolver
lxml
matplotlib
matplotlib
numpy
numpy
opencv-python
pandas
pandas
pillow
Pillow
pycodestyle
pycosat
pycparser
pycryptodomex
pyOpenSSL
pyparsing
PySocks
python-dateutil
pytz
qiskit
requests
requests
ruamel-yaml
scikit-fuzzy
six
sklearn
statsmodels
sympy
tabulate
tensorflow
toml
tornado
tqdm
types-requests
urllib3
xgboost
7 changes: 4 additions & 3 deletions web_programming/crawl_google_results.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
import sys

import requests
from fake_useragent import UserAgent

Expand All @@ -13,11 +14,11 @@ def crawl_and_store_google_search_result(query="potato", filename=""):
filename (str, optional): The name of the file to be saved with the
search results. Defaults to "".
>>> crawl_and_store_google_search_result ("hacktober", "hacktober")
File saved in hacktober.html
File saved as hacktober.html
>>> crawl_and_store_google_search_result ("hacktober", "hacktober.html")
File saved in hacktober.html
File saved as hacktober.html
>>> crawl_and_store_google_search_result ("hacktober")
File saved in hacktober-query.html
File saved as hacktober-query.html
"""
print(f"Searching Google for {query} ....")
if filename == "":
Expand Down