Commits

Breadcrumbs

History for

on3.8

User selector

All users

Datepicker

All time

Commit History

Commits on Jun 16, 2019

• bpo-35922: Fix RobotFileParser when robots.txt has no relevant crawl delay or request rate (GH-11791)

  Show description for 58a1a76
  

  miss-islington

  and

  Rémi Lapeyre

  authored
  58a1a76

  Copy full SHA for 58a1a76

Commits on May 16, 2018

• bpo-21475: Support the Sitemap extension in robotparser (GH-6883)

  

  mcscope

  authored and

  ned-deily

  committed
  5db5c06

  Copy full SHA for 5db5c06

Commits on May 14, 2018

• bpo-32861: urllib.robotparser fix incomplete __str__ methods. (GH-5711)

  Show description for bd08a0a
  

  michael-lazar

  authored and

  serhiy-storchaka

  committed
  bd08a0a

  Copy full SHA for bd08a0a

Commits on Nov 23, 2017

• bpo-31325: Fix usage of namedtuple in RobotFileParser.parse() (#4529)

  

  berkerpeksag

  authored and

  rhettinger

  committed
  3df02db

  Copy full SHA for 3df02db

Commits on Sep 18, 2016

• Issue #25400: RobotFileParser now correctly returns default values for crawl_delay and request_rate

  Show description for 9a7bbb2
  berkerpeksag

  committed
  9a7bbb2

  Copy full SHA for 9a7bbb2

Commits on May 8, 2016

• Merge typo fixes from 3.5

  vadmium

  committed
  1ce738e

  Copy full SHA for 1ce738e

• Fix typos in comments, documentation and test method names

  vadmium

  committed
  f056416

  Copy full SHA for f056416

Commits on Oct 8, 2015

• Issue #16099: RobotFileParser now supports Crawl-delay and Request-rate

  Show description for 960e848
  berkerpeksag

  committed
  960e848

  Copy full SHA for 960e848

Commits on May 13, 2014

• Issue 21469: Minor code modernization (convert and/or expression to an if/else expression).

  Show description for 38acd4c
  rhettinger

  committed
  38acd4c

  Copy full SHA for 38acd4c

• Issue 21469: Mitigate risk of false positives with robotparser.

  Show description for 122541b
  rhettinger

  committed
  122541b

  Copy full SHA for 122541b

Commits on May 29, 2013

• #17403: urllib.parse.robotparser normalizes the urls before adding to ruleline.

  Show description for c70a6ae
  orsenthil

  committed
  c70a6ae

  Copy full SHA for c70a6ae

Commits on Jul 29, 2010

• #4108: the first default entry (User-agent: *) wins.

  birkenfeld

  committed
  0a0fc07

  Copy full SHA for 0a0fc07

Commits on Jul 28, 2010

• Fix Issue6325 - robotparse to honor urls with query strings.

  orsenthil

  committed
  3f8ab96

  Copy full SHA for 3f8ab96

Commits on Jul 31, 2008

• Merged revisions 65209-65216,65225-65226,65233,65239,65246-65247,65255-65256 via svnmerge from

  Show description for d631371
  benjaminp

  committed
  d631371

  Copy full SHA for d631371

Commits on Jul 18, 2008

• Bug 3347: robotparser failed because it didn't convert bytes to string.

  Show description for 73fd46d
  jeremyhylton

  committed
  73fd46d

  Copy full SHA for 73fd46d

Commits on Jun 18, 2008

• Make a new urllib package .

  Show description for 1afc169
  jeremyhylton

  committed
  1afc169

  Copy full SHA for 1afc169

Loading