Improve file URI ergonomics in urllib.request

[barneygale]

Feature or enhancement

I request that we make pathname2url and url2pathname easier to use:

• pathname2url() is made to accept an optional include_scheme argument that sticks file: on the front when true
• url2pathname() is made to strip any file: prefix from its argument.

I think this would go a long way towards making these functions usable, and allow us to remove the scary "This does not accept/produce a complete URL" warnings from the docs.

Linked PRs

• GH-125866: Improve tests for pathname2url() and url2pathname() #125993
• [3.13] GH-125866: Improve tests for pathname2url() and url2pathname() (GH-125993) #126144
• [3.12] GH-125866: Improve tests for pathname2url() and url2pathname() (GH-125993) #126145
• GH-125866: Preserve Windows drive letter case in file URIs #127138
• GH-125866: Deprecate nturl2path module #131432

[serhiy-storchaka]

file:/etc/hosts, file:///etc/hosts and file://localhost/etc/hosts are equivalent URIs. I do not see anything in RFC 8089 that suggests that file:///etc/hosts is more preferable.

Actually, file:/etc/hosts is a new form. It was not allowed in RFC 1738, and only allowed in RFC 8089. So you can say that it is more modern representation.

[barneygale]

I do not see anything in RFC 8089 that suggests that file:///etc/hosts is more preferable.

RFC 8089 is mainly non-normative, but it describes file:/// as "traditional" and file:/ as "minimal"

The freedesktop spec notes:

Some current apps generate URIs of the form "file:/". These
are not correct according to RFC1738, so they should not be
generated. However for backwards compatibility, it is recommended that
such URIs are interpreted as file URIs with an empty hostname.

The wikipedia article lists the file:/// form first, and notes KDE as an exception.

[barneygale]

Also, using file:/// is:

1. More consistent with the Windows version of pathname2url(), which adds an empty authority for DOS drive paths - e.g. C:\foo is converted to ///C:/foo, not /C:/foo
2. Consistent with pathlib.Path.as_uri()

[serhiy-storchaka]

RFC 8089 updates RFC 1738. The file:/// form is RFC 1738 compliant. The file:/ form is not RFC 1738 compliant, but RFC 8089 compliant. So the change does not match the title of this issue. This is why it raises questions.

It may be a justified change, but it needs different justification than RFC 8089 compliance.

[barneygale]

Good point. I'll revise the issue - it's already evolved a bit since I logged it.

[barneygale]

@serhiy-storchaka I've revised the description as follows:

I request that we make pathname2url and url2pathname easier to use:

• pathname2url() is made to accept an optional include_scheme argument that sticks file: on the front when true
• url2pathname() is made to strip any file: prefix from its argument.

Does that seem like a good idea to you?

If we go for these changes, and the user provides a URL with a non-file: scheme to url2pathname(), what should happen in your opinion? (I'm erring towards raising URLError.)

Relatedly, what do you think should happen if the user provides a query or fragment in the URL? Currently it's returned as part of the file path. (This seems under-specified to me, but I'm leaning towards raising URLError here too.)

Thank you

[serhiy-storchaka]

I wonder -- would not be better to add such functions in os.path? The implementation is clearly platform depending, and if we add support for platform with different path format (like old macpath), it will need a completely different implementation. In custom build for exotic platform it is easier to provide a separate os.path implementation than patch urllib.request.

[barneygale]

Good question! It's something that's come up before:

• https://discuss.python.org/t/file-uris-in-python/15600
• Add os.path.fileuri() function #88578

The current Windows + POSIX implementations don't have much in common, but I think that will change when I address this issue and #123599. To solve them, I'm planning to use urlsplit() to parse the URL, replacing parts of the janky implementation in url2pathname(). Here's a rough draft of where I want to end up:

def url2pathname(url):
    scheme, authority, path, query, fragment = urlsplit(url, scheme='file')
    if scheme != 'file':
        raise URLError(f'URL uses non-file scheme: {url!r}')
    if query:
        raise URLError(f'file URL has query: {url!r}')
    if fragment:
        raise URLError(f'file URL has fragment: {url!r}')

    if os.name == 'nt':
        # Windows-specific file URI quirks.
        if authority and authority != 'localhost':
            # e.g. file://server/share/file.txt
            path = '//' + authority + path
        elif path[:3] == '///':
            # e.g. file://///server/share/file.txt
            path = path[1:]
        else:
            if path[:1] == '/' and path[2:3] in ':|':
                # Skip past extra slash before DOS drive in URL path.
                path = path[1:]
            if path[1:2] == '|':
                # Older URLs use a pipe after a drive letter
                path = path.replace('|', ':', 1)
        path = path.replace('/', '\\')
    elif not _is_local_authority(authority):
        # POSIX only: reject URL if authority doesn't resolve to localhost.
        raise URLError(f'file URL has non-local authority: {url!r}')

    encoding = sys.getfilesystemencoding()
    errors = sys.getfilesystemencodeerrors()
    return unquote(path, encoding=encoding, errors=errors)


def pathname2url(pathname, include_scheme=False):
    if os.name == 'nt':
        pathname = pathname.replace('\\', '/')
    encoding = sys.getfilesystemencoding()
    errors = sys.getfilesystemencodeerrors()
    prefix = 'file:' if include_scheme else ''
    drive, root, tail = os.path.splitroot(pathname)

    if drive:
        if drive[:4] == '//?/':
            drive = drive[4:]
            if drive[:4].upper() == 'UNC/':
                drive = '//' + drive[4:]
        if drive[1:] == ':':
            prefix += '///'
        drive = quote(drive, encoding=encoding, errors=errors, safe='/:')
    elif root:
        prefix += '//'

    tail = quote(tail, encoding=encoding, errors=errors)
    return prefix + drive + root + tail

There's clearly some OS-specific code in there, but there's also a fair amount of shared and URL-specific code which might feel out-of-place in os.path.

Besides, if we added functions in os.path we'd need to either live with near-duplicate functionality in urllib, or deprecate the urllib functions (which isn't much fun for their users). I'm not ruling that out, but it seems unnecessarily if there's a reasonable path towards making pathname2url() and url2pathname() a bit more usable.

[barneygale]

FWIW, if we go with the above merger, then we can deprecate the oddball nturl2path module.