Raise FileNotFoundError in read_json if input looks like file path but file is missing

[janosh]

• closes Misleading error messages when opening inexistent json file #29102
• Tests added and passed if fixing a bug or adding a new feature
• All code checks passed.
• Added an entry in the latest doc/source/whatsnew/v1.5.0.rst file if fixing a bug or adding a new feature.

[github-actions]

This pull request is stale because it has been open for thirty days with no activity. Please update and respond to this comment if you're still interested in working on this.

[mroeschke]

LGTM. Will need approval from @jreback to merge. (long string's overflowing the filesystem is handled by file_exists; there's a comment # gh-5874: if the filepath is too long will raise here)

[mroeschke]

Actually, after merging master looks like there's a code path needs updating.

Also @twoertwein, for other file extensions with compression, it doesn't appear we manually raise a FileNotFoundError. Should this automatically be handled by the handler code?

[twoertwein]

Also @twoertwein, for other file extensions with compression, it doesn't appear we manually raise a FileNotFoundError. Should this automatically be handled by the handler code?

Would maybe be better to raise here when the file doesn't exist:

pandas/pandas/io/common.py

Line 692 in b1b788f

if "r" not in mode and is_path:

[janosh]

Is this a known error in TestiLocBaseIndependent.test_iloc_setitem_multicolumn_to_datetime? Looks unrelated to my changes.

[jreback]

lgtm merge on green

[jreback]

cc @mroeschke

[mroeschke]

If you can investigate if this logic can be moved according to #46718 (comment) (such that not just json files raise this) that would be great

[janosh]

What are the safe assumptions to make here about how the file path ends? Is there sth like _extension_to_compression but for file formats?

# Only for write methods
if "r" not in mode and is_path:
    check_parent_directory(str(handle))

_supported_formats = ["json", "csv", "xls", "xlsx", ...]
_allowed_extensions = tuple(itertools.chain(
    (x,) + tuple(f"{x}{c}" for c in _extension_to_compression)
    for x in _supported_formats
))
if (
    is_path
    and handle.lower().endswith(_allowed_extensions)
    and not file_exists(handle)
):
    raise FileNotFoundError(f"File {handle} does not exist")

[twoertwein]

After read_json knows that it is a file, I think you can just raise inside get_handle when the file doesn't exist:

if "r" not in mode and is_path and not file_exists(handle):
    raise ...

[janosh]

You mean this, right?

if "r" in mode and is_path and not file_exists(handle):
    raise ...

[janosh]

Actually read_json doesn't even call get_handle since it's behind a bunch of or cases none of which are true:

if (
    not isinstance(filepath_or_buffer, str)
    or is_url(filepath_or_buffer)
    or is_fsspec_url(filepath_or_buffer)
    or file_exists(filepath_or_buffer)
):
    self.handles = get_handle(...)

[twoertwein]

I think there were attempts at creating an efficient way of checking whether a str is a legit JSON string. I assume these attempts were unsuccessful - we check the reverse. If that's the case, then moving the check into get_handle, will not work.

[janosh]

If that's the case, then moving the check into get_handle, will not work.

So leave PR as is?

[twoertwein]

So leave PR as is?

Probably.

[mroeschke]

Great, thanks @janosh