feat: Use sphinx field-level documentation for hover

[WhyNotHugo]

Given this simple type

class Human:
    #: This person's date of birth
    dob: datetime

And this code:

h = human
h.dob

When fetching the "hover" information (e.g.: textDocument/hover, vim.lsp.buf.hover() on neovim) with the cursor on dob this shows the documentation for the type (e.g.: datetime) in this case.

I think that it would be very useful to show the documentation right above the field, instead of the documentation for type type. This particular format is used by sphinx, so a lot of code-bases (e.g.: Django) use this to document fields. In most contexts, the docs for the field are far more relevant/useful that the docs for the type.

[WhyNotHugo]

Of course, using the current value as fallback makes perfect sense.

[ccordoba12]

I think the right way to document class attributes is to add a docstring below them, like this:

class Human:

    dob: datetime
    """This person's date of birth"""

[WhyNotHugo]

@ccordoba12 I've never seen that before. Sphinx doesn't seem to pick that up. Where have you seen this / which tool uses this format?

[ccordoba12]

See this SO answer for the details:

https://stackoverflow.com/a/69754746/438386

That's the method we use in Spyder too, although it's strange that Sphinx didn't pick it up for you).

[bew]

When fetching the "hover" information (e.g.: textDocument/hover, vim.lsp.buf.hover() on neovim) with the cursor on dob this shows the documentation for the type (e.g.: datetime) in this case.

I also see this behavior when requesting information about any non-function variables, even though there are some information in the function's docstring:

👉 How complex is it to show the parameter's documentation + type instead of the type's documentation?

About documentation of class attributes, which variant do you think we should support? (how difficult are they to support?)

(I might be interested in doing an implementation if it's not too complex)

[krassowski]

There are two issues captured here:

• the attribute documentation is extracted only from strings, but users expect it to be also extracted from comments (as in Sphinx). I understand that the jedi syntax tree parser skips over comments, and while it could extracts the comments from the code position, it currently does not do that. I think this is not too much effort to implement, but it would be best to first check with the jedi maintainer if they would be open to a PR implementing it.
• the parameter documentation is never extracted; I opened Support for documentation of function/method parameters davidhalter/jedi#2061, please see it for more details, but briefly, this is a harder problem as it involves one of:
  • implementing support for PEP 727 (draft!) which proposes to use Annotated[<type>, Doc(<docstring>)]
  • extracting comments (again, possible but not currently implemented)
  • parsing docstrings (which may be in one of many formats)

I looked into whether we could support the parameter description extraction on pylsp level and the answer is maybe. Jedi does not expose all necessary information on the public attributes right now, but for example for completion we could get the "name" object from the private _name attribute and it does contain all we would need.

Edit: another approach is to monkey-patch jedi's ParamNameWrapper with a custom implementation of py__doc__ method, and this would work for all completion/hover, etc.

Any thoughts?

[krassowski]

To extract documentation from comments above attributes the following patch for jedi could be applied:

def find_statement_documentation(tree_node):
    if tree_node.type == 'expr_stmt':
        maybe_name = tree_node.children[0]
        if maybe_name.type != 'name':
            return ''
        maybe_comment = maybe_name.prefix
        if not isinstance(maybe_comment, str):
            return ''
        maybe_comment = maybe_comment.strip()
        if not maybe_comment.startswith('#'):
            return ''
        lines = [
            line.strip().lstrip('#')
            for line in maybe_comment.splitlines()
        ]
        min_indent = min([
            len(line) - len(line.lstrip())
            for line in lines
        ])
        return '\n'.join(
            line.strip().lstrip('#')[min_indent:]
            for line in maybe_comment.splitlines()
        )
    return ''

Additionally #: could be handled too.