gh-130587: Add hand-written docs for non-OP tokens #130588
Conversation
Co-authored-by: Blaise Pabon <blaise@gmail.com>
Co-authored-by: Blaise Pabon <blaise@gmail.com>
|
The rendered docs are at https://cpython-previews--130588.org.readthedocs.build/en/130588/library/token.html |
|
@lysnikolaou, does this look reasonable to you? |
There was a problem hiding this comment.
This looks like a good improvement to me! Thanks @encukou!
I've left some inline comments regarding some specifics in the docs.
Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com> Co-authored-by: Lysandros Nikolaou <lisandrosnik@gmail.com>
|
Thank you for the reviews! I addressed some; I'll continue next week. |
| Note that a token's value may depend on tokenizer options. For example, a | ||
| ``"+"`` token may be reported as either :data:`PLUS` or :data:`OP`, or | ||
| a ``"match"`` token may be either :data:`NAME` or :data:`SOFT_KEYWORD`. | ||
|
|
There was a problem hiding this comment.
I added this note to clarify the “fuzziness” of these categories.
Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com>
|
Thanks @encukou for the PR 🌮🎉.. I'm working now to backport this PR to: 3.13. |
|
Sorry, @encukou, I could not cleanly backport this to |
|
GH-131465 is a backport of this pull request to the 3.13 branch. |
…onGH-130588) (cherry picked from commit 4bced29) Co-authored-by: Petr Viktorin <encukou@gmail.com> Co-authored-by: Blaise Pabon <blaise@gmail.com> Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com> Co-authored-by: Lysandros Nikolaou <lisandrosnik@gmail.com>
…GH-131465) Co-authored-by: Blaise Pabon <blaise@gmail.com> Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com> Co-authored-by: Lysandros Nikolaou <lisandrosnik@gmail.com>
Add hand-written docs for non-OP tokens
Make the automation (
generate_token.py) check that the hand-written docs are present, and only generate docs for the OP tokensSwitch to
list-tablefor the OP tokens, to make their docs more compactAdd
ENDMARKERto the grammar docs where it appears (toplevel components)Add forgotten
versionchangedentry forEXCLAMATIONRemove docs for
NT_OFFSET📚 Documentation preview 📚: https://cpython-previews--130588.org.readthedocs.build/