Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

gh-127833: Add links to token types to the lexical analysis intro #131468

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

encukou
Copy link
Member

@encukou encukou commented Mar 19, 2025

In the first part of the “lexical analysis” document, add links to token type documentation in the token module.

Add a section on the ENDMARKER token.

(I plan to send separate PR(s) for the rest of the document.)

Comment on lines +260 to +261
token, but ``a b`` is two tokens. For another example, ``+a`` is two tokens,
and is equivalent to ``+ a``.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not keen on 'equivalent to', perhaps:

Suggested change
token, but ``a b`` is two tokens. For another example, ``+a`` is two tokens,
and is equivalent to ``+ a``.
token, but ``a b`` is two tokens. However, ``+a`` and ``+ a`` both tokenize to the same output, as ``+a`` is not a valid token.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
awaiting merge docs Documentation in the Doc dir skip news
Projects
Status: Todo
Development

Successfully merging this pull request may close these issues.

2 participants