Skip to content

Conversation

@Lakshya-Upadhyaya
Copy link
Contributor

Description

This PR fixes gh-142968 by adding explicit type checks to token.ISTERMINAL and token.ISNONTERMINAL.

Previously, these functions accepted floats (e.g., 0.5) without error, which was inconsistent with PyPy and the expected behavior of token validation. They now raise a TypeError for non-integer inputs, matching the behavior described in the issue.

Changes Made

  • Modified Tools/build/generate_token.py to inject type checks into the generated Lib/token.py.
  • Regenerated Lib/token.py using the tool.
  • Added a new test file Lib/test/test_token.py with regression tests for these functions.

Verification

  • Validated that token.ISTERMINAL(0.5) now raises TypeError.
  • Ran python -m test -v test_token and confirmed all tests pass.

@picnixz
Copy link
Member

picnixz commented Jan 4, 2026

There is no consensus on this issue and I don't think it's useful. Please don't open PRs when a core developer or a triager has not acknowledged the issue (unless it's straightforward that there is an issue and the fix is simple enough). In addition, acknowledge whetehr an LLM was used for the PR (it's clear that one was used for the PR description); see also https://devguide.python.org/getting-started/generative-ai/.

@picnixz picnixz closed this Jan 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ISNONTERMINAL and ISTERMINAL in token.py accept float arguments in CPython but raise TypeError in PyPy

2 participants