mirrors
/
cpython
mirror of https://github.com/python/cpython.git
2
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
cpython/Tools/cases_generator
History
Mark Shannon f6f4e8a662
GH-132554: "Virtual" iterators (GH-132555) 
* FOR_ITER now pushes either the iterator and NULL or leaves the iterable and pushes tagged zero

* NEXT_ITER uses the tagged int as the index into the sequence or, if TOS is NULL, iterates as before.
 		2025-05-27 15:59:45 +01:00
..
README.md Rename tier 2 redundancy eliminator to optimizer (#115888) 2024-02-26 08:42:53 -08:00
_typing_backports.py gh-104504: cases generator: Add `--warn-unreachable` to the mypy config (#108112) 2023-08-21 00:40:41 +01:00
analyzer.py GH-132554: "Virtual" iterators (GH-132555) 2025-05-27 15:59:45 +01:00
cwriter.py GH-131498: Cases generator: Parse down to C statement level. (GH-131948) 2025-04-02 16:31:59 +01:00
generators_common.py GH-132554: "Virtual" iterators (GH-132555) 2025-05-27 15:59:45 +01:00
interpreter_definition.md gh-133412: amend docs for the `inst` definition (#133708) 2025-05-09 09:06:22 +02:00
lexer.py GH-131498: Cases generator: Parse down to C statement level. (GH-131948) 2025-04-02 16:31:59 +01:00
mypy.ini Replace `strict_concatenate = True` with `extra_checks = True` (#126391) 2025-01-25 12:44:23 +03:00
opcode_id_generator.py GH-122390: Replace `_Py_GetbaseOpcode` with `_Py_GetBaseCodeUnit` (GH-122942) 2024-08-13 14:22:57 +01:00
opcode_metadata_generator.py gh-128045: Mark unknown opcodes as deopting to themselves (#128044) 2025-05-19 10:15:16 -04:00
optimizer_generator.py gh-133273: Keep instruction definitions in `bytecodes.c` and `optimizer_bytecodes.c` in sync (GH-133320) 2025-05-09 01:54:49 +08:00
parser.py GH-131498: Cases generator: Parse down to C statement level. (GH-131948) 2025-04-02 16:31:59 +01:00
parsing.py GH-131498: Cases generator: Parse down to C statement level. (GH-131948) 2025-04-02 16:31:59 +01:00
plexer.py GH-131498: Cases generator: Parse down to C statement level. (GH-131948) 2025-04-02 16:31:59 +01:00
py_metadata_generator.py GH-120024: Tidy up case generator code a bit. (GH-122780) 2024-08-08 10:57:59 +01:00
stack.py GH-131498: Replace single-element arrays with scalars in bytecodes.c (GH-132615) 2025-04-18 07:16:28 -07:00
target_generator.py gh-129989: Properly disable tailcall interp in configure (GH-129991) 2025-02-16 03:01:24 +08:00
tier1_generator.py GH-131498: Cases generator: Allow input and 'peek' variables to be modified (GH-132506) 2025-04-14 12:19:53 +01:00
tier2_generator.py Get rid of ERROR_IF's "label" parameter (GH-132654) 2025-04-29 17:21:53 -07:00
uop_id_generator.py gh-120417: Remove unused imports in cases_generator (#120622) 2024-06-17 21:58:56 +02:00
uop_metadata_generator.py GH-131498: Another refactoring of the code generator (GH-131827) 2025-03-31 13:52:48 +01:00

README.md

Tooling to generate interpreters

Documentation for the instruction definitions in Python/bytecodes.c ("the DSL") is here.

What's currently here:

  • analyzer.py: code for converting AST generated by Parser to more high-level structure for easier interaction
  • lexer.py: lexer for C, originally written by Mark Shannon
  • plexer.py: OO interface on top of lexer.py; main class: PLexer
  • parsing.py: Parser for instruction definition DSL; main class: Parser
  • parser.py helper for interactions with parsing.py
  • tierN_generator.py: a couple of driver scripts to read Python/bytecodes.c and write Python/generated_cases.c.h (and several other files)
  • optimizer_generator.py: reads Python/bytecodes.c and Python/optimizer_bytecodes.c and writes Python/optimizer_cases.c.h
  • stack.py: code to handle generalized stack effects
  • cwriter.py: code which understands tokens and how to format C code; main class: CWriter
  • generators_common.py: helpers for generators
  • opcode_id_generator.py: generate a list of opcodes and write them to Include/opcode_ids.h
  • opcode_metadata_generator.py: reads the instruction definitions and write the metadata to Include/internal/pycore_opcode_metadata.h
  • py_metadata_generator.py: reads the instruction definitions and write the metadata to Lib/_opcode_metadata.py
  • target_generator.py: generate targets for computed goto dispatch and write them to Python/opcode_targets.h
  • uop_id_generator.py: generate a list of uop IDs and write them to Include/internal/pycore_uop_ids.h
  • uop_metadata_generator.py: reads the instruction definitions and write the metadata to Include/internal/pycore_uop_metadata.h

Note that there is some dummy C code at the top and bottom of Python/bytecodes.c to fool text editors like VS Code into believing this is valid C code.

A bit about the parser

The parser class uses a pretty standard recursive descent scheme, but with unlimited backtracking. The PLexer class tokenizes the entire input before parsing starts. We do not run the C preprocessor. Each parsing method returns either an AST node (a Node instance) or None, or raises SyntaxError (showing the error in the C source).

Most parsing methods are decorated with @contextual, which automatically resets the tokenizer input position when None is returned. Parsing methods may also raise SyntaxError, which is irrecoverable. When a parsing method returns None, it is possible that after backtracking a different parsing method returns a valid AST.

Neither the lexer nor the parsers are complete or fully correct. Most known issues are tersely indicated by # TODO: comments. We plan to fix issues as they become relevant.