Commit Graph

6940 Commits

Author SHA1 Message Date
Mark Shannon eecbc7c390
bpo-44338: Port LOAD_GLOBAL to PEP 659 adaptive interpreter (GH-26638)
* Add specializations of LOAD_GLOBAL.

* Add more stats.

* Remove old opcache; it is no longer used.

* Add NEWS
2021-06-14 11:04:09 +01:00
Binbin 17b16e13bb
Fix typos in multiple files (GH-26689)
Co-authored-by: Terry Jan Reedy <tjreedy@udel.edu>
2021-06-12 22:47:44 -04:00
Serhiy Storchaka be8b631b7a
Add more const modifiers. (GH-26691) 2021-06-12 16:11:59 +03:00
Mark Shannon 54cb63863f
bpo-44348: Move trace-info to thread-state (GH-26623)
* Move trace-info to thread state.

* Correct output for pdb when turning on tracing in middle of line
2021-06-10 08:46:59 +01:00
Mark Shannon e117c02837
bpo-44337: Port LOAD_ATTR to PEP 659 adaptive interpreter (GH-26595)
* Specialize LOAD_ATTR with  LOAD_ATTR_SLOT and LOAD_ATTR_SPLIT_KEYS

* Move dict-common.h to internal/pycore_dict.h

* Add LOAD_ATTR_WITH_HINT specialized opcode.

* Quicken in function if loopy

* Specialize LOAD_ATTR for module attributes.

* Add specialization stats
2021-06-10 08:46:01 +01:00
Eric Snow 3e1c7167d8
bpo-43693: Un-revert commit f3fa63e. (#26609)
This was reverted in GH-26596 (commit 6d518bb) due to some bad memory accesses.

* Add the MAKE_CELL opcode. (gh-26396)

The memory accesses have been fixed.

https://bugs.python.org/issue43693
2021-06-08 16:01:34 -06:00
Pablo Galindo 3fe921cd49
Revert "bpo-43693: Add the MAKE_CELL opcode and interleave fast locals offsets. (gh-26396)" (GH-26597)
This reverts commit 631f9938b1.
2021-06-08 13:17:55 +01:00
Pablo Galindo 781dc76577
Fix compiler errors for unused variables (GH-26601) 2021-06-08 13:16:24 +01:00
Eric Snow 631f9938b1
bpo-43693: Add the MAKE_CELL opcode and interleave fast locals offsets. (gh-26396)
This moves logic out of the frame initialization code and into the compiler and eval loop.  Doing so simplifies the runtime code and allows us to optimize it better.

https://bugs.python.org/issue43693
2021-06-07 16:52:00 -06:00
Ryan Hileman 449e6f0ef3
bpo-41299: Reduce lag in Windows threading timeouts by using a higher precision time source (GH-26568) 2021-06-07 21:26:02 +01:00
Eric Snow 2ab27c4af4
bpo-43693: Un-revert commits 2c1e258 and b2bf2bc. (gh-26577)
These were reverted in gh-26530 (commit 17c4edc) due to refleaks.

* 2c1e258 - Compute deref offsets in compiler (gh-25152)
* b2bf2bc - Add new internal code objects fields: co_fastlocalnames and co_fastlocalkinds. (gh-26388)

This change fixes the refleaks.

https://bugs.python.org/issue43693
2021-06-07 12:22:26 -06:00
Mark Shannon 001eb520b5
bpo-44187: Quickening infrastructure (GH-26264)
* Add co_firstinstr field to code object.

* Implement barebones quickening.

* Use non-quickened bytecode when tracing.

* Add NEWS item

* Add new file to Windows build.

* Don't specialize instructions with EXTENDED_ARG.
2021-06-07 18:38:06 +01:00
Pablo Galindo 17c4edc4e0
bpo-43693: Revert commits 2c1e2583fd and b2bf2bc1ec (GH-26530)
* Revert "bpo-43693: Compute deref offsets in compiler (gh-25152)"

This reverts commit b2bf2bc1ec.

* Revert "bpo-43693: Add new internal code objects fields: co_fastlocalnames and co_fastlocalkinds. (gh-26388)"

This reverts commit 2c1e2583fd.

These two commits are breaking the refleak buildbots.
2021-06-04 17:51:05 +01:00
Mark Shannon b2bf2bc1ec
bpo-43693: Compute deref offsets in compiler (gh-25152)
Merges locals and cells into a single array.
Saves a pointer in the interpreter and means that we don't need the LOAD_CLOSURE opcode any more

https://bugs.python.org/issue43693
2021-06-03 18:03:54 -06:00
Max Bélanger 35002aa8f6
bpo-32280: Store _PyRuntime in a named section (GH-4802)
This commit stores the _PyRuntime structure in a section of the same name. This allows a debugging or crash reporting tool to quickly locate this structure at runtime without requiring the symbol table.

Co-authored-by: Pablo Galindo <pablogsal@gmail.com>
2021-06-04 00:37:29 +01:00
Batuhan Taskaya f3491242e4
bpo-11105: Do not crash when compiling recursive ASTs (GH-20594)
When compiling an AST object with a direct / indirect reference
cycles, on the conversion phase because of exceeding amount of
calls, a segfault was raised. This patch adds recursion guards to
places for preventing user inputs to not to crash AST but instead
raise a RecursionError.
2021-06-03 21:01:02 +01:00
Eric Snow 2c1e2583fd
bpo-43693: Add new internal code objects fields: co_fastlocalnames and co_fastlocalkinds. (gh-26388)
A number of places in the code base (notably ceval.c and frameobject.c) rely on mapping variable names to indices in the frame "locals plus" array (AKA fast locals), and thus opargs.  Currently the compiler indirectly encodes that information on the code object as the tuples co_varnames, co_cellvars, and co_freevars.  At runtime the dependent code must calculate the proper mapping from those, which isn't ideal and impacts performance-sensitive sections.  This is something we can easily address in the compiler instead.

This change addresses the situation by replacing internal use of co_varnames, etc. with a single combined tuple of names in locals-plus order, along with a minimal array mapping each to its kind (local vs. cell vs. free).  These two new PyCodeObject fields, co_fastlocalnames and co_fastllocalkinds, are not exposed to Python code for now, but co_varnames, etc. are still available with the same values as before (though computed lazily).

Aside from the (mild) performance impact, there are a number of other benefits:

* there's now a clear, direct relationship between locals-plus and variables
* code that relies on the locals-plus-to-name mapping is simpler
* marshaled code objects are smaller and serialize/de-serialize faster

Also note that we can take this approach further by expanding the possible values in co_fastlocalkinds to include specific argument types (e.g. positional-only, kwargs).  Doing so would allow further speed-ups in _PyEval_MakeFrameVector(), which is where args get unpacked into the locals-plus array.  It would also allow us to shrink marshaled code objects even further.

https://bugs.python.org/issue43693
2021-06-03 10:28:27 -06:00
Mark Shannon 937cebc93b
bpo-44298: Fix line numbers for early exits in with statements. (GH-26513) 2021-06-03 16:45:58 +01:00
Irit Katriel 7b1f527d5b
bpo-17792: more accurate error message for unbound variable access exceptions (GH-24976) 2021-06-02 14:09:04 +01:00
Eric Snow 9f494d4929
bpo-43693: Add _PyCode_New(). (gh-26375)
This is an internal-only API that helps us manage the many values used to create a code object.

https://bugs.python.org/issue43693
2021-05-27 09:54:34 -06:00
Eric Snow 6cc800d363
bpo-43693: Clean up the PyCodeObject fields. (GH-26364)
* Move up the comment about fields using in hashing/comparision.

* Group the fields more clearly.

* Add co_ncellvars and co_nfreevars.

* Raise ValueError if nlocals != len(varnames), rather than aborting.
2021-05-26 20:15:40 +01:00
Gabriele N. Tornetta 90a6c07cb2
bpo-43879: Add native_thread_id field to PyThreadState (GH-25458) 2021-05-26 16:40:14 +02:00
Ken Jin 29669245d4
bpo-44156: Make cached string constants in compile.c subinterpreter compatible (GH-26161) 2021-05-25 15:55:34 +02:00
Mark Shannon af5d497f72
bpo-44032: Delay deletion of stack chunks until thread state is deleted. (GH-26285) 2021-05-24 16:22:02 +01:00
Serhiy Storchaka 8b01067318
bpo-28307: Tests and fixes for optimization of C-style formatting (GH-26318)
Fix errors:
* "%10.s" should be equal to "%10.0s", not "%10s".
* Tuples with starred expressions caused a SyntaxError.
2021-05-23 19:06:48 +03:00
Mark Shannon b11a951f16
bpo-44032: Move data stack to thread from FrameObject. (GH-26076)
* Remove 'zombie' frames. We won't need them once we are allocating fixed-size frames.

* Add co_nlocalplus field to code object to avoid recomputing size of locals + frees + cells.

* Move locals, cells and freevars out of frame object into separate memory buffer.

* Use per-threadstate allocated memory chunks for local variables.

* Move globals and builtins from frame object to per-thread stack.

* Move (slow) locals frame object to per-thread stack.

* Move internal frame functions to internal header.
2021-05-21 10:57:35 +01:00
Victor Stinner 58a993957a
bpo-44131: Py_FrozenMain() uses PyConfig_SetBytesArgv() (GH-26201)
Moreover, Py_FrozenMain() relies on Py_InitializeFromConfig() to
handle the PYTHONUNBUFFERED environment variable and configure C
stdio streams like stdout (make the stream unbuffered).
2021-05-20 12:08:05 +02:00
Pablo Galindo f13a6a2d67
Remove unused function in ceval.c (GH-26246) 2021-05-19 19:12:19 +01:00
Victor Stinner eaede0ded7
bpo-44131: Test Py_FrozenMain() (GH-26126)
* Add test_frozenmain to test_embed
* Add Programs/test_frozenmain.py
* Add Programs/freeze_test_frozenmain.py
* Add Programs/test_frozenmain.h
* Add make regen-test-frozenmain
* Add test_frozenmain command to Programs/_testembed
* _testembed.c: add error(msg) function
2021-05-17 23:48:35 +02:00
Ken Jin f24afda591
bpo-26110: Add ``CALL_METHOD_KW`` opcode to speedup method calls with keywords (GH-26014)
* Add CALL_METHOD_KW

* Make CALL_METHOD branchless too since it shares the same code

* Place parentheses in STACK_SHRINK
2021-05-15 16:15:23 +01:00
Irit Katriel 4aeee0b47b
bpo-28146: Fix a confusing error message in str.format() (GH-24213)
Automerge-Triggered-By: GH:pitrou
2021-05-13 13:55:55 -07:00
Inada Naoki ae3c66acb8
bpo-44094: Remove deprecated PyErr_ APIs. (GH-26011)
These APIs are deprecated since Python 3.3. They are not documented too.
2021-05-13 13:45:26 -07:00
Dong-hee Na 7565586724
bpo-44113: Update fromzenmain not to use Py_SetProgramName (GH-26085) 2021-05-13 10:19:46 +09:00
Victor Stinner ec7c09b9bc
bpo-44113: Fix compiler warning in PySys_AddWarnOption() (GH-26084)
Ignore Py_DEPRECATED() warning on purpose.
2021-05-13 02:27:56 +02:00
Mark Shannon 117bfd2b71
Remove PyTryblock struct (GH-26059) 2021-05-12 14:04:38 +01:00
Mark Shannon cb6f3d7163
bpo-43933: Force RETURN_VALUE bytecodes to have line numbers (GH-26054) 2021-05-12 11:25:44 +01:00
Dennis Sweeney 45862f9f5e
Prevent access outside buffer (GH-26012) 2021-05-10 10:10:22 +01:00
Serhiy Storchaka a0bd9e9c11
bpo-28307: Convert simple C-style formatting with literal format into f-string. (GH-5012)
C-style formatting with literal format containing only format codes
%s, %r and %a (with optional width, precision and alignment)
will be converted to an equivalent f-string expression.

It can speed up formatting more than 2 times by eliminating
runtime parsing of the format string and creating temporary tuple.
2021-05-08 22:33:10 +03:00
Batuhan Taskaya b2ec37a722
bpo-44063: set the missing end locations on the compiler (GH-25956) 2021-05-07 20:49:06 +03:00
Mark Shannon adcd220556
bpo-40222: "Zero cost" exception handling (GH-25729)
"Zero cost" exception handling.

* Uses a lookup table to determine how to handle exceptions.
* Removes SETUP_FINALLY and POP_TOP block instructions, eliminating (most of) the runtime overhead of try statements.
* Reduces the size of the frame object by about 60%.
2021-05-07 15:19:19 +01:00
Inada Naoki 4ebf4a6bfa
bpo-40943: Fix skipitem() didn't raise SystemError (GH-25937)
`convertitem()` raises `SystemError` when '#' is used without `PY_SSIZE_T_CLEAN`.
This commit makes `skipitem()` raise it too.
2021-05-07 11:56:48 +09:00
Dennis Sweeney 80a2a4ed7d
bpo-38530: Refactor and improve AttributeError suggestions (GH-25776)
- Make case-swaps half the cost of any other edit
- Refactor Levenshtein code to not use memory allocator, and to bail early on no match.
- Add comments to Levenshtein distance code
- Add test cases for Levenshtein distance behind a debug macro
- Set threshold to `(name_size + item_size + 3) * MOVE_COST / 6`.
  - Reasoning: similar to `difflib.SequenceMatcher.ratio()` >= 2/3:
```
"Multiset Jaccard similarity" >= 2/3
matching letters / total letters >= 2/3
(name_size - distance + item_size - distance) / (name_size + item_size) >= 2/3
1 - (2*distance) / (name_size + item_size) >= 2/3
1/3 >= (2*distance) / (name_size + item_size)
(name_size + item_size) / 6 >= distance
With rounding:
(name_size + item_size + 3) // 6 >= distance
```

Co-authored-by: Pablo Galindo <pablogsal@gmail.com>
2021-05-03 16:47:27 +01:00
Pablo Galindo 39494285e1
bpo-43754: Fix compiler warning in Python/compile.c (GH-25855)
This fixes the following warning:

'initializing': conversion from 'Py_ssize_t' to 'int', possible loss of data [D:\a\cpython\cpython\PCbuild\pythoncore.vcxproj]
2021-05-03 16:20:46 +01:00
Pablo Galindo c2931d31f8
bpo-43916: Move the _PyStructSequence_InitType function to the internal API (GH-25854) 2021-05-03 15:50:24 +01:00
Batuhan Taskaya ad106c68eb
bpo-42725: Render annotations effectless on symbol table with PEP 563 (GH-25583) 2021-05-03 10:43:00 +03:00
Brandt Bucher 0ad1e0384c
bpo-43754: Eliminate bindings for partial pattern matches (GH-25229) 2021-05-02 13:02:10 -07:00
Mark Shannon 5979e81a21
bpo-43933: Set frame.f_lineno during call to __exit__ (GH-25719)
* Set line number of __exit__ call in a with statement to be that of the with keyword.
2021-04-30 14:32:47 +01:00
Jakub Kulík 9032cf5cb1
bpo-43667: Fix broken Unicode encoding in non-UTF locales on Solaris (GH-25096) 2021-04-30 15:21:42 +02:00
Victor Stinner 4908fae3d5
bpo-43916: PyStdPrinter_Type uses Py_TPFLAGS_DISALLOW_INSTANTIATION (GH-25749)
The PyStdPrinter_Type type now uses the
Py_TPFLAGS_DISALLOW_INSTANTIATION flag to disallow instantiation,
rather than seting a tp_init method which always fail.

Write also unit tests for PyStdPrinter_Type.
2021-04-30 14:56:27 +02:00
Victor Stinner 3bb09947ec
bpo-43916: Add Py_TPFLAGS_DISALLOW_INSTANTIATION type flag (GH-25721)
Add a new Py_TPFLAGS_DISALLOW_INSTANTIATION type flag to disallow
creating type instances: set tp_new to NULL and don't create the
"__new__" key in the type dictionary.

The flag is set automatically on static types if tp_base is NULL or
&PyBaseObject_Type and tp_new is NULL.

Use the flag on the following types:

* _curses.ncurses_version type
* _curses_panel.panel
* _tkinter.Tcl_Obj
* _tkinter.tkapp
* _tkinter.tktimertoken
* _xxsubinterpretersmodule.ChannelID
* sys.flags type
* sys.getwindowsversion() type
* sys.version_info type

Update MyStr example in the C API documentation to use
Py_TPFLAGS_DISALLOW_INSTANTIATION.

Add _PyStructSequence_InitType() function to create a structseq type
with the Py_TPFLAGS_DISALLOW_INSTANTIATION flag set.

type_new() calls _PyType_CheckConsistency() at exit.
2021-04-30 12:46:15 +02:00