Commit Graph

9504 Commits

Author SHA1 Message Date
Irit Katriel ffc2f1dd1c
gh-130080: implement PEP 765 (#130087) 2025-03-17 20:48:54 +00:00
Ken Jin b2ed7a6d6a
gh-131281: Add include for pystats builds (#131369)
Add include to for pystats builds
2025-03-18 00:36:06 +08:00
Mark Shannon a45f25361d
GH-131238: More refactoring of core header files (GH-131351)
Adds new pycore_stats.h header file to help break dependencies involving the pycore_code.h header.
2025-03-17 14:41:05 +00:00
Victor Stinner 978e37bb5f
gh-131238: Add explicit includes to pycore headers (#131257) 2025-03-17 12:32:43 +01:00
Victor Stinner 0453e494b6
gh-131238: Convert pycore_pystate.h static inline to functions (#131352)
Convert static inline functions to functions:

* _Py_IsMainThread()
* _PyInterpreterState_Main()
* _Py_IsMainInterpreterFinalizing()
* _Py_GetMainConfig()
2025-03-17 12:31:55 +01:00
Mark Shannon a1aeec61c4
GH-131238: Core header refactor (GH-131250)
* Moves most structs in pycore_ header files into pycore_structs.h and pycore_runtime_structs.h

* Removes many cross-header dependencies
2025-03-17 09:19:04 +00:00
Chris Eibl f104c19a94
GH-131296: fix clang-cl warning on Windows in pytime.c (GH-131297)
fix warning : integer literal is too large

to be represented in a signed integer type,
interpreting as unsigned [-Wimplicitly-unsigned-literal]
2025-03-15 11:41:34 -07:00
Xuanteng Huang 1821f8f10c
gh-131281: fix compile error due to `BINARY_SUBSCR` (GH-131283)
* fix compile error due to `BINARY_SUBSCR`

* replace stat_inc with `BINARY_OP`
2025-03-15 23:38:46 +08:00
Yan Yanchii 55815a6474
gh-130480: Move duplicate `LOAD_SMALL_INT` optimization from codegen to CFG (#130481) 2025-03-14 21:23:27 +00:00
Victor Stinner a5776639c8
gh-111178: Fix function signatures to fix undefined behavior (#131191) 2025-03-14 09:52:15 +00:00
Victor Stinner 0ddfb5717f
gh-111178: Fix _PyLegacyEventHandler_Type deallocator (#131193)
Don't use PyObject_Free() as tp_dealloc to avoid an undefined
behavior. Instead, use the default deallocator which just calls
tp_free which is PyObject_Free().
2025-03-14 10:23:54 +01:00
Hood Chatham f3e275f1a9
GH-130396: Work around for broken `pthread_get_stackaddr_np` on Emscripten (#131088)
Tests / Ubuntu SSL tests with OpenSSL (3.1.7, ubuntu-24.04) (push) Blocked by required conditions Details
Tests / Ubuntu SSL tests with OpenSSL (3.2.3, ubuntu-24.04) (push) Blocked by required conditions Details
Tests / Ubuntu SSL tests with OpenSSL (3.3.2, ubuntu-24.04) (push) Blocked by required conditions Details
Tests / Ubuntu SSL tests with OpenSSL (3.4.0, ubuntu-24.04) (push) Blocked by required conditions Details
Tests / Hypothesis tests on Ubuntu (push) Blocked by required conditions Details
Tests / Address sanitizer (ubuntu-24.04) (push) Blocked by required conditions Details
Tests / All required checks pass (push) Blocked by required conditions Details
mypy / Run mypy on ${{ matrix.target }} (Lib/_pyrepl) (push) Successful in 14s Details
mypy / Run mypy on ${{ matrix.target }} (Lib/test/libregrtest) (push) Successful in 13s Details
mypy / Run mypy on ${{ matrix.target }} (Tools/build) (push) Successful in 11s Details
mypy / Run mypy on ${{ matrix.target }} (Tools/cases_generator) (push) Successful in 12s Details
mypy / Run mypy on ${{ matrix.target }} (Tools/clinic) (push) Successful in 11s Details
mypy / Run mypy on ${{ matrix.target }} (Tools/jit) (push) Successful in 11s Details
mypy / Run mypy on ${{ matrix.target }} (Tools/peg_generator) (push) Successful in 12s Details
Lint / lint (push) Failing after 1m32s Details
mypy / Run mypy on ${{ matrix.target }} (Tools/wasm) (push) Successful in 11s Details
Tests / Change detection (push) Successful in 26m17s Details
Tests / Docs (push) Has been skipped Details
Tests / Windows ${{ fromJSON(matrix.free-threading) && '(free-threading)' || '' }} (Win32, false, windows-latest) (push) Has been skipped Details
Tests / Windows ${{ fromJSON(matrix.free-threading) && '(free-threading)' || '' }} (arm64, true, windows-latest) (push) Has been skipped Details
Tests / Windows ${{ fromJSON(matrix.free-threading) && '(free-threading)' || '' }} (arm64, false, windows-latest) (push) Has been skipped Details
Tests / Windows ${{ fromJSON(matrix.free-threading) && '(free-threading)' || '' }} (x64, false, windows-latest) (push) Has been skipped Details
Tests / Windows ${{ fromJSON(matrix.free-threading) && '(free-threading)' || '' }} (x64, true, windows-latest) (push) Has been skipped Details
Tests / Windows MSI${{ '' }} (arm64) (push) Has been skipped Details
Tests / Windows MSI${{ '' }} (x64) (push) Has been skipped Details
Tests / Windows MSI${{ '' }} (x86) (push) Has been skipped Details
Tests / macOS ${{ fromJSON(matrix.free-threading) && '(free-threading)' || '' }} (true, ${{ github.repository_owner != 'python' }}, ghcr.io/cirruslabs/macos-runner:sonoma) (push) Failing after 24m9s Details
Tests / macOS ${{ fromJSON(matrix.free-threading) && '(free-threading)' || '' }} (false, ${{ github.repository_owner != 'python' }}, macos-13) (push) Failing after 33m1s Details
Tests / macOS ${{ fromJSON(matrix.free-threading) && '(free-threading)' || '' }} (true, ${{ github.repository_owner != 'python' }}, macos-14) (push) Failing after 33m1s Details
Tests / macOS ${{ fromJSON(matrix.free-threading) && '(free-threading)' || '' }} (false, ${{ github.repository_owner != 'python' }}, macos-14) (push) Failing after 33m3s Details
Tests / macOS ${{ fromJSON(matrix.free-threading) && '(free-threading)' || '' }} (false, ${{ github.repository_owner != 'python' }}, ghcr.io/cirruslabs/macos-runner:sonoma) (push) Failing after 33m5s Details
Tests / Ubuntu ${{ fromJSON(matrix.free-threading) && '(free-threading)' || '' }} ${{ fromJSON(matrix.bolt) && '(bolt)' || '' }} (false, false, ubuntu-24.04-arm) (push) Failing after 26m22s Details
Tests / Ubuntu ${{ fromJSON(matrix.free-threading) && '(free-threading)' || '' }} ${{ fromJSON(matrix.bolt) && '(bolt)' || '' }} (false, false, ubuntu-24.04) (push) Failing after 38m7s Details
Tests / WASI (push) Failing after 29m5s Details
Tests / Ubuntu ${{ fromJSON(matrix.free-threading) && '(free-threading)' || '' }} ${{ fromJSON(matrix.bolt) && '(bolt)' || '' }} (true, false, ubuntu-24.04) (push) Failing after 30m29s Details
Tests / CIFuzz (address) (push) Has been skipped Details
Tests / CIFuzz (memory) (push) Has been skipped Details
Tests / CIFuzz (undefined) (push) Has been skipped Details
Tests / Ubuntu ${{ fromJSON(matrix.free-threading) && '(free-threading)' || '' }} ${{ fromJSON(matrix.bolt) && '(bolt)' || '' }} (false, true, ubuntu-24.04) (push) Failing after 36m25s Details
Tests / Cross build Linux (push) Successful in 7m48s Details
Tests / Ubuntu ${{ fromJSON(matrix.free-threading) && '(free-threading)' || '' }} ${{ fromJSON(matrix.bolt) && '(bolt)' || '' }} (false, true, ubuntu-24.04-arm) (push) Failing after 39m37s Details
Tests / Thread sanitizer ${{ fromJSON(matrix.free-threading) && '(free-threading)' || '' }} (false) (push) Failing after 30m37s Details
Tests / Thread sanitizer ${{ fromJSON(matrix.free-threading) && '(free-threading)' || '' }} (true) (push) Successful in 49m39s Details
Mark stale pull requests / stale (push) Has been skipped Details
Tail calling interpreter / ${{ matrix.target }} (aarch64, 19, macos-14, aarch64-apple-darwin/clang) (push) Has been cancelled Details
Tail calling interpreter / ${{ matrix.target }} (aarch64, 19, ubuntu-24.04-arm, aarch64-unknown-linux-gnu/gcc) (push) Has been cancelled Details
Tail calling interpreter / ${{ matrix.target }} (x64, 19, windows-latest, x86_64-pc-windows-msvc/msvc) (push) Has been cancelled Details
Tail calling interpreter / ${{ matrix.target }} (x86_64, 19, macos-13, x86_64-apple-darwin/clang) (push) Has been cancelled Details
Tail calling interpreter / ${{ matrix.target }} (x86_64, 19, ubuntu-24.04, free-threading) (push) Has been cancelled Details
Tail calling interpreter / ${{ matrix.target }} (x86_64, 19, ubuntu-24.04, x86_64-unknown-linux-gnu/gcc) (push) Has been cancelled Details
Implements a workaround implementation of `pthread_get_stackaddr_np` for Emscripten.
This will be replaced by an implementation that will be included in Emscripten 4.0.6.
2025-03-14 07:04:40 +08:00
Tomasz Pytel c5abded099
gh-130382: add missing `_PyReftracerTrack` to ceval `Py_DECREF` (#130689) 2025-03-13 12:31:49 -04:00
Irit Katriel 4242c2b8d0
gh-130080: move _Py_EnsureArrayLargeEnough to a separate header so it can be used outside of the compiler (#130930) 2025-03-13 16:02:58 +00:00
Victor Stinner 9a63138e09
gh-111178: Fix function signatures in misc files (#131180) 2025-03-13 16:55:08 +01:00
Russell Keith-Magee dd6d24e9d2
gh-130940: Modify default behavior of `PyConfig.use_system_logger` to enable on iOS (#131172)
Modify default behavior of use_system_log to enable on iOS.
2025-03-13 14:28:49 +08:00
Yan Yanchii 3618240624
gh-126835: Avoid creating unnecessary tuple when looking for constant sequence during constant folding (#131054) 2025-03-12 21:45:54 +00:00
Kumar Aditya ea57ffa02e
gh-131141: fix data race in instrumentation while registering callback (#131142) 2025-03-13 00:11:52 +05:30
T. Wouters de2f7da77d
gh-115999: Add free-threaded specialization for FOR_ITER (#128798)
Add free-threaded versions of existing specialization for FOR_ITER (list, tuples, fast range iterators and generators), without significantly affecting their thread-safety. (Iterating over shared lists/tuples/ranges should be fine like before. Reusing iterators between threads is not fine, like before. Sharing generators between threads is a recipe for significant crashes, like before.)
2025-03-12 16:21:46 +01:00
Stan Ulbrych 155c44b901
gh-81267: Correct time.sleep() error message (#131055) 2025-03-12 08:12:57 +00:00
Tian Gao 8b1edae93a
gh-122029: Do not unpack method for legacy tracing anymore (#130898) 2025-03-11 14:04:22 -04:00
Victor Stinner 9d759b63d8
gh-111178: Change Argument Clinic signature for METH_O (#130682)
Use "PyObject*" for METH_O functions to fix an undefined behavior.
2025-03-11 16:33:36 +01:00
Sam Gross 4162bc133b
gh-130396: Fix thread sanitizer crashes on stack overflow tests (gh-130966)
Thread sanitizer will often crash if a test uses more than half the
stack.
2025-03-11 10:33:23 -04:00
Max Bachmann de8818ae23
gh-131082: Add missing guards for WIN32_LEAN_AND_MEAN (#131044)
* Add missing guards for WIN32_LEAN_AND_MEAN

* add missing whitespaces
2025-03-11 12:33:01 +01:00
Pablo Galindo Salgado a931a8b324
gh-117174: Add a new route in linecache to fetch interactive source code (#117500) 2025-03-10 21:54:05 +00:00
Mark Shannon be046ee6e0
GH-123044: Give the `POP_TOP` after a case test a location in the body, not the pattern. (GH-130627) 2025-03-10 17:31:16 +00:00
Mark Shannon 2bef8ea8ea
GH-127705: Use `_PyStackRef`s in the default build. (GH-127875) 2025-03-10 14:06:56 +00:00
Mark Shannon 89df62c120
GH-128534: Fix behavior of branch monitoring for `async for` (GH-130847)
* Both branches in a pair now have a common source and are included in co_branches
2025-03-07 14:30:31 +00:00
Sam Gross a025f27d94
gh-130920: Fix data race in STORE_SUBSCR_LIST_INT (#130923)
The write of the item to the list needs to use an atomic operation in
the free threading build.

Co-authored-by: Tomasz Pytel <tompytel@gmail.com>
2025-03-06 15:59:48 -05:00
Sam Gross 052cb717f5
gh-124878: Fix race conditions during interpreter finalization (#130649)
The PyThreadState field gains a reference count field to avoid
issues with PyThreadState being a dangling pointer to freed memory.
The refcount starts with a value of two: one reference is owned by the
interpreter's linked list of thread states and one reference is owned by
the OS thread. The reference count is decremented when the thread state
is removed from the interpreter's linked list and before the OS thread
calls `PyThread_hang_thread()`. The thread that decrements it to zero
frees the `PyThreadState` memory.

The `holds_gil` field is moved out of the `_status` bit field, to avoid
a data race where on thread calls `PyThreadState_Clear()`, modifying the
`_status` bit field while the OS thread reads `holds_gil` when
attempting to acquire the GIL.

The `PyThreadState.state` field now has `_Py_THREAD_SHUTTING_DOWN` as a
possible value. This corresponds to the `_PyThreadState_MustExit()`
check. This avoids race conditions in the free threading build when
checking `_PyThreadState_MustExit()`.
2025-03-06 10:38:34 -05:00
Sergey B Kirpichev 2352bd418a
gh-130860: Fix width calculation, when separators in fractional part (GH-130865)
This amends f39a07be47
2025-03-06 14:26:29 +02:00
Jamie Phan 10cdd7f91f
GH-130903: typo in optimizer DSL for _GUARD_BOTH_UNICODE (#130904)
Typo introduced in gh-118910.
2025-03-06 12:11:08 +01:00
Tomasz Pytel aeb2327386
gh-130574: renumber RESUME opcode from 149 to 128 (GH-130685) 2025-03-06 08:59:36 +00:00
mpage d7bb7c7817
gh-118331: Fix a couple of issues when list allocation fails (#130811)
* Fix use after free in list objects

Set the items pointer in the list object to NULL after the items array
is freed during list deallocation. Otherwise, we can end up with a list
object added to the free list that contains a pointer to an already-freed
items array.

* Mark `_PyList_FromStackRefStealOnSuccess` as escaping

I think technically it's not escaping, because the only object that
can be decrefed if allocation fails is an exact list, which cannot
execute arbitrary code when it is destroyed. However, this seems less
intrusive than trying to special cases objects in the assert in `_Py_Dealloc`
that checks for non-null stackpointers and shouldn't matter for performance.
2025-03-05 10:42:09 -08:00
Mark Shannon 78d50e91ff
GH-127705: better double free message. (GH-130785)
* Add location information when accessing already closed stackref

* Add #def option to track closed stackrefs to provide precise information for use after free and double frees.
2025-03-05 14:00:42 +00:00
Sam Gross 2f6e0e9f70
gh-130794: Process interpreter QSBR queue in _PyMem_AbandonDelayed. (gh-130808)
This avoids a case where the interpreter's queue of memory to be freed
could grow rapidly if there are many short lived threads.
2025-03-04 18:04:04 -05:00
Amit Lavon 691354ccb0
GH-130415: Narrow str to "" based on boolean tests (GH-130476) 2025-03-04 13:20:17 -08:00
Klaus117 c989e74446
GH-130415: Narrow int to 0 based on boolean tests (GH-130772) 2025-03-04 12:44:09 -08:00
Sergey Miryanov 3a7f17c7e2
gh-130790: Remove references about unicode's readiness from comments (#130801) 2025-03-03 19:18:09 +00:00
Bénédikt Tran 3146a25e97
gh-129173: refactor `PyCodec_BackslashReplaceErrors` into separate functions (#129895)
The logic of `PyCodec_BackslashReplaceErrors` is now split into separate functions,
each of which handling a specific exception type.
2025-03-03 13:58:15 +01:00
Bénédikt Tran f693f84227
gh-129173: simplify `PyCodec_XMLCharRefReplaceErrors` logic (#129894)
Writing the decimal representation of a Unicode codepoint only requires to know the number of digits.

---------

Co-authored-by: Petr Viktorin <encukou@gmail.com>
2025-03-03 11:43:22 +00:00
Ned Batchelder efbc5929ca
docs: internal notes have moved, correct references (#130762) 2025-03-03 06:38:49 -05:00
Brandt Bucher 7afa476874
GH-130415: Use boolean guards to narrow types to values in the JIT (GH-130659) 2025-03-02 13:21:34 -08:00
Hugo Beauzée-Luyssen 214562ed4d
gh-130740: Move some `stdbool.h` includes after `Python.h` (#130738)
Move some `#include <stdbool.h>` after `#include "Python.h"` when `pyconfig.h` is not
included first and when we are in a platform-agnostic context. This is to avoid having
features defined by `stdbool.h` before those decided by `Python.h`.
2025-03-02 09:56:49 +00:00
Mark Shannon 54965f3fb2
GH-130296: Avoid stack transients in four instructions. (GH-130310)
* Combine _GUARD_GLOBALS_VERSION_PUSH_KEYS and _LOAD_GLOBAL_MODULE_FROM_KEYS into _LOAD_GLOBAL_MODULE

* Combine _GUARD_BUILTINS_VERSION_PUSH_KEYS and _LOAD_GLOBAL_BUILTINS_FROM_KEYS into _LOAD_GLOBAL_BUILTINS

* Combine _CHECK_ATTR_MODULE_PUSH_KEYS and _LOAD_ATTR_MODULE_FROM_KEYS into _LOAD_ATTR_MODULE

* Remove stack transient in LOAD_ATTR_WITH_HINT
2025-02-28 18:00:38 +00:00
Sam Gross 038e4d606b
gh-130605: Use relaxed atomics to set the GIL switch interval (gh-130654)
The interval may be concurrently read by a thread attempting to acquire
the GIL.
2025-02-28 09:27:18 -05:00
Hugo Beauzée-Luyssen 830f04b505
Postpone <stdbool.h> inclusion after Python.h (#130641)
Remove inclusions prior to Python.h.

<stdbool.h> will cause <features.h> to be included before Python.h can
define some macros to enable some additional features, causing multiple
types not to be defined down the line.
2025-02-28 10:09:27 +01:00
Petr Viktorin fecf8bc8f2
gh-130595: Fix leak in WITH_EXCEPT_START error case (GH-130626)
Co-authored-by: Ken Jin <kenjin@python.org>
2025-02-28 08:58:50 +00:00
Sam Gross 7aeaa5af2c
gh-130091: Reorder `_PyThreadState_Attach` to avoid data race (gh-130092)
This moves `tstate_activate()` down to avoid a data race in the free
threading build on the `_PyRuntime`'s thread-local `autoTSSkey`. This
key is deleted during runtime finalization, which may happen
concurrently with a call to `_PyThreadState_Attach`.

The earlier `tstate_try/wait_attach` ensures that the thread is blocked
before it attempts to access the deleted `autoTSSkey`.

This fixes a TSAN reported data race in
`test_threading.test_import_from_another_thread`.
2025-02-27 13:57:19 -05:00
Sam Gross d027787c8d
gh-130421: Fix data race on timebase initialization (gh-130592)
Windows and macOS require precomputing a "timebase" in order to convert
OS timestamps into nanoseconds. Retrieve and compute this value during
runtime initialization to avoid data races when accessing the time.
2025-02-27 13:27:54 +00:00